Senior AI Backend Engineer

About the role

Own the evaluation stack for production multi-agent systems: LLM-as-judge, per-PR harnesses, and simulators so agent releases can be gated on measured quality.

What you'll do

• Design and build LLM-as-judge systems and calibrate them against human labels
• Build per-PR eval harnesses and regression detection wired into CI
• Build user simulators for test coverage and adversarial cases
• Pipe real production failures back into evaluation sets

Requirements

• 5+ years of software engineering with recent hands-on LLM or agent work
• Production experience with LLM reliability, latency, cost, and observability
• Nice to have: offline and online eval, LLM-as-judge, or tools such as Arize or LangSmith

AI Alerts shares third-party job opportunities for informational purposes only. We are not the employer and are not involved in the hiring process. Always verify the company and role through official channels before applying, and never pay to apply, train, onboard, process documents, or secure a job offer. Legitimate employers do not ask applicants for money. Read our Terms to learn more.