Quality Analyst

About the role

AI Quality Analyst role evaluating a personalization feature for an AI model. Assess how well it integrates information from past conversations, email, search, and activity. Combines creative prompt design with analytical evaluation.

What you'll do

• Evaluate the quality of personalized responses from an AI model using your own personal experiences as context
• Design creative, multi-turn prompts that test the model’s ability to integrate personal data sources accurately
• Assess model responses for grounding, integration, and helpfulness, identifying incorrect personalization or forced connections
• Review side-by-side comparisons of AI responses to detect subtle differences in relevance and accuracy
• Provide detailed feedback on model behavior to improve personalization performance

Requirements

• English proficiency with a high degree of competence in reading and writing
• Experience in prompt engineering or designing test cases for AI models
• Ability to analyze nuanced and ambiguous AI responses with analytical rigor
• Familiarity with personalization concepts and evaluation metrics in AI systems
• Attention to detail to identify subtle discrepancies in model outputs
• Willingness to use your primary personal Google account and enable personal data sources for assessment
• Full-time availability in your local time zone to support a global, 24-hour operations team

AI Alerts shares third-party job opportunities for informational purposes only. We are not the employer and are not involved in the hiring process. Always verify the company and role through official channels before applying, and never pay to apply, train, onboard, process documents, or secure a job offer. Legitimate employers do not ask applicants for money. Read our Terms to learn more.