AI Software QA Tester

About the role

We are seeking an experienced Automation QA Tester to join our growing team and take ownership of testing our AI avatar and conversational workflows. The primary focus of this role is to ensure that the avatar consistently provides accurate and relevant responses to the correct enquiry, even when users ask the same question in different ways or use varied phrasing. The ideal candidate will have strong experience in test automation, particularly with Playwright or similar tools, and will be comfortable validating both functional behaviour and conversational accuracy across web-based applications. Experience working with Azure DevOps pipelines and testing cloud-based applications is highly desirable. This role is well-suited to someone who has a strong eye for detail, enjoys identifying gaps in user journeys, and can design robust test scenarios to validate how an AI-driven solution performs under different question variations, use cases, and edge cases.

What you'll do

• Design, develop, and maintain automated test scripts for web-based applications and AI avatar interactions.
• Test the AI avatar to ensure it provides the correct answer to the intended enquiry, even when users phrase questions differently or use alternative wording.
• Create and execute test cases covering multiple variations of the same question to validate the avatar’s accuracy, consistency, and contextual understanding.
• Perform end-to-end testing using tools such as Playwright, ensuring strong coverage across functional, regression, and user interaction scenarios.
• Validate conversational flows, response relevance, escalation handling, fallback responses, and overall user experience.
• Work closely with development and product teams to identify defects, inconsistencies, and opportunities to improve the avatar’s response quality.
• Integrate automated testing into CI/CD pipelines using Azure DevOps.
• Test cloud-based applications deployed on Azure App Services and Azure Function Apps.
• Analyse, document, and report test outcomes clearly, including response mismatches, failed scenarios, and conversational gaps.
• Help define best practices for AI/conversational testing, including response validation, intent matching, and variation testing.
• Ensure test environments remain current and aligned with production-like conditions.
• Continuously identify and implement improvements to the automation and testing process.
• Stay up to date with the latest tools, trends, and best practices in automation testing, AI testing, and conversational QA.

Requirements

• Non-Negotiable Requirements
• The QA resource will take full ownership of automation testing
• Playwright is the preferred and required tool, as it integrates well with the existing CI/CD pipeline
• Proficiency in Selenium is acceptable only if there is strong transferability to Playwright
• The candidate must be capable of testing not only technical functionality, but also the accuracy and appropriateness of avatar responses across varied enquiries

AI Alerts shares third-party job opportunities for informational purposes only. We are not the employer and are not involved in the hiring process. Always verify the company and role through official channels before applying, and never pay to apply, train, onboard, process documents, or secure a job offer. Legitimate employers do not ask applicants for money. Read our Terms to learn more.