Automation QA Engineer

Key Accountabilities

• Define and execute quality assurance and testing strategies for AI-enabled applications, agents, and workflows.
• Establish quality criteria and measurable KPIs for accuracy, reliability, latency, safety, compliance, and task success.
• Develop automated evaluation and testing frameworks for AI and agent-based systems.
• Continuously evaluate and monitor system behavior in production environments.
• Identify hallucinations, tool misuse, policy violations, workflow failures, anomalies, and quality degradation.
• Support auditability, risk management, incident response, and continuous quality improvement.
• Collaborate with engineering teams to improve prompts, tools, workflows, and overall system reliability.

Principal Responsibilities

AI & Automation Testing

• Define and implement testing strategies for AI-enabled workflows and agentic applications.
• Build and maintain automated test suites using Playwright for end-to-end UI and workflow testing.
• Develop Python-based automation scripts for API, workflow, and integration testing.
• Create automated evaluation harnesses to measure:
• Task success rate
• Accuracy and relevance
• Hallucination rate
• Response quality
• Tool usage and tool misuse
• Policy and safety violations
• Latency and performance
• Regression and consistency

AI Evaluation & Quality Monitoring

• Apply LLM evaluation frameworks such as DeepEval, RAGAS, and Evidently AI to assess model and application quality.
• Design and maintain evaluation datasets, test scenarios, benchmarks, and regression suites.
• Monitor AI system behavior and quality trends over time.
• Detect model or workflow drift, anomalies, unexpected behavior, and degradation in production.
• Establish automated quality gates and thresholds for AI-enabled releases where appropriate.
• Investigate recurring quality issues and provide evidence-based recommendations for remediation.

Production Monitoring & Reliability

• Implement and maintain production quality monitoring and anomaly detection mechanisms.
• Develop dashboards and reports that communicate quality metrics, trends, risks, and system health.
• Define alerts for critical quality, reliability, performance, and safety issues.
• Support incident investigation and root-cause analysis for AI-related failures.
• Develop and maintain runbooks for common system and AI failure scenarios.
• Participate in incident response and post-incident improvement activities.

Collaboration & Continuous Improvement

• Work closely with developers, architects, product teams, and other stakeholders to improve system quality.
• Provide test results and data-driven recommendations to improve prompts, tool definitions, agent workflows, and system behavior.
• Participate in design and technical reviews to identify potential quality and operational risks early.
• Contribute to CI/CD quality gates and automated regression testing.
• Continuously evaluate new AI testing, evaluation, monitoring, and QA practices.

Governance, Privacy & Compliance

• Ensure testing, logging, evaluation, and monitoring practices align with applicable data privacy, security, audit, governance, and regulatory requirements.
• Maintain sufficient test evidence and documentation to support auditability and operational readiness.
• Identify and escalate AI-related risks that may affect users, business operations, compliance, or trust.
• Support the implementation of AI safety and responsible AI testing practices.

Essential Qualifications

• Minimum 3 years of experience in QA, Test Automation, Software Testing, SDET, DevOps, or related engineering roles.
• Alternatively, 2+ years of experience may be considered where there is strong direct experience testing AI/ML-enabled systems.
• Strong programming skills in Python, particularly for test automation, evaluation frameworks, scripting, and basic data analysis.
• Strong understanding of software QA methodologies, test planning, test execution, defect management, and automation.
• Experience developing automated test frameworks and reusable test scripts.
• Strong analytical and problem-solving skills.
• High attention to detail, particularly for issues that can impact system reliability, safety, compliance, and user trust.
• Ability to work effectively in an evolving environment with rapidly changing AI tools, technologies, and testing practices.
• Strong communication and collaboration skills, with the ability to use evidence and test results to influence engineering and product decisions.