Senior GenAI Quality Engineer and Solution Analyst

Key Responsibilities:

. Define and execute end to end test strategies covering UI workflows, backend APIs, integrations and agentic interfaces.

. Test conversational and agentic behaviour including multi turn context, tool selection, tool inputs and outputs, state transitions, retries, timeouts, handoffs, approvals and recovery from partial failure.

. Validate GenAI responses for task completion, grounding, relevance, consistency, citation behaviour and safe failure, while recognising that outputs can be non deterministic.

. Perform functional, integration, regression, exploratory, negative, resilience and basic performance testing across application layers.

. Design API tests for contracts, authentication, authorisation, validation, error handling, idempotency, rate limits and downstream failures.

. Test UI behaviour across browsers and realistic user journeys, including loading states, interrupted sessions, feedback capture, accessibility basics and clear error communication.

. Create and maintain test data, reusable test scenarios and traceable evidence suitable for enterprise release governance.

. Use logs, traces, request and response payloads and observability tools to isolate defects and distinguish application, model, data, integration and platform issues.

. Automate the tests that materially reduce cycle time, manual effort or production risk, and keep unstable or low value scenarios out of the automation suite.

. Communicate defects and quality risks clearly to engineers, product owners, GenAI specialists, security teams and business stakeholders.

. Provide an evidence based release recommendation, including known limitations, residual risks and areas requiring monitoring.

. Partner with product owners, business users, architects, engineers and GenAI specialists to define the problem, target user journeys and expected business outcomes.

. Analyse proposed GenAI use cases and determine where deterministic application logic, retrieval, workflow orchestration, tool using agents or human approval should be used.

. Translate business requirements into end to end solution flows, functional requirements, interface behaviours, decision rules, acceptance criteria and non functional requirements.

. Map interactions across user interfaces, APIs, models, prompts, retrieval components, enterprise data sources, agent tools and downstream systems.

. Analyse solution options and document tradeoffs relating to quality, complexity, cost, latency, security, data access, maintainability and operational risk.

. Identify unclear ownership, missing controls, integration assumptions, failure scenarios and operational gaps before development begins.

. Support the design of human approval, fallback, escalation and exception handling paths for agentic solutions.

. Define measurable success criteria covering business outcomes, user experience, functional correctness, response quality, latency, reliability and safe failure.

. Maintain traceability from business need through solution requirement, implementation, evaluation scenario and release evidence.

. Facilitate structured design reviews and communicate findings using process flows, sequence diagrams, interface specifications, decision tables and concise solution documentation.

Requirements:
Core Requirements

. 5 to 8 years of experience in software quality engineering, test engineering or a similar hands on role covering complex applications.

. Strong experience testing web user interfaces, backend services and REST APIs.

. Hands on ability with API tools and automation frameworks such as Postman, REST Assured, pytest, Playwright, Cypress, Selenium or equivalent.

. Working knowledge of Java, Python, JavaScript or TypeScript sufficient to build, review and troubleshoot test automation.

. Strong test analysis skills, including requirements review, risk assessment, boundary analysis, negative testing and traceability.

. Experience validating distributed systems and integrations, including asynchronous processing, queues, batch jobs and downstream dependencies.

. Ability to inspect logs, traces, network calls, payloads and database records to identify the actual failure point.

. Experience with Git, pull requests, CI/CD pipelines, test reporting and defect management tools.

. Understanding of security and privacy testing fundamentals, including access control, sensitive data handling, input validation and auditability.

. Strong stakeholder communication and the confidence to challenge weak designs, vague expected outcomes and premature release decisions.

.Ability to work in a fast moving environment where requirements and GenAI behaviour evolve

Solution analysis and design expectations

. Experience analysing complex applications across user journeys, business processes, APIs, data flows and enterprise integrations.

. Ability to facilitate requirements discussions and convert ambiguous business needs into clear functional requirements, acceptance criteria and solution behaviours.

. Experience producing practical analysis artefacts such as process flows, sequence diagrams, context diagrams, interface specifications, decision tables and user stories.

. Ability to analyse solution alternatives and explain tradeoffs involving quality, cost, performance, security, operational support and delivery complexity.

. Understanding of application architecture concepts including synchronous and asynchronous integrations, event flows, authentication, authorisation, failure handling and system boundaries.

. Ability to distinguish problems that require GenAI from those better addressed through deterministic rules, search, workflow automation or conventional application logic.

. Confidence working with product, architecture, engineering, security, data and business stakeholders during discovery and solution design.

GenAI and Agentic Testing Expectations

. Practical understanding of LLM based applications, retrieval augmented generation, prompts, context windows, embeddings and tool using agents.

. Ability to test probabilistic systems using evaluation datasets, repeat runs, quality thresholds and evidence based acceptance criteria rather than brittle exact text matching.

. Experience validating grounded answers, citations, retrieval quality, hallucination risk, prompt injection resistance and safe handling of restricted or unsupported requests.

. Ability to test agent plans and execution paths, tool calls, memory and state, human approval checkpoints, fallback behaviour and termination conditions.

. Awareness of evaluation and observability tooling such as Langfuse, LangSmith, OpenTelemetry, Elastic, Splunk or equivalent.

Nice to Have

. Experience testing applications in banking, finance or another regulated enterprise environment.

. Experience with contract testing, service virtualisation, synthetic monitoring or performance testing tools.

. Experience with Kubernetes, OpenShift, AWS hosted services or containerised deployments.

. Accessibility testing experience and familiarity with WCAG based checks.

. Experience building lean quality dashboards that show release risk, defect escape patterns, flaky tests and cycle time.

.Exposure to red teaming or adversarial testing of GenAI applications

Key Domain/ Technical Skills
. UI and API Quality Engineering

. Agentic and GenAI Application Testing

.Risk Based Test Automation and Observability