QA Test Consultant- AI

Description We are seeking a highly motivated Quality Engineering (QE) Consultant with 4–6 years of experience in software testing, automation, and emerging AI technologies. The ideal candidate will have hands-on experience testing AI/GenAI applications , APIs , and web applications , along with strong expertise in Playwright automation and AI evaluation frameworks . This role will focus on ensuring the quality, reliability, safety, and performance of AI-powered solutions by designing comprehensive test strategies, implementing automation frameworks, validating AI model behavior, and establishing evaluation mechanisms for large language model (LLM)-based applications. The candidate will collaborate closely with Product Owners, Developers, Data Scientists, and AI Engineers to drive quality throughout the software development lifecycle. Responsibilities AI & GenAI Testing - Design and execute test strategies for AI/GenAI applications, including LLM-powered solutions, AI agents, copilots, and conversational systems. - Validate AI outputs for: - Accuracy - Relevance - Groundedness - Consistency - Hallucination detection - Toxicity and safety compliance - Create and maintain prompt test suites and benchmark datasets. - Perform functional, regression, performance, and reliability testing for AI-enabled applications. - Define and execute AI evaluation metrics and quality gates. Automation & Quality Engineering - Develop and maintain test automation frameworks using Playwright . - Automate UI, API, and end-to-end business workflows. - Create reusable automation assets and testing accelerators. - Integrate automated tests into CI/CD pipelines. - Support shift-left testing and quality engineering practices. API Testing - Design and execute API test cases using tools such as: - Postman - REST Assured - Playwright API Testing - Validate API contracts, authentication, authorization, error handling, and performance. - Conduct integration testing across distributed systems and third-party services. AI Evaluation & Quality Metrics - Establish AI evaluation frameworks and quality scorecards. - Measure model quality using evaluation techniques such as: - Precision/Recall - Relevancy scoring - Semantic similarity - Groundedness validation - Human-in-the-loop evaluation - Analyze AI quality trends and recommend improvements. - Support Responsible AI and model governance requirements. Collaboration & Stakeholder Management - Work closely with engineering, product, and AI teams to identify quality risks early. - Participate in requirement reviews and design discussions. - Communicate testing progress, risks, and quality metrics to stakeholders. - Contribute to QE best practices, standards, and reusable playbooks. Qualifications Experience - 4–6 years of experience in Quality Engineering, Software Testing, or Test Automation. - Hands-on experience testing AI/Generative AI applications. - Experience with modern web application testing and API testing. Technical Skills - Strong expertise in Playwright automation. - Hands-on experience with API testing and automation. - Experience validating AI/LLM-based applications and AI agents. - Good understanding of AI evaluation methodologies and testing techniques. - Familiarity with: - M365 Copilot - OpenAI/Azure OpenAI - Claude - Gemini - Agentic AI frameworks - Knowledge of test management tools such as Azure DevOps (ADO), Jira, or similar platforms. - Experience working with CI/CD pipelines and DevOps practices. Preferred Skills - Exposure to Python or TypeScript for automation and AI testing. - Experience with AI evaluation frameworks such as: - RAG evaluation - Prompt evaluation - LLM benchmarking - Human feedback-based evaluation - Knowledge of Responsible AI principles, AI governance, and model risk management. - Experience testing Retrieval-Augmented Generation (RAG) systems and AI agents. Education - Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline. Key Competencies - Analytical thinking and problem-solving - Strong communication and stakeholder management - Quality-first mindset - Continuous learning and innovation - Collaboration and teamwork - Attention to detail - Adaptability to emerging AI technologies