Quality Analyst

Description We are looking for a Quality Assurance (QA) Engineer with 3-7 years of experience in testing, QA, and reporting, with exposure to Conversational AI and Agentic AI systems. The ideal candidate will have a strong understanding of standard testing practices and be passionate about ensuring the highest quality standards in AI-driven applications. This role involves validating conversational experiences, AI-powered workflows, autonomous agent behaviors, integrations with external tools and APIs, and ensuring the reliability, safety, and performance of AI systems in production environments. Responsibilities - Develop and execute test cases, scripts, and procedures for Conversational AI and Agentic AI applications. - Collaborate with product managers, developers, business analysts, and other stakeholders to ensure quality standards are met throughout the development lifecycle. - Perform functional, regression, integration, and end-to-end testing across conversational AI platforms and agent-based AI systems. - Validate AI agent decision-making, task execution, multi-step workflows, reasoning paths, and interactions with external tools, APIs, and knowledge sources. - Test LLM-powered applications for accuracy, relevance, consistency, hallucinations, guardrail compliance, and response quality. - Evaluate agent orchestration capabilities, including multi-agent collaboration, task decomposition, context management, memory retention, and tool utilization. - Identify, document, and track defects using JIRA and provide detailed reports on testing results. - Analyze requirements and design comprehensive test strategies covering conversational flows, autonomous agent behaviors, integrations, and edge-case scenarios. - Conduct prompt, response, and workflow validation to ensure optimal performance and user experience. - Perform security, privacy, bias, and responsible AI testing to validate compliance with organizational and regulatory standards. - Generate test reports, dashboards, and quality metrics for stakeholders, ensuring clear and actionable insights. - Contribute to the continuous improvement of AI testing processes, frameworks, methodologies, and automation capabilities. Qualifications - 3-7 years of experience in software testing and quality assurance, with exposure to Conversational AI, Generative AI, Agentic AI, or similar AI technologies. - Strong knowledge of QA methodologies, SDLC, STLC, defect management, and validation techniques. - Experience testing conversational AI platforms, chatbots, voicebots, virtual assistants, or AI-driven applications. - Understanding of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), prompt engineering concepts, and AI agent architectures. - Experience with JIRA for defect tracking and project management. - Strong proficiency in Excel for data analysis, reporting, and test metrics. - Familiarity with API testing tools such as Postman, Swagger, or equivalent. - Experience in functional, integration, performance, security, and user acceptance testing. - Understanding of AI evaluation metrics, model behavior analysis, and conversational testing frameworks. - Ability to work effectively in cross-functional teams and adapt to fast-paced environments. - Strong analytical, troubleshooting, and problem-solving skills.