Our Customer: A technology company developing an AI-based application for enterprise users. The project focuses on delivering accurate, reliable, secure, and user-friendly AI-powered functionality.
You will join a cross-functional team and work closely with developers, AI engineers, product owners, and business stakeholders to ensure the quality of AI features throughout the product lifecycle. Your Tasks: * Analyze project requirements, user stories, and AI-related business scenarios. * Create and execute test cases and test scenarios for AI features and workflows. * Perform functional, system, integration, regression, and end-to-end testing. * Validate AI-generated responses for accuracy, relevance, completeness, clarity, and consistency. * Test different types of user inputs, including valid, invalid, incomplete, and ambiguous queries. * Identify hallucinations, incorrect or inconsistent responses, bias, and gaps in AI understanding. * Prepare prompt-based test scenarios and test different prompt variations. * Verify that AI follows defined instructions, guardrails, business rules, and expected tone. * Validate AI responses against approved knowledge sources and expected outcomes. * Test integrations between AI components, APIs, databases, knowledge bases, and third-party or enterprise systems. * Verify authentication, authorization, role-based access, and data correctness. * Perform regression testing after changes to prompts, models, APIs, or business logic. * Maintain reusable regression test suites. * Test AI behavior in edge cases and unexpected user scenarios. * Log defects with clear reproduction steps, test data, expected and actual results, and severity. * Track defects using Jira, Azure DevOps, or similar tools. * Work closely with developers, AI engineers, product owners, and business stakeholders to resolve issues. * Provide testing results and release sign-off input.
Required Experience and Skills: * 3+ years of commercial experience in software testing / QA. * Strong understanding of functional, regression, system, integration, and end-to-end testing. * Experience creating test cases, test scenarios, and test data. * Experience testing AI-based applications, chatbots, APIs, enterprise applications, or similar solutions. * Experience validating AI-generated outputs for correctness, relevance, and quality. * Understanding of AI concepts such as prompts, LLM responses, hallucinations, context, and knowledge retrieval. * Experience with defect tracking tools such as Jira, Azure DevOps, or similar. * Good analytical and problem-solving skills. * Strong attention to detail and ability to identify inconsistencies in AI-generated responses. * Good communication skills and ability to work effectively with cross-functional teams.
Would be a Plus: * Hands-on experience testing Generative AI, NLP, ML, chatbot, or conversational AI solutions. * Experience testing RAG-based or knowledge-based AI applications. * Basic understanding of prompt engineering and LLM behavior. * Experience with API testing tools such as Postman or Swagger. * Experience with automation testing tools such as Selenium, Playwright, or similar. * Knowledge of Azure OpenAI, Azure AI, Microsoft Copilot Studio, Power Platform, or similar AI platforms. * Understanding of prompt injection, data leakage, access control, and responsible AI testing. * Experience testing AI integrations with databases, knowledge bases, or third-party services.
