We’re seeking a Senior QA Engineer with strong experience in GenAI and agentic systems to lead testing for our AI-driven products. You’ll focus on automation, evaluation, and ensuring the reliability and performance of advanced AI/ML technologies.

Key Responsibilities

• Lead testing and evaluation of Generative AI and agentic systems.
• Build automated test suites using Python, Pytest, and Playwright for API, UI, unit, and integration testing.
• Validate AI model behavior, outputs, data quality, and prompts.
• Use tools like MLFlow, LangFuse, and LangFlow for model monitoring and evaluation.
• Work with data scientists, engineers, and product teams to define quality goals.
• Investigate failures, triage issues, and drive resolution.
• Maintain clear documentation and improve test automation and CI/CD workflows.

Required Skills

• Strong Python and test automation experience (Pytest, Playwright).
• API (REST/GraphQL) and UI testing expertise.
• Solid understanding of GenAI and agentic system evaluation.
• Experience with JSON-based APIs and AI tools (MLFlow, LangFuse, LangFlow).
• Hands-on with LLMs, prompt testing, and AI model evaluation.
• Prior QA experience in AI/ML or GenAI systems preferred.

Nice to Have

• Experience with LangChain or chatbot testing.
• Exposure to cloud environments and CI/CD pipelines.

Personal Attributes

• Strong ownership and attention to detail.
• Clear communicator and collaborative team player.
• Comfortable presenting results and giving/receiving feedback.

Further Information:

• Duration: Long term contract (B2B)
• Workload: Remote

Show more
Show less