AI QA Specialist Needed

We're looking for an AI QA / Agent Testing Specialist to help us rigorously test and validate a portfolio of AI agents — including AI voice agents and other LLM-based agents we're actively developing and deploying for clients. What we need help with: - Designing and running end-to-end test plans for AI voice agents and text-based AI agents (conversation flows, edge cases, error handling, handoffs/escalations) - Building repeatable test suites to catch hallucinations — factual accuracy checks, out-of-scope response detection, and grounding validation against source data/knowledge bases - Verifying agents stay within defined parameters/guardrails (tone, scope, business rules, prohibited topics, compliance requirements) - Load/stress testing and regression testing as agents are updated, so we catch when a change breaks existing behavior - Setting up ongoing evaluation frameworks (not just one-time testing) so we can track quality over time and support continuous improvement - Testing integration points — APIs, CRM/data connections, voice pipeline (ASR/TTS), and any tool calls the agent makes - Documenting bugs/failure patterns clearly with reproducible steps, and recommending fixes or prompt/logic adjustments Ideal experience: - Hands-on QA experience specifically with LLM-based or conversational AI agents (not just traditional software QA) - Familiarity with voice AI platforms (e.g., Vapi, Retell, Bland, or similar) a strong plus - Experience with eval frameworks/tools for LLMs (e.g., promptfoo, Ragas, LangSmith, or custom scripted evals) - Understanding of hallucination detection, prompt engineering basics, and RAG-based grounding - Comfortable writing test scripts/automation (Python preferred) as well as manual/exploratory testing - Clear communicator who can document findings for a non-technical audience when needed