QA Lead (Automation+Performance)- Dallas, TX

Job description

We are seeking a QA Automation Lead who is ready to move beyond traditional "Pass/Fail" testing. In this role, you will design and build automation frameworks specifically for Agentic AI products . You will focus on evaluating the performance of autonomous agents, ensuring they follow logical reasoning paths, call the correct tools, and provide accurate, safe outputs. Your mission is to build the "evaluations" (Evals) that define what high-quality AI behavior looks like, moving the needle from unpredictable experiments to production-grade software. Key Responsibilities: - Non-Deterministic Testing: Develop automation strategies for probabilistic outputs, using model-based evaluation to "test the tester." - Building "Eval" Pipelines: Create and maintain "Golden Datasets" to benchmark agent performance across different versions of prompts and models. - Tool-Use Validation: Build automated tests to verify that agents call the correct functions/APIs with the right parameters in complex multi-step workflows. - Regression Testing for Prompts: Monitor how subtle changes in prompt engineering or model updates (e.g., moving from GPT-4 to Claude 3.5) affect the product’s reliability. - Laten...