Owns the eval harness and quality gate from the beginning. This role replaces the old late‑stage “Evals Specialist” model with a standing owner for measurable agent quality.

Key Responsibilities

- Build and maintain the MVP eval harness: golden tasks, exception tasks, scorecard metrics, and regression packs.

- Wire evals into CI so quality regressions fail builds and releases.

- Define and maintain release‑gate thresholds with Product and the Tech Lead.

- Lay the path for later adversarial and drift‑testing expansion without overbuilding MVP scope.

Requirements

- Experience evaluating ML, LLM, or non‑deterministic systems.

- Strong test and benchmark design capability.

- Comfort working with noisy metrics, thresholds,



and probabilistic behavior.

- Good scripting and automation skills.

- Uses AI to generate candidate eval cases and failure hypotheses, but never confuses generated tests with validated quality.

- Approaches AI quality as an operating system, not a QA afterthought.

What Success Looks Like in the First 90 Days

- The first reference agent has a published scorecard and gated eval path.

- Golden and exception tests run automatically.

- The team can explain what “good enough to ship” means in measurable terms.

📌 Agent Quality / Evals Engineer 1754 (Colombia)
🏢 Softgic S.
📍 Colombia