Plutus21 is seeking an AI Engineer to join our core platform team and take ownership of agent evaluation and governance. The AI Engineer will build the systems that test, approve, and monitor AI agents before and after they go live, and will ensure that governance rules are enforced in code at runtime. This role works closely with the Principal Platform Engineer, the Cloud Engineer, and product and client engagement teams, and is critical to preparing AI capabilities for use by external clients.
Key Responsibilities
• Build and maintain an evaluation framework covering datasets, metrics, agreed thresholds, and a go-live gate for each agent capability.
• Run re-evaluations whenever an agent's model or instructions change, and identify regressions before release.
• Build a governed agent runtime that assigns an accountable owner to each action, requires human approval where needed, and applies deterministic checks before any irreversible action.
• Implement rule-level and campaign-level approval for high-volume actions,
including statistically sized review samples before release and automatic suspension when quality thresholds are breached.
• Design review and evaluation workflows for AI-generated content that is sent outside the client organization.
• Ensure every model call is recorded with the model used, instruction version, data provenance, and quality checks applied.
• Keep the platform independent of any single model provider.
• Work with the Principal Platform Engineer to convert written governance rules into executable validation.
• Collaborate with product owners and client engagement teams to define capability requirements, evaluation datasets, and quality thresholds.
• Document design decisions clearly before implementation begins.
Qualifications
• Proven experience shipping tool-using, multi-step LLM agents to production environments, beyond prototypes.
• Demonstrated experience building evaluation sets and metrics, setting thresholds, an
📌 AI Engineer (Bogotá)
🏢 Plutus21 Partners
📍 Bogotá
Postulate a este anuncio
Muestra tus habilidades a la empresa, rellenar el formulario y deja un toque personal en la carta, ayudará el reclutador en la elección del candidato.