Postúlate en Kit Empleo: kitempleo.com.co/empleo/1at2ba

Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.
Role Overview
We are looking for highly analytical engineers and technical domain experts to contribute to advanced AI evaluation and benchmarking projects focused on realistic terminal-based and infrastructure-heavy workflows. In this role, you will design technically challenging tasks that evaluate how AI systems reason through debugging, operational failures, complex workflows, and multi-step problem-solving scenarios.
The idóneo candidate has strong experience working with production systems, debugging, automation, or large-scale engineering workflows, and can design realistic technical challenges that simulate real-world engineering environments.
This role is particularly well suited for professionals with backgrounds in backend engineering, infrastructure, DevOps, data systems, MLOps,



cybersecurity, or platform engineering.
CONTRACT: Contractor assignment (5 weeks)
COMMITMENT: Full-time (40h/week) or Part-time (20h/week) with minimum 4h PST overlap
LOCATION: Remote — Bangladesh, Brazil, Colombia, Egypt, Ghana, India, Pakistan, Indonesia, Kenya, Nigeria, Turkey, Vietnam
PROCESS: One technical assessment/interview (~45 min)
Responsibilities:
Design realistic terminal-based benchmark tasks for AI evaluation systems
Create technically deep debugging and investigation scenarios
Develop task specifications involving infrastructure, workflows, pipelines, or operational failures
Write clear solution approaches and deterministic evaluation criteria
Identify realistic edge cases, failure modes, and system constraints
Design multi-step reasoning challenges across complex technical environments
Contribute expertise across one or more engineering or operational domains
Review and refi

Postúlate en Kit Empleo: kitempleo.com.co/empleo/1at2ba

📌 AI Evaluation Engineer (Colombia)
🏢 Importante Empresa Del Sector
📍 Colombia