Job Description

Insight Integral is looking for a senior data engineer to design, build, and maintain production-grade batch and streaming pipelines using Databricks, Lakeflow Declarative Pipelines, and dbt for our client. This person will participate in sprint planning, standups, retrospectives, and backlog grooming. You will monitor and troubleshoot pipeline reliability, performance, and cost, implement centralized data governance and access controls with Unity Catalog and tools such as Immuta, and build DataOps standards, CI/CD automation, and automated data-quality controls. You will be responsible for partnering with cross-functional stakeholders to deliver curated, analytics-ready datasets and data models, and review pipeline designs and code while guiding mid-level and junior engineers. Lastly, you will evaluate emerging Databricks capabilities, including Lakeflow, AI/BI, and SQL AI Functions and recommend improvements to platform architecture and engineering practices. This is a 3 month contract with yearly anticipated extensions remote in Brazil or Colombia.

Skills and Requirements

- 7+ years of data engineering and architecture experience
- Deep, hands-on production experience with Databricks and modern lakehouse platforms
- Strong Apache Spark experience, including PySpark and Spark SQL
- Experience building and operating scalable batch and streaming pipelines
- Experience with declarative pipeline frameworks, including Lakeflow Declarative Pipelines
- Strong experience with dbt, Databricks Workflows,



and Apache Airflow for data transformation and orchestration
- Advanced SQL and strong Python programming skills
- Solid understanding of lakehouse architecture, medallion/layered data modeling, and data warehousing
- Production experience with Databricks Unity Catalog
- Cloud Experience (AWS, Azure, GCP)
- Scala or Java experience
- Databricks Certified Data Engineer Professional certification
- Confluent Kafka and Fivetran experience
- Immuta experience alongside Unity Catalog
- Alation or similar data catalog and lineage tooling
- Declarative Automation Bundles
- Photon, Liquid Clustering, and Databricks Serverless compute

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process,

#J-18808-Ljbffr

📌 Médico General – Consulta Externa, Jornada Tarde (San Gil) (Colombia)
🏢 Fundacion Avanzar Fos
📍 Colombia