We are seeking a Senior Data Engineer with proven expertise in Azure Data Platform technologies, Databricks, and PySpark to strengthen our growing data engineering organization.

In this position, you will craft and deliver scalable, cloud-native data solutions that drive enterprise analytics, reporting, and business intelligence efforts.

You will manage large-scale datasets, develop reusable ingestion frameworks, refine data pipelines, and play a key role in evolving modern data platforms built on Azure and Databricks.

The right candidate pairs strong hands-on engineering capabilities with a background in designing robust, metadata-driven data solutions, along with a genuine drive to deliver high-quality, maintainable data products.

Responsibilities

Design, build, and sustain scalable data ingestion, transformation, and delivery pipelines

Deliver cloud-native data solutions using Azure and Databricks technologies

Create and refine ETL/ELT processes with PySpark, Spark SQL, and Delta Lake

Apply metadata-driven and configuration-based ingestion frameworks to boost scalability and maintainability

Build and sustain batch and incremental data processing pipelines

Set up data quality controls, validation frameworks, monitoring, logging, and error handling mechanisms

Work closely with business stakeholders, architects, and engineering teams to capture requirements and deliver data solutions

Address production issues and push forward continuous improvements in platform reliability and performance

Track and fine-tune Databricks workloads for efficiency, scalability, and cost management

Uphold engineering best practices including code reviews, automated testing, CI/CD, and infrastructure automation

Contribute to data governance, security, and access management initiatives using Unity Catalog and Azure services

Guide team members and shape engineering standards and best practices





Requirements3+ years of experience in Data Engineering or Software Engineering

Expertise in Azure Databricks, Apache Spark, and PySpark

Proficiency in Spark SQL and a track record of building solutions on Azure cloud platforms

Advanced Python programming skills paired with strong SQL development and optimization experience

Background in Unity Catalog and enterprise data governance practices

Familiarity with Git-based source control workflows

Experience designing and maintaining CI/CD pipelines

Knowledge of orchestration tools such as Databricks Workflows and Azure Data Factory (ADF)Strong analytical, troubleshooting, and problem-solving skills

Excellent communication and stakeholder management abilities

English proficiency at B2 level or higher

Nice to have

Knowledge of Terraform and Infrastructure as Code (IaC)Experience with Databricks Asset Bundles

Skills in building reusable data engineering frameworks and accelerators

Expertise in advanced Spark performance tuning and optimization

Familiarity with enterprise-scale lakehouse architectures

Exposure to data security, compliance, and governance initiatives

Experience working within Agile delivery teams

We offer

International projects with top brands

Work with integral teams of highly skilled, diverse peers

Healthcare benefits

Employee financial programs

Paid time off and sick leave

Upskilling, reskilling and certification courses

Unlimited access to the Linked

In Learning library and 22,000+ courses

Global career opportunities

Volunteer and community involvement opportunitiesEPAM Employee Groups

Award-winning culture recognized by Glassdoor, Newsweek and Linked

InEPAM is an Equal Opportunity Employer.

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, sexual orientation, gender identity or expression, disability, protected veteran status, or any other characteristic protected by applicable law. #J-18808-Ljbffr

📌 Senior Data Engineer (Colombia)
🏢 EPAM Systems
📍 Colombia