Description

-
- Design, build, and maintain end-to-end data pipelines for ingestion, transformation, and delivery of large-scale data.
- Develop and optimize data processing logic using PySpark on Databricks (Apache Spark).
- Implement ETL/ELT pipelines integrating data from multiple structured and semi-structured sources.
- Contribute to the design and implementation of lakehouse architectures (Delta Lake, Medallion architecture).
- Ensure data quality, reliability, performance, and observability across pipelines.
- Optimize Spark jobs through partitioning, caching, and performance tuning techniques.
- Collaborate with data architects, analysts, and business stakeholders to translate requirements into scalable data solutions.
- Implement best practices in CI/CD, version control, and pipeline automation.
- Support the evolution of modern data platforms and analytics capabilities.
- Work with orchestration tools (ADF, Airflow, or similar) to manage workflows.
- Communicate technical solutions and trade-offs effectively to both technical and non-technical stakeholders.





Qualifications

-
-
- BS in Computer Science, Engineering, or related technical field, or equivalent practical experience.
- 4+ years of experience in Data Engineering or similar roles.
- Strong experience with Databricks and PySpark for large-scale data processing.
- Solid knowledge of Spark architecture and performance optimization.
- Strong proficiency in Python and SQL.
- Experience designing and building data pipelines and ETL/ELT processes.
- Understanding of modern data architectures (lakehouse, distributed systems, big data processing).
- Experience with version control (Git) and CI/CD practices.
- Familiarity with cloud platforms (Azure preferred, AWS also acceptable).
- Experience with orchestration tools (Azure Data Factory, Airflow, Fabric).
- Strong communication skills and ability to work with cross-functional teams.
- Advanced English (written and spoken).
- Proactive, analytical mindse

📌 Senior Data Engineer (Colombia)
🏢 Perficient
📍 Colombia