AgileEngine, LLC. is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake while applying a medallion architecture for analytics and ML workloads.
You will collaborate with cross-functional teams, optimize Spark jobs for performance and cost, and drive governance and data quality across the platform using Unity Catalog and automated checks.