Sr Data Engineer, Python + Spark (Data Federation skillset - Data Lakehouse - Eg: Starburst) - New York

Job description

Senior Data Engineer (Data Federation & Lakehouse) As a Senior Data Engineer , you will be responsible for breaking down data silos. This role focuses on building a unified, high-performance data layer using Data Federation techniques. You won't just move data; you will architect a Data Lakehouse environment where disparate sources feel like a single, cohesive database for our analytics and AI teams. ### Core Responsibilities ### Technical Requirements ### Preferred "Bonus" Skills Background in Data Mesh or Data Fabric methodologies. - Data Federation Architecture: Design and implement federated query layers (e.g., Starburst/Trino ) to allow high-speed analytics across distributed data sources without unnecessary data movement. - ETL/ELT Pipeline Development: Build scalable, distributed data processing pipelines using Python and Apache Spark (PySpark). - Lakehouse Implementation: Manage and optimize modern table formats like Delta Lake , Apache Iceberg , or Hudi to bring ACID transactions to our data lake. - Performance Tuning: Optimize Spark jobs and SQL queries across the federation layer to minimize latency and manage compute costs. - Governance & Security: Implement fine-graine...