Manager - Databricks Platform Engineer

Job description

Databricks Platform Engineer Overview: Works with the Data Platform team on the design, build, and optimization of the Databricks ecosystem, including Fivetran ingestion and dbt transformation pipelines. Ensures scalable, reliable compute environments, seamless AWS integration, and automated deployments using Terraform and GitHub Actions. This role is focused on developing new capabilities in machine learning, advanced analytics, and unified data pipelines. Key Responsibilities: - Design, deploy, and optimize Databricks workspaces and clusters on AWS to support data science and modeling workloads. - Manage Fivetran connectors and dbt projects for end-to-end data ingestion, transformation, and orchestration across Databricks, S3, and Snowflake. - Automate infrastructure provisioning and configuration using Terraform integrated with GitHub Actions. - Architect and implement high availability (HA) and disaster recovery (DR) strategies, including cross-region replication and failover. - Implement job orchestration, cluster policies, autoscaling, and CI/CD workflows for notebooks, models, and releases. - Establish monitoring and alerting for cost, usage, pipeline health, data freshness,...