Data Engineer - Hybrid / Remote

Job description

Data Engineer - Hybrid / Remote Opportunity This role requires a highly technical Data Engineer with expert-level proficiency in Azure Databricks, distributed data pipelines, and large-scale healthcare data processing. This role focuses on designing and implementing high-throughput ingestion pipelines, transactional lakehouse layers, and secure PHI data flows using Azure-native services and Databricks runtime optimizations. You will build and operate production-grade data pipelines that meet rigorous requirements for security, lineage, compliance (HIPAA), observability, and operational SLAs, supporting analytics, AI, and clinical insights across the organization. - Hybrid for candidates in Nashville and surrounding areas. - Remote option available for candidates outside of surrounding areas. Core Responsibilities: Platform & Architecture: - Architect and implement scalable data processing pipelines using: - Databricks Runtime (Apache Spark, Spark SQL, MLflow, Delta Lake) - Delta Lake ACID transactions, Z-Ordering, OPTIMIZE, and Change Data Feed (CDF) - Unity Catalog for governance, lineage, RBAC, and audit controls - Design and enforce a medallion (Bronze/Silver/Gold) architecture...