Resident Solution Architect Databricks

Job description

- Design and implement large-scale Azure and Databricks Data Lakehouse solutions. - Build and optimize ETL/ELT pipelines for batch and real-time streaming workloads using Azure Data Factory, Databricks, and Apache Spark. - Develop scalable data ingestion frameworks and integrate diverse structured and unstructured data sources. - Optimize data storage and query performance using Delta Lake, Parquet format, partitioning, and Spark performance tuning techniques. - Implement robust data governance, security, and access control using Unity Catalog, Azure Key Vault, and least-privilege principles. - Build and maintain data quality frameworks using Great Expectations or similar validation tools for batch and streaming data. - Automate pipeline deployment and CI/CD processes using Azure DevOps, Git, and configuration management tools. - Enable advanced analytics and ML workflows by preparing curated datasets and integrating with ML platforms like MLflow. - Develop real-time streaming solutions using Kafka, Azure Event Hubs, and Databricks Structured Streaming. - Create reusable PySpark libraries and frameworks for data curation, reconciliation, notifications, and Delta table automation. -...