Data Engineer

Job description

Position Summary: An experienced data engineer with strong Azure Databricks expertise is required for a 6 month contract to develop and deliver a scalable data pipeline that ingests files and transforms them into standardised, high-quality datasets for reporting. This is a hands-on delivery role focused on building a production-grade pipeline, suited to someone comfortable working independently, defining technical solutions, and rapidly delivering high-quality data engineering components. Location: London - UK - Fenchurch What will your job entail?: Key Responsibilities: - Implement a scalable ETL pipeline in Azure Databricks, that transforms raw data into standardised schemas to support reporting and analytics, following a medallion architecture in line with internal design standards. - Leverage an existing email ingestion tool to extract multiple format source files from a shared mailbox. - Develop notebooks, jobs and pipelines in Databricks for ETL orchestration. - Design and implement robust validation rules, monitor pipeline runs and troubleshoot failures, with logging and notifications to support robust data processes. - Integrate with CI/CD flows across environments using Az...