Senior Data Engineer

Job description

Job details The Department of Agriculture, Fisheries and Forestry (DAFF) is looking for a Data Engineer to join the Digital Transformation Program in theAustralian Bureau of Agricultural and Resource Economics and Sciences(ABARES) to work across several data and analytics platforms. The candidate will develop and optimise data pipelines in Azure Databricks , with a strong focus on Python and SQL . The candidate will have expertise in Azure Data Factory, Azure DevOps, CI/CD , and Git version control , as well as a deep understanding of Kimball dimensional modelling and Medallion architecture . This role requires strong collaboration skills to translate business requirements into effective technical solutions. Key duties and responsibilities: Key Responsibilities: - Develop, optimise, and maintain data pipelines using Python and SQL within Azure Databricks Notebooks . - Design and implement ETL/ELT workflows in Azure Data Factory , ensuring efficient data transformation and loading. - Apply Kimball dimensional modelling and Medallion architecture best practices for scalable and structured data solutions. - Collaborate with team members and business stakeholders to understand data req...