Lead Data Engineer

Job description

Become a part of our caring community We are seeking an experienced and strategic Lead Data Engineer to join our ITSM Applied AI and Analytics team. This individual will lead the development and evolution of our enterprise-facing data store built on Azure Databricks, driving architectural decisions, mentoring team members, and ensuring scalable, high-quality data solutions across the organization. Key Responsibilities: Lead the design, development, and optimization of ETL/ELT pipelines using Databricks, Apache Spark, Unity Catalog, and Delta Lake. Architect data workflows following Medallion Architecture (Bronze, Silver, Gold layers). Guide the integration of external APIs and systems, including ServiceNow and Kafka for real-time data streaming. Establish and enforce best practices in CI/CD, GitHub workflows, and code quality. Manage compute appropriately to ensure cost control. Collaborate with stakeholders to define data strategy, governance, and platform enhancements. Provide technical leadership and mentorship to data engineers and analysts. Conduct code reviews and ensure adherence to engineering standards. Communicate effectively with technical and non-technical stakeholders....