Data Engineer (Spark/Scala)

Job description

Location: Northampton (Hybrid 2-3 Days / Week) About the Job you are considering: This role involves designing, building, and supporting large‑scale Big Data solutions using Hadoop and Spark technologies. You will primarily develop and debug Spark jobs in Scala, with opportunities to use Java and Python where appropriate. The position focuses on creating scalable data pipelines, optimizing distributed processing, and supporting analytics and data science teams. It also offers exposure to cloud‑based Big Data platforms and modern data engineering practices in a fast‑paced, data‑driven environment. Hybrid working: The places that you work from day to day will vary according to your role, your needs, and those of the business; it will be a blend of Company offices, client sites, and your home; noting that you will be unable to work at home 100% of the time. Your Role: - Design and develop Hadoop‑based applications and scalable data pipelines. - Debug, develop, and optimize Spark jobs primarily using Scala. - Build, operate, monitor, and troubleshoot Hadoop clusters. - Develop robust ETL workflows using Spark, Hive, and Pig. - Create and manage data ingestion pipelines using Sqoop, Flu...