Senior Data Engineer

Job description

SUMMARY OF THE JOB: We are seeking a seasoned Senior Data Engineer to design, build, and optimize our next-generation data platform. You will be responsible for architecting scalable data pipelines, managing large-scale distributed systems, and ensuring our data infrastructure in AWS and Databricks is robust and efficient. The ideal candidate is a Spark expert with a deep understanding of the AWS ecosystem and a passion for automation. MAIN ACTIVITIES / RESPONSIBILITIES: Pipeline Architecture: Design and implement complex batch and streaming ETL/ELT pipelines using Python, SQL, and Spark to process massive datasets. Cloud Infrastructure: Leverage AWS Data Analytics services to build scalable, secure, and cost-effective data solutions. Orchestration & DevOps: Manage and automate data workflows using Airflow , while utilizing Docker and ECS for containerized application deployment. System Optimization: Monitor and tune the performance of distributed systems (Spark Cluster) to ensure high availability and low latency. Infrastructure as Code: Utilize AWS CloudFormation or Terraform to manage data infrastructure, ensuring repeatable and version-controlled environments. Cost Optimization...