Data Engineer

Job description

The AI & Data Analytics Team is looking for a Senior Data Engineer to join our team . In this role, you will be responsible for designing, building, and optimizing robust data pipelines that process massive datasets in both batch and real-time. You will work at the intersection of software engineering and data science, ensuring that our data architecture is scalable, reliable, and follows industry best practices. Priorities can change in a fast-paced environment like ours, so this role includes, but is not limited to, the following responsibilities: - Pipeline Development: Design and implement complex data processing pipelines using Apache Spark. - Architectural Leadership: Build scalable, distributed systems that handle high-throughput data streams and large-scale batch processing. - Infrastructure as Code: Manage and provision cloud infrastructure using Terraform. - CI/CD & Automation: Streamline development workflows by implementing and maintaining GitHub Actions for automated testing and deployment. - Code Quality: Uphold rigorous software engineering standards, including comprehensive unit/integration testing, code reviews, and maintainable documentation. - Collaboration: Work...