Software Development Manager - AWS Glue, Glue and GenAI for Data Processing
Amazon.com · East Palo Alto, California, US
Do you enjoy working on large-scale distributed systems and Agentic AI ? The team is responsible for the Apache Spark Upgrade Agent and broader SQL Migration...
Job description
Do you enjoy working on large-scale distributed systems and Agentic AI ? The team is responsible for the Apache Spark Upgrade Agent and broader SQL Migration Agents for Analytics services (Glue, EMR, Athena), the Remote MCP Service (SageMaker Unified Studio MCP), and the AWS Glue Python Service - Product and Runtime. The Remote MCP service is a stepping stone to building broader Migration Agents in the next 2-3 years to enable customers migrate their SQL workloads to AWS Analytics services (Athena, SparkSQL and Redshift). Apache Spark Upgrade Agent[1], is a new generative Agentic AI capability that enables customers to upgrade their Apache Spark applications to newer versions. Spark upgrades traditionally require 6-12 months of engineering effort for customers due to analyzing API changes, resolving dependency conflicts, and validating functional correctness across hundreds of enterprise applications. Apache Spark Upgrade Agent, reduces upgrade timelines from months to weeks through conversational interfaces, automated code transformation, and rigorous data quality validation, allowing engineers to use natural language for upgrades while maintaining approval control over changes an...