Principal Software Engineer - CoreAI Model Inference & Serving

Job description

Overview Join our team within CoreAI , where we are building the AI data-plane that powers all LLM inferencing workloads across Microsoft and Azure customers—from cutting-edge startups to Fortune 500 enterprises. Our converged AI fabric delivers inference capabilities for all LLMs in Microsoft catalog , including OpenAI, Anthropic, Mistral, Cohere, Llama, and more. As a Principal Software Engineer , you will shape the future of one of the largest and fastest-growing services in Azure, foundational to Microsoft’s AI strategy. Our mission is to serve models at scale—reliably, efficiently, and with ultra-low latency—enabling a rich set of AI-powered product experiences. This is a rapidly evolving space with immense opportunities to learn, innovate, and drive industry-wide impact! Responsibilities: - Be a hands-on technical leader, designing, coding, and shipping core serving systems, smart routing, and request distribution for a broad portfolio of LLMs, including OpenAI, Mistral, Grok, DeepSeek, and others. - Build large-scale AI services and platform capabilities that power new products and customer experiences. - Drive cutting-edge innovation in AI systems alongside world-class engi...