At Microsoft, we’re a community of passionate innovators driven by curiosity and purpose. We collaborate to imagine what’s possible and accelerate our careers in a cloud-powered world where openness and innovation unlock limitless potential.
Artificial Intelligence is central to Microsoft’s strategy—and the Azure AI Platform is leading the charge. As part of our team, you’ll contribute to cutting-edge projects that solve real-world challenges using transformative technologies.
We are looking for a Senior Software Engineer to join our fast-paced, agile team at the core of Microsoft’s AI infrastructure. This team builds and operates the Next Generation Traffic Shaping, Scheduling, and Optimization Platform—a foundational control-plane service responsible for intelligent routing, capacity allocation, quota enforcement, and workload optimization for OpenAI models and other large-scale AI workloads across Azure.
In this role, you will help define and build the systems that determine how AI traffic is routed and served globally. You will work on the Traffic Shaper, the critical platform that dynamically balances demand across thousands of AI endpoints, optimizes utilization of premium AI accelerators, and ensures reliable, low-latency experiences for Microsoft Copilot, Azure OpenAI Service, and strategic enterprise customers.
You will design and develop large-scale distributed systems responsible for:
Intelligent request routing and workload placement across global regions.
Capacity management and allocation across models, tenants, and customer offerings.
Real-time traffic shaping, prioritization, and load balancing.
Quota management, throttling, and fairness enforcement.
Fleet-wide optimization to maximize GPU utilization and service efficiency.
High-fidelity telemetry, observability, and automated operational controls.
Resiliency, failover, and incident mitigation mechanisms for mission-critical AI services.
As a senior engineer, you will provide technical leadership across projects, drive architecture decisions, influence platform strategy, and partner closely with teams across Azure AI, OpenAI, Infrastructure, Networking, and Core AI Services. You will own services operating at global scale and help shape the future of Microsoft's AI infrastructure.
astructure.
Impact
This team sits on the critical path of Microsoft's AI strategy. The systems you build will directly influence how capacity is allocated, how traffic is routed, and how efficiently some of the world's largest AI workloads are served. Your work will help power Azure OpenAI Service, Microsoft Copilot experiences, and the next generation of AI products used by millions of customers worldwide.
Responsibilities
Design and implement highly scalable, reliable, and performant distributed systems that power Azure AI traffic management and scheduling.
Drive architecture and technical direction for routing, capacity allocation, and workload optimization services.
Develop algorithms and control-plane systems that intelligently distribute AI workloads across global infrastructure while meeting latency, availability, and cost objectives.
Collaborate with partner teams to onboard new models, regions, and customer offerings while maintaining operational excellence.
Build observability and operational tooling to monitor service health, capacity utilization, traffic patterns, and customer impact in near real-time.
Lead investigations into complex production issues and drive systematic improvements to reliability, resiliency, and platform automation.
Identify and execute opportunities to improve fleet efficiency, reduce operational overhead, and optimize utilization of premium AI infrastructure.
Mentor engineers, review designs and code, and raise engineering standards across the organization.
Drive engineering excellence through testing, automation, documentation, and operational best practices.
Influence long-term platform strategy and contribute to roadmap planning for next-generation AI infrastructure.
Qualifications
Required:
Preferred:
3+ years of experience in software engineering, preferably in distributed systems or cloud infrastructure.
Strong coding skills in C#, Python, or Go.
Experience with telemetry, metrics pipelines, or resource scheduling systems.
Familiarity with cloud platforms (Azure, AWS, GCP) and container orchestration (Kubernetes, Service Fabric).
Experience operating large-scale services with stringent reliability and availability requirements.
Experience with distributed storage systems, telemetry platforms, and real-time data processing.
Familiarity with AI/ML infrastructure, GPU fleets, inference serving systems, or large-scale model deployment platforms.
Experience leading cross-team initiatives and driving projects from design through production deployment and operation.
#AIPLATFORM, #AzureOpenAI, #llm, #distributedsystems, #MicrosoftAI, #AICoreServices
Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.
Software Engineering IC4 - The typical base pay range for this role across the U.S. is USD $119,800 - $234,700 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $160,200 - $261,000 per year.
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:
https://careers.microsoft.com/us/en/us-corporate-pay
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.
Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.
Every company has a mission. What's ours? To empower every person and every organization to achieve more. We believe technology can and should be a force for good and that meaningful innovation contributes to a brighter world in the future and today. Our culture doesn’t just encourage curiosity; it embraces it. Each day we make progress together by showing up as our authentic selves. We show up with a learn-it-all mentality. We show up cheering on others, knowing their success doesn't diminish our own. We show up every day open to learning our own biases, changing our behavior, and inviting in differences. Because impact matters.
Microsoft operates in 190 countries and is made up of approximately 228,000 passionate employees worldwide.
