Job description

Overview Microsoft Silicon and Cloud Hardware Infrastructure Engineering (SCHIE) is the team behind Microsoft’s expanding Cloud Infrastructure and responsible for powering Microsoft’s “Intelligent Cloud” mission. CHIE delivers the core infrastructure and foundational technologies for Microsoft's over 200 online businesses including Bing, MSN, Office 365, Xbox Live, Skype, OneDrive and the Microsoft Azure platform globally with our server and data center infrastructure, security and compliance, operations, globalization, and manageability solutions. Our focus is on smart growth, high efficiency, and delivering a trusted experience to customers and partners worldwide and we are looking for passionate, high-energy engineers to help achieve that mission.    As Microsoft's cloud business continues to grow the ability to deploy new offerings and HW infrastructure on time, in high volume with high quality and lowest cost is of paramount importance. To achieve this goal, the Silicon Cloud Hardware Infrastructure Engineering (SCHIE) team is instrumental in defining and delivering measures of success for hardware design, qualification, fleet support, scale, and sustainability related to Microsoft cloud hardware.  Azure Memory and Storage Center of Excellence (AMS CoE) is part of the SCHIE organization focusing on Memory and Storage devices going into the Cloud hardware servers. AMS provide memory and storage solutions to Azure, drive memory and storage suppliers to deliver high quality products, meeting our requirements.   Every hour a server sits unhealthy is an hour of lost customer capacity. We build the software that catches hardware failures before they take a node down and repairs the ones that do, without a human in the loop. Production AI agents reasoning over fleet telemetry, the data platform beneath them, and the tooling engineers use to ship them safely. Across millions of nodes. We are looking for a Senior Cloud Hardware Engineer to scale Azure’s Fault Self‑Healing and Failure Prediction systems . Responsibilities What you'll do • Build the platform behind Azure's fault self-healing and failure-prediction system with telemetry pipelines, prediction services, decision logic, and automated repair workflows. • Ship AI agents to production: prompt and tool design, retrieval over diagnostics data, evaluation harnesses, guardrails, and the CI/CD path that deploys new agent skills safely. • Close the loop safely: automated remediation with staged rollout, blast-radius limits, and verification. • Own the developer experience: SDKs, APIs, and dashboards that let engineers across the org author and deploy new detection and repair logic themselves. • Build and monitor measurements: prediction precision/recall, false-repair rate, action success rate, and regression gates that block a bad model from shipping. What you'll bring • 8+ years building and operating production software, distributed services, data platforms, or large-scale automation. • Python and C# (or C++/Rust), plus cloud-scale data pipelines at high volume. • AI/ML in production, not just experimentation: agent or model serving, evaluation, versioning and rollback, drift and regression monitoring. • LLM application patterns: agent/tool-calling, RAG, structured output and judgment about when an LLM is the wrong answer. • A track record of automation that takes real actions on real infrastructure, with the safety engineering that requires. • Bonus: anomaly detection on time-series data; Kusto/SQL; server hardware, firmware, or datacenter operations. Why this role • Your impact shows up in fleet availability metrics and customer uptime, quarter over quarter. The AI agents you ship take autonomous action on production infrastructure at a scale very few teams can offer. Qualifications • Required/minimum qualifications • Master's Degree in Electrical Engineering, Computer Engineering, Mechanical Engineering, or related field AND 3+ years technical engineering experience OR Bachelor's Degree in Electrical Engineering, Computer Engineering, Mechanical Engineering, or related field AND 5+ years technical engineering experience OR equivalent experience • Other Requirements: • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings: • Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter. • Preferred Qualifications: • • Bachelor's Degree in Computer Engineering, Computer Science, Electrical Engineering, or related field AND 8+ years of firmware or embedded systems engineering experience OR Master's Degree in Computer Engineering, Computer Science, Electrical Engineering, or related field AND 6+ years of firmware or embedded systems engineering experience OR equivalent experience • 6+ years developing SSD or storage device firmware, including 4+ years working directly with NVMe and PCIe protocols • Demonstrated depth in storage device resiliency and fault analysis — failure mode characterization, error handling and recovery paths, and root-cause investigation of field failures • Experience supporting live-site operations for storage at fleet scale, including on-call ownership and production incident resolution • Track record of owning end-to-end technical design across the full reliability lifecycle: detection, prediction, mitigation, and repair • Proven experience building automation-heavy systems that operate safely at hyperscale, with the guardrails, staged rollout, and blast-radius controls that requires Hardware Engineering IC4 - The typical base pay range for this role across the U.S. is USD $119,800 - $234,700 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $160,200 - $261,000 per year. Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: careers.microsoft.com This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled. Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.

Requirements

Must-haves

We are looking for a Senior Cloud Hardware Engineer to scale Azure’s Fault Self‑Healing and Failure Prediction systems .
Build the platform behind Azure's fault self-healing and failure-prediction system with telemetry pipelines, prediction services, decision logic, and automated repair workflows.
Ship AI agents to production: prompt and tool design, retrieval over diagnostics data, evaluation harnesses, guardrails, and the CI/CD path that deploys new agent skills safely.
Close the loop safely: automated remediation with staged rollout, blast-radius limits, and verification.
Own the developer experience: SDKs, APIs, and dashboards that let engineers across the org author and deploy new detection and repair logic themselves.
Build and monitor measurements: prediction precision/recall, false-repair rate, action success rate, and regression gates that block a bad model from shipping.
8+ years building and operating production software, distributed services, data platforms, or large-scale automation.
Python and C# (or C++/Rust), plus cloud-scale data pipelines at high volume.
AI/ML in production, not just experimentation: agent or model serving, evaluation, versioning and rollback, drift and regression monitoring.
LLM application patterns: agent/tool-calling, RAG, structured output and judgment about when an LLM is the wrong answer.
A track record of automation that takes real actions on real infrastructure, with the safety engineering that requires.
Why this role
Your impact shows up in fleet availability metrics and customer uptime, quarter over quarter. The AI agents you ship take autonomous action on production infrastructure at a scale very few teams can offer.
Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter.
6+ years developing SSD or storage device firmware, including 4+ years working directly with NVMe and PCIe protocols
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.

Nice-to-haves

Bonus: anomaly detection on time-series data; Kusto/SQL; server hardware, firmware, or datacenter operations.
Demonstrated depth in storage device resiliency and fault analysis — failure mode characterization, error handling and recovery paths, and root-cause investigation of field failures
Experience supporting live-site operations for storage at fleet scale, including on-call ownership and production incident resolution
Track record of owning end-to-end technical design across the full reliability lifecycle: detection, prediction, mitigation, and repair
Proven experience building automation-heavy systems that operate safely at hyperscale, with the guardrails, staged rollout, and blast-radius controls that requires
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

azureaici/cdpythonrustmlllmsql

Platform: eightfold
Effort: medium