Job description

AWS runs the world's largest fleet of AI/ML accelerator servers. When a model with billions of parameters trains across a large scale of GPUs, every minute of downtime costs real progress. We are building the automation, diagnostics, and predictive intelligence that keeps this fleet running at peak. If you want to work at the intersection of hardware, software, and scale — where your code directly prevents customer-impacting failures — this is the role. We are seeking a Systems Development Engineer to build automation software, diagnostic tooling, and fleet health infrastructure for our accelerated compute platforms. You will work across multiple teams and organizations to design scalable, reliable systems for our accelerated compute fleet. What You Will Do You will tackle problems no one has fully defined yet — spanning hardware, firmware, kernel, and software simultaneously. You will own systems end to end, writing code that prevents failures rather than reacts to them, and building automation that replaces manual toil with intelligent self-healing. You will work across PCIe topology, GPU diagnostics, Linux drivers, and telemetry pipelines to correlate signals and isolate faults at fleet scale. When your system catches a failing GPU before a training job crashes, that is your impact. Why You Will Love It Your automation runs at a large scale across servers in the cloud. When you ship, you see failure rates move within days. The team is small enough that your decisions shape the architecture, and large enough that you will always have experts to learn from across hardware, firmware, and software. The Ideal Candidate You know the full stack from bare-metal to userland. You debug at the intersection of components, not just within them. You build at cloud scale and care how your systems decisions impact customers. You are an excellent communicator who can drive alignment across hardware, software, and operations teams. Key job responsibilities Fleet Health & Predictive Infrastructure 1. Build and own the automation infrastructure for accelerator (AI/ML) fleet health at a large scale of servers, driving toward zero-touch operations that detect, diagnose, triage, and remediate faults without human intervention 2. Design and develop test frameworks, test coverage strategies, and diagnostic tooling to validate hardware functionality, detect faults, and ensure qualification coverage across the platform lifecycle. 3. Design predictive failure detection using telemetry, sensor data, error trending, and log correlation to identify degrading components before customer impact 4. Develop monitoring dashboards and alerting for real-time fleet health visibility across manufacturing, lab, and production environments 5. Define and track fleet health metrics: failure rates, mean time to detect and resolve issues, first-time fix rate, test dwell time, and predictive accuracy Debugging & Troubleshooting 1. Debug complex system-level issues across compute, GPU, and networking in production — including Linux boot/runtime failures, PCIe, power, NIC, NVMe, and GPU subsystems on x86 and ARM 2. Perform root cause analysis correlating across firmware, kernel, driver, and physical layer; feed findings into manufacturing quality and design improvements Systems Development & Automation 1. Design scalable test automation for hardware bring-up, regression, and qualification — reducing manufacturing test cycle times without sacrificing coverage through intelligent test sequencing and parallel execution 2. Build data pipelines correlating test results, sensor telemetry, and component-level data to identify systemic yield issues and drive upstream fixes 3. Develop and maintain Linux device drivers on ARM and x86; work with OS internals and accelerator/GPU software stacks 4. Build and manage tests covering all functional aspects of the system and CI/CD pipelines for rapid deployment to manufacturing lines and production fleet Cross-Team Collaboration 1. Work across engineering teams and internal customers to ensure new accelerated compute hardware meets data path, control path, and onboarding requirements 2. Engage with ODMs and design partners on testability, diagnostic coverage, and automation requirements during hardware design and bring-up phases — influencing functional and performance readiness of the platform 3. Partner with datacenter operations to close the loop between field failures, manufacturing escapes, and design improvements Operational Excellence 1. Participate in post-incident reviews, identify contributing causes and drive permanent fixes that eliminate whole classes of risk 2. Produce clear, maintainable documentation for systems, runbooks, and automation to enable others to operate and extend your work 3. Drive process improvements that increase team agility — reducing development friction, eliminating unnecessary gates, and improving delivery velocity May require occasional ( amazon.jobs for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner. The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience, qualifications, and location. Amazon also offers comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage), 401(k) matching, paid time off, and parental leave. Learn more about our benefits at amazon.jobs . USA, CA, Cupertino - 148,700.00 - 201,200.00 USD annually USA, TX, Austin - 129,200.00 - 174,800.00 USD annually USA, WA, Seattle - 129,200.00 - 174,800.00 USD annually

Requirements

Must-haves

Why You Will Love It
The Ideal Candidate
Key job responsibilities
Fleet Health & Predictive Infrastructure
Design and develop test frameworks, test coverage strategies, and diagnostic tooling to validate hardware functionality, detect faults, and ensure qualification coverage across the platform lifecycle.
Design predictive failure detection using telemetry, sensor data, error trending, and log correlation to identify degrading components before customer impact
Develop monitoring dashboards and alerting for real-time fleet health visibility across manufacturing, lab, and production environments
Define and track fleet health metrics: failure rates, mean time to detect and resolve issues, first-time fix rate, test dwell time, and predictive accuracy
Debugging & Troubleshooting
Debug complex system-level issues across compute, GPU, and networking in production — including Linux boot/runtime failures, PCIe, power, NIC, NVMe, and GPU subsystems on x86 and ARM
Perform root cause analysis correlating across firmware, kernel, driver, and physical layer; feed findings into manufacturing quality and design improvements
Systems Development & Automation
Design scalable test automation for hardware bring-up, regression, and qualification — reducing manufacturing test cycle times without sacrificing coverage through intelligent test sequencing and parallel execution
Build data pipelines correlating test results, sensor telemetry, and component-level data to identify systemic yield issues and drive upstream fixes
Develop and maintain Linux device drivers on ARM and x86; work with OS internals and accelerator/GPU software stacks
Build and manage tests covering all functional aspects of the system and CI/CD pipelines for rapid deployment to manufacturing lines and production fleet
Cross-Team Collaboration
Work across engineering teams and internal customers to ensure new accelerated compute hardware meets data path, control path, and onboarding requirements
Engage with ODMs and design partners on testability, diagnostic coverage, and automation requirements during hardware design and bring-up phases — influencing functional and performance readiness of the platform
Partner with datacenter operations to close the loop between field failures, manufacturing escapes, and design improvements
Operational Excellence
Participate in post-incident reviews, identify contributing causes and drive permanent fixes that eliminate whole classes of risk
Produce clear, maintainable documentation for systems, runbooks, and automation to enable others to operate and extend your work
Drive process improvements that increase team agility — reducing development friction, eliminating unnecessary gates, and improving delivery velocity

Nice-to-haves

None flagged.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

awsaimlci/cd

Platform: amazon
Effort: medium

Benefits

Benefits detected in the listing.

Health Insurance

Dental

Vision

401(K)

Paid Time Off

Parental Leave