Job description

What You Will Do This position is open for external candidates only to apply. The High Performance Computing (HPC) Division at Los Alamos National Laboratory provides scientific computing resources consisting of some of the largest HPC systems in the world as well as numerous large commodity clusters. Our HPC Systems Group (HPC-SYS) is seeking a Team Leader for the AIOps team to make better use of our vast amounts of HPC System and Application data, this team will add AI solutions to our HPC infrastructure. The new team will work alongside the other Teams in HPC-SYS, Monitoring, Web Services and Cybersecurity. The Monitoring Team is responsible for collecting all data from our Data Centers, everything from Facilities to Clusters, and implementing operational dashboard, alerts and reports using tools like Splunk. Our Web Servers team runs our admin and user facing web sites, including user Documentation, Ticketing systems and Gitlab. Our Cybersecurity Team monitors and implements cybersecurity policies on our HPC systems. The AIOps team will have three major focus areas, LLMs, System Data Analysis and System Automation. We have a large set of HPC specific documentation for both users and admins that will be integrated into LLMs, this team will be responsible for designing, building, and running the LLMs, as well as adding agentic capabilities to increase the usefulness to the user. We have massive amounts of system data. This team will implement ML and Data Science techniques to perform deeper analysis of the data to improve performance analysis, event correlation and anomaly detection. Finally, the team will investigate AI driven workflows for task automation within the Data Center. You will work closely with members of the AIOps team and System Matter Experts (SMEs) in different HPC areas to design and develop these tools using our on-prem analysis and AI systems. The successful candidates' scope will include monitoring and analyzing system performance to identify anomalies. You will analyze large volumes of data to identify patterns and trends using ML and Data Science techniques with the goal of developing automation scripts and workflows to implement proactive measures. You and the team will maintain the AIOps platforms and tools including the user-facing LLMs. The successful candidate will continue actively growing their technical skills and keeping up to date with the latest technologies in the field. In addition, the selected candidate will have the opportunity to develop technical products such as technical documentation, presentations, technical papers, and reports, to communicate findings internally and at conferences. This position is full-time and is located at Los Alamos National Laboratory in Los Alamos, New Mexico. What You Need Minimum Job Requirements: • HPC Systems & Linux Administration: Strong understanding of compute nodes, schedulers, high-speed networks, parallel file systems, accelerators, containers, and common HPC workloads. Proficient with the Linux command line and hardware and software security practices. • Technical Leadership & AIOps Strategy: Experience leading multidisciplinary teams of HPC engineers, data scientists, ML engineers, and developers. Able to define and execute a roadmap for applying AI and ML to improve reliability, efficiency, performance, and user support. • Python & Software Development: Strong programming skills with Python and AI frameworks for data analysis, model development, automation, APIs, testing, and production-quality services, including Git-based version control. • Machine Learning & Statistical Analysis: Strong statistical skills and working knowledge of ML algorithms for anomaly detection, classification, clustering, forecasting, time-series analysis, event correlation, and predictive maintenance. • Monitoring & Observability: Extensive experience analyzing system logs, telemetry, metrics, and traces, along with alerting and observability platforms. • Root Cause & Performance Analysis: Able to correlate failures and anomalous behavior across compute, storage, network, scheduler, facility, and application data. Experienced in identifying bottlenecks, resource contention, and inefficient workloads. • Data Engineering: Able to design pipelines that collect, normalize, enrich, store, and retrieve large volumes of HPC telemetry and operational data. • MCP & Systems Integration: Experience developing secure MCP or similar interfaces that let AI assistants query schedulers, monitoring systems, documentation, and ticketing platforms. • LLM, RAG & AI Agent Architecture: Knowledge of retrieval-augmented generation, tool-using agents, workflow orchestration, and LLM evaluation, including accuracy, retrieval quality, hallucination risk, latency, and cost. • Security, Responsible AI & Communication: Understanding of authentication and authorization, least-privilege access, secrets management, and prompt-injection defenses. Sound judgment on when AI should recommend, require human approval, or act autonomously. Able to explain technical findings and AI limitations to diverse stakeholders. Education/Experience at HPC Engineer 3: Position Requires a bachelor's in Computer Science or Computer Engineering or a related field and 6 years of relevant experience in high performance computing or scalable AI computing, or data center environments or equivalent combination of education and experience in related field. Desired Qualifications: • Production RAG Experience: Hands-on experience building and running RAG-based LLM systems using documentation, tickets, knowledge bases, and operational history. • End-to-End AIOps Implementation: Experience implementing AIOps tools and workflows from ML analysis through system automation and configuration management. • MLOps & LLMOps: Experience with model deployment, versioning, monitoring, drift detection, reproducibility, and lifecycle management. • Monitoring Tool Experience: Experience integrating operational metrics into Splunk and working with Syslog, Telegraf, Prometheus, Grafana, or similar tools. • HPC User Support: Familiarity with common user issues, including job failures, resource requests, environments, software dependencies, storage, and performance. • Knowledge Management: Able to turn documentation, runbooks, incident reports, and expert knowledge into reliable content for retrieval and automated assistance. • Professional Effectiveness: Effective at working with customers and vendors. Detail-oriented, organized, and able to multitask in a fast-paced environment, with a commitment to evaluating emerging AI technologies for measurable value. • Active DOE Q Clearance Work Location: This position will be located in Los Alamos, NM, with the potential for a hybrid work arrangement (60% onsite/40% offsite) from a location within 2 hours ground commute of this location. Reporting onsite will be required. Hybrid is at the discretion of management and can change at any time with appropriate notice. Position commitment: Regular appointment employees are required to serve a period of continuous service in their current position in order to be eligible to apply for posted jobs throughout the Laboratory. If an employee has not served the time required, they may only apply for Laboratory jobs with the documented approval of their Division Leader. The position commitment for this position is 1 year. Note to Applicants: For consideration, applicants should submit a cover letter addressing how their knowledge, skills and abilities meet the minimum requirements along with a resume. Due to federal restrictions contained in the current National Defense Authorization Act, citizens of the People's Republic of China-including the special administrative regions of Hong Kong and Macau-as well as citizens of the Islamic Republic of Iran, the Democratic People's Republic of Korea (North Korea), and the Russian Federation, who are not Lawful Permanent Residents ("green card" holders) are prohibited from accessing facilities that support the mission, functions, and operations of national security laboratories and nuclear weapons production facilities, which includes Los Alamos National Laboratory. Where You Will Work Located in beautiful northern New Mexico, Los Alamos National Laboratory (LANL) is a multidisciplinary research institution engaged in strategic science on behalf of national security. Our generous benefits package includes: § PPO or High Deductible medical insurance with the same large nationwide network § Dental and vision insurance § Free basic life and disability insurance § Paid childbirth and parental leave § Award-winning 401(k) (6% matching plus 3.5% annually) § Learning opportunities and tuition assistance § Flexible schedules and time off (PTO and holidays) § Onsite gyms and wellness programs § Extensive relocation packages (outside a 50 mile radius) Additional Details Directive 206.2 - Employment with Triad requires a favorable decision by NNSA indicating employee is suitable under NNSA Supplemental Directive 206.2 . Please note that this requirement applies only to citizens of the United States. Foreign nationals are subject to a similar requirement under DOE Order 142.3A. Clearance: Q (Position will be cleared to this level). Selected applicants will be subject to a background investigation conducted by or on behalf of the Federal Government, and must meet eligibility requirements* for access to classified matter. This position requires a Q clearance, and obtaining such clearance requires US Citizenship except in extremely rare circumstances. Dependent upon the position, additional authorization to access classified information may be required, which may or may not be available to dual citizens. Receipt of a Q clearance and additional access authorization ultimately is a decision of the Federal Government and not of Triad. *Eligibility requirements: To obtain a clearance, an individual must be at least 18 years of age; U.S. citizenship is required except in very limited circumstances. See DOE Order 472.2 for additional information. New-Employment Drug Test: The Laboratory requires successful applicants to complete a new-employment drug test and maintains a substance abuse policy that includes random drug testing. Although New Mexico and other states have legalized the use of marijuana, use and possession of marijuana remain illegal under federal law. A positive drug test for marijuana will result in termination of employment, even if the use was pre-offer. Regular position: Term status Laboratory employees applying for regular-status positions are converted to regular status. Internal Applicants: Regular appointment employees who have served the required period of continuous service in their current position are eligible to apply for posted jobs throughout the Laboratory. If an employee has not served the required period of continuous service, they may only apply for Laboratory jobs with the documented approval of their Division Leader. Please refer to Policy Policy P701 for applicant eligibility requirements. Equal Opportunity: Los Alamos National Laboratory is an equal opportunity employer. All employment practices are based on qualification and merit, without regard to protected categories such as race, color, national origin, ancestry, religion, age, sex, gender identity, sexual orientation, marital status or spousal affiliation, physical or mental disability, medical conditions, pregnancy, status as a protected veteran, genetic information, or citizenship within the limits imposed by federal, state, and local laws and regulations. The Laboratory is also committed to making our workplace accessible to individuals with disabilities and will provide reasonable accommodations, upon request, for individuals to participate in the application and hiring process. To request such an accommodation, please send an email to applyhelp@lanl.gov or call (505)-664-6947.

Requirements

Must-haves

This position is open for external candidates only to apply.
What You Need
Monitoring & Observability: Extensive experience analyzing system logs, telemetry, metrics, and traces, along with alerting and observability platforms.
Data Engineering: Able to design pipelines that collect, normalize, enrich, store, and retrieve large volumes of HPC telemetry and operational data.
MCP & Systems Integration: Experience developing secure MCP or similar interfaces that let AI assistants query schedulers, monitoring systems, documentation, and ticketing platforms.
LLM, RAG & AI Agent Architecture: Knowledge of retrieval-augmented generation, tool-using agents, workflow orchestration, and LLM evaluation, including accuracy, retrieval quality, hallucination risk, latency, and cost.
For consideration, applicants should submit a cover letter addressing how their knowledge, skills and abilities meet the minimum requirements along with a resume.
Where You Will Work
§ Free basic life and disability insurance
§ Learning opportunities and tuition assistance
§ Flexible schedules and time off (PTO and holidays)
§ Onsite gyms and wellness programs
§ Extensive relocation packages (outside a 50 mile radius)
Additional Details
Eligibility requirements: To obtain a clearance, an individual must be at least 18 years of age; U.S. citizenship is required except in very limited circumstances. See DOE Order 472.2 for additional information.
Regular position: Term status Laboratory employees applying for regular-status positions are converted to regular status.

Nice-to-haves

Production RAG Experience: Hands-on experience building and running RAG-based LLM systems using documentation, tickets, knowledge bases, and operational history.
End-to-End AIOps Implementation: Experience implementing AIOps tools and workflows from ML analysis through system automation and configuration management.
MLOps & LLMOps: Experience with model deployment, versioning, monitoring, drift detection, reproducibility, and lifecycle management.
Monitoring Tool Experience: Experience integrating operational metrics into Splunk and working with Syslog, Telegraf, Prometheus, Grafana, or similar tools.
HPC User Support: Familiarity with common user issues, including job failures, resource requests, environments, software dependencies, storage, and performance.
Knowledge Management: Able to turn documentation, runbooks, incident reports, and expert knowledge into reliable content for retrieval and automated assistance.
Active DOE Q Clearance

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aimlpythonmachine learningllm

Platform: careers-html
Effort: medium

Benefits

Benefits detected in the listing.

Dental

Vision

401(K)

Pto

Parental Leave

Wellness