Job description

Required Qualifications • Active TS/SCI clearance with CI Polygraph. • BS/BA and 5 years of experience, an additional 4 years of experience may be considered in lieu of a degree. • Approximately 5–8 years of experience in platform operations, site reliability engineering, cloud engineering, software operations, applied AI/ML operations, or production application support. • Experience operating and troubleshooting production software applications or cloud services. • Experience with monitoring, logging, metrics, incident response, and configuration or change management. • Experience evaluating software behavior through repeatable test cases, regression testing, quality measures, or operational performance analysis. • Working knowledge of foundation-model or LLM-based applications, including prompts, model endpoints, tool calls, and retrieval-augmented applications. • Experience with AWS or comparable cloud platforms. • Strong analytical, troubleshooting, documentation, and communication skills. Desired Qualifications • Experience with AWS Bedrock or comparable managed foundation-model services. • Familiarity with LLM or agent evaluation frameworks, automated evaluation harnesses, or AI application testing. • Experience with OpenTelemetry, Amazon CloudWatch, OpenSearch, Prometheus, Grafana, or comparable observability technologies. • Familiarity with ServiceNow, IT service management, and incident-management processes. • Experience with GitLab, CI/CD pipelines, and controlled configuration promotion. • Familiarity with Kubernetes and containerized application environments. • Experience monitoring cloud utilization, model consumption, and service costs. • Knowledge of risk management frameworks, continuous monitoring, or security evidence collection. • Experience supporting classified, restricted, or highly regulated environments. We are seeking an AI/ML Operations & Assurance Engineer to monitor, evaluate, and maintain production AI-enabled applications, foundation-model integrations, and agentic workflows. This role focuses on operational reliability, workflow performance, evaluation and regression testing, configuration control, usage and cost monitoring, and the safe introduction of AI-related changes into production environments. The ideal candidate combines cloud and application operations experience with an understanding of large language model (LLM) applications, agent behavior, and AI-specific operational risks. The engineer will collaborate with AI/ML developers, site reliability engineers, cybersecurity teams, and platform operations personnel to maintain reliable, auditable, and secure AI-enabled services. ResponsibilitiesProduction Monitoring and Operations • Monitor availability, latency, error rates, model and token usage, service costs, tool-call failures, and workflow completion metrics for production AI services. • Integrate AI service health and agent-workflow telemetry into observability dashboards, monitoring systems, and alerting processes. • Investigate failed, stalled, repetitive, or abnormal AI workflows and coordinate remediation with software, platform, and AI engineering teams. • Maintain operational runbooks, escalation procedures, troubleshooting guidance, and recovery processes for AI-enabled applications. • Distinguish infrastructure and application failures from issues associated with model behavior, prompts, retrieval sources, or agent orchestration. AI Evaluation and Operational Assurance • Develop and maintain repeatable evaluation scenarios for critical AI workflows, developer assistants, and model-service updates. • Assess task completion, output quality, error patterns, inappropriate tool use, and other indicators of operational suitability. • Coordinate regression testing before changes to models, prompts, tools, workflow logic, or retrieval sources are deployed to production. • Analyze user feedback and operational results to identify where workflows require refinement, additional controls, or restrictions on automated execution. • Document evaluation results, limitations, and recommendations to support evidence-based deployment decisions. Change and Configuration Management • Maintain version control and change records for prompts, agent instructions, tool definitions, model selections, evaluation datasets, and workflow configurations. • Coordinate model and service updates with AI engineering, cybersecurity, platform operations, and architecture stakeholders. • Support rollback to validated configurations when changes negatively affect reliability, performance, or workflow quality. • Ensure AI-related changes follow established review, testing, approval, and deployment procedures. • Monitor configuration drift and verify that production services remain aligned with approved configurations. Security, Audit, and Compliance Support • Coordinate with cybersecurity and identity teams to verify access controls, tool permissions, logging, and auditability for production AI services. • Support investigations involving anomalous AI behavior, unauthorized actions, or potential data-handling concerns. • Collect and maintain operational evidence supporting security assessments, continuous monitoring, and service reviews. • Verify that production workflows use approved models, endpoints, tools, and data sources in accordance with established policies. • Support the implementation and validation of operational safeguards for AI-enabled functions. Operational Improvement and Reporting • Identify repetitive operational tasks and support activities suitable for controlled AI-assisted automation. • Measure the impact of deployed workflows on processing time, support workload, service quality, and operational efficiency. • Collaborate with service delivery, Tier 2 support, SRE, and platform engineering teams to improve workflows based on operational demand. • Report trends in service health, model usage, operating costs, evaluation results, and unresolved issues. • Recommend improvements to monitoring, testing, configuration control, and operational procedures. Location: This position is fully onsite in Chantilly, VA. We are seeking an AI/ML Operations & Assurance Engineer to monitor, evaluate, and maintain production AI-enabled applications, foundation-model integrations, and agentic workflows. This role focuses on operational reliability, workflow performance, evaluation and regression testing, configuration control, usage and cost monitoring, and the safe introduction of AI-related changes into production environments. The ideal candidate combines cloud and application operations experience with an understanding of large language model (LLM) applications, agent behavior, and AI-specific operational risks. The engineer will collaborate with AI/ML developers, site reliability engineers, cybersecurity teams, and platform operations personnel to maintain reliable, auditable, and secure AI-enabled services. Responsibilities Production Monitoring and Operations • Monitor availability, latency, error rates, model and token usage, service costs, tool-call failures, and workflow completion metrics for production AI services. • Integrate AI service health and agent-workflow telemetry into observability dashboards, monitoring systems, and alerting processes. • Investigate failed, stalled, repetitive, or abnormal AI workflows and coordinate remediation with software, platform, and AI engineering teams. • Maintain operational runbooks, escalation procedures, troubleshooting guidance, and recovery processes for AI-enabled applications. • Distinguish infrastructure and application failures from issues associated with model behavior, prompts, retrieval sources, or agent orchestration. AI Evaluation and Operational Assurance • Develop and maintain repeatable evaluation scenarios for critical AI workflows, developer assistants, and model-service updates. • Assess task completion, output quality, error patterns, inappropriate tool use, and other indicators of operational suitability. • Coordinate regression testing before changes to models, prompts, tools, workflow logic, or retrieval sources are deployed to production. • Analyze user feedback and operational results to identify where workflows require refinement, additional controls, or restrictions on automated execution. • Document evaluation results, limitations, and recommendations to support evidence-based deployment decisions. Change and Configuration Management • Maintain version control and change records for prompts, agent instructions, tool definitions, model selections, evaluation datasets, and workflow configurations. • Coordinate model and service updates with AI engineering, cybersecurity, platform operations, and architecture stakeholders. • Support rollback to validated configurations when changes negatively affect reliability, performance, or workflow quality. • Ensure AI-related changes follow established review, testing, approval, and deployment procedures. • Monitor configuration drift and verify that production services remain aligned with approved configurations. Security, Audit, and Compliance Support • Coordinate with cybersecurity and identity teams to verify access controls, tool permissions, logging, and auditability for production AI services. • Support investigations involving anomalous AI behavior, unauthorized actions, or potential data-handling concerns. • Collect and maintain operational evidence supporting security assessments, continuous monitoring, and service reviews. • Verify that production workflows use approved models, endpoints, tools, and data sources in accordance with established policies. • Support the implementation and validation of operational safeguards for AI-enabled functions. Operational Improvement and Reporting • Identify repetitive operational tasks and support activities suitable for controlled AI-assisted automation. • Measure the impact of deployed workflows on processing time, support workload, service quality, and operational efficiency. • Collaborate with service delivery, Tier 2 support, SRE, and platform engineering teams to improve workflows based on operational demand. • Report trends in service health, model usage, operating costs, evaluation results, and unresolved issues. • Recommend improvements to monitoring, testing, configuration control, and operational procedures. Location: This position is fully onsite in Chantilly, VA. Required Qualifications • Active TS/SCI clearance with CI Polygraph. • BS/BA and 5 years of experience, an additional 4 years of experience may be considered in lieu of a degree. • Approximately 5–8 years of experience in platform operations, site reliability engineering, cloud engineering, software operations, applied AI/ML operations, or production application support. • Experience operating and troubleshooting production software applications or cloud services. • Experience with monitoring, logging, metrics, incident response, and configuration or change management. • Experience evaluating software behavior through repeatable test cases, regression testing, quality measures, or operational performance analysis. • Working knowledge of foundation-model or LLM-based applications, including prompts, model endpoints, tool calls, and retrieval-augmented applications. • Experience with AWS or comparable cloud platforms. • Strong analytical, troubleshooting, documentation, and communication skills. Desired Qualifications • Experience with AWS Bedrock or comparable managed foundation-model services. • Familiarity with LLM or agent evaluation frameworks, automated evaluation harnesses, or AI application testing. • Experience with OpenTelemetry, Amazon CloudWatch, OpenSearch, Prometheus, Grafana, or comparable observability technologies. • Familiarity with ServiceNow, IT service management, and incident-management processes. • Experience with GitLab, CI/CD pipelines, and controlled configuration promotion. • Familiarity with Kubernetes and containerized application environments. • Experience monitoring cloud utilization, model consumption, and service costs. • Knowledge of risk management frameworks, continuous monitoring, or security evidence collection. • Experience supporting classified, restricted, or highly regulated environments.

Requirements

Must-haves

Required Qualifications • Active TS/SCI clearance with CI Polygraph.
BS/BA and 5 years of experience, an additional 4 years of experience may be considered in lieu of a degree.
Approximately 5–8 years of experience in platform operations, site reliability engineering, cloud engineering, software operations, applied AI/ML operations, or production application support.
Experience operating and troubleshooting production software applications or cloud services.
Experience with monitoring, logging, metrics, incident response, and configuration or change management.
Experience evaluating software behavior through repeatable test cases, regression testing, quality measures, or operational performance analysis.
Working knowledge of foundation-model or LLM-based applications, including prompts, model endpoints, tool calls, and retrieval-augmented applications.
Experience with AWS or comparable cloud platforms.
Strong analytical, troubleshooting, documentation, and communication skills.
Desired Qualifications • Experience with AWS Bedrock or comparable managed foundation-model services.
Familiarity with LLM or agent evaluation frameworks, automated evaluation harnesses, or AI application testing.
Experience with OpenTelemetry, Amazon CloudWatch, OpenSearch, Prometheus, Grafana, or comparable observability technologies.
Familiarity with ServiceNow, IT service management, and incident-management processes.
Experience with GitLab, CI/CD pipelines, and controlled configuration promotion.
Familiarity with Kubernetes and containerized application environments.
Experience monitoring cloud utilization, model consumption, and service costs.
Knowledge of risk management frameworks, continuous monitoring, or security evidence collection.
Experience supporting classified, restricted, or highly regulated environments.
ResponsibilitiesProduction Monitoring and Operations • Monitor availability, latency, error rates, model and token usage, service costs, tool-call failures, and workflow completion metrics for production AI services.
Integrate AI service health and agent-workflow telemetry into observability dashboards, monitoring systems, and alerting processes.
Investigate failed, stalled, repetitive, or abnormal AI workflows and coordinate remediation with software, platform, and AI engineering teams.
Maintain operational runbooks, escalation procedures, troubleshooting guidance, and recovery processes for AI-enabled applications.
Distinguish infrastructure and application failures from issues associated with model behavior, prompts, retrieval sources, or agent orchestration.
AI Evaluation and Operational Assurance • Develop and maintain repeatable evaluation scenarios for critical AI workflows, developer assistants, and model-service updates.

Nice-to-haves

None flagged.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aimlllmawsci/cdkubernetes

Platform: peraton
Effort: medium