Job description

ABOUT ALTERION Alterion is building the control plane for the agentic enterprise - enabling organizations to safely deploy, monitor, and scale autonomous AI agents with full visibility and governance. Our platform unifies semantic observability, dynamic policy enforcement, and distributed monitoring to ensure agent ecosystems remain trustworthy, compliant, and efficient at scale. As part of this mission, we are building an Intelligence Platform that is responsible for understanding, scoring, and optimizing agent behavior across the enterprise. Including accuracy, security, cost, drift, anomalous behavior, and runtime dynamics. Role Overview We are looking for a Senior Applied AI Engineer to join our on-site Research & Intelligence team in San Francisco. This role is for an experienced engineer who enjoys working close to research problems but is primarily motivated by building, shipping, and improving production systems. You will work on applied GenAI problems including agent evaluation, observability, and safety; translating proven techniques into scalable, reliable components within Alterion’s platform. This is not a pure research role. Success is measured by what ships and what improves in production, not by publications. What You’ll Work On - Build and maintain pipelines for model evaluation, scoring, and monitoring in production environments. - Apply fine-tuning, optimization, and lightweight model adaptation techniques where appropriate. - Develop embeddings, classifiers, and scoring services that support natural language analysis. - Run structured experiments to evaluate model performance, tradeoffs, and regressions. - Partner with engineering and product teammates to translate ideas into working systems. - Contribute to internal tooling, dashboards, and prototypes that improve agent visibility and control. Minimum Qualifications - 5+ years of experience in software engineering, ML engineering, or applied AI roles. - Strong Python skills and experience building production services or data pipelines. - Hands-on experience working with LLMs (APIs or open-source models) in real systems. - Familiarity with model evaluation, monitoring, and performance tradeoffs. - Experience using ML frameworks such as PyTorch or similar. Preferred Qualifications - Experience with GenAI evaluation, observability, or safety tooling. - Familiarity with embeddings, vector search, or semantic similarity systems. - Exposure to fine-tuning, PEFT, or cost-aware inference strategies. - Experience working in early-stage or fast-moving product teams. Who Thrives Here - Researchers who like to build, iterate, and see real impact quickly. - People who enjoy working in-person with a small team on hard, novel problems. - Builders who combine scientific depth with strong engineering instincts. - Individuals passionate about defining how autonomous AI agents are evaluated, governed, and optimized.

Requirements

Must-haves

You will work on applied GenAI problems including agent evaluation, observability, and safety; translating proven techniques into scalable, reliable components within Alterion’s platform.
This is not a pure research role. Success is measured by what ships and what improves in production, not by publications.
What You’ll Work On
Build and maintain pipelines for model evaluation, scoring, and monitoring in production environments.
Apply fine-tuning, optimization, and lightweight model adaptation techniques where appropriate.
Develop embeddings, classifiers, and scoring services that support natural language analysis.
Run structured experiments to evaluate model performance, tradeoffs, and regressions.
Partner with engineering and product teammates to translate ideas into working systems.
Contribute to internal tooling, dashboards, and prototypes that improve agent visibility and control.
5+ years of experience in software engineering, ML engineering, or applied AI roles.
Strong Python skills and experience building production services or data pipelines.
Hands-on experience working with LLMs (APIs or open-source models) in real systems.
Familiarity with model evaluation, monitoring, and performance tradeoffs.
Experience using ML frameworks such as PyTorch or similar.

Nice-to-haves

Experience with GenAI evaluation, observability, or safety tooling.
Familiarity with embeddings, vector search, or semantic similarity systems.
Exposure to fine-tuning, PEFT, or cost-aware inference strategies.
Experience working in early-stage or fast-moving product teams.
Who Thrives Here
Researchers who like to build, iterate, and see real impact quickly.
People who enjoy working in-person with a small team on hard, novel problems.
Builders who combine scientific depth with strong engineering instincts.
Individuals passionate about defining how autonomous AI agents are evaluated, governed, and optimized.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aimlpython

Platform: ashby
Effort: quick