Senior Machine Learning / Data Engineer at VTI Aerospace

Job description

As a Senior Machine Learning / Data Engineer - Evaluation, you will design and implement systems that measure and monitor the performance of our computer vision, automatic speech recognition (ASR), and small language model (SLM) systems. You will develop evaluation methodologies, benchmarking pipelines, and monitoring tools that ensure our AI systems perform reliably in real-world environments. You will help establish the evaluation standards and performance benchmarks that guide the development of all AI systems at the company. You will work closely with machine learning and data engineering teams to evaluate model performance, identify failure modes, and guide improvements to data collection and model training, helping bring AI-powered aviation tools to market. This role is ideal for someone who enjoys designing robust evaluation systems and uncovering insights from model performance data, and wants to have a direct impact on the quality and reliability of our AI systems. LOCATION Work arrangement: On-site Primary location: Seattle, WA or Bozeman, MT About VTI Aerospace VTI Aerospace builds AI-powered perception and pilot assist technologies to unlock the future of aviation. With offices in Bozeman, MT and Seattle, WA, we are a team of engineers and technologists from Boeing, Airbus, Aurora, and beyond. We are passionate about pushing the boundaries of autonomy and aviation safety. Our company is dedicated to advancing the field of aviation with cutting-edge solutions What You'll Do Define and implement evaluation methodologies for computer vision, ASR, and language systems Identify and track key performance indicators (KPIs) that measure system and model effectiveness Perform dataset coverage analysis to understand strengths, gaps, and biases in training and evaluation data Identify model deficiencies and collaborate with ML engineers to improve training data and model performance Build scalable model evaluation pipelines in Python for automated benchmarking and regression testing Design and maintain systems for monitoring model performance and drift in production environments Architect data models and storage systems for evaluation results using relational and time-series databases Build dashboards and reporting tools to visualize model performance and evaluation metrics Collaborate with ML and data engineering teams to improve evaluation workflows and data pipelines Contribute to ground-truth data pipelines and processes that support reliable evaluation What We're Looking For 5+ years of experience developing production software systems in Python Strong experience designing and implementing data processing or analysis pipelines Experience building systems for machine learning evaluation, experimentation, or benchmarking Experience working with large datasets and building scalable data workflows Familiarity with statistical methods, experimental design, and model performance analysis Experience building automated pipelines using tools such as Airflow or similar orchestration frameworks Experience working with evaluation or experiment tracking tools such as MLflow or similar systems Experience designing data models and working with relational databases and time-series data Strong collaboration skills and ability to work closely with machine learning and data engineering teams Bonus Points Experience working with computer vision, ASR, or language models in production environments Experience using PyTorch or other deep learning frameworks Experience evaluating multimodal or large-scale AI systems Experience building monitoring systems for ML models in production Experience designing dataset analysis or data quality tooling Experience building dashboards or monitoring tools using Grafana or similar platforms Why You'll Love Working Here Opportunity to work on meaningful problems in aviation and deliver products to industry at a rapid pace Small team with significant impact on company direction Collaborative and innovative environment Competitive compensation and benefits Opportunities for growth and leadership VTI offers a fast-paced and dynamic work environment that focuses on technical excellence and collaboration. Additionally, we foster a culture of work-life balance to prevent burn-out. Compensation packages are comprised of salary and incentive stock options. Salaries may range from $80,000 to $175,000 per year depending on role and individuals experience. VTI additionally offers 401K with a 4% match, an unlimited PTO policy, and health care reimbursement up to the IRS allowable limit through QSEHRA. VTI is an equal opportunity employer and values diversity. We do not discriminate on the basis of race, religion, color, national origin, sex, gender, gender expression, sexual orientation, age, marital status, veteran status or disability status. All job offers are contingent upon the candidate passing background, and reference checks. At this time, we are unable to provide visa sponsorship or consider candidates who require visa transfers. Applicants must be authorized to work in the United States without the need for visa sponsorship now or in the future. APPLY TODAY! Didn't find the position you're looking for? Send us your resume! OPEN SUBMISSIONS Learn More About VTI Aerospace ABOUT US bottom of page <link href="siteassets.parastorage.com

Requirements

Must-haves

This role is ideal for someone who enjoys designing robust evaluation systems and uncovering insights from model performance data, and wants to have a direct impact on the quality and reliability of our AI systems.
Work arrangement: On-site
Primary location: Seattle, WA or Bozeman, MT
Define and implement evaluation methodologies for computer vision, ASR, and language systems
Identify and track key performance indicators (KPIs) that measure system and model effectiveness
Perform dataset coverage analysis to understand strengths, gaps, and biases in training and evaluation data
Identify model deficiencies and collaborate with ML engineers to improve training data and model performance
Build scalable model evaluation pipelines in Python for automated benchmarking and regression testing
Design and maintain systems for monitoring model performance and drift in production environments
Architect data models and storage systems for evaluation results using relational and time-series databases
Build dashboards and reporting tools to visualize model performance and evaluation metrics
Collaborate with ML and data engineering teams to improve evaluation workflows and data pipelines
Contribute to ground-truth data pipelines and processes that support reliable evaluation
5+ years of experience developing production software systems in Python
Strong experience designing and implementing data processing or analysis pipelines
Experience building systems for machine learning evaluation, experimentation, or benchmarking
Experience working with large datasets and building scalable data workflows
Familiarity with statistical methods, experimental design, and model performance analysis

Nice-to-haves

Experience working with computer vision, ASR, or language models in production environments
Experience using PyTorch or other deep learning frameworks
Experience evaluating multimodal or large-scale AI systems
Experience building monitoring systems for ML models in production
Experience designing dataset analysis or data quality tooling
Experience building dashboards or monitoring tools using Grafana or similar platforms

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

machine learningaimlpython

Platform: careers-html
Effort: medium

Benefits

Benefits detected in the listing.

Vision

401k

Pto