AI/ML Evaluation Reviewer- Healthcare AI at Mpathic2

Job description

About the Role mpathic provides independent, multidisciplinary evaluations of healthcare AI technologies seeking deployment in real-world clinical settings. As part of our work supporting Utah's Office of Artificial Intelligence Policy (OAIP) regulatory sandbox, we are recruiting experienced machine learning (ML) and data science professionals to serve as technical reviewers on expert evaluation panels. In this role, you will independently assess the technical evidence supporting healthcare AI applications, including model performance, validation methodology, generalizability, reliability, and proposed monitoring and safeguards. Your assessments will help inform recommendations regarding whether and under what conditions an AI technology may be appropriate for pilot deployment. You will work alongside clinical and other subject-matter experts to evaluate whether applicants' technical claims are supported by credible evidence and whether their proposed evaluation and monitoring plans are appropriate for the intended healthcare setting and patient population. This is an evaluation and scientific review role, not a model development position. We are looking for experts who can critically assess existing evidence, identify limitations, and communicate actionable recommendations to technical and non-technical stakeholders. What You'll Be Doing Responsibilities may include: Evaluate model performance and validation: Assess the appropriateness of validation methods, reference standards, sample sizes, statistical analyses, performance metrics, operating thresholds, and reported uncertainty. Assess data quality and representativeness: Review whether training, testing, and validation datasets are appropriate for the intended clinical application, including potential data leakage, subgroup performance differences, and generalizability across patient populations and healthcare settings. Evaluate technical reliability: Assess evidence of model robustness, reproducibility, known failure modes, and performance under atypical or out-of-distribution inputs. Review proposed evaluation plans: Determine whether applicants have established scientifically defensible methods, performance benchmarks, and success criteria for evaluating AI technologies during pilot deployment, particularly when pre-deployment evidence is limited. Assess technical safeguards and human oversight: Evaluate whether proposed technical safeguards, escalation mechanisms, and human-in-the-loop processes are operationally feasible and supported by appropriate testing. Evaluate monitoring and change management: Assess applicants' plans for detecting performance degradation, monitoring model behavior, managing updates, and establishing criteria for intervention or rollback. Collaborate with multidisciplinary reviewers: Work alongside clinicians and other technical experts to identify concerns that require cross-disciplinary judgment, particularly where model performance may affect clinical outcomes. Produce structured evaluation findings: Document strengths, limitations, missing evidence, and technical concerns using a standardized review framework. Provide clear, evidence-based recommendations and identify what additional evidence or changes would resolve identified concerns. Participate in reviewer calibration and quality assurance: Contribute to discussions intended to improve consistency, transparency, and defensibility across evaluations. Maintain independence and confidentiality: Follow conflict-of-interest, confidentiality, and data-handling requirements when reviewing proprietary or unpublished materials. What We're Looking For We are seeking technically rigorous, independent reviewers who can distinguish between a promising technology and evidence that adequately supports its proposed use. Strong candidates are comfortable questioning unsupported performance claims, identifying limitations in study design, and recognizing when additional evidence is needed before meaningful conclusions can be drawn. They understand that healthcare AI evaluation is context-dependent: technical performance must be considered alongside the intended population, clinical application, degree of autonomy, and potential consequences of model error. Candidates should also be able to translate complex technical findings into clear, actionable recommendations for multidisciplinary audiences, including clinicians and state regulatory stakeholders. Basic Qualifications At least 3 years of professional or applied research experience in machine learning, data science, biostatistics, clinical informatics, or a closely related field, with direct experience evaluating or validating predictive models or AI systems. Demonstrated understanding of model evaluation methodologies, including appropriate performance metrics, validation design, statistical uncertainty, threshold selection, and common sources of bias or methodological error. Experience assessing model performance on independent or held-out data and interpreting whether reported performance generalizes beyond the evaluation dataset. Hands-on experience evaluating at least one relevant class of AI systems, such as clinical natural language processing, large language models, conversational AI, medical imaging, computer vision, or tabular predictive models. Ability to critically assess technical research, evaluation reports, or validation studies and identify methodological limitations or unsupported conclusions. Strong analytical writing skills, including the ability to communicate complex findings clearly to non-technical audiences. Ability to work independently while contributing to multidisciplinary expert review and calibration processes. Willingness to complete confidentiality agreements, conflict-of-interest disclosures, and reviewer training requirements. Preferred Qualifications Experience evaluating AI or ML systems used in healthcare, clinical decision support, digital health, or other high-stakes environments. Familiarity with clinical AI evaluation and reporting frameworks, such as TRIPOD+AI, STARD-AI, CONSORT-AI, DECIDE-AI, or model cards. Experience evaluating generative AI, LLM-based systems, or autonomous AI agents, including multi-turn behavior, reliability, hallucinations, and technical safeguards. Experience with robustness testing, adversarial evaluation, red teaming, or systematic failure-mode analysis. Knowledge of production monitoring, distribution shift, model drift, version control, and change-management practices. Experience designing or evaluating human-in-the-loop systems or comparing model performance against human expert judgments. Familiarity with FDA guidance for AI-enabled medical devices, clinical decision support, or software as a medical device. Experience conducting independent technical reviews, scientific peer reviews, grant reviews, or formal technology evaluations. Experience collaborating with clinicians, researchers, or regulatory stakeholders on AI evaluation or deployment. Engagement Structure This is a contract, per-review opportunity. Reviewers will be assigned to applications based on their technical expertise and the specific AI systems being evaluated. Assignments will involve reviewing applicant materials, documenting findings using mpathic's standardized evaluation framework, and participating in review discussions or calibration activities as needed. Reviewers will receive orientation to the evaluation framework and project-specific requirements prior to their first assignment.

Requirements

Must-haves

What You'll Be Doing
Evaluate model performance and validation: Assess the appropriateness of validation methods, reference standards, sample sizes, statistical analyses, performance metrics, operating thresholds, and reported uncertainty.
Evaluate technical reliability: Assess evidence of model robustness, reproducibility, known failure modes, and performance under atypical or out-of-distribution inputs.
Assess technical safeguards and human oversight: Evaluate whether proposed technical safeguards, escalation mechanisms, and human-in-the-loop processes are operationally feasible and supported by appropriate testing.
Evaluate monitoring and change management: Assess applicants' plans for detecting performance degradation, monitoring model behavior, managing updates, and establishing criteria for intervention or rollback.
Participate in reviewer calibration and quality assurance: Contribute to discussions intended to improve consistency, transparency, and defensibility across evaluations.
Maintain independence and confidentiality: Follow conflict-of-interest, confidentiality, and data-handling requirements when reviewing proprietary or unpublished materials.
We are seeking technically rigorous, independent reviewers who can distinguish between a promising technology and evidence that adequately supports its proposed use.
Candidates should also be able to translate complex technical findings into clear, actionable recommendations for multidisciplinary audiences, including clinicians and state regulatory stakeholders.
Experience assessing model performance on independent or held-out data and interpreting whether reported performance generalizes beyond the evaluation dataset.
Ability to critically assess technical research, evaluation reports, or validation studies and identify methodological limitations or unsupported conclusions.
Strong analytical writing skills, including the ability to communicate complex findings clearly to non-technical audiences.

Nice-to-haves

Experience evaluating AI or ML systems used in healthcare, clinical decision support, digital health, or other high-stakes environments.
Familiarity with clinical AI evaluation and reporting frameworks, such as TRIPOD+AI, STARD-AI, CONSORT-AI, DECIDE-AI, or model cards.
Experience evaluating generative AI, LLM-based systems, or autonomous AI agents, including multi-turn behavior, reliability, hallucinations, and technical safeguards.
Experience with robustness testing, adversarial evaluation, red teaming, or systematic failure-mode analysis.
Knowledge of production monitoring, distribution shift, model drift, version control, and change-management practices.
Experience designing or evaluating human-in-the-loop systems or comparing model performance against human expert judgments.
Familiarity with FDA guidance for AI-enabled medical devices, clinical decision support, or software as a medical device.
Experience conducting independent technical reviews, scientific peer reviews, grant reviews, or formal technology evaluations.
Experience collaborating with clinicians, researchers, or regulatory stakeholders on AI evaluation or deployment.
Engagement Structure
This is a contract, per-review opportunity. Reviewers will be assigned to applications based on their technical expertise and the specific AI systems being evaluated.
Assignments will involve reviewing applicant materials, documenting findings using mpathic's standardized evaluation framework, and participating in review discussions or calibration activities as needed.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aimachine learningmlllm

Platform: bamboohr
Effort: medium

Benefits

Benefits detected in the listing.

Vision