Job description

Position Summary: Magpie Health Analytics is seeking an experienced AI Engineer to design, build, and operate secure, trustworthy generative AI solutions that help our healthcare clients deliver faster, more consistent access to authoritative program guidance. Key Responsibilities: Our clients are government health payers. Your primary responsibility is to design, develop, test, deploy, and maintain an AI-enabled decision-support capability that allows users to search and navigate large bodies of program documentation (statutes, regulations, rules, and guidance) using natural language. The solution will first support internal customers and will later be extended to external, public-facing users. You will work in collaboration with project managers, data engineers, full stack developers, analysts, and client stakeholders to ensure the successful implementation of the client solution. The AI Engineer duties include: • Design, develop, and maintain generative AI solutions that ground large language model (LLM) responses in authoritative source documentation. • Build and maintain document ingestion pipelines, including parsing, chunking, metadata tagging, embedding generation, and vector/semantic search indexing. • Collaborate with cross-functional teams and subject matter experts to understand user needs and translate them into requirements, prompts, and evaluation criteria. • Implement responses with source citations and direct links to the underlying documentation to ensure traceability. • Design and implement guardrails, including content filtering, scope restrictions, PII/PHI protections, and refusal handling for out-of-scope questions. • Develop evaluation frameworks and test sets to measure retrieval quality, answer accuracy, groundedness, and consistency; monitor and tune performance over time. • Deploy and operate solutions within approved AWS services and federal security controls, supporting documentation needed for security authorization and AI governance reviews. • Implement logging, monitoring, and audit trails to support transparency, cost management, and responsible AI oversight. • Write clean, reusable, and well-documented code, and participate in code reviews, testing, and deployment processes. • Stay up to date with emerging AI technologies, models, and federal AI guidance. Educational Requirements: • A master’s degree in computer science, data science, engineering, or another relevant field. Qualifications: • 5+ years of experience developing software and cloud-based solutions, with demonstrated recent experience designing and deploying LLM or generative AI applications. • Strong proficiency in Python and experience with modern LLM APIs, SDKs, orchestration frameworks, and AI/ML libraries (e.g., Hugging Face, LangChain, LlamaIndex, or equivalent). • Hands-on experience designing and deploying generative AI solutions, including embeddings, vector databases, and hybrid/semantic search. • Experience with AWS AI and data services such as Amazon Bedrock, Amazon OpenSearch, Amazon Kendra, SageMaker, Lambda, or S3. • Experience with prompt and context engineering, LLM evaluation, and techniques for reducing hallucinations and ensuring answers are grounded in source material. • Designing AI applications to support multiple foundation models/providers and evaluating models based on accuracy, latency, cost, security, and use-case requirements. • Experience with Agile software development methodologies. • Experience with building and maintaining CI/CD pipelines, testing, and deploying infrastructure using services such as GitHub Actions, CloudFormation, Terraform, and/or Jenkins. • Excellent problem-solving and communication skills, including the ability to explain AI capabilities and limitations to non-technical stakeholders. • Ability to work independently and in a team, with strong attention to detail and presentation. Preferred Qualifications: • Experience supporting federal health programs, including program policy, regulatory guidance, and customer or stakeholder support operations. • Experience deploying AI solutions in federal environments, including FedRAMP-authorized services, ATO/security authorization processes, and NIST 800-53 controls. • Familiarity with responsible AI and federal AI governance frameworks (e.g., NIST AI Risk Management Framework, OMB AI guidance). • Understanding of security and compliance regulations in healthcare, such as HIPAA and HITECH. • Experience building conversational or search interfaces, including Section 508 accessibility compliance. • Relevant AWS certification (e.g., AWS Certified Machine Learning Engineer – Associate, AI Practitioner, Generative AI Developer – Professional, Solutions Architect, or equivalent)

Requirements

Must-haves

Design, develop, and maintain generative AI solutions that ground large language model (LLM) responses in authoritative source documentation.
Build and maintain document ingestion pipelines, including parsing, chunking, metadata tagging, embedding generation, and vector/semantic search indexing.
Collaborate with cross-functional teams and subject matter experts to understand user needs and translate them into requirements, prompts, and evaluation criteria.
Implement responses with source citations and direct links to the underlying documentation to ensure traceability.
Design and implement guardrails, including content filtering, scope restrictions, PII/PHI protections, and refusal handling for out-of-scope questions.
Develop evaluation frameworks and test sets to measure retrieval quality, answer accuracy, groundedness, and consistency; monitor and tune performance over time.
Deploy and operate solutions within approved AWS services and federal security controls, supporting documentation needed for security authorization and AI governance reviews.
Implement logging, monitoring, and audit trails to support transparency, cost management, and responsible AI oversight.
Write clean, reusable, and well-documented code, and participate in code reviews, testing, and deployment processes.
Stay up to date with emerging AI technologies, models, and federal AI guidance.
A master’s degree in computer science, data science, engineering, or another relevant field.
5+ years of experience developing software and cloud-based solutions, with demonstrated recent experience designing and deploying LLM or generative AI applications.
Strong proficiency in Python and experience with modern LLM APIs, SDKs, orchestration frameworks, and AI/ML libraries (e.g., Hugging Face, LangChain, LlamaIndex, or equivalent).
Hands-on experience designing and deploying generative AI solutions, including embeddings, vector databases, and hybrid/semantic search.
Experience with AWS AI and data services such as Amazon Bedrock, Amazon OpenSearch, Amazon Kendra, SageMaker, Lambda, or S3.
Experience with prompt and context engineering, LLM evaluation, and techniques for reducing hallucinations and ensuring answers are grounded in source material.
Designing AI applications to support multiple foundation models/providers and evaluating models based on accuracy, latency, cost, security, and use-case requirements.
Experience with Agile software development methodologies.

Nice-to-haves

Experience supporting federal health programs, including program policy, regulatory guidance, and customer or stakeholder support operations.
Experience deploying AI solutions in federal environments, including FedRAMP-authorized services, ATO/security authorization processes, and NIST 800-53 controls.
Familiarity with responsible AI and federal AI governance frameworks (e.g., NIST AI Risk Management Framework, OMB AI guidance).
Understanding of security and compliance regulations in healthcare, such as HIPAA and HITECH.
Experience building conversational or search interfaces, including Section 508 accessibility compliance.
Relevant AWS certification (e.g., AWS Certified Machine Learning Engineer – Associate, AI Practitioner, Generative AI Developer – Professional, Solutions Architect, or equivalent)

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aillmawspythonmlagileci/cdaccessibilitymachine learning

Platform: breezy
Effort: medium