Staff Machine Learning Engineer - Agentic AI at ELLKAY

Job description

ELLKAY started out providing connectivity solutions to laboratories and within a few years, grew to also provide data management solutions to ambulatory organizations. ELLKAY is now a trusted data management partner in five healthcare segments. ELLKAY’s solutions continue to serve laboratories and ambulatory practices and have expanded to empower hospitals and health systems, healthcare IT vendors, ambulatory practices, health plans, and other healthcare organizations with cutting-edge technologies and solutions that drive their growth and interoperability strategies. Today, ELLKAY remains true to our core values, building strong partner relationships and offering unparalleled service and support while providing innovative, scalable solutions to the challenges our customers face in today’s data-rich world. ELLKAY's experience, customer-focused approach, and reputation for innovation, speed, and accuracy differentiate ELLKAY as a premier partner for your interoperability needs and data management strategy. Job Description: We're looking for a versatile Full Stack AI/ML Engineer who can architect, build, and deploy production-grade AI/ML and Agentic AI systems on AWS infrastructure. This role combines deep expertise in foundation models, agentic frameworks, healthcare data standards, and serverless cloud architecture to create scalable, secure, and compliant AI solutions for healthcare organizations. Essential Duties & Responsibilities: AI/ML & Agentic Engineering • Agentic Frameworks & Architectures: Architect, implement, and deploy end-to-end Agentic AI systems utilizing tool calling, stateful multi-agent collaboration, dynamic task routing, reflection loops, and self-correction workflows using frameworks like LangGraph, AutoGen, CrewAI, or Bedrock Agents. • Prompt & Tool Engineering: Design and implement production foundation model applications using advanced prompt engineering, structured output design, function-calling patterns, dynamic tool retrieval, and hallucination mitigation strategies. • Memory & Vector Search: Build and optimize vector search and long-term memory solutions using OpenSearch Serverless with k-NN, including HNSW/IVF parameter tuning, hybrid retrieval (lexical + semantic), and multi-stage ranking pipelines (BM25 + dense + re-ranking) for RAG and agent context retrieval. • Bedrock & Guardrails: Develop and deploy Amazon Bedrock and Bedrock Agent solutions leveraging Claude family models, Titan Embeddings v2, and Guardrails for PHI/PII detection while optimizing for provisioned vs. on-demand throughput. • Classical ML: Implement classical ML pipelines including clustering algorithms (HDBSCAN, k-means), confidence score calibration (Platt scaling, isotonic regression), and model evaluation frameworks. Healthcare Data & Standards • Design and implement systems that work with FHIR R4 resources, HL7 v2 message structures, and USCDI data elements. • Build ontology retrieval and agent tool interfaces against LOINC, SNOMED, and RxNorm terminologies to support clinical decision-making and data normalization. • Ensure HIPAA compliance and implement appropriate security controls for protected health information (PHI) across LLM inputs, tool payloads, and agent memory states. Backend & Infrastructure • Build high-performance APIs and agent execution backends using FastAPI and Pydantic v2 with async I/O patterns, robust retry/backoff mechanisms, state management, and structured error handling. • Design and deploy AWS serverless architectures including Lambda (Python + container), API Gateway (REST + HTTP), Step Functions, EventBridge, SQS, and SNS. • Implement container-based workloads for agent microservices on ECS Fargate with auto-scaling policies, service discovery, and Fargate Spot optimization. • Architect data and state solutions using Aurora PostgreSQL Serverless v2, DynamoDB, and S3 with appropriate indexing, partitioning, caching, and lifecycle policies. • Write production-grade Infrastructure as Code using AWS CDK (TypeScript or Python) supporting multi-tenant provisioning and cross-account deployments. Observability & Security • Implement comprehensive observability using AWS X-Ray distributed tracing, CloudWatch metrics/logs/alarms, agent execution trajectory tracing, structured JSON logging, and correlation ID propagation. • Apply security engineering best practices including IAM least-privilege roles for tool executions, KMS envelope encryption, Secrets Manager rotation, JWT validation, tenant isolation via session tags, and OWASP API Top-10 awareness. Evaluation & Quality • Design rigorous evaluation frameworks for foundation models and AI agents, including tool selection precision, execution success rates, benchmark creation, per-class metrics, held-out test sets, label leakage prevention, and statistical significance testing. • Implement prompt and agent strategy versioning, human-in-the-loop evaluation pipelines, and A/B testing frameworks to continuously improve performance. Qualifications: Technical Skills • Python Stack: Proficiency with Pydantic v2, Pandas/Polars, FastAPI, Boto3, Asyncio, and Agent/LLM frameworks (e.g., LangGraph, AutoGen, LlamaIndex). • Agentic AI & Foundation Models: Proven experience designing stateful agents, function/tool calling interfaces, autonomous reasoning loops, structured output generation, and evaluation harnesses. • AWS Services: Production experience with Lambda, API Gateway, Step Functions, EventBridge, SQS, SNS, ECS Fargate, Aurora PostgreSQL, DynamoDB, S3, and AWS CDK. • Vector Search & Memory: Practical knowledge of OpenSearch Serverless, k-NN algorithms, hybrid retrieval, and ranking optimization for RAG and agent memory stores. • Machine Learning: Experience with scikit-learn, clustering algorithms, and confidence calibration techniques. • Healthcare Standards (preferred): Working knowledge of FHIR R4, HL7 v2, USCDI, LOINC, SNOMED, and RxNorm. • Security: Good understanding of IAM, KMS, secrets management, and OWASP security principles. Experience • 10+ years of software engineering experience with at least 3 years focused on production AI/ML and GenAI/Agentic systems. • Proven track record of deploying production ML and agentic applications at scale. • Experience with healthcare data or regulated industries strongly preferred. • Demonstrated ability to design and implement end-to-end ML and agent evaluation frameworks. • Well versed in AI development leveraging tools such as Codex/Claude Code Benefits: ELLKAY offers a comprehensive and competitive benefit package that starts day one! Including: • Medical, Dental, and Vision benefits • Employer-paid Life and LTD • 401k w/ matching – once eligibility is met • Work/life balance • Paid Volunteer Program • Flexible working hours • Generous FTO • Remote work options • Employee Discounts • Parental Leave • Gym membership / Exercise class stipends Our awesome culture includes: • Working with talented, collaborative, and friendly people who love what they do • Professional growth within • Innovation environment • On site in HQ Free daily lunches Awards: • 2025 Top Workplaces Employee Appreciation • 2025 Top Workplaces Employee Well-Being • 2025 Top Workplaces Professional Development • 2025 Top Workplaces Leadership • 2026 USA Today Top Workplaces • 2026 NJ Top Workplaces Additional information At ELLKAY, we are committed to operating as a hybrid workplace, offering employees flexibility in how they structure their time between in-office and remote work. We recognize the significance of fostering connections, collaboration, and creativity within our office culture and its positive impact on our business. Our philosophy of operating as a hybrid workplace underscores our dedication to enabling employees to tailor work-life balance to their individual preferences. For those who do not live within 40 miles of one of our offices, we are open to considering remote work for candidates whose skills and experience strongly align with the role. While we prioritize a hybrid work environment for most roles, we understand the importance of flexibility and are open to remote work for specific positions and specialized skill sets. For more information on our company, visit www.ELLKAY.com. ELLKAY LLC is a Smoke-Free Workplace. ELLKAY, LLC provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws. Compensation Range: $170,000 - $190,000

Requirements

Must-haves

AI/ML & Agentic Engineering
Classical ML: Implement classical ML pipelines including clustering algorithms (HDBSCAN, k-means), confidence score calibration (Platt scaling, isotonic regression), and model evaluation frameworks.
Healthcare Data & Standards
Design and implement systems that work with FHIR R4 resources, HL7 v2 message structures, and USCDI data elements.
Build ontology retrieval and agent tool interfaces against LOINC, SNOMED, and RxNorm terminologies to support clinical decision-making and data normalization.
Ensure HIPAA compliance and implement appropriate security controls for protected health information (PHI) across LLM inputs, tool payloads, and agent memory states.
Backend & Infrastructure
Build high-performance APIs and agent execution backends using FastAPI and Pydantic v2 with async I/O patterns, robust retry/backoff mechanisms, state management, and structured error handling.
Design and deploy AWS serverless architectures including Lambda (Python + container), API Gateway (REST + HTTP), Step Functions, EventBridge, SQS, and SNS.
Implement container-based workloads for agent microservices on ECS Fargate with auto-scaling policies, service discovery, and Fargate Spot optimization.
Architect data and state solutions using Aurora PostgreSQL Serverless v2, DynamoDB, and S3 with appropriate indexing, partitioning, caching, and lifecycle policies.
Write production-grade Infrastructure as Code using AWS CDK (TypeScript or Python) supporting multi-tenant provisioning and cross-account deployments.
Observability & Security
Implement comprehensive observability using AWS X-Ray distributed tracing, CloudWatch metrics/logs/alarms, agent execution trajectory tracing, structured JSON logging, and correlation ID propagation.
Evaluation & Quality
Implement prompt and agent strategy versioning, human-in-the-loop evaluation pipelines, and A/B testing frameworks to continuously improve performance.
Technical Skills
Python Stack: Proficiency with Pydantic v2, Pandas/Polars, FastAPI, Boto3, Asyncio, and Agent/LLM frameworks (e.g., LangGraph, AutoGen, LlamaIndex).
Agentic AI & Foundation Models: Proven experience designing stateful agents, function/tool calling interfaces, autonomous reasoning loops, structured output generation, and evaluation harnesses.
AWS Services: Production experience with Lambda, API Gateway, Step Functions, EventBridge, SQS, SNS, ECS Fargate, Aurora PostgreSQL, DynamoDB, S3, and AWS CDK.
Vector Search & Memory: Practical knowledge of OpenSearch Serverless, k-NN algorithms, hybrid retrieval, and ranking optimization for RAG and agent memory stores.
Machine Learning: Experience with scikit-learn, clustering algorithms, and confidence calibration techniques.

Nice-to-haves

Healthcare Standards (preferred): Working knowledge of FHIR R4, HL7 v2, USCDI, LOINC, SNOMED, and RxNorm.
Experience with healthcare data or regulated industries strongly preferred.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aimlawsllmpythonapiresttypescriptmachine learning

Platform: adp-workforce-now
Effort: medium

Benefits

Benefits detected in the listing.

Dental

Vision

401k

Parental Leave

Professional Development