ML Engineer - LLM Evaluation & Automation at Grid Dynamics Holdings, Inc

Job description

US Remote, US Apply We are seeking a highly skilled Machine Learning Engineer who specializes in leveraging Large Language Models (LLMs) for automated evaluation and quality assessment. In this role, you will design and build systems that automatically measure and improve the accuracy, relevance, and consistency of model outputs. You will lead initiatives to create evaluation pipelines, develop metrics, and deliver actionable insights for continuous improvements. This position requires strong technical expertise, analytical problem-solving abilities, and the capacity to manage projects across multiple cross-functional teams. Essential functions • Design and implement automated systems and pipelines for evaluating LLM outputs. • Develop metrics and KPIs to measure output quality, accuracy, and consistency using LLM-based evaluations • Collaborate with Engineering teams to create automated logic checks and validation tools. • Partner with Data Scientists to analyze evaluation results and optimize prompt and task structures. • Provide feedback loops to ensure evaluation guidelines align with LLM-based assessments. • Investigate how LLM-derived evaluations can enhance product reliability and user experience. • Recommend refinements to prompt engineering, evaluation strategies, and automation tools. • Stay informed on emerging trends in LLM evaluation, automated quality assessment, and AI toolchains. • Continuously improve and expand automated evaluation processes based on industry best practices. Qualifications • 5+ years of experience in ML engineering, NLP, or AI/ML automation. • Hands-on experience in prompt engineering and designing LLM-based evaluation systems is preferred • Strong understanding of machine learning principles with focus on NLP and advanced LLM capabilities (e.g., Chain-of-Thought, agentic workflows) • Expertise in building automated evaluation or QA pipelines. • Excellent analytical and problem-solving skills with experience in root cause and error pattern analysis. • Proven project management and cross-functional collaboration experience. • Excellent communication skills to convey complex insights to technical and non-technical audiences. • Detail-oriented mindset with a focus on evaluation metrics, prompt design, and automation. • Ability to quickly adapt to new business rules and evaluation guidelines across diverse product domains. • Strong programming skills in Python and SQL. • Experience with big data technologies like PySpark for data aggregation and sampling is a strong plus • Bachelor’s/Master’s degree in Computer Science/ Engineering or a related field. We offer • Opportunity to work on cutting-edge projects • Work with a highly motivated and dedicated team • Competitive salary • Flexible schedule • Benefits package - medical insurance, vision, dental, etc. • Corporate social events • Professional development opportunities • Well-equipped office About us Grid Dynamics (NASDAQ: GDYN) is a leading provider of technology consulting, platform and product engineering, AI, and advanced analytics services. Fusing technical vision with business acumen, we solve the most pressing technical challenges and enable positive business outcomes for enterprise companies undergoing business transformation. A key differentiator for Grid Dynamics is our 8 years of experience and leadership in enterprise AI , supported by profound expertise and ongoing investment in data , analytics , cloud & DevOps , application modernization and customer experience . Founded in 2006, Grid Dynamics is headquartered in Silicon Valley with offices across the Americas, Europe, and India. \n \n Grid Dynamics California Job Applicant Privacy Notice \n \n \n \n Grid Dynamics Holdings, Inc. and its affiliates (collectively, \u201cGrid Dynamics,\u201d \u201cwe,\u201d \u201cus,\u201d or \u201cour\u201d)\n respect your privacy. This California Job Applicant Privacy Notice (\u201cNotice\u201d) applies to Personal\n Information that we collect as a Business from or about job applicants who are California residents\n (collectively, \u201cyou,\u201d or \u201cjob applicants\u201d). Please see our California Resident\n Privacy Notice for\n information about the information we process when you visit our website.\n \n \n For the purposes of this notice, Personal Information means information that identifies, relates to,\n describes, is reasonably capable of being associated with, or could reasonably be linked, directly or\n indirectly, with a particular consumer or household, or as otherwise defined by the California Consumer\n Privacy Act of 2018 as amended by the California Privacy Rights Act of 2020 (California Civil Code \u00a7\u00a7\n 1798.100 to 1798.199) and its implementing regulations, as amended or superseded from time to time (\u201cCCPA\u201d).\n \n \n Personal Information does not include information that is:\n \n \n • Lawfully made available from government records. \n • Deidentified or aggregated. \n • Otherwise excluded from the scope of the CCPA. \n \n \n \n 1. Sources of Personal Information We Collect \n \n We collect Personal Information from you when you provide it to us or automatically through your devices. We\n may also obtain information about you from other sources, such as former employers, including managers and\n colleagues, [other employment or personal references, educational institutions,] background check providers,\n social media services, recruitment agencies, affiliates, consumer data brokers, and publicly available\n databases.\n \n \n \n 2. Personal Information Collection, Disclosure \n \n The chart below provides the categories of Personal Information collected from or about job applicants for\n the business purpose(s) described in this Notice.\n \n \n \n \n Category \n \n \n Categories of Third Parties to Whom We Disclosed Personal Information \n \n \n \n \n A. Identifiers \n \n Examples: Name, address, email address, and government-issued IDs (social security number,\n driver\u0027s license number, passport number, or other similar\n identifiers).\n \n \n \n • Affiliates \n • Service providers and vendors \n \n \n \n \n \n \n B. Categories of Personal\n Information in Cal. Civ. Code Section 1798.80(e)\n \n \n Examples: Name, signature,\n social security number, address, telephone number, passport\n number, driver\u0027s license or state identification card number,\n education, employment, and\n employment history.\n \n \n \n • Affiliates \n • Service providers and vendors \n \n \n \n \n \n \n C. Characteristics of Protected Classifications under California or Federal Law\n \n \n Examples: Age (over 40), race or color, ancestry or national origin, mental or physical disability,\n sexual orientation, sex (including gender), gender identity or\n expression, marital status,\n military and veteran status.\n \n \n \n • Affiliates \n • Service providers and vendors \n \n \n \n \n \n D. Internet or Other Electronic Network Activity Information \n \n Examples: Usage metrics and information regarding your\n interaction with an internet\n website, application or\n advertisement.\n \n \n \n • Affiliates \n • Service providers and vendors \n \n \n \n \n \n E. Audio and Visual Information \n \n \n \n \n \n F. Biometric Information \n \n \n \n \n \n G. Education Information (as defined in 20 U.S.C. 1232g; 34 C.F.R. Part 99) \n \n Examples: Records that are\n directly related to a student\n maintained by an educational agency or institution or by a party acting for the agency or\n institution.\n \n \n \n • Affiliates \n • Service providers and vendors \n \n \n \n \n \n H. Inferences Drawn from Personal Information \n \n \n \n \n \n I. Sensitive Personal Information \n \n Examples: Government-issued IDs (Social Security numbers, visa and passport information), and\n ethnicity.\n \n \n \n • Affiliates \n • Service providers and vendors \n \n \n \n \n \n We do not sell or share (for cross-context behavioral advertising purposes) any of the above categories of\n Personal Information. We do not use or disclose your Sensitive Personal Information for purposes other than\n the limited purposes permitted by the CCPA.\n \n \n Over the past 12 months, we have collected and disclosed each of the categories of Personal Information as\n described in the chart above.\n \n \n \n 3. Use of Personal Information \n \n We collect your Personal Information for the following specific business and commercial purposes:\n \n \n \n • \n Recruitment Purposes: Undertake activities related to recruitment and processing your job\n application, such as evaluating job applications, determining qualifications for employment and reaching\n a hiring decision, contacting references, verifying your identity or other information related to your\n application, protecting against fraud, and communicating with you.\n \n • \n Diversity and Inclusion-Related Purposes: Monitor equal employment opportunities and uphold our\n commitment to diversity and inclusion.\n \n • \n Health, Safety, and Security Purposes: Protect the health and safety of our job applicants,\n employees, and other personnel, and to assess and maintain the security of our offices, facilities, and\n online services.\n \n • \n Analytics Purposes: Analyzing your interactions with our services.\n \n • \n Marketing Purposes: Marketing purposes, such as developing and providing promotional and\n advertising materials that may be useful, relevant, valuable or otherwise of interest to you.\n \n • \n Law-Related Purposes: Comply with laws and regulations, including collecting and disclosing job\n applicants\u0027 Personal Information as required by law, under judicial authorization, or in exercising or\n defending our legal rights.\n \n • \n Notified Purposes: For other purposes for which we provide specific notice at the time the\n information is collected, and\/or to which you consent.\n \n \n \n \n \n 4. Data Retention \n \n We retain Personal Information for as long as needed or permitted in light of the purpose(s) for which it\n was obtained and consistent with applicable law. The criteria used to determine our retention periods for\n each category of Personal Information include:\n \n \n \n • \n The length of time we have an ongoing relationship with you and the length of time thereafter during\n which we may have a legitimate need to reference your Personal Information to address issues that may\n arise consistent with the purpose(s) for which it was collected;\n \n • \n Whether there is a legal obligation to which we are subject that requires us to retain information for,\n or delete information after, a certain period of time; and\n \n • \n The length of time we use the information to detect and prevent fraud and security incidents.\n \n \n \n \n \n 5. Your California Privacy Rights \n \n If you are a California resident, you may exercise the following rights:\n \n \n \n • \n The right to know what Personal Information we have collected about you, including the categories of\n Personal Information, the categories of sources from which the Personal Information is collected, the\n business or commercial purpose for collecting, selling, or sharing Personal Information, and the\n categories of third parties to whom we disclose Personal Information.\n \n • \n The right to access the specific pieces of Personal Information we have collected about you in a\n portable and, to the extent technically feasible, readily usable format.\n \n • \n The right to correct inaccurate Personal Information that we maintain about you.\n \n • \n The right to request deletion of the Personal Information that we have collected about you.\n \n • \n The right to opt out of the sale of Personal Information, or the sharing of Personal\n Information for cross-context behavioral advertising. We do not sell or share (for cross-context\n behavioral advertising purposes) your Personal Information, including if you are under 16 years old.\n \n • \n The right to limit our use or disclosure of your sensitive Personal Information. We do not use or\n disclose your Sensitive Personal Information for purposes other than the limited purposes permitted by\n the CCPA.\n \n • \n The right to be free from discriminatory treatment for the exercise of your privacy rights.\n \n \n \n \n To submit a request to exercise your privacy rights, you may email us at dpo@griddynamics.com. Such requests\n are subject to our ability to reasonably verify your identity, and we may request additional information\n from you in order to do so. To verify your request, please follow the instructions provided in our response\n to your request. After we validate your request, we will process your request in the timeline set forth in\n the CCPA.\n \n \n If you are acting as an authorized agent of a California resident to make a request on their behalf, please\n provide us with written authorization signed by the resident, and we will contact you and the resident with\n further steps required to verify the resident\u0027s identity.\n \n \n \n 6. Contact Information \n \n If you have any questions, comments, or concerns about our processing activities, please email us at\n dpo@griddynamics.com\n \n \n "}]' data-id='58170' data-recaptcha='6LfbTHsoAAAAAFoatK-U3moWm-rkpo1pm611q5QF'> Apply to the position Country of application* I decline I accept Information on personal data processing You cannot apply for a position without accepting “INFORMATION ON PERSONAL DATA PROCESSING” Read again Submitting Applications for this job are no longer accepted. Please explore other open opportunities on our platform. Thank you! You applied for the position ML Engineer - LLM Evaluation & Automation successfully. We will get back to you soon. Have a great day! Something went wrong... There are possible difficulties with connection or other issues. Please try to use another browser (it's recommended to use the latest version of Google Chrome browser). If the problem still persists, please send your application to cv@griddynamics.com Retry Something went wrong... Please double-check the information filled in the form, and make sure to provide valid data. Retry Don’t see the right opportunity? Contact us anyway and let’s talk! To apply, send your resume and cover letter to jobs@griddynamics.com Grid Dynamics is an equal opportunity employer. We are committed to creating an inclusive environment for all employees during their employment and for all candidates during the application process. All qualified applicants will receive consideration for employment without regard to, and will not be discriminated against based on, age, race, gender, color, religion, national origin, sexual orientation, gender identity, veteran status, disability or any other protected category. All employment is decided on the basis of qualifications, merit, and business need. Grid Dynamics Privacy Policy and E-verify

Requirements

Must-haves

US Remote, US
Design and implement automated systems and pipelines for evaluating LLM outputs.
Develop metrics and KPIs to measure output quality, accuracy, and consistency using LLM-based evaluations
Collaborate with Engineering teams to create automated logic checks and validation tools.
Partner with Data Scientists to analyze evaluation results and optimize prompt and task structures.
Provide feedback loops to ensure evaluation guidelines align with LLM-based assessments.
Investigate how LLM-derived evaluations can enhance product reliability and user experience.
Recommend refinements to prompt engineering, evaluation strategies, and automation tools.
Stay informed on emerging trends in LLM evaluation, automated quality assessment, and AI toolchains.
Continuously improve and expand automated evaluation processes based on industry best practices.
5+ years of experience in ML engineering, NLP, or AI/ML automation.
Strong understanding of machine learning principles with focus on NLP and advanced LLM capabilities (e.g., Chain-of-Thought, agentic workflows)
Expertise in building automated evaluation or QA pipelines.
Excellent analytical and problem-solving skills with experience in root cause and error pattern analysis.
Proven project management and cross-functional collaboration experience.
Excellent communication skills to convey complex insights to technical and non-technical audiences.
Detail-oriented mindset with a focus on evaluation metrics, prompt design, and automation.
Ability to quickly adapt to new business rules and evaluation guidelines across diverse product domains.
Strong programming skills in Python and SQL.
Experience with big data technologies like PySpark for data aggregation and sampling is a strong plus
Bachelor’s/Master’s degree in Computer Science/ Engineering or a related field.
Opportunity to work on cutting-edge projects
Work with a highly motivated and dedicated team

Nice-to-haves

Hands-on experience in prompt engineering and designing LLM-based evaluation systems is preferred

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

machine learningllmaimlpythonsql

Platform: careers-html
Effort: medium

Benefits

Benefits detected in the listing.

Dental

Vision

Professional Development