Applied Research Scientist, LLM Evaluation & Post-Training

Position SummaryInnodata is expanding its GenAI research capability to advance state-of-the-art evaluation and post-training methods for LLM and multimodal systems. As an Applied Research Scientist, LLM Evaluation & Post-Training, you will lead research and experimentation on how evaluation design, measurement strategies, and feedback signals influence model improvement.This role is ideal for a technically rigorous researcher who is deeply fluent in modern LLM evaluation and post-training, and who can turn research insight into practical methods for customer solutions and internal platform innovation. You will work across human-in-the-loop and AI-augmented workflows, partnering with Language Data Scientists and AI/ML Research Engineers to design and validate evaluation frameworks that drive measurable model gains.The ideal candidate combines strong experimental and statistical judgment with hands-on technical ability and can engage as a peer with research and engineering stakeholders at leading AI companies.
Who We're Looking ForYou have at least 5+ years of relevant experience (including graduate research) in applied ML research, research science, or advanced ML experimentation, with significant experience in LLM evaluation, benchmarking, alignment, or post-training. You have a track record of designing high-quality experiments, interpreting results rigorously, and translating findings into practical improvements.You are comfortable working across research and product/customer contexts. You can identify important methodological questions, build a research agenda, and collaborate with engineers and data experts to execute. You understand that evaluation is not only about metrics, but about measurement validity, robustness, stress testing, and alignment to real-world usage.You are excited by frontier challenges including long-context, cross-modal, and dynamic multi-turn evaluations, and by the opportunity to build new benchmark datasets and evaluation frameworks that become strategic assets for Innodata and its customers.You bring an implementation-minded approach to experimentation and are comfortable collaborating closely with engineers to productionize methods and research outputs when appropriate.
Tell me moreAs an Applied Research Scientist, LLM Evaluation & Post-Training, you will help define the next generation of evaluation-driven model improvement workflows. You will study how different evaluation approaches (human, automated, hybrid) shape model selection and post-training outcomes, and you will design experiments that produce credible, actionable conclusions.Your work may include designing benchmark datasets, developing evaluation taxonomies and protocols, defining metrics and scoring methodologies, analyzing failure modes, and testing how changes in evaluation setup affect downstream fine-tuning results. You will also support customer engagements by bringing scientific rigor to evaluation strategy, methodology review, and technical recommendations.This is a highly collaborative role that sits at the intersection of research, engineering, and language/data operations.
ResponsibilitiesDefine and execute a research agenda focused on LLM evaluation and post-training, especially evaluation-driven model improvementDesign rigorous experiments to study how evaluation methodologies impact fine-tuning and post-training outcomesDevelop and validate evaluation frameworks for LLM and multimodal systems, including:benchmark/task designscoring methodsjudge/model-assisted evaluationhuman evaluation protocolsrobustness/stress testingLead research on advanced evaluation domains, including long-context, cross-modal, and dynamic multi-turn evaluationsStudy the effectiveness and limitations of existing evaluation techniques, and propose improved methodologies with clear validity and scalability tradeoffsAnalyze model behavior and failure patterns; generate actionable recommendations for model improvement and evaluation redesignCollaborate with AI/ML Research Engineers to translate research methods into scalable evaluation and post-training pipelinesCollaborate with Language Data Scientists to integrate human-in-the-loop and synthetic data/evaluation strategies into research programsEngage with customer technical stakeholders to understand evaluation goals, review methodologies, and provide expert recommendationsContribute to internal benchmark datasets, evaluation frameworks, and reusable research assetsProduce high-quality technical documentation, internal research reports, and client-facing materials explaining methods, results, assumptions, and limitationsContribute to thought leadership and best practices in LLM evaluation, post-training, and GenAI quality measurement
RequirementsMS/PhD in Computer Science, Machine Learning, Statistics, Applied Mathematics, AI, or a related quantitative scientific field (PhD strongly preferred)5+ years of relevant experience in applied research / research science in ML/AI, with substantial work in LLMs or foundation modelsDemonstrated experience with LLM evaluation, benchmarking, alignment, post-training, or model quality researchStrong foundation in experimental design, statistical analysis, and scientific reasoning for ML systemsStrong coding skills in Python for research experimentation and analysis (e.g., data processing, evaluation pipelines, statistical analysis, visualization)Experience working with modern ML tooling/frameworks (e.g., PyTorch, Hugging Face, JAX/TensorFlow as applicable) sufficient to design and execute model/evaluation experimentsAbility to evaluate and compare human and automated evaluation methods, including tradeoffs in cost, reliability, validity, and scalabilityExperience designing evaluation studies and protocols that are reproducible across datasets, model versions, and evaluation runsAbility to collaborate directly with technical stakeholders including research scientists, ML engineers, data scientists, and customer technical counterpartsStrong communication skills and ability to present nuanced technical conclusions, assumptions, and limitations clearlyTechnical skillsEvaluation Science & BenchmarkingExperience designing benchmark datasets, test suites, or evaluation frameworks for language or multimodal modelsDeep understanding of metric design, scoring reliability, and measurement validityExperience with human evaluation methods and quality assurance considerations (e.g., rubric design, inter-rater reliability, adjudication frameworks)LLM / Post-TrainingUnderstanding of post-training methods and how training objectives interact with evaluation outcomesAbility to reason about model behavior, failure modes, and tradeoffs across tasks/domainsFamiliarity with alignment and robustness considerations in model evaluationQuantitative AnalysisStrong statistical analysis skills (sampling, uncertainty, significance testing where appropriate, error analysis, metric interpretation)Ability to synthesize complex experimental findings into actionable recommendationsPreferred SkillsHands-on experience running or supporting fine-tuning/post-training experiments (SFT, preference optimization, RLHF/RLAIF-style workflows)Experience with multimodal evaluation (e.g., text-image, audio, video)Experience with long-context benchmarking/evaluation and real-world context management challengesExperience designing multi-turn, interactive, or agentic evaluation protocolsPublished research and/or open-source benchmark contributions in LLM evaluation, post-training, alignment, or related areasExperience in customer-facing applied research, technical consulting, or cross-functional product/research collaborationsFamiliarity with safety, trustworthiness, and governance considerations in GenAI evaluationHow this role partners with the teamThis role works closely with:Language Data Scientists, who bring deep expertise in language data, human evaluation workflows, multilingual/multimodal process design, and data quality operationsAI/ML Research Engineers, who implement scalable training/evaluation systems and connect research methods to production-grade pipelinesBusiness and Customer Teams, who rely on Innodata for expert consultation and credible, technically rigorous GenAI solutionsInternal R&D and Platform Teams, to transform research outputs into reusable frameworks, benchmarks, and differentiated offerings
DEIB CommitmentInnodata is committed to building an inclusive and diverse workplace where everyone is respected, valued, and empowered to contribute. We welcome applicants from all backgrounds, regardless of age, gender, sexual orientation, gender identity or expression, disability, ethnicity, religion, civil status, nationality, or any other characteristic protected by law.We believe that diverse perspectives strengthen our teams, our culture, and the work we deliver to our clients. All employment decisions are based on qualifications, merit, business needs, and alignment with the role.