Research Scientist – LLM Evaluation & Benchmarking

About the role

Anyone AI Labs is seeking a Research Scientist to lead the design and execution of evaluation benchmarks for frontier large language models. This remote position focuses on creating reliable, high‑impact measurements that inform both internal labs and the broader AI community.

Key responsibilities

• Conduct evaluation research and translate measurement goals into original benchmark designs.
• Develop comprehensive evaluation packages covering reasoning, coding, agentic tool use, and multimodal tasks, ensuring expert‑verified ground truth and multi‑model validation.
• Recruit, calibrate, and manage subject‑matter experts, serving as the final arbiter of correctness and difficulty.
• Act as the technical liaison for partner labs, translating their measurement needs into robust evaluation designs.
• Deliver pilot evaluation packages, and when applicable, turn successful work into public benchmarks and peer‑reviewed publications.

Required profile

• Research background in machine‑learning evaluation or benchmarking with a record of published or widely used benchmarks.
• Deep expertise in LLM and frontier‑model benchmarking, especially for code and agentic evaluations.
• Strong understanding of measurement concepts such as construct validity, psychometrics, rubrics, headroom, and contamination.
• Interest in safety‑oriented evaluation, capability elicitation, and robustness.
• Proven ability to lead expert pools and maintain rigorous standards throughout the research loop.
• Fluent English; Spanish is a plus.

Required skills

What we offer

• Opportunity to shape the future of AI evaluation on a global scale.
• Collaboration with top AI labs and researchers.
• Support for publishing in leading conferences such as NeurIPS, ICLR, and ACL.