Expert Software Engineer (10+ Years) - AI Evaluation Tasks, Prior RL Data Work Required (Remote)

Patronus AI is a frontier lab building simulation research and infrastructure to accelerate progress toward human-aligned AGI. We built some of the earliest research in AI evaluation. That includes FinanceBench, the first large-scale benchmark of how language models handle financial analysis; CopyrightCatcher, which tested leading models for copyright infringement; Humanity's Last Exam; SimpleSafetyTests; and Lynx, a model that detects hallucinations in AI output. Our team comes from Meta AI, Amazon AGI and Google. Our customers include foundation model labs and Fortune 500 companies, and we are backed by Lightspeed Venture Partners, Notable Capital and others. At Patronus AI we are looking for highly experienced software engineers (task-based contract) to design realistic, challenging tasks for frontier AI models and to judge how well those models handle them. You'll apply your day-to-day engineering judgment to show where AI gets code, reasoning and technical instructions right, and where it falls short. This posting is for engineers who have already done this kind of work professionally. At least 6 months of paid data work with an RL / AI training-data company is a hard requirement; applications without it will not be considered. What you'll do (flexible, typically 15-20 hours per week) - Design realistic, multi-step coding and technical tasks that test an AI model's ability to reason, write code and follow precise instructions. - Build tasks around realistic workspace materials such as codebases, technical specs, architecture docs and datasets. - Evaluate AI-generated code and technical content for correctness, efficiency, clarity and alignment with real-world engineering practice. - Grade model outputs against rubrics and ground-truth answers, and write clear, structured feedback. - Help improve how AI understands programming, system design, data workflows and engineering best practices. - Work independently and asynchronously, meet deadlines, and collaborate with the Patronus team to keep quality high. What we're looking for - 10+ years of professional experience as a software engineer (internships excluded). - MANDATORY: at least 6 months of paid data work with an RL / AI training-data company such as Mercor, Handshake, Surge AI, Turing or micro1 (or a comparable company), for example task authoring, code review or rubric grading, RLHF comparisons or model evaluation. Tell us where, for how long and what you did in the screening question. - Strong coding skills in at least one major language, such as Python, Java, C++, JavaScript, TypeScript or Go. - Experience building, reviewing and debugging production-grade code. - Ability to follow detailed technical guidelines and work independently. - Excellent written communication and close attention to detail. - A bachelor's degree in computer science, engineering or a related field, or equivalent experience. - Experience with system design, data pipelines, ML or LLM-based applications. - Prior work in code review, technical writing, AI training or model evaluation. Our standards - All tasks, project files and solutions are human-authored. LLM-generated code, content or text is not permitted. - Tasks and solutions must be original. Nothing may be copied from published or easily searchable sources, including tutorials and marketplace content. - Never include code, assets or project files from a current or former employer, client or studio, or anything covered by an NDA. All companies, projects and scenarios are fictional, and all code and content is your own. - Quality is judged on accuracy, rubric validity, difficulty and reviewer agreement, not volume. Rate and logistics - $250 to $350 USD per hour depending on experience. Task-based contract, remote, flexible, typically 15-20 hours per week. - Please answer the screening question yourself, without AI assistance; we are hiring for your judgment. How hiring works 1. Apply here with your resume and links to your GitHub, LinkedIn or portfolio. 2. Complete a short AI interview. 3. Onboarding and your first project.