Senior Software Engineer – AI Coding Agent Evaluation & Task Development (Long-Term)

We're looking for experienced software engineers to help create realistic coding tasks used to evaluate AI coding agents. The work involves taking existing codebases and designing practical engineering tasks, such as implementing new features or fixing bugs. You'll write working solutions, build comprehensive tests, and evaluate how well AI agents handle these tasks. If you have strong coding, debugging, and testing skills and are comfortable working with unfamiliar codebases this is for you. Hands-on experience with AI coding agents (Claude Code, Codex, or similar) is required. Familiarity with SWE-bench, Terminal-Bench, or similar evaluation frameworks is highly preferred. Typical Workflow for Each Task: 1. Choose an existing codebase (any programming language or stack) and design a realistic feature or bug fix that builds on it. 2. Write clear instructions for an AI coding agent, including relevant context and numbered acceptance criteria. 3. Implement a working reference solution and create a hidden, black-box test suite to verify correctness. 4. Run AI coding agents on the task, review their code changes, and identify cases where incorrect solutions pass the tests. Improve the tests to catch these issues. 5. Document your findings and provide feedback on how the agents performed. This is a long-term opportunity with ongoing projects and flexible working hours. We're looking for reliable engineers who can work independently, pay close attention to detail, and consistently deliver quality work. Please share: - A brief overview of your engineering background and strongest programming languages. - Your experience with AI coding agents or evaluation frameworks. - Your weekly availability and how soon you can start.