Senior Software / Data Engineer for AI Coding Agent Evaluation (Long-term, 40 hrs/week)

We're putting together a small team of experienced engineers to test AI coding agents before frontier labs release them. This is real engineering work. You get real tickets on real codebases, you check what the agent produces, and you explain clearly where it went wrong and why. This is not a short gig. Work continues week after week, 40 hours a week is available, and strong performers can be approved for more. You choose your own hours. What the work looks like - Review the agent's output like a PR from a teammate. Run it, test it, try to break it. Then write up what's wrong, what's missing, and what's just sloppy. - Pair with the agent on a problem you know well. Work through it turn by turn and call out each point where the model goes off track, bluffs, or misses something. - Write tasks that stump the agent. Build realistic engineering problems with tests that prove whether a solution is actually correct. Labs use these to measure their models. - Write rubrics and prompts. Define what a good answer looks like and make the grading criteria clear enough that someone else can apply them consistently. You don't need an AI or ML background. What we need is your engineering judgment, written in your own words and backed by things you actually ran. Requirements - 4+ years of professional software engineering experience. Data engineers who write production code (Python, SQL, Scala, Java, etc.) are welcome. - A degree in Computer Science, Computer Engineering, Software Engineering, or a closely related field. Unrelated degrees don't qualify, and this one is firm. - Any main language is fine: Python, Java, Go, Kotlin, C++, TypeScript, or anything else. - You read code critically and don't just accept what a tool gives you. You'll use AI tools every day, and the point is knowing when they're wrong. - Strong written English. Most of what you deliver is written explanation. You must pass the assessment We only move forward with candidates who pass a technical assessment, and you must take it yourself. It's 3 hours total, and you can do the sections in any order: 1. Coding (DSA): one data structures and algorithms problem, about 30 minutes. 2. AI coding agent evaluation: one short scenario, 15 to 30 minutes. You'll either flag where a model's response went wrong and compare it against a second response, or write a clear task brief for a coding task. 3. Code review: a PR-style diff, about 20 minutes. We grade correctness and overall quality (style, maintainability, design), not just bug-finding. 4. Rubric and prompt writing: 60 to 75 minutes. We grade on clarity, correctness, and reasoning. A right answer with no explanation scores lower than a well-reasoned one. Ground rules (no exceptions) - Your own words only. No AI-written text in your reviews, write-ups, or assessment answers. - Evidence for every claim. If you say something is broken, show what you ran and what happened. One confirmed breach of any of these ends the engagement. How the contract works - Contract renews weekly as long as your work meets the quality bar. - Up to 40 hrs/week, with more available for strong performers. - Fully remote, flexible schedule. To apply, answer the screening questions below. Generic or AI-generated proposals will be declined. We'd rather read three honest sentences than a polished page that says nothing.