Software / AI Evaluation Engineer (Terminal-Bench, Docker & Agentic Evals)

We are looking for an experienced Software Engineer / AI Evaluation Specialist to design, author, and validate terminal-agent evaluation tasks for Terminus 3 project (built on the open Terminal-Bench 3.0 / Harbor format). In this role, you will create reproducible, isolated environments and complex real-world software engineering challenges to evaluate state-of-the-art AI agents. Key Responsibilities: Author end-to-end tasks containing agent Dockerfiles, verifier test suites, prompt specifications (instruction.md), and oracle reference implementations (solution/solve.sh). Configure multi-container verifier isolation to eliminate reward-hacking, ensure strict artifact sharing, and verify zero data leakage of tests or solutions into the agent's environment. Write deterministic, outcome-based grading tests that inspect the final environment state. Calibrate task difficulty across predefined tiers (Base, Core, Advanced, Frontier) by benchmarking pass rates against leading LLM agents. Run local validations using stb harbor CLI to ensure reproducible builds, pinned dependencies, and clean checklist runs before submission. Required Qualifications & Skills: * Linux & Containerization: Advanced expertise in Docker, multi-stage builds, shell scripting (Bash), and isolated container environments. * Testing & Verification: Proven experience building deterministic Python test suites (e.g., PyTest) that evaluate system states rather than trivial outputs. * Domain Breadth: Strong proficiency in at least one key domain:Software & Systems: Systems programming, OS internals, Databases, Data Engineering, or Compilers. Target Languages: Python, C/C++, Rust, Go, TypeScript, Java, or C#. Specialized Domains: Cybersecurity (AppSec, Forensics, Reverse Engineering), ML Engineering, or Science/Operations. * Prompt Engineering & Specification: Ability to write clear, real-world ticket-style requirements stating the goal rather than the step-by-step solution. * Reproducibility Mindset: Deep attention to detail regarding version locking, deterministic execution, and debugging platform vs. test issues. Preferred Qualifications: ~ Prior experience contributing to Terminal-Bench, SWE-bench, or similar LLM agent evaluation frameworks. ~ Direct familiarity with the stb CLI toolchain. Key Candidate Screening Questions“ * Have you developed benchmarks or evaluation tasks using Docker and Python for autonomous coding agents (like Terminal-Bench or SWE-bench)?” * How do you design a deterministic test suite that verifies a task was truly solved without allowing an LLM agent to reward-hack or leak test data?” * Which programming languages and domain specialties (e.g., Systems, Security, ML, Algorithms) are your strongest?”