AI Developers Needed: Evaluate New Foundation Model on Real-World Projects

OVERVIEW Based Models is an AI research lab recruiting 5 experienced AI developers for a closed beta. We’re evaluating a causal reasoning layer that boosts mid-tier open model performance. We're looking to benchmark our model against frontier models (Claude, GPT) on real-world coding projects and agentic workflows. We've already tested on public benchmarks (SWE-bench, Terminal-Bench, tau²). Now we want to measure performance on actual projects developers have already built, using existing automated tests and evaluation pipelines. API credentials and a short integration note will be provided upon selection. TWO EVALUATION TRACKS Track A: AI Coding Agents Rerun coding tasks from real projects you've completed using coding agents (Claude Code, Cline, Aider, etc.). Compare our model against the frontier model originally used on the same tasks. Use existing unit tests, integration tests, or other automated tests to measure correctness. Report task success rates, failure modes, token usage, execution time, and cost. Track B: Agentic Workflows Evaluate our model within existing agentic workflows you've built or deployed for clients. Replace the current frontier model with our model while keeping the workflow and tools consistent. Use existing automated evals to compare task completion, reliability, efficiency, and cost. Identify where our model succeeds, fails, or requires additional retries. REQUIREMENTS - Must have existing real-world projects with automated tests or evals that can be reused for benchmarking. This is a hard requirement. - Hands-on experience building projects with AI coding agents (Track A) or developing LLM-powered agentic workflows (Track B). - Ability to run the same tasks with different models and objectively compare results. - Experience integrating LLM APIs into coding agents or workflows. - You may use non-sensitive or personal projects. - Expected time commitment is approximately 2–4 hours per developer (swap API endpoint, re-run existing workflow, and write a concise summary). We are NOT looking for developers to create new benchmarks or synthetic exercises. The goal is to evaluate our model on real projects with established success criteria. DELIVERABLES Each developer will provide: **Evaluation Setup**: Brief description of the real-world project or workflow, existing tests/evals, coding agent (Track A) or agentic pipeline (Track B), and frontier model baseline. **Comparative Results**: Side-by-side benchmark of Based Models’ model vs. the frontier model on the same tasks, using identical evaluation criteria. Metrics include task success rate, token usage, execution time, and cost per successful task. **Execution Logs**: Reproducible test results, model outputs, and execution traces documenting successes, failures, and retries. **Evaluation Summary**: Concise analysis of where Based Models’ model outperforms or underperforms the frontier model, key failure modes, and actionable recommendations. All evaluations must use existing real-world projects and automated tests/evals. Deliverables should provide objective, reproducible evidence of model performance - not subjective impressions. A concise write-up is sufficient. Using an agent to help draft the summary is fine, provided the metrics, logs, and side-by-side comparison are accurate and reproducible. HOW TO APPLY Please briefly answer: - Which track are you applying for - A (Coding Agents) or B (Agentic Workflows)? - What real-world project or workflow have you built that could be used for evaluation? - What existing automated tests or evals can you reuse? Please provide specific examples. - Which coding agent or frontier model did you originally use? - Can you rerun the same tasks with a new model and objectively compare results? Applications without existing projects and automated evaluation capabilities will not be considered. COMPENSATION Fixed price of $250 per completed evaluation, paid upon delivery of the four listed deliverables. Limited to five developers.