Test case catalogue and evaluation tooling for a form-filling AI

We are a non-profit research organisation in Germany building an AI assistant that helps people with limited German complete government forms. The project is publicly funded by the German Federal Ministry of Labour and Social Affairs and runs until mid 2027. This contract is about measurement. We need a test case catalogue with reference solutions, plus an evaluation routine that reports precision, recall and F1 across the test set, so we can tell whether the assistant actually gets the fields right. DELIVERABLES, FOUR MILESTONES 30 Sep 2026: test case concept, first catalogue, metric definitions 31 Oct 2026: evaluation routine running, first measurement report 30 Nov 2026: catalogue extended, edge cases and failure modes covered 31 Dec 2026: consolidation, documentation, handover Scope per milestone is agreed with us at the start of each period, so there is room to adapt as the project learns. IMPORTANT This contract goes to someone who did not produce the datasets being measured. Whoever creates the material does not write the tests it is judged by. If you are also proposing on our data schema or dialogue dataset contracts, please tell us, as we cannot award both to the same person. WHAT WE ARE LOOKING FOR Evaluation experience for machine learning or structured extraction tasks, comfort defining metrics, and clean reproducible tooling. BUDGET AND DEADLINE Budget: 2,800 EUR net, fixed price, paid per accepted milestone. Please quote in EUR. Deadline for proposals: 12 August 2026, 12:00 CEST. How we will decide: price 40 percent, experience with structured extraction from forms or PDFs 20 percent, experience with multilingual datasets 20 percent, availability across all four milestones 20 percent. PLEASE ADDRESS THE FOLLOWING IN YOUR PROPOSAL 1. Describe one project where you built an evaluation or test harness. What did you measure, and how? 2. Have you worked with multilingual datasets? Which languages, and what was your role? 3. Can you commit to four deliverables between now and 31 December 2026? 4. What is your fixed price for the full scope described? TERMS Fixed price contract under German law, payment per accepted milestone, all results transfer to us. The work will later be published under an open source licence.