Optimize a Decision-Tree AI Agent to Run on a Local Open-Source LLM (Qwen)

Freelance Mission: Optimize a Decision-Tree AI Agent to Run on a Local Open-Source LLM (Qwen) Context We built an AI assistant POC powered by a decision tree that encodes a complex regulatory framework (general instructions on foreign exchange operations). The POC is built with Lovable (React, TypeScript, Supabase) and works very well with cloud models (OpenAI and others). The end client requires on-premise execution with an open-source model that fits on a single 32 GB VRAM GPU. Our current prompt sends far too many tokens for a model of this size, and that is the main blocker. Your mission Make the POC work with Qwen (or any open-source model that fits in 32 GB of VRAM) while hitting these targets: Accuracy: over 80% correct answers on our test dataset Latency: under 5 seconds per response, under 3 seconds ideally Controlled context: a significant reduction in the tokens sent to the model Ideas to explore (feel free to propose better) Node-by-node or sub-branch traversal of the tree instead of sending it whole in the prompt Pre-filtering or retrieval of the relevant branches before the LLM call Prompt compression and restructuring, constrained structured outputs (JSON) Model and quantization choice (Qwen 14B, quantized 32B, MoE...), prefix caching, serving stack (vLLM or equivalent) Building a small reproducible evaluation harness (accuracy and latency) What we provide Access to the GitHub repository Access to DeepInfra for model testing Test dataset with expected answers Demo of the existing POC and a context handover session with us Deliverables Working POC on the selected model, meeting the accuracy and latency targets Short report: benchmark comparison of the models tested, choices made, limitations Code merged into the repo, with documentation to reproduce the tests Profile Strong LLM engineering experience: advanced prompt engineering, context optimization, evaluation Hands-on with open-source models (Qwen, Llama, Mistral...), quantization and serving (vLLM, Ollama or equivalent) Comfortable with TypeScript / Node (the codebase is React / Supabase / TypeScript); Python is a plus Autonomous and rigorous about measurement: we decide on numbers, not gut feeling Terms Estimated duration: 16 to 24 hours Rate: $40 to $50/hour depending on profile Start: as soon as possible Remote NDA signed before access to data To apply: send your profile and an example of an LLM optimization project (context, latency or cost)