RAG/LLM Engineer for Pinecone

RAG/LLM Engineer – Pinecone Optimisation, AI Chatbot Debugging & Accuracy Improvement Technology: Retrieval-Augmented Generation (RAG), Pinecone, LLMs About LawNeeds LawNeeds is a UK-based LegalTech company operating a mobile-first platform that combines AI-powered UK immigration information with access to regulated legal professionals. Our mobile application is commercially live, and our AI chatbot uses Retrieval-Augmented Generation (RAG), with Pinecone as its vector database, to provide informational guidance on UK immigration matters. We are looking for an experienced RAG/LLM Engineer to take independent ownership of a focused four-week project aimed at improving the legal accuracy, relevance, consistency and reliability of our existing chatbot. This is not a project to build a new chatbot. We need someone who can diagnose and fix problems within an existing RAG system. Project Background (Screening questions at the bottom) Our in-house legal counsel recently conducted an audit of the chatbot and identified recurring inaccuracies affecting response quality. These include: - Outdated immigration requirements and financial thresholds. - Confusion between different immigration routes and eligibility requirements. - Incorrect or irrelevant legal citations. - Failure to distinguish current immigration rules from proposed policy changes. - Incomplete responses that omit important exceptions or transitional provisions. - Incorrect interpretation of user questions and circumstances. - Failure to ask necessary clarifying questions. - Inconsistent responses to related immigration scenarios. We have also engaged a UK immigration law specialist who will conduct structured legal accuracy testing throughout the four-week project. We therefore need a technical specialist who can work directly with the legal tester, investigate the underlying technical causes of identified inaccuracies, implement improvements and verify the technical performance of the updated system. Key Responsibilities 1. Conduct a Technical Audit of the Existing RAG System Review and investigate the chatbot's existing architecture, including: Document ingestion, parsing, chunking and indexing. Embedding generation and vector database configuration. Pinecone indexes, namespaces and metadata structures. Retrieval strategies, similarity matching and relevance ranking. Source selection, filtering and retrieval of outdated information. Context construction and how retrieved information is passed to the LLM. System prompts, response-generation instructions and conversational context handling. Citation generation and grounding of chatbot responses in authoritative sources. Identify the likely technical causes of the recurring inaccuracies and develop a prioritised remediation plan. 2. Implement RAG and Pinecone Optimisation Based on the technical audit, independently implement appropriate improvements, potentially including: Optimising Pinecone retrieval queries and search configurations. Improving document chunking, embeddings and indexing strategies. Refining metadata filtering to distinguish immigration categories, legal effective dates and document versions. Improving retrieval relevance and reducing the use of outdated or irrelevant information. Introducing or improving hybrid search and reranking where technically justified. Strengthening how retrieved sources are selected and incorporated into chatbot responses. Improving citation reliability and ensuring generated answers are supported by relevant retrieved material. Refining prompts and contextual instructions to improve interpretation of user questions. Improving the chatbot's handling of ambiguous questions, missing information and follow-up conversations. Reducing hallucinations, unsupported statements and inconsistent responses. We expect the engineer to investigate and address root causes rather than relying solely on prompt modifications. 3. Work Directly With Our Legal Quality Assurance Specialist A UK immigration law specialist will independently test the chatbot throughout the engagement. The legal specialist will provide structured findings identifying inaccurate responses, the correct legal position and supporting authoritative sources. The engineer will be expected to: Review legal audit findings as they are submitted. Investigate the technical causes of identified inaccuracies. Implement appropriate technical corrections. Return corrected scenarios for legal retesting. Address any recurring issues identified during retesting. Maintain a clear record of technical changes and outstanding issues. Proactively communicate progress and technical limitations. The project will operate as a continuous testing, correction and retesting process rather than a one-off technical review. The legal specialist will be responsible for verifying legal accuracy. The engineer will be responsible for diagnosing and implementing technical improvements. 4. Take Independent Ownership of the Technical Work stream We already have a lead developer responsible for the wider application. The successful freelancer will work alongside our lead developer, who will provide necessary access, technical context and support with integration or deployment where required. However, we are specifically seeking someone who can independently manage the RAG optimisation process. The freelancer should be able to: Assess the existing system without requiring continuous technical supervision. Identify and prioritise technical issues. Propose and implement appropriate solutions. Manage the remediation workflow. Coordinate directly with the legal QA specialist. Maintain technical documentation and progress reports. Escalate blockers promptly. Our lead developer will retain responsibility for access controls, security oversight and production deployment approval. 5. Implement Evaluation and Regression Testing Establish a practical, repeatable process for assessing chatbot performance, including: Creating a baseline from previously identified inaccurate responses. Evaluating retrieval relevance and source selection. Testing whether responses are grounded in appropriate retrieved material. Comparing results before and after technical changes. Running regression tests to ensure improvements do not introduce new problems. Tracking corrected, unresolved and recurring issues. Where appropriate, use established RAG evaluation techniques or frameworks. 6. Documentation and Handover Provide clear documentation of: Technical problems identified. Root-cause findings and supporting evidence. Changes implemented. Evaluation results and improvements achieved. Outstanding technical limitations. Recommendations for ongoing RAG maintenance and optimisation. All agreed code changes and configurations must be handed over to LawNeeds. Required Skills and Experience Applicants must demonstrate: 1) Strong practical experience developing, debugging and optimising RAG-based AI applications. 2) Hands-on experience using Pinecone in production or comparable real-world systems. 3) Experience with vector search, embeddings, indexing, metadata filtering and retrieval optimisation. 4) Experience diagnosing inaccurate, hallucinated or poorly grounded LLM responses. 5) Strong knowledge of LLM APIs, prompt engineering and context management. 6) Experience with backend development and debugging, preferably Python or the language used by the existing system. 7) Experience implementing structured RAG evaluation and regression testing. 8) Ability to work independently within an existing codebase. 9) Strong communication, documentation and problem-solving skills. Preferred Experience - Previous experience improving RAG systems used in LegalTech, regulatory, financial or other knowledge-intensive applications. - Experience handling frequently updated information sources. - Familiarity with document versioning, temporal filtering and effective-date metadata. - Experience with hybrid search, reranking and retrieval evaluation. - Experience with LangChain, LlamaIndex or equivalent frameworks. - Previous collaboration with legal professionals or subject-matter experts. - Experience optimising production systems without disrupting existing users. UK immigration law expertise is not required, as legal accuracy will be assessed by our immigration law specialist. Four-Week Project Structure Week 1 – Technical Assessment and Initial Corrections Review the existing RAG architecture and Pinecone configuration. Examine the initial legal audit findings. Identify and prioritise technical issues. Establish baseline evaluation tests. Begin addressing critical issues where feasible. Submit a technical assessment and remediation plan. Week 2 – Retrieval and Response Optimisation Implement priority corrections. Improve retrieval relevance, source selection and response grounding. Address findings submitted by the legal QA specialist. Conduct initial regression testing. Document technical changes and progress. Week 3 – Continued Remediation and Retesting Investigate additional issues identified through legal testing. Implement further improvements. Coordinate retesting with the legal QA specialist. Refine retrieval and response-generation processes. Assess whether earlier corrections remain effective. Week 4 – Final Optimisation, Validation and Handover Complete agreed priority corrections. Conduct final regression and performance testing. Review outstanding findings with the legal QA specialist. Produce before-and-after evaluation results. Submit technical documentation and a final handover report. Expected Deliverables By the end of the four-week engagement, we expect: - A documented technical assessment of the existing RAG and Pinecone architecture. - A prioritised technical remediation plan. - Implemented improvements addressing agreed high-priority inaccuracies. - An updated technical issue tracker with correction and retesting status. - A repeatable evaluation and regression testing process. - Evidence of improvements in retrieval relevance, source grounding and response reliability. - Documented code changes and configurations. - A final technical report and complete handover. The objective is to achieve measurable improvements within the four-week period, prioritising the most consequential problems rather than undertaking an unnecessary complete rebuild. Working Arrangements - Remote, four-week fixed-price engagement. - Flexible working hours, subject to agreed milestones and collaboration requirements. - Direct communication with our appointed legal QA specialist. - Limited support from our existing lead developer for access, technical clarification and deployment. - Regular progress updates and transparent issue tracking. - All changes must be tested and approved before deployment to production. - Access will be restricted to what is necessary for the engagement, and confidentiality and data protection requirements will apply. How to Apply Please provide: - A brief description of your experience with Pinecone-based RAG applications. - Examples of existing AI chatbots you have debugged or optimised, particularly where responses were inaccurate, outdated or irrelevant. - An explanation of how you would investigate the technical causes of inaccurate retrieval and response generation. - Details of your experience implementing RAG evaluation and regression testing. - Your proposed approach to managing this four-week project independently while collaborating with a legal QA specialist. - Your availability. - We are particularly interested in candidates who can demonstrate practical experience identifying and resolving RAG performance issues. Please do not apply if your experience is limited to basic chatbot setup, API integration or prompt engineering without hands-on RAG troubleshooting experience. Screening Questions 1) Describe a Pinecone-based RAG application you have personally developed, debugged or optimised. What specific problems did you resolve? 2) A chatbot retrieves outdated immigration rules even though newer documents exist in the knowledge base. How would you diagnose the cause and implement a reliable solution? 3) How would you determine whether inaccurate responses originate from document ingestion, embeddings, Pinecone retrieval, context construction, prompts or the LLM? 4) What metrics or evaluation methods would you use to demonstrate that retrieval relevance, citation accuracy and response quality have improved? 5) How would you independently manage technical corrections while collaborating with a legal specialist who submits ongoing accuracy reports and an existing lead developer who controls deployment?