AI Engineer / DevOps: Self-Hosted LLM Platform + AI Agents with LiteLLM and Third-Party Model APIs
Preferred qualifications
- Experience: Expert
- Job Success: 90%+
- Min. earnings: $10,000+
We are a growing digital company with an internal team. We want our own private AI workspace with agents, a "Grok / ChatGPT / Claude"-style experience that we control, for our team.
The app layer and agent harness run on our own server. The models do NOT run on our server: everything talks to third-party LLM APIs (DeepSeek, Qwen, Kimi/Moonshot, GLM, MiniMax, OpenRouter and similar) through a gateway such as LiteLLM.
THE MOST IMPORTANT PART: it must not feel worse than a fully managed product like Grok, ChatGPT or Claude. Fast, streaming, good default models and system prompts, routing and fallbacks, web search that works, file handling, memory, and agents that actually finish tasks.
WHAT YOU WILL DO
- Recommend and set up the stack: LLM gateway (e.g. LiteLLM), chat UI (e.g. Open WebUI, LibreChat, LobeChat, AnythingLLM, Onyx) and agent harness (e.g. OpenHands, Dify, n8n, MCP servers). We want your expert opinion.
- Users, roles and permissions: who can use which models, tools and agents.
- Per-user / per-team budgets and spend limits, API key handling, usage logging.
- Auth / SSO, secure deployment (Docker), backups, monitoring.
- Knowledge base / RAG so the team can upload our own files and get good answers.
- Agents with tools (web search, files, code, MCP) tuned to really complete tasks.
- Tune models, prompts, routing and fallbacks so speed and quality feel like a managed product.
- Full documentation and handover to our in-house developer.
- Training sessions for the team so we can load our files and run it ourselves.
DONE MEANS
- Workspace live on our server, team logged in with correct roles and model access.
- Spend limits and logging working per user.
- At least 5 providers connected with fallbacks; web search, file upload/RAG and at least 2 working agents.
- Side-by-side test vs a managed product on our real tasks with no clear drop in quality or speed.
- Written docs (setup, config, how to add models/users/agents, troubleshooting).
- Handover call(s) with our developer, who can then maintain it alone.
- Team training session(s) completed.
ABOUT YOU
- You have set up self-hosted LLM workspaces or agent platforms for real teams before.
- Hands-on with LiteLLM or similar gateways, Open WebUI / LibreChat or similar, Docker, Linux, Python.
- Strong reviews and a clear track record. Clear English and good at explaining.
Part time / as needed, about 1-3 months. Please answer the screening questions and share examples of past setups.
Open job
AI proposal draft
Generate a short cover letter to copy into the offer. Says you are interested and ready to work.
Sign in to generate an AI proposal draft.
Log in