AI Desktop Automation Contractor for Supervised ERP Writeback Assistant

Overview Bob.ai is looking for an experienced AI automation / desktop automation contractor to help us prototype and build a supervised desktop assistant that can write data back into ERP systems used by Public Housing Agencies. Today, some ERP writeback workflows are handled through RPA. We want to explore a better AI-assisted option where the work happens entirely on the customer’s own Windows desktop session. This is especially important for systems like Elite, where customers access the product through Elite Cloud / hosted desktop environments, and we do not want to depend on the ERP vendor installing Power Automate, UiPath, or other tooling inside their cloud environment. The goal is to build a supervised local desktop assistant that can observe the user’s screen, interact with an authorized ERP session, fill approved fields, pause before final save/submit, and capture an audit trail. What We Are Building We want a local Windows desktop application or prototype agent that can: Run on the customer’s desktop or workstation Observe an ERP session running locally, through Citrix, RDP, virtual app, hosted desktop, or similar Receive structured writeback tasks from Bob.ai Navigate the ERP UI using mouse, keyboard, clipboard, screen understanding, OCR, and/or Windows UI Automation Fill specific fields based on Bob.ai-approved data Stop before final Save / Submit Ask the user for approval before committing changes Capture screenshots and logs for before/after auditability Detect common failure states such as missing fields, validation errors, permission issues, session timeout, unexpected popups, or mismatch between expected and visible values This should be a supervised assistant, not an unattended bot. Example Workflow User logs into their ERP system normally. User opens the Bob.ai Desktop Writeback Assistant. Bob.ai sends a structured task, such as fields to update for a tenant, household, unit, inspection, certification, or transaction. The desktop assistant navigates the ERP screen. The assistant fills the fields. The assistant pauses before final save. The user reviews and approves. The assistant captures proof of completion and sends status back to Bob.ai. Responsibilities The contractor will help us: Recommend the best technical approach for a supervised local desktop AI assistant Build an MVP or proof of concept on Windows Evaluate OpenAI computer-use, Claude computer-use, Windows UI Automation, OCR, image recognition, and/or traditional desktop automation approaches Design the local action-execution harness for mouse, keyboard, clipboard, and screenshot capture Implement guardrails so the assistant only acts within approved windows or applications Add approval checkpoints before high-risk actions such as Save, Submit, Delete, or Finalize Create a structured logging/audit mechanism Build error handling and exception routing Help define what should be deterministic automation versus AI-driven screen understanding Document the architecture, limitations, risks, and recommended next steps Required Experience Please apply only if you have strong experience with at least several of the following: Windows desktop automation AI computer-use agents OpenAI computer-use API, Claude computer-use, or similar agentic UI-control systems Microsoft UI Automation / WinAppDriver / pywinauto / FlaUI / AutoHotkey / SikuliX / OCR-based automation Desktop app development in .NET, Python, Electron, or similar Automating Citrix, RDP, hosted desktop, or virtual app environments Screen capture, OCR, computer vision, and UI state detection Secure local agents that communicate with a cloud backend RPA tools such as UiPath, Power Automate Desktop, Automation Anywhere, or Blue Prism Human-in-the-loop workflow design Audit logging, approval workflows, and exception handling Nice to Have Experience with public sector, housing, healthcare, finance, or other regulated workflows Experience with ERP systems Experience automating legacy desktop applications Experience with Yardi, Elite, MRI, Emphasys, Tenmast, or similar systems Experience building safe automation where the user supervises the agent Knowledge of Citrix/RDP automation challenges Experience designing agent guardrails and action policies Important Constraints The solution should run on the customer’s desktop or workstation. The solution should not require the ERP vendor to install automation software in their cloud environment. The assistant should operate under the customer’s existing authenticated ERP session. The user must remain in control. The assistant must pause before final save/submit unless explicitly configured otherwise. The solution must be auditable. The AI model should not be trusted blindly; deterministic validation and policy checks are required. Deliverables We are open to either a short discovery project or an MVP build. Ideally, the first engagement would include: Technical recommendation and architecture proposal MVP desktop automation harness Prototype showing supervised field entry into a sample Windows app, Citrix/RDP session, or test ERP-like interface Screenshot/action logging User approval checkpoint before final save Error detection and exception handling approach Documentation of risks, limitations, and production roadmap Engagement Type This can begin as a paid discovery / prototype project. If successful, there may be follow-on work to build a production-grade version. Please include in your response: Relevant desktop automation or computer-use agent projects you have built Your recommended approach for this project Tools/frameworks you would use Any experience with Citrix, RDP, hosted desktop, or legacy ERP automation How you would design guardrails so the assistant does not take unsafe actions Estimated timeline for a small MVP Whether you are available for follow-on production work About Bob.ai Bob.ai builds AI-powered tools for Public Housing Agencies and related housing workflows. We help automate complex administrative processes while keeping accuracy, compliance, and human oversight at the center of the product.