Senior Full-Stack Engineer Needed to Stabilize or Rebuild Internal Data Automation Platform

Job Description: We have a working internal data automation platform, but it was built quickly and now has technical debt. The system works, but it is not as stable, clean, or product-grade as we need it to be. We process large datasets, with the goal of handling millions of records per month. We need someone who understands scale, queues, database performance, scraping reliability, and operational safety. We are looking for one strong engineer who can come in, understand the current codebase, and help us decide whether to clean it up, rebuild parts of it, or rebuild the system properly from the ground up. This is not a landing page or simple CRUD app. It involves backend workflows, web scraping, third-party APIs, database state, background jobs, AI/vendor integrations, exports, and an internal dashboard used to monitor and operate the system. Important: Detailed Loom Video Required: Please include a detailed Loom video made specifically for this job. Proposals without a job-specific Loom video will be rejected outright. We do not want a generic introduction or a reused sales video. In the Loom, please show or clearly explain relevant past work, especially around scraping, large datasets, backend workflows, queues, dashboards, and production systems. The Loom should cover: - A real project you worked on that is similar in complexity - The scale of the data handled - The scraping or workflow architecture - How failures, retries, blocked pages, and queues were handled - How you approached database performance and large exports - What you would check first in a working but messy codebase like ours Please hide any private client data, credentials, or sensitive details. Core Work: - Review the existing codebase and architecture - Identify what should be refactored, rebuilt, or left alone - Improve backend workflows and background jobs - Improve scraping reliability, retries, extraction quality, and failure handling - Build safer dashboard controls for monitoring, retrying, pausing, and reviewing jobs - Improve database queries, pagination, exports, and status tracking for large datasets - Add better logging, monitoring, and operational visibility - Help create a proper production/staging setup - Document the system clearly so future work is easier Current Stack: - Node.js / TypeScript - Postgres / Supabase - Trigger.dev or similar workflow/job orchestration - Vercel/serverless API routes - GitHub - AI/vendor APIs - Internal dashboard/admin UI - Web scraping/browser automation tools Web Scraping Experience Needed: We need real scraping experience, not just basic HTTP requests. Useful experience includes: - Playwright, Puppeteer, Crawlee, Apify, Browserless, Cheerio, or similar - Dynamic pages, redirects, blocked pages, timeouts, and bad HTML - Proxies, rate limits, retries, backoff, and queue-based scraping - RSS, sitemap, and URL discovery - Content extraction and quality checks - Logging failures so they can be reviewed and retried safely Large Dataset Experience Needed: We need someone comfortable working with high-volume systems, including: - Millions of records per month - Batch processing and queue-based workflows - Efficient Postgres queries and indexing - Pagination, chunking, deduplication, and idempotency - Exporting large filtered datasets safely - Avoiding memory-heavy frontend or backend patterns - Monitoring throughput, failures, and cost Required Experience: - Strong Node.js and TypeScript - Strong Postgres experience - Experience with background jobs, queues, or workflow orchestration - Experience building internal dashboards/admin tools - Production debugging experience - Good judgment around credentials, databases, and live systems - Clear communication and ability to work independently Helpful Experience: - Supabase - Trigger.dev - Vercel - AI API integrations - Data enrichment pipelines - Scraping infrastructure - Cost/usage tracking - Building simple operator tools for non-technical users Not a Good Fit If: - You mainly build landing pages - You only use no-code tools - You have only built demo AI apps - You cannot work safely around production data - You need every task fully specified before investigating the codebase First Milestone: We want to start with a paid technical audit. Deliverables: - Review the current codebase and workflow setup - Identify major reliability, scraping, database, dashboard, and deployment gaps - Recommend whether to refactor, partially rebuild, or fully rebuild - Propose the first 2-3 implementation milestones - Optionally complete one small low-risk improvement Please Include In Your Proposal: - A Loom video as described above - One relevant project example - The volume of data that project handled - Your experience with Node.js, TypeScript, and Postgres - Your scraping stack and how you handle failures/blocks - Your experience with queues or workflow tools - How you would approach a working but messy codebase Suggested Upwork Skills: TypeScript, Node.js, PostgreSQL, Web Scraping, Playwright