PostgreSQL + Node.js Reliability Engineer (Replit) – Diagnose Production DB & Job Queue Issues

Audilo — Database & Backend Reliability EngineerProject OverviewAudilo is a finished AI music-creation app. We need a senior backend/reliability engineer to diagnose and resolve production database and background-job reliability issues so we can confidently launch publicly.This is a focused investigation and stabilization engagement — not a redesign, rewrite, schema overhaul, or migration off Replit.BackgroundThe app sends music-generation requests to external paid providers and processes the results through background jobs. Two things must be protected: Customer work must not silently disappear or get stuck. A restart or retry must not accidentally trigger another paid generation. We experienced a production incident involving database connection timeouts, connection-pool saturation, and a significant Node.js event-loop delay. The root cause is still unproven.Scope of Work Review existing evidence Reconstruct the incident timeline, confirm what code and configuration were running at the time, and clearly separate confirmed facts from hypotheses. Investigate the database ↔ application boundary Connection pool sizing, lifecycle, checkout/release, and timeouts (especially pg) Long transactions, locks, connection leaks, and worker concurrency Event-loop stalls and application resource pressure Whether recovery logic makes outages worse Review background-job recovery Leases, heartbeats, stale-job detection, and safe ownership transfer Ensure an old worker cannot continue after another takes over Ensure known provider requests resume safely (no duplicate paid submissions) Treat unknown outcomes (e.g. timeouts) conservatively Recommend the smallest safe fix Provide a narrow, evidence-based correction or containment approach. Test in an isolated environment first and define a clear production acceptance check (owner approval required before any production change). Required Skills Strong PostgreSQL production troubleshooting Node.js / TypeScript Connection pooling (pg) PostgreSQL-backed job queues (ideally pg-boss) Production incident analysis and working within managed hosting constraints (Replit) Someone who only designs schemas or writes reports is not a fit. This work sits at the intersection of the database, application runtime, and background workers.Constraints Start with code review and read-only access. No production writes, schema changes, publications, or induced outages without explicit approval. Least-privilege access only. No migration off Replit as a default troubleshooting step. Deliverables Diagnosis — verified findings, most likely failure mechanism, and remaining uncertainties. Minimal remediation plan — proposed changes, risks, and recovery procedure. Verification evidence — tests demonstrating safe recovery and protection against duplicate paid provider requests. Specific questions for Replit Support (infrastructure evidence only they can provide). Clear release recommendation: Ready / Conditionally ready / Blocked, with reasons.