Senior Data Engineer (Data Ops) - Go, PostgreSQL & Production Data Pipelines

*Overview* Launch Lab is supporting an established organisation building an AI-powered recruitment platform. The engineering team is strengthening its in-house capability and needs a senior, hands-on Data Engineer to take ownership of ingestion and the data feeding production search and matching. This is a practical delivery role: the immediate priority is to make existing pipelines more reliable, recoverable and easier to operate, while continuing to develop connectors, source onboarding and downstream data flows. You’ll work alongside specialist data/ML expertise, technology leadership, Product and DevOps, with clear ownership of routine pipeline delivery. *Engagement Details* - Start date: ASAP. - Initial term: 6 months, with potential for extension. - Location: Remote; candidates must be based in India or Pakistan. - Working hours: Monday–Friday, 8 hours per day (40 hours/week), 9am-5pm GMT. - Engagement: One dedicated individual contributor working with an established engineering team. - Equipment: BYOD *About the Project* The platform brings together approved source data and production search/matching capabilities. Existing ingestion processes have become a delivery bottleneck, and the team wants to reduce reliance on specialist knowledge for everyday operation and recovery. The current technology includes Go orchestration and data migrations, PostgreSQL/pgvector, and VoyageAI embeddings via Amazon Bedrock. The engineer will work within this existing architecture, improve operational reliability and contribute to agreed changes to the matching data interfaces. The wider pipeline tooling will be explained during the technical handover. *Must-Have Experience & Skills* Technical - Demonstrable hands-on ownership of production data ingestion pipelines, including failure investigation, recovery and ongoing operation. - Practical Go (Golang) development: able to maintain or extend existing orchestration services, backend components or migration logic. - Strong SQL and PostgreSQL, including modelling, query work and reliable handling of production data. - APIs, data integration, batch/asynchronous processing, transformations, deduplication, incremental loads and source/schema changes. - Production embedding or vector-search experience supporting search, retrieval or matching workflows. - Understanding of retries, checkpoints, idempotency, replay/backfill, quarantine, reconciliation and avoiding silent loss or duplicate effects. - Production testing, deployment, monitoring, troubleshooting and appropriate data access/privacy controls. Business & Delivery - Comfortable owning day-to-day technical delivery as an individual contributor, while collaborating with specialist engineers and technical leadership. - Able to translate data-quality, source-mapping and downstream requirements into clear, testable engineering work. - Experience delivering reliable data flows for production products, ideally involving search, matching or ML-enabled features. Soft Skills - Pragmatic, independent problem-solver who investigates issues, documents decisions and escalates blockers promptly. - Clear communication with Engineering, Product, Data/ML and DevOps colleagues. - Comfortable joining an evolving platform, understanding an existing system and improving it incrementally. Desirable Experience - Direct experience with pgvector, Amazon Bedrock embedding integration or VoyageAI. - Python for data processing alongside Go. - Recruitment/HR technology, entity resolution, search relevance, embedding evaluation or sensitive-data governance. *Responsibilities* - Map the existing ingestion flow during handover, including sources, interfaces, transformations, storage, manual steps and failure modes. - Maintain and improve ingestion connectors and Go orchestration; automate repeatable source onboarding and incremental updates. - Implement data validation, schema-change detection, deduplication, metadata and source-to-downstream lineage. - Improve reliability with safe retries, checkpointing, replay/backfill and recovery procedures; reconcile processed, rejected, blocked and searchable records. - Maintain data interfaces and embedding/vector indexing flows supporting production search and matching. - Help identify and resolve processing bottlenecks, distinguish technical failures from records blocked by approved consent/access rules, and keep downstream representations consistent. - Implement monitoring, useful logs and actionable alerts for freshness, processing failures, throughput, latency and cost. - Write runbooks and handover notes so another engineer can run and recover pipelines safely. *Success Criteria* - Existing production ingestion is easier to run, diagnose and recover, without silent record loss or duplicate downstream effects. - Records are validated, traceable and accounted for across source, processing and search/matching outcomes. - Changes to sources and schemas can be handled through documented, repeatable processes. - Technical versus approval/consent-related blockers are reported clearly, with actionable operational monitoring. - Another engineer can follow the documented procedure to operate and recover the pipeline. *Working Practices* - Collaborate with the existing engineering group, specialist data/ML support, Product and DevOps. - Participate in relevant delivery stand-ups, backlog planning and a regular data progress/decisions review; schedules will be agreed. - Equipment, approved development environment, access and software requirements will be confirmed before onboarding. *Assessment Process* - CV/Github Review, evidence of production ingestion ownership, Go, PostgreSQL and production search/matching pipelines. - R1 Technical Interview, approximately 30 minutes. - R2 Live Paired Technical Exercise, 60 minutes. - Client interview, practical fit, collaboration and delivery expectations. *Next Steps* Please apply through Upwork with an up-to-date CV tailored to this role. The CV should describe the projects you worked on, technologies you personally used, responsibilities held and outcomes achieved. A skills list without project context is unlikely to demonstrate the required fit. - Highlight a production ingestion pipeline you personally delivered or operated, including how failures and recovery were handled. - Identify specific Go and PostgreSQL implementation work you have completed. - Describe any production embedding, vector retrieval or matching integration you contributed to. - Confirm your current working location, earliest start, UK overlap availability and proposed Upwork hourly rate. - Provide a GitHub/code repository or relevant code sample if available. *Please do not apply without a CV. We are looking for directly relevant production engineering experience, not just familiarity with the listed tools.*