Senior Data Engineer

About the role We are looking for a hands-on Senior Data Engineer to join a team building an AI-powered search and matching platform. You will improve data ingestion and production pipelines that feed search and matching, combining production data engineering with ML, embeddings, and matching integration. The goal is to remove ingestion bottlenecks and make the flow from approved data sources to search and matching reliable. You will own routine data delivery, improve reliability and data quality, and work alongside Data and ML specialists, Platform Engineering, Product, and DevOps. Tech Stack: Go orchestration, PostgreSQL with pgvector, and VoyageAI embeddings served through Amazon Bedrock. Key Responsibilities Pipeline Ownership and Ingestion - Document existing data flows, source interfaces, transformations, and failure modes during handover. - Automate ingestion, validation, deduplication, and incremental updates from approved sources. - Make source onboarding repeatable and reduce the validated backlog. Recovery, Data Quality and Lineage - Implement safe retries, checkpoints, replay/backfill, and quarantine without silent data loss or duplication. - Reconcile record counts and maintain metadata, source identifiers, and lineage. - Detect schema changes, apply quality checks, and investigate failures. Production Operation and Matching - Maintain matching-service interfaces and contribute to agreed ML/matching changes. - Test embedding and indexing consistency, compatibility, and matching quality. - Work with DevOps on logs and alerts for freshness, failures, throughput, latency, and cost. - Maintain tests, deployment discipline, and runbooks for independent operation and recovery. Privacy and Collaboration - Implement approved consent, access, separation, retention, and deletion rules, and flag unresolved mappings. - Clarify requirements, report progress, and escalate open decisions. Essential Requirements - Production data pipeline ownership, including failure investigation and recovery. - Strong SQL, PostgreSQL data modelling, and practical Go for orchestration and migrations. - APIs, batch and asynchronous processing, schema evolution, metadata, and lineage. - Idempotency, reconciliation, and safe recovery or reruns. - Production vector search and embedding pipelines for search or matching. - Ability to diagnose input, embedding, indexing, and downstream quality issues. - Cloud operations, automated testing, code review, and deployment discipline. - Monitoring, alerting, runbooks, privacy-aware data modelling, and access controls. - Experience building data pipelines for production search, matching, or ML products. Desirable Skills - Hands-on pgvector and Amazon Bedrock experience (strongly preferred). - VoyageAI embedding integrations and embedding evaluation. - Python for data processing. - Entity resolution and search relevance. Soft Skills - Clear communication with Product, Engineering, and DevOps. - Independent investigation, practical problem-solving, and attention to detail. - Ownership of implementation, operational reliability, and follow-through. - Comfortable clarifying ambiguity, explaining decisions, verifying results, and escalating blockers. Assessment Process 1. CV Screening 2. First-Stage Technical Interview 3. Pair-Programming Challenge using Synthetic Data 4. Final Interview covering fit, scope, communication, and availability. How to Apply Please share the following: - Updated CV detailing your pipeline, Go, PostgreSQL, and search/matching projects, including your individual contribution and impact. - Code repositories, where available, plus examples of ingestion recovery work and production ML/search/matching support. - Your location, earliest start date, and interview availability.