AI Data Automation Engineer: Source, Build, QA and Ship Data Products End to End

We are a B2B data intelligence platform. We sell signals to commercial teams: which companies are hiring, spending, moving people and warming up as buyers. Customers include large media and agency groups. They come to us for one thing: fresh data they can export. Every dataset in the app is judged on three numbers: coverage, accuracy and recency. This role exists to push all three up, one dataset at a time. You will run whole epics on your own. That means working out what customers need, finding where the data lives, building or buying the pipeline, landing it in BigQuery, building models on top of it, checking it with LLMs and shipping it to production down to the column and cell. Then you keep it fresh. You will not be handed a spec. You will write it. We are AI-native. Claude Code, connected to our warehouse over MCP, is how we explore data, write dbt models and pipelines, and QA output. We run our own MCP servers. If you still do by hand what an agent could do, this isn't the role for you. THE LOOP YOU ARE BUILDING A customer asks for something we don't have, or have only thinly. You size the demand and decide whether it's worth building. You find the precursor data, whether that's public sources, vendors, APIs or our own engines. You scrape it or license it and land it in BigQuery. You transform it in dbt and build data science models on top of it, such as entity matching, classification and enrichment, to a very high accuracy bar. You QA every model with LLM checks against ground truth. You deploy the columns and cells that pass, through ClickHouse and Elasticsearch, into the app. You set a refresh schedule that costs as little as possible for the freshness the customer needs. Coverage, accuracy and recency go up, and you report all three. That loop is the job. A dataset that's 90% accurate but six months stale is worth less than one that's 85% accurate and refreshed weekly. You will be the one deciding where that trade-off sits. WHAT YOU BUILD: FOUR LAYERS 1. Demand → dataset. Turn customer requests, support tickets, product usage in PostHog and sales calls into a ranked backlog of data models to build. You write the case for each one: who wants it, what it unlocks and what it costs to keep fresh. 2. Sourcing: build or buy. Custom scrapers for public sources, written within each site's terms and with privacy in mind. They're built with Playwright, Apify or Bright Data and run on GCP. Vendor evaluation where buying beats building: sample pulls, match rates against our entity base, cost per useful row and refresh SLAs. You bring us a recommendation with numbers, not a list of vendors. 3. Modelling and LLM-driven QA. dbt models on BigQuery, from raw data through staging to the marts the app reads. You build the data science models on top: matching, deduplication, classification and enrichment. Automated QA uses LLMs to check, reconcile and score records before they reach a customer. You prototype in Claude Code against the warehouse and prove the model there. Only then does it go to production. 4. Storage, refresh and deployment. You decide where each dataset lives and how it reaches the app: BigQuery, ClickHouse, Elasticsearch, Postgres and Redis. You also decide what to fetch at request time and how often to re-pull, through dbt Cloud jobs, Dagster, Pub/Sub and Cloud Run. Deployment happens at column and cell level, so a weak field never drags a strong dataset down with it. Lightweight endpoints and internal tools ship on Vercel, and failures alert through Sentry and Datadog. WHAT YOU OWN END TO END - The data backlog, ranked by customer demand and cost to serve - Scrapers, vendor integrations and the pipelines that join them - dbt models and the data science models built on top of them - Vendor evaluations and build-vs-buy recommendations - LLM QA processes and the accuracy benchmarks behind them, running continuously rather than once - Storage and refresh architecture, with GCP cost per dataset - The weekly coverage, accuracy and recency report, by dataset and by field 30 / 60 / 90 - Day 30: Current datasets audited for coverage, accuracy and recency. First epic picked and scoped. LLM QA running on a schedule against at least one existing dataset. - Day 60: First new dataset sourced, modelled in dbt, QA'd and deployed to the app at column level. First vendor evaluation delivered with a recommendation. Refresh and storage cost per dataset measured. - Day 90: Two epics shipped end to end, each on a refresh schedule with ongoing QA. Measurable gains in coverage, accuracy or recency on the datasets you touched. A demand-ranked backlog for the next quarter. SUCCESS METRICS - Coverage, accuracy and recency, by dataset. This is the north star. Every other metric feeds it. - Model accuracy against ground truth. It stays high after launch, not just on launch day. - Cost per fresh record. It goes down as you optimise storage and refresh across GCP. - QA pass rate before production. Nothing ships to a customer without passing. - Epics shipped end to end. Shipped means live in the app, not stuck in a notebook. COMPENSATION Full time, 40 hours a week, $24–42/hr depending on experience. Tooling, vendor sample and compute costs are paid separately from your fee. REQUIREMENTS - You have done the full loop yourself, more than once. You sourced raw data, pulled it into a database, built data science models on it to a very high accuracy, deployed those models into a production app, then maintained and refreshed them with ongoing QA. This is the most important requirement. Your GitHub or Loom should show it. - AI-native. You use Claude Code every day, including with MCP connected to a data warehouse, to write code, build pipelines and QA data. - dbt on BigQuery in production. You have worked with incremental models, tests, sources and the cost of a careless full refresh. - ClickHouse. You know when to serve from ClickHouse rather than query BigQuery, and the cost and latency trade-offs between them. - Orchestration. You have used dbt Cloud plus Dagster or a similar orchestrator, and you think in assets, schedules, sensors and freshness. - GCP. You are comfortable with Cloud Run, Pub/Sub, Cloud Storage, IAM and keeping a GCP bill under control. - Production scrapers you built yourself with Playwright, Apify, Bright Data or similar, including rate limiting, anti-bot handling, monitoring and fixes when they break. - Strong Python and SQL. You are at home with entity matching, deduplication and schema design. - LLMs for real data work through the Anthropic or OpenAI APIs, such as extraction, classification and validation, and you can show how you measured accuracy. - Vendor evaluation. You have evaluated or integrated third-party data vendors and can explain match rate, fill rate and freshness without looking them up. - Autonomy. You can take a vague goal, turn it into a plan and ship it without daily check-ins. - GDPR awareness. You work with UK and EU data, and scraping and personal data both sit inside privacy review. - Clear written English. You will write the specs, vendor memos and weekly reports. Nice to have: Elasticsearch. Postgres and Redis. Building MCP servers in Python. TypeScript, GraphQL (Apollo Federation) or Next.js, so you can follow your data into the app. Vercel and Supabase. Terraform, GKE and Cloud Build. Clay and Make. Product management experience with data products. Our stack: Claude Code + MCP, Anthropic and OpenAI APIs, Python, SQL, dbt (dbt Cloud), BigQuery, ClickHouse, Elasticsearch, Postgres, Redis, Dagster, GCP (Cloud Run, Pub/Sub, GKE, Cloud Build), Terraform, Playwright, Apify, Bright Data, GraphQL, Vercel, Supabase, PostHog, Sentry, Datadog, Make, Clay, Linear and Slack. How we work: Async-first, any timezone. You need to be reachable on Slack and you need to ship. Hacky and shipped over polished and planned. If the first version of a new dataset is a scraper, a Claude QA prompt and one table of cells that passed, that is correct. Make it work, show us the numbers, then automate it. HOW TO APPLY: READ THIS CAREFULLY Proposals without one of these will not be read: - A GitHub link to a scraper, pipeline or data model you built yourself, or - A Loom of 5 minutes or less walking through one dataset you took through the full loop: where you sourced it, how you stored it, the model you built on it and its accuracy, how it got into production and how you kept it fresh and QA'd. Start your proposal with the word "Recency" so we know you read this far. Proposals that look like templates or are written entirely by AI will be rejected.