Data mining: build an automated web scraping + ETL pipeline for summer camp data

We need a data engineer to build the pipeline that feeds CampSanity's camp database: a repeatable system that pulls camp and session information from public sources, starting with one pilot metro, normalizes it into one schema, deduplicates it, and loads it into Postgres on a schedule. This is not a one-off scrape. By mid-December the pipeline must run on its own and we must be able to add new sources without rebuilding it. Coverage: we start with one pilot metro, New York City, and want as many of its camps as possible. If the pilot goes well, we plan to add Washington, DC and then Seattle as separately priced extensions. Build the pipeline so that adding a metro means adding its sources, not rebuilding. Source types • Camp directories: state licensing lists, city and county parks-and-rec catalogs. • Registration platforms that host many camps on similar page templates (CampMinder, Amilia/SmartRec, Sawyer, ActivityHero, Jackrabbit). • Franchise and multi-site operators (sports academies, coding camps, language schools, museum and university programs). • Individual camp websites in each metro we cover, with a manual-review step for sites that cannot be parsed. • Standard lists of organizations that tend to run camps: public and private schools, colleges and universities, YMCAs and community centers, museums and similar, checked one by one for camp programs. We are looking to you to suggest ways of extracting camp data: better sources, tools or methods than the ones listed here are welcome. Deliverables • A camp and session data schema, starting from the draft we provide (see attachment), finalized with us in week 1. • A custom scraper for each type of source listed above, plus an AI-based tool that reads individual camp websites, fills in our data fields, and flags anything it's unsure about for human review. • Normalization: addresses geocoded to lat/long; ages and grades mapped to a single age range; dates parsed to ISO; prices parsed to a weekly figure with the original text kept; activities tagged against a controlled vocabulary we agree on together. • Deduplication across sources (same camp from a directory and from its own site is one record) with a stable camp ID. • Loading into Postgres with full history. • Scheduled runs (daily during Jan–Mar registration season, weekly otherwise), with a run log and a simple QA report: records added, changed, failed, and fields with low confidence. • A review queue: a CSV or lightweight admin view listing low-confidence or failed records so a human can correct them. • Documentation and a handover call so a second engineer can add a new source in under a day. Milestones Data fields agreed; first camps pulled from directory lists and saved to the database All NYC camps from the directory lists saved, with at least 95% placed on the map Registration-platform extractors + dedup + normalization Session dates, ages and prices populated on ≥ 80% of records; manual spot-check of 50 random records AI-based scraper for individual camp websites working; NYC camps from all source types saved to the database Review queue working; camp count per source type Scheduling, QA report, docs, handover Pipeline runs unattended for 7 days; a new source added in the handover call Extensions (optional, priced separately): after the pilot, Washington, DC, then Seattle. Each one covers finding that metro's sources, running them through the pipeline, and delivering a camp count and review queue for the new metro. Paid on delivery, one metro at a time. Required skills: Python; Scrapy or Playwright; Postgres; working with messy semi-structured data; geocoding APIs; experience running scrapers responsibly (rate limiting, robots.txt, caching) and handling site changes. Nice to have: LLM-based extraction (structured output from OpenAI or Anthropic APIs), dbt or similar, prior directory or marketplace data projects. Constraints: public data only, no logging into camp accounts, no collection of any child or parent data. Code and data are owned by CampSanity and delivered in our GitHub and cloud account (we will set these up with the developer).