Python Data Engineer for Public Regulatory Register Extraction

We are developing a commercial intelligence platform that structures publicly available information from official regulatory sources. We require an experienced Python data engineer or web scraping specialist to extract a complete public regulatory register and the detailed verification pages linked to each record. The register contains regulated entities, authorisation numbers, licence categories, statuses, approved activities, approved service provider relationships and associated websites. The objective is to create a reliable and reusable extraction process that can regularly collect, structure and load the data into an existing Supabase PostgreSQL database. This is not a manual data entry project or a request for a one time spreadsheet. The identity of the target organisation, website and wider commercial context will be disclosed to shortlisted candidates after their relevant experience has been reviewed. Project objectives The selected contractor will be expected to: Inspect how the public register loads and retrieves its data. Determine whether the website uses public JSON requests, API endpoints, server rendered HTML or browser based requests. Document any available endpoints, request parameters, pagination, filters and stable record identifiers. Extract all publicly visible entity and authorisation records. Open and extract information from all accessible linked verification pages. Normalise entities, licences, approved activities, service providers, verticals and websites into relational data structures. Preserve the original source values alongside any normalised values. Include the official source URL and extraction timestamp for every record. Build a process that can be rerun to identify new, changed and removed records. Propose a Supabase compatible relational structure for the extracted data. Create a documented Python process for loading the data into a Supabase staging environment. Use inserts and upserts correctly so repeated runs do not create duplicate records. Maintain relationships between entities, licences, activities, approved service providers and websites. Deliver documented source code that our technical team can operate independently. Where a public JSON request or structured endpoint is available, this should normally be preferred to full browser automation. Data to be extracted The publicly available records may include the following information. Entity information • Legal entity name • Company or registration number • Registered address • Public contact information • Overall regulatory status • Verification page URL • Official source URL • Date last checked Authorisation information • Authorisation or licence number • Authorisation type • Licence class • Current status • Effective date • Termination or expiry date • Related regulated activities • Associated entity • Official source URL Each licence or authorisation must be stored as a separate record and linked to the relevant entity. Approved activities • Activity or service type • Approved product category • Approved vertical • Associated licence number • Associated entity • Status where available Multiple activities or verticals must be represented separately rather than combined into one text field. Approved service providers • Service provider name • Service provider authorisation number • Related entity • Related licence • Related activity • Regulatory relationship type • Source URL Each approved service provider must be stored as a separate relationship record. Lists of providers must not be combined into a single text field. These records represent a publicly disclosed regulatory relationship. They should not automatically be classified as confirmed commercial deployments. Websites and domains • Published website URL • Normalised domain • Associated entity • Associated licence • Source URL • First observed date • Last observed date • Current status where available The original published URL must be retained alongside the normalised domain. Change history The extraction process should support change tracking for material fields, including: • Record identifier • Field changed • Previous value • New value • Date the change was detected • Source URL • Extraction run identifier Records that disappear from the public source should be flagged for review rather than automatically deleted from the database. Supabase integration The extracted information will be stored in an existing Supabase PostgreSQL environment. The contractor will be expected to: • propose the relational tables required for the extracted data; • provide SQL migration files or table definitions; • map public source records to stable internal identifiers; • create a Python loading script for Supabase; • use upserts to update existing records; • prevent duplicate entities, licences, domains and relationships; • maintain foreign key relationships between the relevant records; • record when each record was first seen, last checked and last changed; • maintain a history of material changes; • log failed inserts, unmatched relationships and validation errors; • store credentials through environment variables; and • document how another developer can run the extraction and database loading process. The contractor will initially work against a staging or test environment. Production database access will not be provided during the initial discovery stage. Any changes to the existing production schema must be reviewed and approved by our technical team. The loading process must be idempotent. Rerunning the same extraction should not create duplicate entities, licences, domains, activities or service provider relationships. Required deliverables The contractor must provide: • Python source code • Complete CSV exports • Complete JSON exports • Normalised relational data files • Supabase compatible relational schema • SQL migration files or table definitions • Python Supabase loading script • Field mapping between source data and database fields • Upsert and deduplication logic • Basic change detection process • Data dictionary • Technical discovery report • Error and failed record report • Duplicate and exception report • Record count summary • Database loading error log • Setup and installation instructions • Instructions for running future extractions • Instructions for running database updates • Quality assurance results • Successful staging import demonstration • Source code delivered through GitHub or an equivalent repository The solution must not depend on the contractor’s personal accounts, private infrastructure or proprietary software unless agreed in advance. All code, documentation and project outputs must be transferred to the client following payment. Project milestones Milestone 1: Technical discovery and sample The contractor will: • inspect the public register and linked verification pages; • identify how the website retrieves its information; • document requests, endpoints, identifiers, filters and pagination; • assess whether extraction can be completed reliably; • recommend the most appropriate extraction method; • propose an initial relational data structure; and • deliver a sample of approximately 10 records. The discovery report should also identify: • any technical access restrictions; • inconsistent record structures; • possible duplicate entities; • missing identifiers; • data quality concerns; and • any paid services that may be required. Only the first milestone will be funded initially. Milestone 2: 50 record pilot The pilot should include a varied sample of records, including: • different authorisation types; • entities with multiple licences; • multiple approved activities; • multiple approved service provider relationships; • multiple websites; • active and inactive records where available; and • unusual or incomplete record structures. The pilot must include: • CSV output; • JSON output; • normalised relational tables; • proposed Supabase schema; • SQL migration files; • Supabase loading script; • source URLs; • extraction timestamps; • error log; • data dictionary; and • manual verification of at least five records. The 50 record pilot must be successfully loaded into a Supabase staging environment. Milestone 3: Complete extraction Following approval of the pilot, the contractor will: • extract the complete public register; • extract all accessible linked verification pages; • provide complete normalised outputs; • load the full dataset into the approved Supabase staging structure; • document incomplete or failed records; • identify duplicates and exceptions; • provide final record counts; and • deliver reusable extraction and loading code. Milestone 4: Change detection, quality assurance and handover The contractor will: • detect new records between runs; • detect changed records between runs; • identify records that are no longer visible; • maintain field level change history; • add logging and retry handling; • provide database validation checks; • provide final operating documentation; • complete source code handover; and • demonstrate that another developer can run the extraction and Supabase update process independently. Recurring monitoring, production deployment, frontend work and long term maintenance may be commissioned separately. Technical requirements Applicants should have substantial experience with: • Python • Web scraping and data extraction • Browser developer tools and network inspection • JSON and API extraction • Playwright, Selenium, Scrapy or Beautiful Soup • Relational data modelling • PostgreSQL • Supabase • CSV and JSON processing • ETL workflows • Database migrations • Upserts and deduplication • Change detection • Logging and exception handling • Git and technical documentation Experience extracting public government, corporate, licensing or regulatory registers would be particularly valuable. Project constraints The work must comply with the following requirements: • Publicly accessible pages only • No authenticated portals • No CAPTCHA circumvention • No bypassing technical access controls • No attempts to access confidential or unpublished information • Conservative request frequency • Caching of unchanged content where appropriate • Collection limited to information required for the project • Official source URLs retained • Original source values preserved • Credentials stored through environment variables • No production database access during the initial discovery stage • Any proxy, hosting or third party service costs disclosed in advance The contractor should raise any technical, legal or access concerns during the discovery stage rather than continuing without approval. Acceptance criteria The work will only be approved where: • sampled records match the official public source; • entities with multiple licences are not duplicated incorrectly; • licences are stored as separate records; • approved activities and verticals are represented separately; • approved service providers are stored as separate relationship records; • original regulatory terminology is preserved; • every record includes an official source URL; • every record includes an extraction timestamp; • failed or incomplete records are clearly reported; • the 50 record pilot loads successfully into Supabase staging; • rerunning the same dataset does not create duplicates; • changed source records update the correct database records; • removed records are flagged rather than automatically deleted; • multiple licences and provider relationships remain linked to the correct entity; • database credentials are not stored in the repository; • the extraction can be rerun by another developer; • the Supabase loading process can be run by another developer; • normal records do not require manual processing; and • the source code is readable and documented. Budget The indicative fixed price is $1,200. Applicants should provide a proposed milestone breakdown and identify any elements that may require a separate budget. The final scope and milestone values may be adjusted following technical discovery. Only the first technical discovery milestone will be funded initially. The indicative budget includes: • technical discovery; • pilot extraction; • complete public register extraction; • linked verification page extraction; • reusable source code; • Supabase staging integration; • basic change detection; • documentation; and • handover. The initial scope does not include: • frontend development; • production platform changes; • extensive restructuring of the existing Supabase database; • cloud hosting costs; • ongoing scraper maintenance; • long term monitoring; or • manual commercial enrichment of the records. Successful delivery may lead to additional work involving production integration, scheduled monitoring and other public regulatory datasets. Screening questions Please answer all of the following questions in your proposal. How would you determine whether a public register uses a JSON endpoint or API behind its website interface? How would you model an entity with multiple licences, approved activities, service providers and websites? How would you detect records that have changed or disappeared between extraction runs? How would you prevent duplicate records when loading repeated extraction runs into Supabase? How would you maintain relationships between entities, licences, activities, service providers and domains? How would your solution respond if the website changed its HTML structure? Please describe a similar public register, government database or structured website you have extracted. Please describe your experience loading scraped or extracted data into Supabase or PostgreSQL. Which tools would you expect to use for this project, and why? How would you validate that the complete public register has been extracted? Will all source code, SQL files, documentation and operating instructions be transferred to us? Are any paid proxies, browser services, hosting platforms or third party subscriptions required? Please provide your proposed fixed price and milestone breakdown. Please confirm that you are willing to begin with a separately funded technical discovery milestone. Confidentiality The target organisation, website and full commercial context will be shared only with shortlisted candidates. Applicants must agree not to disclose the project, source website, extracted data or technical findings to third parties. All project outputs, source code, SQL files, schemas and documentation will be owned by the client following payment.