Data Acquisition and Extraction Engineer

# Data Acquisition and Extraction Engineer ## About the Role We are hiring a Data Acquisition and Extraction Engineer to build, operate, and improve a large-scale public web data acquisition platform. The platform collects information from a broad range of public websites and document repositories on a recurring schedule. Sources include static HTML, JavaScript-driven applications, PDFs, office documents, and websites whose structure changes over time. This role is focused on implementing reliable acquisition workflows, expanding source coverage, improving extracted outputs, and ensuring failures can be diagnosed and recovered without compromising downstream data. The successful candidate will work closely with a principal engineer who owns the broader acquisition, validation, lineage, and trust framework. This role remains an independent engineering position with substantial responsibility for production implementation and operational quality. ## Responsibilities * Build and maintain source-specific and provider-based web acquisition workflows. * Implement reusable scraper templates, adapters, parsers, and extraction components. * Acquire data from static pages, JavaScript-driven applications, APIs exposed through public websites, PDFs, and office documents. * Add validation checks that confirm expected content was acquired, not merely that a request completed. * Instrument acquisition and extraction stages with structured logs, metrics, statuses, and diagnostic context. * Detect and diagnose changes in website structure, page content, document layout, and extracted outputs. * Implement retry, replay, reprocessing, and backfill workflows. * Ensure acquisition jobs are safe to rerun and do not create duplicate or inconsistent downstream data. * Investigate failed, empty, incomplete, or anomalous acquisition results. * Maintain and improve provider templates shared across multiple source websites. * Expand acquisition coverage by onboarding new sources and improving source matching. * Implement extraction rules, selectors, classifiers, and normalization logic. * Improve the completeness and consistency of extracted data. * Parse and normalize structured and semi-structured content from web pages and documents. * Support geospatial and address normalization where required. * Build test fixtures and regression tests for source-specific and shared extraction behavior. * Validate proposed changes against current and historical source samples. * Support dashboards and reporting for source health, execution status, coverage, and data quality. * Execute targeted backfills and recovery operations. * Participate in root-cause analysis for recurring production issues. * Convert human-review findings into durable tests and extraction improvements. * Document source behavior, failure modes, implementation decisions, and operational procedures. ## What Success Looks Like * New sources can be onboarded through consistent, documented patterns. * Existing source integrations remain maintainable as websites change. * Failures include enough evidence to identify the affected source, stage, and likely cause. * Shared provider failures can be distinguished from source-specific failures. * Empty or malformed output is detected before reaching downstream systems. * Failed jobs can be retried or reprocessed safely. * Extraction changes are tested against representative fixtures before deployment. * Coverage increases without introducing unmeasured regressions. * Human review results lead to reusable validation checks rather than repeated manual corrections. * Common failure patterns are handled by shared tooling instead of repeated one-off fixes. ## Required Experience * Professional experience building and maintaining web scrapers, crawlers, or public web acquisition systems. * Strong Python development skills. * Working knowledge of SQL for data validation and investigation. * Experience with HTML parsing and structured data extraction. * Experience with JavaScript-rendered websites and browser automation. * Experience parsing or processing PDFs, office documents, or other semi-structured files. * Experience building production batch pipelines or scheduled data-processing workflows. * Familiarity with retries, idempotency, replay, and backfill patterns. * Experience debugging production failures using logs, metrics, source captures, and historical comparisons. * Ability to write automated tests for scraper and extraction behavior. * Experience working with changing external systems that are outside the engineering team’s control. * Ability to distinguish transport success, content success, and extraction success. ## Relevant Technical Experience The current environment includes technologies such as: * Python * SQL * TypeScript * AWS Lambda * AWS Glue * AWS Step Functions * Amazon SQS * Amazon S3 * ECS or EKS * Relational databases * Parquet-based storage * Browser automation frameworks * HTML and document-processing libraries * Monitoring and dashboarding systems Candidates are not expected to have used every component. Experience with comparable cloud-based acquisition and data-processing systems is sufficient. ## Preferred Experience * Large-scale or distributed scraping systems * Provider-template or adapter-based scraper architectures * Source and schema drift detection * Data quality and pipeline observability * Statistical or historical anomaly detection * Source discovery and classification * Human-review or annotation workflows * Geospatial data and address normalization * CI pipelines for scraper and parser regression testing * Containerized workloads * Public-sector, market, pricing, or location-based datasets * Agentic or LLM-assisted developer tooling Familiarity with rate limits, session handling, or access constraints on public websites is useful but not required. Anti-bot work is not a primary focus of the role. ## Working Style * You build repeatable source integrations rather than disposable scripts. * You treat validation and instrumentation as part of implementation. * You investigate why a scraper produced incorrect data, not only why it crashed. * You design jobs to be rerun safely. * You use historical outputs and fixtures to prevent regressions. * You are comfortable working within an existing framework while improving it incrementally. * You communicate failure modes and implementation constraints clearly. * You can execute against established technical direction while making sound independent engineering decisions. ## Role Structure This role reports to the CTO and works closely with the Principal Data Acquisition and Trust Engineer. The principal engineer owns the broader technical framework for acquisition trust, lineage, validation, semantic standards, and architectural direction. The Data Acquisition and Extraction Engineer is responsible for implementing and operating the workflows that make those standards effective in production. This is not a subordinate execution-only role. The engineer will own significant production systems, contribute to technical design, and participate directly in decisions affecting reliability, coverage, and extraction quality. ## Why This Role Matters Public websites change continuously. A reliable acquisition platform must adapt without silently losing data, corrupting outputs, or requiring extensive manual repair. This role will directly improve the platform’s source coverage, extraction quality, operational visibility, and ability to recover from failure. The work will affect the reliability of every downstream product built on the acquired data.