Senior Data Engineer for AWS Ingestion

Senior Data Engineer — connector build on an existing AWS ingestion framework Engagement: Fixed price, milestone-based · Remote · EU-based or EU-working-hours strongly preferred Duration: 4–6 weeks for the first package Background We are a B2B analytics company in the sports and sponsorship sector. We have just completed a data-foundation build on AWS and are now extending it with additional source connectors. The foundation already exists and is not in scope to rebuild. It comprises: a reusable ingestion framework, a landing and standardisation layer, a canonical entity model, a dbt/PySpark transformation framework, scheduled orchestration with retries and backfill, an automated data-quality and lineage layer, CI/CD, and Redshift as the analytical warehouse of record. Raw data lands on S3 in Parquet with full history retained. Everything runs inside our own AWS environment. The framework was built to a specific standard: onboarding an additional source should be a matter of configuration against the existing pattern, not writing a new pipeline. Your work will test that claim. If you find the framework falls short of it, saying so early is part of the job and will not count against you. The gap to close We need additional sources connected and their fields modelled into the curated layer. Two work packages, each independently scoped. This posting covers Package A only; Package B is described so you can see the trajectory. Package A — licensed and public third-party sources (this engagement) Source class Access Notes Licensed company firmographics & funding provider REST API Export caps apply on the current tier; incremental extraction required Licensed search & web-traffic analytics provider REST API Already connected — extension only, monthly history per entity Licensed social-listening / audience & sentiment platform REST API Already connected — extension to a second entity type, 12-month aggregates Licensed consumer-research / audience panel platform API or platform export New connector, highest field yield of the package Employer-review source No usable public API Bot/collector fallback, low volume, static attributes Public no-credential web sources (3–4) Open HTTP / public endpoints Low complexity, includes a sports-statistics source and a search-interest source Volume and shape: approximately [A] company-level entities per run at [C] runs per month, and [B] property-level entities refreshed on a slower cadence. Roughly 60 fields at company level and 70 at property level, of which a significant share are already satisfied by existing connectors or by replication of our own platform tables — your scope is the remainder. Static attributes must not be re-pulled per run; time-series values refresh on schedule with [D] of history on first load. Entity resolution: every external source must link to our internal entity ID. Exact matches resolve automatically; ambiguous matches queue for human confirmation, and every confirmation is recorded. No linking by file name or by free-text company name alone. Package B — contact-level enrichment (not in this posting). Person-level data from a B2B contact database with an enrichment orchestration layer. Handled separately under a data-processing agreement; mentioned only so you understand it is deliberately excluded here. Required stack Non-negotiable, because it must match what exists: AWS (S3, Redshift, Secrets Manager, IAM, Lambda), Parquet, dbt for SQL transformations, tests and lineage, PySpark for heavy transforms, Python, Git with PR-based review and CI. Credentials in Secrets Manager — never in code, config or environment files committed to the repo. API-first extraction; bots only where no usable API exists. Deliverables and acceptance criteria Each is independently testable. We will verify all of them before final payment. Connectors. Every in-scope source lands in the S3 raw layer on schedule through the existing ingestion framework, implemented as configuration of that framework rather than a standalone pipeline. Authentication, rate-limit handling, incremental extraction and retry behaviour all demonstrated. History retention. Nothing is overwritten. Every record carries its source and extraction timestamp. A re-run on identical inputs produces identical outputs. Entity resolution. One table links our internal ID to every external key, with automatic exact matching, a queue for ambiguous cases, and an audit record of every confirmation. Curated fields. All in-scope fields queryable in Redshift at the specified grain, each traceable to its source. Quality gate. Automated tests on keys, completeness and value ranges run on every load and block bad data from reaching the curated layer. Failures alert to a channel we nominate. Documentation and transfer. A field-level catalogue mapping each field to source, key, refresh and owner; a runbook per connector; and a recorded walkthrough. Final acceptance requires one of our own engineers to onboard a new source unassisted using your framework configuration and playbook, and to run the pipeline for one full cycle without you. This is a gate, not a formality. Constraints All work runs inside our AWS environment, on our repositories, under our CI standards. No data leaves it beyond the lookups each source itself requires. We own every artifact produced — code, schemas, pipeline definitions, configuration, documentation — with no reverse-engineering restrictions on our own property. Declare in writing any pre-existing tooling of yours used during delivery, and confirm nothing of yours is required to operate the result afterwards. No component of yours may sit in the critical path. EU/GDPR. All processing stays in EU regions. You will sign a DPA and be listed as a sub-processor. Out of scope and not disclosed: our scoring and valuation logic, our weighting models, our product APIs and application screens, and the commercial terms attached to any entity. You do not need them and you will not receive them. Source subscriptions and API credits are contracted and paid by us. How to quote — read this carefully Quotes structured as “N weeks × a team of M” will not be compared. Give us: Total effort in person-days, split by role and seniority Price per connector, by complexity band Price per modelled field beyond the connector Your day rate per role The milestone schedule you propose and what triggers each payment Fixed price against milestones. We do not engage on open-ended time and materials.