Data Pipeline Engineer (SQL/Python)

**Title:** Junior Data Pipeline Engineer (Python/SQL) **Read before applying:** This is a **junior-level, fixed-scope assignment estimated at 32 hours** (fixed contract - only). Please apply only if you are comfortable completing the defined deliverables within this junior-level scope, budget, and timeline. We need a junior data pipeline engineer to complete a defined ETL/ELT data ingestion service for a project-management platform's analytics layer. - PostgreSQL staging schemas and migration scripts for raw Person, Employment, and Classification ingestion - Python ETL jobs (using Pandas or PySpark) to extract from source APIs/databases, transform (clean, deduplicate, normalize) - Load into a target data warehouse (PostgreSQL/BigQuery) - Incremental load logic (upsert/merge) with change-data-capture (CDC) awareness for daily updates - Orchestration DAG (using Prefect, Airflow, or Dagster) to schedule, monitor, and retry failed pipeline runs - Data quality validation rules (null checks, uniqueness, referential integrity, range checks) with logging and alerting on failures - Simple data lineage documentation (source → transform → target) for analytics team handoff - Basic performance optimization (batch sizing, indexing, query tuning) for pipeline efficiency - Clear setup, environment variable, and handoff documentation ### Required Skills - Python (Pandas or PySpark) - PostgreSQL / SQL (complex joins, aggregations, window functions) - FastAPI / REST API integration (extracting from external sources) - At least one orchestration tool (Prefect, Airflow, Dagster, or similar) - Git and basic data validation testing - Understanding of incremental vs. full-load strategies and idempotent pipeline design In your proposal, begin with: **“SCOPE CONFIRMED.”** To prove you read the job description. Then share one relevant data pipeline example (e.g., a Python script that extracts from an API, transforms with Pandas, and loads into PostgreSQL) and briefly confirm your availability.