← Zakázky

Senior Data Engineer - AWS Lakehouse (Iceberg, Trino, Airflow, dbt)

Rozpočet: $12.0 - $20.0 HOURLY / FULL_TIME ⭐ 4.36 (8) Germany

apache-airflow-platform, python, apache-spark, snowflake, kubernetes, etl-pipelines, amazon-web-services, terraform

Preferred qualifications

  • Experience: Intermediate
We run a production data platform on AWS and are looking for an experienced data engineer to work on it with us on an ongoing basis. THE STACK YOU WOULD ACTUALLY TOUCH - Orchestration: Apache Airflow (managed / MWAA) - DAG authoring, scheduling, backfills - Lakehouse: Apache Iceberg on S3 with the Glue catalog - table maintenance, compaction, snapshot expiry, orphan-file cleanup - Query engines: Trino / Starburst, Athena, Snowflake - Transformation: dbt (Trino and Snowflake adapters) - Processing: Spark / PySpark, EMR - Infrastructure: Terraform on AWS, EKS + Helm, ArgoCD - Services: Python, FastAPI TYPICAL WORK - Build and fix Airflow DAGs, including hourly and backfill runs that must be safely re-runnable - Iceberg table maintenance and the retention / compaction jobs around it - dbt models against Trino and Snowflake - Terraform changes for the AWS resources these pipelines depend on - Debug pipeline failures end to end, from S3 and the Glue catalog through Iceberg and Trino out to the downstream API WHAT WE NEED - Real production experience with Iceberg and at least one of Trino / Starburst / Athena. This is the part most applicants do not have - please be honest about it. - Airflow in production, not just tutorials - dbt beyond the basics - Terraform and AWS IAM competence - You use Claude / Claude Code (or a comparable AI coding assistant) as a normal part of your daily workflow. We work this way and expect you to be genuinely fast with it - including knowing where these tools get things wrong and how you verify their output. This is not optional for us. - Clear written English, works independently, asks when something is unclear NICE TO HAVE Snowflake, Kubernetes/Helm, Grafana/Prometheus, German. HOW WE WORK We start with one small paid task so both sides can see whether it fits. If that works, ongoing part-time collaboration. Our budget for this role is up to $20/hour - please only apply if that works for you. PLEASE ANSWER THESE IN YOUR PROPOSAL (short, concrete answers - no marketing copy) 1. How do you handle orphan-file cleanup and snapshot expiry for Iceberg tables on S3 with the Glue catalog? What goes wrong if the retention is configured incorrectly? 2. Describe one Airflow DAG you made safely re-runnable (idempotent). What exactly did you change? 3. How do you use Claude (or a similar AI coding assistant) in your daily work? Give one concrete example where it was wrong and how you caught it. 4. Your hourly rate, and how many hours per week you can commit. Proposals that do not answer these four questions will not be reviewed.
Otevřít na Upwork

AI proposal draft

Generate a short cover letter for this job. Edit before sending.

Sign in to generate an AI proposal draft.

Přihlásit