Senior Data Engineer: Design & Build Automated Python ETL Pipelines with Power BI Reporting Layer
Bütçe: $300.0
FIXED /
⭐ 5.00 (3)
Egypt
pyspark, apache-airflow-platform, python, etl-pipelines, microsoft-power-bi, data-modeling, sql, data-visualization, pandas, postgresql
We’ve grown fast and our data is a mess — CRM exports, a PostgreSQL production database, and weekly CSV drops from 3 vendors. Nothing is connected. Our team makes decisions from stale spreadsheets. We need a senior data engineer to design a proper pipeline, automate the data flow, and give our leadership team a Power BI dashboard that actually reflects reality every morning.
This is not a junior task. We’ve tried before and got unmaintainable spaghetti code. We need someone who thinks in schemas, writes clean modular Python, documents as they go, and treats data quality as a first-class concern — not an afterthought.
SCOPE OF WORK
• Audit all 3 data sources and design a unified star-schema analytical model
• Build modular Python ETL scripts: extract from API + PostgreSQL + CSV, transform, load to analytical layer
• Implement data quality checks at every stage: null handling, deduplication, type validation, anomaly alerts
• Set up automated daily scheduling (cron / Airflow) so data refreshes without manual intervention
• Connect final analytical tables to Power BI; build 2–3 core dashboards (sales, ops, pipeline)
• Deliver full documentation: data dictionary, pipeline diagram, runbook for the internal team
• 2-week handover support period after delivery — answer questions, fix edge cases
Upwork'te aç