Data Engineer (Mid-Senior) – Fabric-to-AWS Migration

# Data Engineer (Mid-Senior) – Fabric-to-AWS Migration **Type:** Long-term, 4–5 hours/week ## About the Role We are moving our client data platform from Microsoft Fabric to AWS, and we need a mid-to-senior data engineer to help build and deliver that migration. ## The Migration at a Glance - **Scale:** Several terabytes of data across roughly 12 clients, each with its own Fabric workspace. - **Source (today):** A Fabric lakehouse on OneLake with raw, canonical, and semantic layers stored as Delta tables, fed by three pipeline types per client: two that ingest from SQL Server and one that loads JSON event data landed in ADLS Gen2. - **Target:** An AWS data lake using Iceberg on S3, queried through Athena and the Glue Data Catalog, with Lake Formation and IAM for access control. Metadata and logging data moves from Azure SQL to RDS Postgres. - **Approach:** Build one parameterized, metadata-driven template (Terraform plus Glue jobs keyed by client, layer, and table), prove it on one pipeline type, run Fabric and AWS in parallel with automated reconciliation, then cut over client by client with a rollback window. ## What You Will Do - Build metadata-driven AWS Glue (PySpark) jobs that read OneLake Delta data and write Iceberg tables, handling details such as deletion vectors, partitioning, and version pinning so results can be reconciled. - Stage and move historical data from ADLS Gen2 and OneLake to S3 using the right tool for each job (DataSync, AzCopy, rclone, Glue, DMS). - Rebuild Fabric pipeline logic (copy and lookup activities, scheduling, email notifications) on MWAA/Airflow or Step Functions, with Lambda, SNS, and SES where they fit. - Port Fabric notebooks to Glue: replace notebookutils, abfss:// paths, and notebook chaining with boto3, Secrets Manager, S3 paths, the Glue Data Catalog, and workflow orchestration. Translate T-SQL where needed. - Migrate relational metadata and logging data from SQL Server to Postgres, including schema conversion and data type handling. - Write Terraform modules and CI/CD (GitHub Actions) so that a new client environment is created from a template rather than by hand. - Build and run reconciliation checks (row counts, aggregates, schema and data comparisons) for each table during parallel runs, and document the results for sign-off. - Monitor cost and performance, and recommend improvements as workloads settle on AWS. - Document patterns and runbooks so that onboarding the next client is repeatable. ## What We Are Looking For ### Required - 4+ years of professional data engineering experience, with at least 2 years building production pipelines on AWS. - Strong Python and SQL, plus hands-on PySpark experience. - Working experience with AWS Glue, S3, Athena, and the Glue Data Catalog. - Experience with an open table format (Iceberg, Delta, or Hudi) and a solid grasp of the differences between them. - Experience with a workflow orchestrator such as Airflow/MWAA or AWS Step Functions. - Experience delivering data migrations or platform rebuilds, including validation and cutover planning. - Clear written and verbal communication; you can explain a trade-off to both engineers and non-technical stakeholders. ### Strongly Preferred - Experience with Microsoft Fabric, Azure Data Factory, Synapse, or Azure Data Lake Storage Gen2. - Experience with AWS DMS and schema conversion (SQL Server to Postgres). - Familiarity with Lake Formation and IAM-based data access control. - Experience building data quality or reconciliation frameworks. - Experience with event or JSON ingestion at volume (Kafka, Event Hubs, Kinesis). ### Nice to Have - Interest in using AI-assisted tooling to speed up repetitive conversion work, with a human review step.