Microsoft Fabric Data Pipeline Engineer (40 Hour Contract)

We're piloting a pipeline that moves SharePoint Online content and metadata into Microsoft Fabric (OneLake) using a medallion architecture. This is a hands-on engineering role: we need someone to test and document how the connector and pipelines behave (metadata, permissions, versions, refreshes, duplicates, deleted content) rather than assume. What you'll do • Configure and test the SharePoint connection for a defined pilot scope • Build pipelines that land documents and metadata in the Bronze layer • Implement incremental refresh, change detection, retry, logging, and exception handling • Develop Silver-layer cleansing, de-duplication, version handling, and metadata mapping • Prepare Gold-layer outputs for validated information products • Capture lineage from SharePoint sources through to validated outputs • Test document additions, updates, versions, duplicates, and other change scenarios • Build dashboards, alerts, and run histories for monitoring • Document technical constraints, workarounds, and production considerations Required skills • University degree in Computer Science • 5+ years of development on the Microsoft platform • Hands-on Microsoft Fabric experience: data pipelines, OneLake, lakehouses, notebooks, Spark or SQL transformations, orchestration • Strong data engineering skills: incremental loads, schema handling, data quality, observability, recoverability • Experience integrating REST APIs, files, document metadata, or Microsoft 365 data sources • Proficient in SQL and Python or PySpark • Experience with source control, deployment, testing, and documentation • Ability to design repeatable tests that show actual platform behaviour Nice to have • SharePoint Online, Microsoft Graph, content types, versioning, and permissions • Medallion architecture, Delta formats, semantic models, or AI-ready data preparation • Microsoft Fabric Analytics Engineer, Azure Data Engineer, or similar certifications • Experience building pilots that were hardened for production Deliverables • Working SharePoint-to-OneLake ingestion pipeline • Bronze, Silver, and Gold layer components • Tests, monitoring, logs, runbooks, and deployment documentation • Technical findings and production-hardening recommendations