Senior Data Platform Engineer(s) : (Airtight Git Hygiene, PySpark, Custom Python Frameworks)

Job Description We are looking for a high-caliber Senior Data Platform Engineer / Software Engineer (Data Expertise) to help design, build, and scale a next-generation, generic data infrastructure platform. This is not a traditional role for building isolated, one-off ETL pipelines. Our architecture relies on a highly abstract, metadata-driven framework where a single, unified engine orchestrates system-wide data operations via content parameters. A single code change you make will have a global, system-wide impact. We are open to Full-Time (FTE), Contract-to-Hire (C2H), or Direct Contract engagements. Deep Technical Stack & Core Responsibilities 1. Advanced Python & Package Lifecycle Theory • Custom Package Ecosystems: Experience or deep conceptual understanding of designing, versioning, and deploying custom internal Python packages. • Private Distribution Mechanics: Understanding of how shared enterprise libraries are hosted on private PyPI/Artifact servers, and how local environment settings are configured to securely pull and install these internal dependencies. • Dependency & Conflict Management: Mastery over managing complex Python dependency trees, resolving environment-level package conflicts, and structuring robust setup.py / pyproject.toml configurations. 2. Rigorous Version Control & Git Hygiene • Branching & Merging: Exceptional Git hygiene is mandatory. You must have clean habits regarding when to pull, which branches to target, and what critical checks to perform before merging code. • State Management: Deep comfort with advanced Git commands, specifically how to cleanly revert to specific historical checkpoints or roll back production-breaking changes without disrupting the tree. 3. Foundational & Specialized Data Engineering • Delta Live Tables (DLT) & Delta Lake: Hands-on experience leveraging Delta Live Tables for building reliable, maintainable, and testable data processing pipelines. High proficiency in handling Delta table mechanics (ACID transactions, time travel, schema evolution, optimization). • Infrastructure vs. Governance: Clear understanding of the boundary between Infrastructure-as-Code (using Terraform to provision cloud resources, service principals, and roles) and Data Governance frameworks (like Databricks Unity Catalog for managing asset volumes and data access controls). • Core PySpark Optimization: Ability to tune and optimize Spark distributed computing jobs without relying heavily on a visual Spark UI, specifically handling massive table ingestions, configuring JDBC connection parameters for speed, and managing partitioning, ordering, and sorting strategies. Our Technical Philosophy & Interview Approach • Architectural Abstraction: We look for engineers who naturally build for the future. When you implement a feature for a new ingestion source, it must be generic enough to automatically benefit existing and future pipelines. • Honesty over AI Fluency: We have a zero-tolerance policy for AI-assisted cheating during our screening process. We value engineering hygiene, a solid problem-solving attitude, and the confidence to say "I don't know, but here is how I would test it" over robotic, script-read responses. Mandatory Interview Security Notice To ensure fairness and technical integrity, all technical interview rounds will be strictly proctored. Candidates must be comfortable using our designated secure system environment, and depending on location, in-person technical screening may be requested.