Senior Data Engineer: PySpark, S3/Delta Lakehouse for Tick-Level Futures Data

Quantum Signals is a Paris and San Francisco fintech building AI trading signals from CME limit-order-book data. Hedge funds and bank desks use our signals intraday. We are looking for a senior data engineer to take ownership of our market-data pipeline. What you will own • A batch medallion lakehouse (Bronze, Silver, Gold) that ingests Databento CME Level II (MBP-10) data and produces per-minute order-book features used to train our models. • PySpark 4 jobs on our self-managed Spark standalone cluster (Hetzner), reading and writing Parquet and Delta Lake on AWS S3. • Manifest-driven incremental runs, backfills across years of tick data, coverage checks and row-count gates. • Nightly orchestration: today Jenkins, moving to Prefect 3 (Docker Compose, Infisical secrets, Slack alerts). You will finish and harden that move. • Futures-contract handling: roll calendars, CME sessions and holidays, trade-date partitioning. • Later: a Kafka plus Spark Structured Streaming path for live data. Must have • 5+ years data engineering, with production PySpark (not only notebooks): partitioning, shuffle tuning, memory sizing, S3A. • Delta Lake or another table format on S3; Parquet internals. • Strong Python (3.12+), pytest, uv or Poetry, clean CLI tools. • Running Spark outside managed platforms: standalone or Kubernetes, Java/JVM troubleshooting. • A workflow orchestrator in production (Prefect preferred; Airflow or Dagster fine). • Comfort reading an existing 8,000-line codebase and improving it rather than rewriting it. Strong plus • High-frequency or tick market data: order books, MBP/MBO, Databento, CME, futures rolls. • DuckDB for validation; Kafka; Docker; Linux server administration. • Experience taking over a system from a departing owner. How we work GitHub PRs with code review, Linear tickets, Slack. Small senior team across Paris, SF and remote. Overlap with CET business hours for at least 3 hours a day. Engagement Hourly, 20 to 30 hours per week, initial 3 months with a view to long-term. A short paid trial task comes first.