Data Engineer for a high-frequency analytics platform

Project: We are developing an analytics platform to process real-time telemetry data for the optimisation and trading platform. This solution supports 24x7 high-frequency control and monitoring operations. Some data pipelines process at 0.2 seconds; others run at lower frequencies. Profile: Data Engineer (Python) We need additional engineering support to fast-track the hardening and production deployment. Some tasks in this profile include: - Migrate the platform to production in GCP. - Build/harden the data pipelines from open-source data. - Evaluate existing implementation, build/harden the edge telemetry data pipeline. - Schedule jobs via Dagster. - ETL/ELT, data modelling and storage in a cloud platform. - CI/CD automation with GitHub Actions. (and everything else that a data engineering profile demands) Acceptance Criteria: - All pipelines must be scalable and fault-tolerant for both batch and real-time streaming data. - TDD with reproducible code. Duration: 8 weeks (fixed monthly payment) Availability: full-time; 40 hours/week with overlapping CET hours. Required Skills: - Industrial hands-on experience in Python programming. (min 5 years) - (Must) Hands-on experience implementing SQL and data wrangling: ETL/ELT (min 5 years) - (Must) Hands-on experience in event-driven architecture/messaging queue frameworks such as MQTT, Kafka, Pub/Sub, RabbitMQ, etc. (min 5 years) - Secure API development. - Experience in Docker/GKE, Airflow or Dagster, Git, CI/CD and TDD. NDA applies.