Data Engineer: Video + Sensor QC Pipeline for Robot Training Data

We're an early-stage team collecting real-world manipulation data for robot learning. Operators record tasks with a handheld gripper: a GoPro (4K, 60 fps, built-in IMU) plus force and gripper-width sensors, each running on its own clock. We already have a Python QC prototype: data loaders, 18 automated checks (video, force, sync, gripper, SLAM pose), an HTML report and an evaluation script. We also have a written spec for how data should be stored, checked and reviewed. What we don't have yet is the production system around it. That's this job. Milestone 1: storage and ingest - Cloud object storage (S3 or GCS) with three zones: raw (never modified; versioning and delete lock), derived, curated - An offload tool for field laptops: copy SD cards, checksum every file, upload, read back and verify before the card is wiped - A per-session manifest and a Postgres schema for sessions, files and QC results - An event-driven queue that runs our existing checks when a session finishes uploading - Working end to end on public UMI-style sample data Milestone 2: review - A simple web page where reviewers watch flagged clips and confirm or overturn the result - A form for non-technical teammates to flag problems - A daily summary (email or Slack) and an alert when one device keeps failing the same check Milestone 3: after our pilot data comes in - Calibrate thresholds against human labels and harden the checks Data volume starts at a few TB and will grow to tens of TB. One hour of video is about 45 GB per camera. You're likely a fit if you: - Have personally built and run a pipeline for large video or sensor data, not only contributed to one inside a big infra team - Know S3/GCS (lifecycle, versioning, object lock, IAM), Postgres schema design, Python, and a job queue or workflow tool (SQS, Pub/Sub, Celery, Prefect, Airflow or similar) - Are comfortable with FFmpeg and video metadata - Can set this up on your own, document it, and hand it over cleanly - Write clear English and can overlap 3-4 hours a day with New York time Nice to have: ROS bags, MCAP, HDF5 or LeRobot; GoPro GPMF; time-series alignment; Docker and Terraform. This is engineering work. No ML research or model training. How we'll work: shortlisted candidates get the full spec and prototype under NDA. We start with a short paid scoping task (read the spec, propose architecture, stack and an estimate for Milestone 1), then go milestone by milestone. If it goes well, there is more work after this. Rate is open. Tell us your hourly rate and how you would estimate Milestone 1. To apply, start your proposal with the word "manifest". Then tell us about the largest video or sensor pipeline you built yourself: data volume, stack, and one thing that broke in production and how you fixed it.