Data Engineer - PDF Processing Pipeline

We're a mid-sized company processing dozens of legal contracts daily, and we need a Data Engineer to take our AI-powered PDF pipeline from "barely working" to "boringly reliable." Right now, PDFs land in S3, we run them through regex + Claude for risk extraction, and email results to admins. But the extraction breaks on weird table layouts, Airflow tasks randomly timeout, and there's zero visibility into what's failing. What we need you to own: - Text extraction: Use pdfplumber/pymupdf to pull text, then clean it with `re` and Pandas. Handle scanned docs with OCR fallbacks. The goal: feed Claude clean, structured text so we stop wasting tokens on garbage. - Orchestration: Rewrite our Airflow DAGs with proper retries, task grouping, and failure alerts. We need to know which PDF broke and why without digging through CloudWatch for 20 minutes. - Simple internal UI: Build a lightweight web interface (Streamlit, Flask, or FastAPI) where admins can see a list of processed contracts, view flagged risks, override AI decisions, and manually re-run failed documents. This isn't a public-facing app—keep it functional and fast. - Future-proofing: This UI and its backend APIs must be built cleanly because it will eventually be embedded into our main company web portal. So no spaghetti code, think modular, well-documented endpoints. Tech stack: Python (re, pandas, numpy), AWS S3, Airflow, Claude API, and a basic web framework of your choice. You are: Someone who gets defensive about data quality. You write code that assumes the PDF will be corrupted, the schema will change, and the API will rate-limit you. You also communicate clearly. We don't need daily essays, just honest updates when something breaks.