Senior AI Pipeline Engineer – PDF-to-JSON Data Processing Platform

Project Overview We are seeking an experienced AI Pipeline Engineer / Data Processing Architect to review, redesign, and optimize an existing PDF-to-JSON-to-PostgreSQL data extraction platform. Our current system processes large volumes of structured PDF documents from multiple sources, extracts tabular and semi-structured data using LLM-powered workflows, converts that information into JSON, and loads it into PostgreSQL. While a functional prototype exists, onboarding new document formats remains time-intensive and the current architecture does not scale efficiently for production requirements. We are looking for an expert who can evaluate the existing implementation, identify architectural weaknesses, and lead the design of a scalable, maintainable solution capable of rapidly supporting new PDF document formats with minimal engineering effort. Technical Environment Core Technologies • Python • PostgreSQL • LLM-powered document extraction • JSON-based data models • Structured PDF documents (non-scanned) Areas of Expertise Required • LLM prompt engineering • AI-driven data extraction pipelines • PDF parsing and document processing • ETL/ELT architecture • Data classification and normalization • PostgreSQL schema design • Workflow orchestration and pipeline optimization Current Challenges 1. Scaling New Document Types The existing pipeline relies heavily on document-specific extraction templates and mappings. Supporting new document types requires significant manual effort and creates scalability challenges. The ideal solution should either: • Greatly reduce the amount of template maintenance required, or • Replace the template-based architecture with a more generalized and adaptable mapping framework. 2. Monolithic LLM Processing The current extraction workflow attempts to perform too many tasks within a single LLM interaction, resulting in: • Increased processing costs • Slower execution times • Dependence on more advanced models • Reduced maintainability We believe the workflow should be restructured into multiple processing stages that leverage simpler models and deterministic logic wherever possible. 3. Data Classification & Normalization Equivalent data elements are often labeled differently across document sources, requiring significant manual review and maintenance. The platform needs a robust framework for: • Standardizing extracted data • Managing aliases and terminology variations • Automatically classifying extracted content into standardized schemas • Reducing manual intervention 4. Database Ingestion Architecture The current process for loading validated data into PostgreSQL requires redesign. A scalable ingestion framework should be architected and implemented to support production-level processing volumes. Objectives The redesigned platform should: • Extract data from PDFs with high fidelity and minimal content drift from source documents. • Produce clean, structured JSON output from extracted content. • Support documents containing multiple logical entities and generate separate outputs when appropriate. • Automatically classify and map extracted data to standardized schema categories. • Normalize aliases and terminology variations across document sources. • Rapidly support new PDF structures without requiring major code changes. • Route validated data through the processing pipeline and load it into PostgreSQL. • Provide robust auditing, validation, and exception handling capabilities. Scope of Work Architecture Assessment • Review the existing codebase and pipeline design. • Identify bottlenecks, technical debt, and scalability concerns. • Recommend architectural improvements and implementation strategies. Pipeline Refactor • Redesign the extraction workflow. • Implement a modular, multi-stage processing architecture. • Improve accuracy, throughput, maintainability, and operational costs. Classification & Mapping Framework • Design a generalized system for data normalization and classification. • Implement code-driven routing and schema mapping logic. • Reduce dependency on document-specific configurations. Validation & Quality Controls • Build validation processes for: - Unclassified data - Mapping failures - Extraction inconsistencies - Exception handling workflows PostgreSQL Integration • Design and implement a production-ready data loading framework. • Support schema evolution, auditing, and future extensibility. Administrative & Maintenance Tools • Create tools to: - Review extracted data - Manage mappings and classifications - Update schema definitions - Audit pipeline exceptions - Support continuous onboarding of new document formats Ideal Candidate • Senior Python Engineer, Data Engineer, or Solutions Architect • Experience building AI-driven document processing systems • Strong background in LLM prompt engineering and extraction workflows • Experience designing scalable ETL/ELT pipelines • Expertise with PostgreSQL and data modeling • Experience with classification, normalization, and schema mapping systems • Proven track record refactoring complex systems into scalable production architectures • Comfortable providing both architectural guidance and hands-on implementation Bonus Experience • Building high-volume document processing platforms • Designing systems that support rapidly changing document formats • Developing data quality, auditing, and exception management frameworks • Optimizing AI workflows for both accuracy and cost efficiency We are looking for a partner who can not only implement solutions but also help define the long-term architecture needed to support large-scale document processing, rapid onboarding of new document types, and reliable production operations.