AI and Data Engineering: Building an LLM-Integrated Data Pipeline

An end-to-end data pipeline that blends core data engineering with applied AI, built to show how the two work together in a real production-style workflow, not as separate skill sets, but as one integrated system. On the data engineering side: ingestion from CSV, database, or API sources, schema validation, deduplication, and orchestration with Airflow, with dbt/Python handling transformations and loading into a clean, query-ready table. On the AI side: an LLM is embedded directly into the pipeline to classify records, tag and enrich free-text fields, flag anomalies, and generate structured summaries from unstructured data, work that traditional rule-based ETL can't handle on its own. Stack: Python · Airflow · dbt · OpenAI/Anthropic API · PostgreSQL Why it matters: most teams today aren't looking for a pure data engineer or a pure AI engineer, they need someone who can build reliable pipelines and know where and how to layer AI into them. This project is a working example of exactly that combination