Senior Data Engineer – AI Data Processing & Privacy Infrastructure

Senior Data Engineer – Data Processing & Off-the-Shelf Dataset Curation We are looking for an experienced Senior Data Engineer to help us build an AI-powered data anonymization and privacy-preserving data processing platform. The ideal candidate should have strong experience in data engineering, large-scale data processing, ETL pipelines, and sourcing and preparing off-the-shelf (OTS) datasets for AI/ML applications. You will work closely with our engineering and AI teams to develop scalable data pipelines, prepare high-quality datasets, and implement secure data processing workflows. Key Responsibilities: 1. Data Engineering & Processing - Design, develop, and maintain scalable ETL/ELT pipelines. - Build data ingestion and processing workflows for CSV, JSON, Excel, Word, PDF, emails, Slack messages, and database exports. - Implement data cleaning, normalization, transformation, deduplication, and validation. - Develop efficient pipelines for processing large datasets, including terabyte-scale workloads. - Work with Python, SQL, Pandas, PyArrow, and Apache Spark. - Optimize pipeline performance, reliability, and processing costs. 2. Off-the-Shelf Data Sourcing & Preparation - Research and identify publicly available, open-source, and commercially licensed datasets. - Find suitable datasets for AI training, PII detection, Named Entity Recognition (NER), and data anonymization. - Evaluate dataset quality, completeness, accuracy, licensing, and usability. - Clean, organize, and prepare datasets for machine learning applications. - Create structured training, validation, and testing datasets. - Maintain documentation covering dataset sources, permissions, and transformations. - Ensure appropriate data governance and licensing compliance. 3. AI/ML Dataset Preparation - Prepare training datasets for PII/PHI detection and anonymization models. - Work with AI engineers to define dataset schemas and annotation requirements. - Support synthetic data generation and data labeling workflows. - Develop automated dataset quality checks. - Prevent duplication and data leakage between training and evaluation datasets. - Support benchmark dataset development and model evaluation. 4. Privacy & Secure Data Processing - Integrate sensitive data detection and anonymization components into data pipelines. - Support data masking, pseudonymization, tokenization, and synthetic data replacement. - Develop secure data processing workflows within customer-controlled cloud environments. - Implement encryption, access controls, and data handling best practices. - Support GDPR, HIPAA, and other applicable privacy requirements. 5. Cloud & Infrastructure - Work with AWS or GCP for data storage and processing. - Develop containerized data processing workloads using Docker. - Work with PostgreSQL, S3, and distributed processing systems. - Support secure, self-hosted and customer-VPC deployments. - Implement data pipeline monitoring, logging, error handling, and recovery. Required Skills: - 4+ years of professional data engineering experience. - Strong expertise in Python and SQL. - Hands-on experience with ETL/ELT pipeline development. - Experience with Pandas, PyArrow, Polars, or Apache Spark. - Experience processing structured and unstructured datasets. - Experience sourcing, cleaning, and preparing off-the-shelf datasets. - Strong understanding of data quality, data modeling, and data validation. - Experience with AWS or GCP. - Familiarity with AI/ML training data preparation. - Understanding of secure data processing and data privacy principles. Nice to Have: - Experience with PII detection and data anonymization. - Experience with NER, NLP, or Hugging Face datasets. - Knowledge of Microsoft Presidio or spaCy. - Experience with synthetic data generation. - Familiarity with Apache Airflow or Temporal. - Experience with large-scale data processing and distributed computing. - Understanding of GDPR, HIPAA, SOC 2, or ISO 27001. Expected Deliverables: - Scalable data ingestion and processing pipelines. - Clean, validated datasets ready for AI/ML training. - A curated catalog of suitable off-the-shelf datasets with documented licensing. - Automated data preparation and validation workflows. - Support for processing multiple document formats. - Secure integration with data anonymization workflows. - Proper documentation and reproducible data pipelines. Who We're Looking For: We are looking for a hands-on engineer who can take ownership of data engineering tasks, work independently, and collaborate with our AI and infrastructure teams. The ideal candidate should understand the complete data lifecycle, from sourcing existing datasets to building production-ready data processing pipelines. Experience working with privacy-sensitive data or AI/ML infrastructure will be a strong advantage. To Apply: Please include the following in your proposal: 1. Briefly describe your experience building large-scale data processing pipelines. 2. Have you worked with off-the-shelf datasets for AI/ML training? Please share examples. 3. Which technologies have you used for ETL pipelines and distributed data processing? 4. Have you worked with PII detection, anonymization, or privacy-sensitive datasets? 5. Share one or two relevant projects, preferably involving large datasets or AI/ML pipelines. We are looking for someone who can contribute to the project immediately and work with us on building a scalable, secure, and reliable data processing platform.