Senior Data Engineer | Airbyte, Airflow, dbt, Redshift, Superset & AI Data Platforms

## About the Role We are seeking an experienced **Senior Data Engineer** to design, build, and maintain a modern cloud-based data platform for a US-based professional services organization. This role involves developing scalable ELT pipelines, transforming data into analytics-ready models, and enabling business intelligence and AI-driven analytics. The ideal candidate is highly autonomous, capable of rapidly integrating new data sources, and experienced with modern data engineering tools including **Airbyte, Apache Airflow, dbt, Amazon Redshift, and Apache Superset**. Experience preparing enterprise data for AI and machine learning use cases is highly desirable. Given the project's accelerated timeline, we are looking for someone who can contribute immediately and take ownership of the end-to-end data engineering lifecycle. ## Key Responsibilities * Design, develop, and maintain scalable ELT pipelines using **Airbyte** to ingest data into **Amazon Redshift**. * Configure and manage native Airbyte connectors (e.g., QuickBooks, Salesforce) as well as develop custom connectors for third-party applications and REST APIs. * Build custom data extraction solutions using Python where native connectors are unavailable. * Orchestrate and monitor data workflows using **Apache Airflow**, ensuring reliable scheduling, dependency management, and operational visibility. * Consolidate data from multiple instances of the same source system into unified warehouse models. * Ingest, validate, and normalize Excel and spreadsheet-based datasets into structured warehouse tables. * Develop robust **dbt** models for staging, transformation, testing, documentation, and business-ready data marts. * Implement incremental loading strategies, schema evolution handling, data quality validation, and automated pipeline monitoring. * Create semantic datasets and dashboards using **Apache Superset** to support business reporting and executive analytics. * Collaborate with business stakeholders to translate reporting and analytical requirements into scalable data models. * Design data architectures that support advanced analytics, AI initiatives, predictive modeling, and future machine learning workloads. ## Required Skills & Experience * Strong hands-on experience with **Airbyte**, including developing or customizing connectors using the Airbyte CDK or Low-Code Connector Builder. * Solid experience with **Apache Airflow** for workflow orchestration and production pipeline management. * Advanced Python programming skills, including REST API integrations, authentication mechanisms, pagination, rate limiting, retries, and error handling. * Strong SQL expertise with **Amazon Redshift** or equivalent cloud data warehouse technologies. * Extensive experience using **dbt** for data transformations, testing, documentation, and modular data modeling. * Experience integrating heterogeneous enterprise systems into centralized analytical platforms. * Ability to quickly understand unfamiliar APIs and build production-ready ingestion pipelines within short delivery timelines. * Strong understanding of data modeling, warehouse architecture, and ELT best practices. * Experience implementing monitoring, alerting, logging, and pipeline reliability practices. * Excellent written communication skills in English with availability for partial overlap with US Eastern business hours. ## Preferred Qualifications * Experience with AWS services including IAM, S3, CloudWatch, Secrets Manager, and networking related to Amazon Redshift. * Familiarity with QuickBooks, Salesforce, and professional services data models. * Experience building executive dashboards using **Apache Superset**. * Previous experience working with consulting firms, agencies, or professional services organizations. * Knowledge of AI-ready data architectures, feature engineering, vector databases, LLM data pipelines, or supporting Generative AI and machine learning applications. * Experience implementing data governance, lineage, metadata management, and data quality frameworks. ## Technical Stack * **Data Ingestion:** Airbyte, Python * **Orchestration:** Apache Airflow * **Transformation:** dbt * **Data Warehouse:** Amazon Redshift * **Business Intelligence:** Apache Superset * **Cloud Platform:** AWS (S3, IAM, CloudWatch) * **Languages:** SQL, Python * **Data Sources:** REST APIs, QuickBooks, Salesforce, Excel, ERP, CRM, Time Tracking, Expense Management Systems ## What Success Looks Like * Reliable, production-grade data pipelines with automated monitoring and alerting. * Clean, documented, analytics-ready data models powering business intelligence and AI initiatives. * Rapid onboarding of new enterprise data sources with minimal operational overhead. * High-quality, scalable data infrastructure that supports reporting, forecasting, and future AI-driven decision-making.