Python + GCP + Terraform + Harness Pipeline Expert Needed (Data Integration & Automation)
Бюджет: $30.0 - $33.0
HOURLY / FULL_TIME
⭐ 4.88 (89)
Canada
python, bigquery, google-cloud-platform, python-script
We are looking for an experienced engineer with strong expertise in Python, Google Cloud Platform (GCP), Terraform, and Harness CI/CD pipelines to help build and automate a cloud-based data integration solution.
Project Overview
We need to build an automated pipeline that:
* Fetches user data from one or more REST APIs using Python.
* Cross-references the user data with existing team/organizational datasets.
* Identifies and fills missing attributes by merging multiple data sources.
* Uses an LLM to intelligently categorize users based on predefined business rules.
* Stores the enriched dataset in Google BigQuery.
* Automates the entire workflow using Harness pipelines.
* Provisions and manages cloud infrastructure using Terraform.
* Delivers clean, analytics-ready data for the Power BI team to build dashboards.
Required Skills
* Strong Python development experience
* Google Cloud Platform (BigQuery, Cloud Storage, IAM, Cloud Functions/Cloud Run or similar services)
* REST API integration and data processing
* Terraform (Infrastructure as Code)
* Harness CI/CD pipeline automation
* SQL and BigQuery optimization
* Experience working with LLMs (OpenAI, Gemini, Vertex AI, or similar) for data enrichment and categorization
* Git and version control best practices
Nice to Have
* Experience with Power BI data models
* Data engineering and ETL pi
Отвори в Upwork