← Lavori

Python + GCP + Terraform + Harness Pipeline Expert Needed (Data Integration & Automation)

Budget: $30.0 - $33.0 HOURLY / FULL_TIME ⭐ 4.88 (89) Canada

python, bigquery, google-cloud-platform, python-script

We are looking for an experienced engineer with strong expertise in Python, Google Cloud Platform (GCP), Terraform, and Harness CI/CD pipelines to help build and automate a cloud-based data integration solution. Project Overview We need to build an automated pipeline that: * Fetches user data from one or more REST APIs using Python. * Cross-references the user data with existing team/organizational datasets. * Identifies and fills missing attributes by merging multiple data sources. * Uses an LLM to intelligently categorize users based on predefined business rules. * Stores the enriched dataset in Google BigQuery. * Automates the entire workflow using Harness pipelines. * Provisions and manages cloud infrastructure using Terraform. * Delivers clean, analytics-ready data for the Power BI team to build dashboards. Required Skills * Strong Python development experience * Google Cloud Platform (BigQuery, Cloud Storage, IAM, Cloud Functions/Cloud Run or similar services) * REST API integration and data processing * Terraform (Infrastructure as Code) * Harness CI/CD pipeline automation * SQL and BigQuery optimization * Experience working with LLMs (OpenAI, Gemini, Vertex AI, or similar) for data enrichment and categorization * Git and version control best practices Nice to Have * Experience with Power BI data models * Data engineering and ETL pi
Apri su Upwork