Lead Data & AI Platform Engineer (Agentic Analytics)


Lead Data & AI Platform Engineer (Agentic Analytics) 
Location: UAE / Remote
Employment Type: Full-time / Remote
Experience: 5 to 8+ years
Level: Senior / Lead (Hands-on) About the Role We are looking for an experienced Lead Data & AI Platform Engineer to architect, build and evolve a production-grade, AI-powered analytics platform. Our vision is to create an intelligent data platform where business users can ask questions in natural language, explore trusted data, generate meaningful insights and initiate controlled business actions through specialized AI agents. The platform brings together modern data engineering, cloud-native infrastructure, semantic search and agentic AI to deliver analytics across business, marketing and product domains. We are looking for someone who can take ownership of this initiative—from architectural decisions and hands-on development to production deployment, security and continuous improvement. This is a hands-on engineering leadership role, not a research-only or management-only position. Key Responsibilities 1. Data Platform & Warehouse Engineering * Design, implement and maintain scalable data ingestion and transformation pipelines from databases, third-party APIs, SaaS platforms and event streams. * Build and optimize ELT workflows using Python, SQL, Apache Airflow, dlt and dbt. * Develop structured warehouse models, including staging, intermediate, fact and dimension tables. * Implement reliable incremental ingestion, Change Data Capture (CDC), schema evolution and data reconciliation. * Establish a governed semantic layer with reusable business metrics and consistent definitions. * Ensure data quality, freshness, observability and recoverability across pipelines. 2. Agentic AI & LLM Engineering * Design and develop production-grade AI agents using frameworks such as LangGraph. * Implement supervisor-driven multi-agent workflows capable of coordinating specialized analytical agents. * Integrate LLMs such as Anthropic Claude with enterprise data, APIs and analytical tools. * Develop Retrieval-Augmented Generation (RAG) capabilities using semantic and hybrid search. * Build secure, controlled text-to-SQL capabilities for natural-language data analysis. * Implement tool calling, conversational context management, persistent agent state, bounded execution and error recovery. * Establish agent evaluation, tracing, performance monitoring and LLM cost optimization. 3. Search, Streaming & Analytics * Implement knowledge retrieval and event analytics using OpenSearch or equivalent technologies. * Design hybrid search combining keyword retrieval, vector embeddings and similarity search. * Develop real-time or near-real-time event ingestion pipelines using AWS streaming services. * Integrate structured warehouse data and unstructured knowledge sources for AI-assisted analytics. * Enable analytical outputs including business KPIs, trends, funnels, cohorts, charts and actionable recommendations. 4. Cloud Architecture, Security & Reliability * Design and deploy scalable platform components on AWS using containerized services and Kubernetes. * Collaborate with DevOps engineers on infrastructure automation, CI/CD, monitoring and deployment strategies. * Implement secure multi-tenant data access, least-privilege permissions and controlled execution of AI-generated queries. * Establish strong safeguards against unauthorized data access, unsafe queries and unintended AI actions. * Design human-in-the-loop approval workflows for actions affecting external systems. * Ensure production readiness through automated testing, logging, auditability, resilience and resource optimization. 5. Technical Ownership & Leadership * Own the platform's technical roadmap and drive implementation from architecture to production. * Translate business requirements into scalable data models, services and AI workflows. * Make informed architectural decisions balancing scalability, security, performance, maintainability and cost. * Conduct code reviews, establish engineering standards and mentor team members. * Work closely with backend, frontend, DevOps and business stakeholders. * Evaluate new technologies pragmatically and take accountability for the quality and reliability of delivered solutions. Required Qualifications & Technical Skills Professional Experience * 8+ years of professional software engineering experience, including substantial experience in backend systems or data engineering. * Proven experience designing and delivering production-grade data platforms, distributed systems or analytical applications. * Demonstrable hands-on experience integrating LLMs or AI agents into real applications. * Experience taking technical ownership of complex, multi-component engineering initiatives. Core Technical Competencies * Programming: Advanced proficiency in Python and SQL, including API development, asynchronous processing and data-intensive applications. * Data Engineering: Strong knowledge of ELT/ETL, workflow orchestration, data modeling, data warehouses and data quality. * Databases: Strong practical experience with PostgreSQL, query optimization, schema design and analytical workloads. * AI Engineering: Practical understanding of LLM APIs, prompt engineering, function/tool calling, RAG, embeddings and AI agent orchestration. * Search & Retrieval: Experience with OpenSearch, Elasticsearch or comparable search/vector retrieval systems. * Cloud & Containers: Working experience with AWS, Docker, Kubernetes and cloud-native application deployment. * System Design: Ability to design secure, scalable, observable and fault-tolerant distributed architectures. * Security: Understanding of authentication, authorization, tenant isolation, database permissions and secure API integrations. Engineering Mindset * Strong problem-solving and debugging capabilities. * Ability to independently investigate unfamiliar technologies and deliver production-quality implementations. * Clear technical communication and documentation skills. * Strong sense of ownership, engineering discipline and accountability. Preferred / Additional Experience Experience with the following technologies will be considered advantageous: Area Relevant Technologies Data Orchestration Apache Airflow 3, dlt, dbt, Cosmos AI & Agents LangGraph, LangChain, Anthropic Claude, Langfuse Search & Embeddings Amazon OpenSearch, vector databases, Amazon Bedrock AWS Data Services Kinesis, Firehose, S3, EKS Backend FastAPI, REST APIs, streaming/SSE Infrastructure OpenTofu/Terraform, Helm, GitHub Actions, Istio Analytics Lightdash, Metabase, dbt Semantic Layer Agent Interfaces AG-UI, CopilotKit, Next.js Familiarity with finance, fintech, marketing analytics or product analytics is a plus. We value strong engineering fundamentals and proven delivery capabilities over familiarity with every framework listed above. Who We're Looking For We are particularly interested in engineers who have built real systems—not just prototypes—and who enjoy solving complex architectural challenges through hands-on implementation. You should be equally comfortable writing Python code, designing data pipelines, integrating AI agents, reviewing SQL execution security and discussing cloud architecture. If you are passionate about bringing together data engineering and agentic AI to build meaningful enterprise products, we'd like to hear from you. How to Apply Please submit your updated CV, relevant GitHub profile or technical portfolio, and a brief description of a data engineering or AI-powered system you have designed or delivered. Shortlisted candidates will participate in a technical discussion and practical architecture assessment.