Senior Platform Engineer — Kubernetes HA Migration & Production Operations (Long-Term)

Engagement: Full-time contract (40 hrs/week), long-term — this is a build-then-own role, not a short project Timezone: Must overlap at least 4 hours with both US Eastern and Central European business hours Budget: [hourly range / monthly retainer — fill in] Start: Immediate About us We're an established enterprise Identity Governance and Administration (IGA) software company (~120 people, 20+ years in market) shipping a modern microservices platform: identity provider, policy decision point (AuthZEN), workflow orchestration, identity graph, agent governance, and analytics. The stack is Python/FastAPI services fronted by Traefik, with PostgreSQL (pgvector), Valkey, Redpanda, Neo4j, ClickHouse, OpenBao, and object storage. Today the full platform runs as a large Docker Compose stack. We have a detailed, reviewed, gated operationalization plan to take it to production Kubernetes — AKS for our SaaS, plus a portable (RKE2-class / on-prem) profile for enterprise customers — with genuine three-zone HA, external roots of trust, GitOps delivery, and certified failover drills. Your job is to execute that plan and then run what you built. You will work directly with the CEO/Chief Architect and the development leads. The architecture and sequencing are decided; we need an engineer who can implement precisely, challenge us where reality disagrees with the plan, and operate the result to measurable SLOs. What you'll do Phase 1 — Harden the current stack (immediate): credential rotation, closing exposed data ports, fail-closed health checks, splitting Redis cache vs. coordination planes, image digest pinning, tested backup/restore for Postgres, Neo4j, ClickHouse, and OpenBao. Phase 2 — Build the production platform: Terraform/Bicep for AKS, VNets, private endpoints, Workload Identity, Key Vault, zone-redundant PostgreSQL Flexible Server, Front Door, load balancers A Helm library chart (workload archetypes: web, fenced-singleton pair, scalable worker, migration Job) plus per-service charts, composed via Argo CD ApplicationSets; digest-pinned, signed release manifests Tier-1 data layer in both profiles: two PostgreSQL clusters (Flexible Server / CloudNativePG), two Valkey planes, 3-broker Redpanda (RF=3, per-topic partition plan), OpenBao 3-node Raft with Azure Key Vault / external-KMS auto-unseal and Kubernetes auth, replicated ClickHouse (Altinity operator + Keeper), Neo4j Enterprise 3-core causal cluster Traefik v3 ingress (IngressRoute/Middleware CRDs, ForwardAuth, cert-manager), LDAPS TCP passthrough via dedicated LB Security defaults enforced in charts: restricted Pod Security Admission, non-root, NetworkPolicy segmentation, private endpoints only, isolated node pools for untrusted execution, image-signature admission, SBOM/vuln scanning in CI Observability: kube-prometheus-stack, Loki, Tempo, OpenTelemetry; SLO burn-rate alerting; consumer-lag, lease-age, sealed-vault, cert-expiry alerts Phase 3 — Cutover and certification: per-store migration runbooks (logical replication, snapshot restore, Kafka drain-and-redrive from transactional outboxes) with rollback cutoffs; then scripted HA certification drills — zone loss, store failovers, lease-holder kill under load, rolling upgrades, key rotation under live traffic, clean-environment restores — measured against ratified SLOs. Ongoing: own production operations — on-call posture, capacity, upgrades, backup verification, drill cadence, runbooks. Required skills (be ready to demonstrate all of these) 5+ years production Kubernetes, including at least one migration of a real multi-service platform from single-host/Compose to multi-zone Kubernetes — you can describe what broke AKS specifics: Workload Identity, private clusters, availability zones, Key Vault CSI, managed PostgreSQL — plus non-Azure Kubernetes (RKE2, kubeadm, or similar) so the portable profile isn't an afterthought Terraform (or Bicep) and Helm at the "author the library chart" level, not just consumer level GitOps with Argo CD or Flux, including ApplicationSets and sync-wave-ordered migrations HashiCorp Vault or OpenBao in HA: Raft, auto-unseal, Kubernetes auth, policy design, Transit — and you can explain why static unseal keys on a shared volume are disqualifying Operating at least three of: Kafka/Redpanda (operator, RF, partition/ordering design), CloudNativePG or equivalent Postgres HA, ClickHouse replication with Keeper, Neo4j clustering, Redis/Valkey HA Distributed-systems literacy: you can explain fencing tokens vs. advisory locks, transactional outboxes, idempotent consumers, and why "exactly-once" is a claim you refuse to make Backup/DR discipline: you have restored production systems into clean environments, not just scheduled backup jobs Strong written English; you will write runbooks, prerequisites docs, and migration procedures that other engineers execute Nice to have Python (our services and tooling), Traefik v3, cert-manager, KEDA, Prometheus/Grafana authoring, supply-chain tooling (cosign, SBOM, admission controllers), identity/OAuth/OIDC domain familiarity How we'll evaluate you Short technical interview with the Chief Architect, then a paid pilot task (Gate 1 hardening items on a copy of our stack). We keep engineers who are precise, verify their own work, and say "this part of the plan is wrong, here's why" with evidence. To apply, answer these in your proposal (applications without them are skipped) Describe a Compose-or-VM-to-Kubernetes migration you led: stack size, what broke during cutover, and what you'd do differently. How do you run Vault/OpenBao HA with no static unseal material, and how do applications authenticate to it? A singleton worker writes to a database and must never run twice concurrently. Describe your fencing design and its failure modes (network partition, GC pause, DB failover). Which three of the Tier-1 stores listed above have you operated in production, and at what scale? Your timezone and weekly availability.