IT Systems Engineer / Infrastructure Administrator

About the Project You will be joining Techspace Software Inc. to build and operate the on-premises infrastructure for our client — a multi-tenant AI agent platform spanning six business verticals: a SaaS recruitment platform (PrimeRoster), real estate automation (Slytherin Estates), luxury fleet management (Luxify Elite Fleet), algorithmic trading (TrendCore/Finesse Capital), content intelligence (CIMN), and a suite of standalone business automation agents. The infrastructure is bare-metal, on-prem, with no managed cloud underneath it. The full architecture is already designed and documented across build documents. Your job is to execute that build, then own it. Engagement structure Phase 1 (Contract): Execute the Day 0 → Day 90 build runbook — hardware reconfiguration, Proxmox cluster, pfSense networking, PKI, identity, secrets management, HA layer, Cloudflare edge, observability, and security operations setup. Phase 2 (Retained, performance-based): Ongoing maintenance, patch management, DR verification, IaC evolution, capacity planning, compliance evidence, and on-call for P1 incidents. What You Will Build (Phase 1) Configure and operate a 4-node Proxmox VE 8 cluster (NUC14, EliteDesk G3/G6, MSI GE66) with ZFS local storage, cloud-init VM templates, and Proxmox Backup Server with GCM-256 encryption Build the pfSense network foundation: VLANs 210-215 and per-tenant VLANs 220/470-473, full inter-VLAN firewall rule matrix, egress allowlists per tenant, NAT, DHCP, and Wazuh syslog forwarding Stand up enterprise core services: split-horizon Unbound DNS, chrony NTP, EJBCA offline Root CA + per-spoke step-ca intermediates + ACME auto-renewal for all ~43 certificate profiles, HashiCorp Vault with auto-unseal and AppRole/OIDC auth Build the HA layer: HAProxy + Keepalived VIP failover, Patroni + etcd for Postgres, Redis Sentinel — documented failover test for each Deploy the Cloudflare edge: Zero Trust Tunnel (cloudflared on DMZ VLAN), WAF rules, Access policies, Origin CA — with explicit admin panel isolation verified at 4 independent layers Stand up observability and security: Prometheus + Grafana + Alertmanager, Vector log shipping, Wazuh SIEM with custom detection rules, OpenVAS vulnerability scanner, CIS L1 hardening baked into the base VM template Write the Ansible + Terraform IaC codebase so VMs are reproducible and configuration drift is eliminated Own change management, patch cycles, backup/restore verification, DR plan, and capacity planning from day one What You Will Own Long-Term (Phase 2) Monthly patch cycles across Ubuntu VMs, Proxmox hosts, pfSense, and Docker images Quarterly PBS restore verification and annual full DR drill Wazuh SIEM tuning, incident response (P1: 15-min acknowledge / 4-hour resolve SLAs), post-incident reports Ongoing IaC maintenance as new tenant hosts are procured (Phase 1: 9 new hosts, Phase 2: Dev-PR host) Compliance evidence collection for SOC 2 / GDPR / CCPA controls (Stage 10 control framework) Capacity planning reports and procurement recommendations as agent workloads grow Acting as the escalation point for the Developer and n8n Automation Specialist when infrastructure issues arise Required Skills and Experience Hypervisor and Virtualisation Proxmox VE 8+ — cluster formation (Corosync), ZFS tuning, cloud-init template lifecycle, qm command-line proficiency Proxmox Backup Server — datastore setup, GCM-256 encryption, retention policies, restore procedures VM template design: Ubuntu 24.04 golden image with CIS hardening, node_exporter, chrony, and Wazuh agent baked in Networking pfSense — VLAN creation and interface assignment, firewall rule matrix, NAT, DHCP, pfBlockerNG, config backup and git versioning 802.1Q VLAN trunking, managed switch configuration DNS (Unbound — split-horizon zones, DNSSEC, local overrides), NTP (chrony) Tailscale VPN — subnet route advertising, ACL policies Cloudflare — Zero Trust Tunnels, WAF custom rules, Access policies, Origin CA, rate limiting PKI, Identity and Secrets EJBCA Community Edition — offline Root CA build, key generation, ceremony runbook, CRL publication Smallstep step-ca — per-spoke intermediate CAs, ACME provisioners, policy-based cert issuance, systemd auto-renewal HashiCorp Vault — Raft storage, transit auto-unseal, KV v2, AppRole, OIDC, Vault Agent template injection Trust store distribution across VMs, Proxmox hosts, and Docker containers Google Workspace OIDC integration for Grafana, Proxmox, and Vault admin access High Availability and Disaster Recovery HAProxy configuration (frontends, backends, ACLs, health checks) + Keepalived VRRP Patroni + etcd — Postgres HA cluster with synchronous replication (RPO = 0) Redis Sentinel — primary/replica + 3-node Sentinel quorum Documented failover testing and DR runbooks Security Operations Wazuh SIEM — all-in-one deploy, agent registration, group management, custom detection rules, active response CIS Level 1 hardening (SSH, fail2ban, auditd, AppArmor, sysctl, UFW, unattended-upgrades) OpenVAS/Greenbone vulnerability scanning — authenticated scans, remediation SLA tracking TLS everywhere — Postgres verify-full, Redis TLS, internal service-to-service, mTLS planned for Phase 2 Observability Prometheus — scrape config, alerting rules (14+ rules), 30-day retention Grafana — dashboard import, custom panels, Alertmanager routing to Slack/email Vector log shipping — dual sink (Wazuh + Loki), tenant-tagged structured logs Exporters: node_exporter, postgres_exporter, redis_exporter, haproxy, blackbox, patroni Infrastructure as Code Ansible — inventory design, group_vars, roles (base-hardening, node-exporter, wazuh-agent, docker, step-client, UFW), site/harden/deploy/patch playbooks Terraform — Proxmox provider (telmate), reusable VM modules, state management Git — repository strategy, pfSense XML versioning, CI/CD design (GitHub Actions, Phase 2) Linux and General Ubuntu 24.04 (noble) — systemd, ZFS, TLS/PKI internals, iptables/nftables, strong bash and Python scripting Docker / docker-compose — image registry operation, not application code authorship MinIO S3 — bucket setup, TLS, lifecycle policies, Prometheus integration RabbitMQ — cluster admin, vhost setup, Prometheus plugin, TLS, management UI PgBouncer — connection pooling configuration, transaction mode, per-tenant auth Nice to Have Experience procuring and racking mini-PC server hardware (HP EliteDesk, Intel NUC, MSI-class builds) NVIDIA GPU passthrough (VFIO-PCI, IOMMU group management) — relevant for the CIMN Phase 2 build Cluster shared storage (Ceph or NFS) — current design uses application-level HA; this may come in later phases Compliance frameworks: SOC 2, GDPR, CCPA evidence collection and audit preparation What We Are Not Looking For Important — please read before applying This is NOT a cloud infrastructure role. There is no AWS, Azure, or GCP underneath this. Everything runs on bare-metal Proxmox. This is NOT an entry-level sysadmin position. The environment spans PKI ceremonies, multi-node HA clustering, SIEM operations, and IaC — all at once. This is NOT a role where you will be handed a finished system. Phase 1 is a build from scratch against detailed documentation. You need to be comfortable reading a 140-document build pack and executing against it systematically. This IS a role for a hands-on engineer who takes ownership, writes and follows runbooks, and communicates clearly with a non-technical business owner.