Fix MongoDB M30 Performance Spikes & System Instability Under AI Bot Traffic

We are facing a critical scalability and stability issue in our production system and need an experienced backend/database performance engineer to help us stabilize our architecture under heavy and unpredictable load. Current Stack MongoDB (M30 tier on Atlas) Redis caching layer Next.js frontend and api response (ISR implemented) Cloudflare Static assets caching rules, overall caching rules + Argo routing Indexed and query-optimized MongoDB (Performance Advisor applied) Reduced DB connections and implemented caching strategies Problem Summary Despite multiple optimizations, our system still experiences severe performance degradation during traffic spikes (especially late-night spikes caused by AI bots and automated traffic and don't want to make bots disabled). During these spikes: Website becomes extremely slow or completely unresponsive Outages last around ~5 minutes MongoDB becomes the primary bottleneck despite indexing and query tuning Redis caching helps but does not fully prevent overload We want to avoid upgrading MongoDB beyond M30 due to cost constraints Important Constraints We do NOT want to upgrade MongoDB tier (must stay within M30 limits) We do NOT want to disable or block bot traffic at this stage We cannot fully decouple the architecture into separate databases per module right now We already implemented most standard optimizations (indexes, caching, ISR, CDN, query tuning) What We Need We are looking for someone who can: Analyze our current architecture and pinpoint hidden bottlenecks Design a strategy to handle traffic spikes within M30 limits Improve system resilience without relying on vertical scaling Suggest architectural improvements (queueing, throttling, request shaping, caching layers, read/write separation, etc.) Help prevent MongoDB saturation during burst traffic Improve stability so short traffic spikes do NOT cause downtime Ideal Expert Strong experience with MongoDB performance under load Deep knowledge of caching strategies (Redis, edge caching, etc.) Experience with high-traffic Node.js / Next.js systems Familiar with rate-limiting, queue systems, and traffic shaping Experience handling bot-heavy traffic environments Goal We want a stable system where traffic spikes do not cause downtime, while staying within current infrastructure limits.