Rework Scrapy price spiders blocked on 12 auto-parts sites

We track competitor prices for auto parts across 12 retailer websites. The scraper is a Scrapy project (about 14 spiders) that has run on one server via cron for roughly two years. In the last six weeks, 7 of the 12 sites started blocking us: 403s, Cloudflare challenges on a couple, and empty product grids on others because the content moved to client-side rendering. Current setup - Python 3.10, Scrapy 2.x, data goes into PostgreSQL - Roughly 180k product pages per day - Single IP, simple user-agent rotation, no proxies What we need - Rotating residential or ISP proxy integration (we have an account with a provider, can switch if you suggest better) - Playwright (scrapy-playwright or equivalent) for the pages that need JS rendering, only where necessary to keep costs down - Smarter request behavior: retry logic, backoff, realistic headers, session handling - Reliable scheduling with run logs, and an alert when a spider's item count drops sharply - Short documentation so our one in-house developer can add a new site Acceptance - At least 95% of target pages scraped successfully on all 12 sites over a 7-day test run - Cost per 1,000 pages reported, so we can compare proxy options Other notes - The code lives in a private Git repo, and you'd get a staging box that mirrors production - Two of the blocked sites also changed their page layout recently, so selectors need a refresh there - Some product pages show prices only after choosing a vehicle (make/model/year), which the current spiders handle poorly We stay on the right side of terms of use for public pricing data only. No logins, no personal data. To confirm you've read the description, please start your proposal with the word "otter".