Build a near-autonomous, low-maintenance web scraper with AI — architecture and best practices

We're building a crawler/scraper that continuously collects listings-style data from many external sites and keeps it updated, storing it in a structured database that a programmatic SEO site is built on top of. You'd own both the crawling and the database structure. Two core goals: build it with AI, and make it run as autonomously as possible once live — minimal manual checks, minimal ongoing coding, fast to fix when it breaks. I want someone with hands-on experience who has actually built this. Be technical and concrete, and explain your reasoning — I'm comparing how different experts think. Please walk me through how you would solve the following: 1. AI in the build — central. Where exactly would you use LLMs (parsing, adapting to layout changes, classifying data) to make it self-healing and low-maintenance, and where would you avoid them as unreliable or too costly? 2. Reliability — keeping it from breaking every time a source changes layout. Self-healing selectors, schema detection, AI-assisted parsing. 3 Architecture — stack, frameworks, infrastructure for pulling structured data from many differently-built source sites. 4. Database — how you'd structure the stored data so it's clean, deduplicated, and ready for a programmatic SEO layer to build pages from. 5. Anti-bot — handling rate limits, IP blocks, CAPTCHAs, and JS-rendered content reliably and responsibly. 6. Data quality — accuracy, deduplication, and detecting when a source breaks or returns bad data. 7. Monitoring & self-running — alerting and error-handling so it's hands-off, and fast to diagnose when something breaks. 8. Infrastructure & cost — realistic hosting, storage and bandwidth needs, rough monthly cost as source volume grows. Cheapest sensible setup that still scales, and where not to cut costs? 9. Replication — how you'd structure it so the whole crawler can be relaunched for a new market/country with minimal rework. 10. Legal — ToS, robots.txt, data-usage considerations. 11. Your track record — what you've built, how autonomous it really ended up, actual maintenance load, what you'd do differently. Priority is lightweight, low-cost, low-maintenance while handling real volume.