Senior Python Engineer – Real Estate Listings Scraper & Price-History Pipeline

What you will build - Production-grade scrapers for immobiliare.it and idealista.it, covering Milan (city + metropolitan area) for sale listings, running daily - Incremental crawling: catch new listings every day, re-check active listings for price or content changes, and reliably detect removed or unpublished listings (with removal date) - A listing history model: one record per listing, with versioned snapshots (price, surface, rooms, floor, energy class, description, photos count, advertiser type: private vs agency, agency name) - Cross-portal deduplication: identify the same property listed on both portals and by multiple agencies (address, geolocation, features, image similarity) - Anti-blocking layer: rotating residential proxies, realistic browser fingerprints, rate limiting, retry/backoff, session handling, and graceful handling of commercial anti-bot protection - Monitoring and alerting: daily metrics (pages fetched, listings parsed, block rate, empty-field rate) and alerts when a scraper breaks - Data stored in PostgreSQL (PostGIS a plus), with a clean schema and documented export or API - Geography as configuration: adding a new city must be a config change, not a code change Code standards (mandatory) Our code review is AI-assisted, so the code must be easy to read and verify: - Code lives in our GitHub repository from day one. We own all IP - Small pull requests, each with a clear description of what changed and why. No large single-commit dumps - Typed Python (type hints everywhere), ruff + mypy clean, enforced in CI - pytest suite where parsers are tested against saved HTML/JSON fixtures, so parsing can be verified without hitting the portals - Modular per-portal adapters behind a common interface - README, ARCHITECTURE.md and a runbook (“scraper broke: what to check”), kept up to date with each PR Requirements - 5+ years of Python. Strong with Scrapy and/or Playwright, httpx, asyncio - Proven experience scraping heavily protected sites at scale. Experience with idealista or similar real estate portals is a strong plus: please show examples - Solid data modeling: change-data-capture, versioned records, deduplication - Docker, CI/CD (GitHub Actions), scheduling (Prefect, Airflow or cron-based workers), cloud deployment (AWS/GCP/Hetzner) - Good written English. Overlap with CET working hours Nice to have - Proxy provider experience (Bright Data, Oxylabs, Smartproxy or similar) and cost optimization - Image hashing for duplicate detection - Experience with Italian addresses and geocoding Engagement Phase 1 – Paid PoC: One portal (immobiliare.it), Milan. Acceptance: 7 consecutive daily runs with new-listing capture, price-change detection and removal detection working, tests passing in CI, documentation in place Phase 2 – Full Milan build: second portal, cross-portal deduplication, monitoring. Scope and price agreed after Phase 1 Phase 3 – Ongoing maintenance retainer (monthly), with an agreed response time when a scraper breaks. Expansion to other cities To apply Start your proposal with the word “HISTORY” so we know you read this. Then answer: - Which heavily protected sites have you scraped at scale, and how did you handle the anti-bot protection? - How would you detect that a listing was removed, as opposed to temporarily missing from results? - How would you deduplicate the same apartment listed by 3 different agencies across 2 portals? - Estimated monthly infrastructure and proxy cost to track all Milan sale listings daily on both portals? - Share a link to a public repository (or a code sample) that shows your testing and code-structure standards. Proposals without concrete examples will not be considered.