End-to-End Web Crawling, Scraping & Data Extraction Pipeline Developer

I run an automotive business and need to track real-time car market trends. I am looking for a skilled Web Scraping & Data Engineering Developer to build an automated end-to-end data pipeline. The goal of this project is to collect publicly available automotive data, including car categories, listing prices, customer reviews, and market news, and export clean, formatted datasets (Excel / CSV) for business analysis. Project Scope & Milestones Trial Task (1–2 Days): - Recommend 2–3 suitable automotive source websites covering pricing, reviews, categories, and industry news. - Build a lightweight, functional end-to-end pipeline that collects publicly available data from these sites and extracts structured sample data. - Automatically clean and format the output into a structured .xlsx (Excel) file. Full Project (Long-Term): - Expand pipeline scale to monitor multiple platforms concurrently. - Implement scheduled runs and reliable data collection processes. - Maintain and optimize data delivery pipelines long-term. Technical Requirements - Strong experience in Python (Scrapy, Playwright, BeautifulSoup, Pandas) or Node.js (Crawlee, Puppeteer). - Proven track record handling dynamic JavaScript pages, pagination, and common data collection challenges. - Clean output formatting directly to Excel (.xlsx) or CSV. - Ability to advise on the best technical architecture for performance and stability.