Build a Reusable Product Data Extraction Workflow Using n8n or Python

We are looking for an experienced web scraping and data automation specialist to build a reusable system that extracts complete product information from a large U.S.-based consumer electronics and automotive audio e-commerce website. The website name and exact URLs will be shared privately with shortlisted candidates. ## Project Objective The final solution must allow us to extract and periodically update the website’s complete product catalog. We need to be able to run the workflow ourselves on a weekly basis without requiring the developer’s ongoing involvement. Our preferred solution is an **n8n workflow**, provided n8n can reliably support the required extraction process. We are also open to a hybrid solution in which n8n triggers and manages a Python, Playwright, Puppeteer, Apify, Browserless, or similar scraping service. ## Data to Extract For every available product, the system should capture: * Product title * Full product description * Brand * Model number * SKU or product identifier * Current price * Original or previous price, when available * Availability or stock status * Product category * Product subcategory * Product page URL * All product images in the highest available resolution * Product highlights and key features * Full specifications * Customer reviews * Review rating * Total review count * “What’s in the Box” information * Compatible accessories * Recommended accessories * Related products * Product variations or options, when applicable * Any additional structured product data available on the page Reviews should ideally be captured individually, including: * Reviewer name or display name * Review date * Rating * Review title * Review text * Verified purchase status, when available Accessories and related products should include their names, URLs, prices, SKUs, and relationship to the primary product whenever possible. ## Technical Requirements The workflow must: 1. Discover and process the full product catalog, including products located across categories, subcategories, pagination, or dynamically loaded pages. 2. Avoid duplicate products. 3. Support JavaScript-rendered content when necessary. 4. Include reasonable request delays, rate limiting, retries, and error handling. 5. Resume from the last successful position if the workflow stops or fails. 6. Record failed URLs so they can be retried. 7. Distinguish between new, updated, unchanged, and unavailable products. 8. Be suitable for weekly execution. 9. Allow us to run the workflow manually from n8n. 10. Provide clear execution logs and a summary report after each run. 11. Respect applicable website terms, robots directives, and reasonable access limits. 12. Avoid methods that depend on bypassing authentication, access controls, CAPTCHAs, or other security protections. ## Weekly Update Logic After the initial full catalog extraction, weekly runs should preferably use incremental update logic rather than reprocessing everything unnecessarily. The weekly process should identify: * Newly added products * Price changes * Description or specification changes * Availability changes * Newly added images * Newly added reviews * Newly added accessories * Products that have been removed or discontinued The system should retain historical information where practical, particularly for price and availability changes. ## Preferred Technology Our first preference is: * n8n as the workflow interface and scheduler The workflow may integrate with: * Python * Playwright * Puppeteer * Apify * Browserless * ScrapingBee * Bright Data * Firecrawl * PostgreSQL * Supabase * Google Sheets * Airtable * Amazon S3 or compatible storage Please recommend the most reliable and cost-effective architecture. We do not want a fragile workflow that only works during the initial demonstration. ## Data Output The extracted data should be available in a structured format suitable for importing into another website or product database. Preferred formats include: * CSV * JSON * Excel * PostgreSQL or Supabase database Because each product may contain multiple images, reviews, specifications, and accessories, the database or exported files should use an appropriate relational or structured format. A suggested structure may include: * Products * Product Images * Product Specifications * Product Reviews * Product Accessories * Product Categories * Price History * Scraping Run Logs ## Required Deliverables The selected freelancer must provide: * Complete working n8n workflow * All supporting Python, JavaScript, Playwright, or Puppeteer scripts * Database structure or export format * Configuration instructions * Setup documentation * Weekly execution instructions * Error-handling and retry logic * Duplicate-prevention logic * Incremental update logic * Sample output for validation * One complete initial catalog extraction * A successful test of the weekly update process * Source code and ownership of all custom work * Any required environment variables clearly documented * A short recorded walkthrough showing how to run and maintain the system The workflow should not contain undocumented dependencies, developer-owned accounts, or credentials that would prevent us from operating it independently. ## Qualifications Please apply only if you have demonstrated experience with: * Large-scale e-commerce data extraction * n8n workflow development * Python, Playwright, Puppeteer, or similar browser automation * Structured data extraction * Dynamic and JavaScript-rendered websites * Pagination and category discovery * Incremental scraping and change detection * Database design * Retry logic and failure recovery * Image extraction and storage * Product review extraction Experience building reusable catalog-monitoring systems is strongly preferred. ## Application Questions Please answer the following: 1. Have you completed a similar e-commerce catalog extraction project? Please describe it. 2. Would you build this entirely in n8n or use n8n with an external scraping script or service? Explain why. 3. How would you discover the complete product catalog? 4. How would you handle JavaScript-rendered product information? 5. How would you prevent duplicates? 6. How would you detect changes during weekly runs? 7. How would you structure products, specifications, reviews, images, and accessories? 8. How would you handle failed pages and resume interrupted runs? 9. What third-party services, proxies, databases, or paid tools would be required? 10. What recurring monthly costs should we expect? 11. Please provide examples of relevant n8n or scraping projects. 12. Please confirm that all source code, workflows, and documentation will be transferred to us upon completion. ## Important Please do not submit a generic proposal. Begin your application with the phrase: **“REUSABLE PRODUCT WORKFLOW”** Applications that do not include this phrase or do not answer the questions above may not be reviewed.