Web scraping

## Overview I am looking for an experienced **web scraping / data engineering developer** to build a robust automated system that monitors multiple UK business\-for\-sale websites **every day**, extracts the complete information from every relevant listing, structures the data into defined fields, removes duplicates, tracks changes and feeds the information into an **Excel database and searchable web dashboard**\. This is intended to become a long\-term **business acquisition intelligence system**, not a one\-off scraping project\. ### Websites to scrape The initial sources are: 1. **Rightbiz** https://www\.rightbiz\.co\.uk/search/?more\_category=none§or=businesses&location=uk&sortby=new&noindex=1 2. **Business Sale Report** https://www\.business\-sale\.com/companies\-for\-sale?page=14 3. **BusinessesForSale UK** https://uk\.businessesforsale\.com/uk/search/businesses\-for\-sale 4. **Daltons Business** https://www\.daltonsbusiness\.com 5. **Deal Opportunities** https://www\.dealopportunities\.co\.uk The system must be designed so that additional websites can easily be added later\. --- # What I need the system to do ### 1\. Scrape automatically every day The system should: - Visit each website automatically on a daily schedule\. - Crawl through all relevant listing/search pages\. - Identify new listings\. - Extract existing listings\. - Detect listings that have been updated\. - Detect listings that have been removed/sold/withdrawn where possible\. - Record the date the listing was first discovered\. - Record the date it was last checked\. - Record the date it was last updated\. - Continue monitoring listings rather than simply scraping them once\. The system should be capable of running automatically in the cloud/server without requiring my computer to be switched on\. --- # 2\. Extract the COMPLETE listing For each individual business listing, I want the system to follow the listing through to its individual advert page and extract as much information as is publicly available\. Potential fields should include: ### Basic information - Business name/title - Listing URL - Source website - Date first discovered - Date last checked - Date last updated - Listing status - Business category - Sub\-category - Sector - Location - County - Region - Postcode where available - UK\-wide / relocatable - Leasehold/freehold - Franchise/non\-franchise - New/established business - Seller/broker name - Broker/company name - Contact information where publicly displayed ### Financial information Extract wherever available: - Asking price - Guide price - Offers invited - Turnover - Gross profit - EBITDA - Adjusted EBITDA - Net profit - SDE - Gross margin - Cash - Assets - Net assets - Rent - Business rates - Stock value - Working capital - Any other financial information contained within the advert The scraper must **not attempt to manufacture figures that aren’t provided**\. If a field isn’t available, it should be recorded as: `Not disclosed` rather than guessed\. --- # 3\. Extract the full advert description I want the complete listing description captured and stored\. This is important because I want to subsequently use the information to identify acquisition opportunities based on things such as: - Owner retirement - Ill health - Relocation - Lifestyle change - Partnership breakdown - Lack of succession - Other seller motivation - Property ownership - Freehold premises - Management team - Number of employees - Recurring revenue - Long\-term contracts - Customer concentration - Growth potential - Acquisition potential - Seller finance - Deferred consideration - Negotiable price - Offers invited The original advert text should be retained in a separate field so that no information is lost\. --- # 4\. Intelligent data extraction I don’t just want raw scraping\. I want the system to take the advert and populate structured fields\. For example, if an advert says: “Long-established family-owned engineering company. Current owner is retiring. The company employs 24 people and owns its freehold premises.” The system should extract: **Sector:** Engineering **Employees:** 24 **Owner retirement:** Yes **Freehold property:** Yes **Family owned:** Yes **Seller motivation:** Retirement Similarly, the system should identify acquisition characteristics such as: - Owner retiring - Owner selling due to health - Freehold property - Property included - Management team in place - Recurring revenue - Contracted income - Established business - Long trading history - Seller financing mentioned - Offers invited - Price negotiable - Confidential sale - Relocatable - Leasehold - Franchise - Non\-franchise Please build this using a sensible combination of **rules/regex \+ NLP/AI where appropriate**, rather than relying entirely on an LLM API for every field\. --- # 5\. Duplicate detection This is extremely important\. The same business may appear on multiple websites\. The system should attempt to identify duplicate listings using combinations of: - Business name - Advert title - Location - Phone number - Website - Broker - Asking price - Turnover - Description similarity - Other identifying information I want to be able to see: **One underlying business → multiple source listings** rather than having the same opportunity appear five times\. However, I still want the original source URLs retained\. --- # 6\. Change tracking If a business originally appears at: **£1,500,000** and later changes to: **£1,250,000** the system should retain the history\. For example: |Date |Asking Price| |----------|-----------:| |04/09/2026|£1,500,000 | |18/09/2026|£1,400,000 | |03/10/2026|£1,250,000 | The same should apply to: - Asking price - Turnover - Profit - Description - Status - Location - Other important fields This will allow me to identify businesses where the asking price has been reduced or the advert has been sitting on the market for a long period\. --- # 7\. Excel database I want the underlying data exported/synchronised to **Excel**\. Ideally, the workbook should contain structured sheets such as: ### MASTER DATABASE All businesses\. ### NEW LISTINGS Businesses discovered within the last 24 hours / 7 days\. ### PRICE CHANGES Businesses where the asking price has changed\. ### HIGH PRIORITY Businesses meeting predefined acquisition criteria\. ### RETIREMENT / HEALTH Businesses where seller motivation indicates retirement or health issues\. ### PROPERTY BACKED Businesses where freehold/commercial property appears to be included\. ### FINANCIALS Financial information extracted from adverts\. ### DUPLICATES Potential duplicate listings across different websites\. ### CHANGE HISTORY Historical changes to each listing\. --- # 8\. Searchable web dashboard I also want a **proper usable dashboard**, not just an Excel file\. The dashboard should allow me to search and filter the entire database\. For example: **Show me:** - Businesses in England - EBITDA \ greater than £250,000 - Asking price less than £1m - Freehold property - Owner retiring - 20\+ employees - Non\-franchise - Established \ greater than 10 years Or: **Show me:** - Engineering businesses - Turnover \ greater than £2m - EBITDA \ greater than£300k - Asking price undisclosed - Freehold - Owner retiring Or: **Show me:** - All businesses added in the last 7 days - Sorted by EBITDA - With price reductions highlighted --- # 9\. Dashboard features I would like the dashboard to include: - Global search - Advanced filtering - Sorting - Saved searches - New listing indicator - Price reduction indicator - Source website - Location map if practical - Financial summary - Business category - Seller motivation - Property ownership - Number of employees - EBITDA - Turnover - Asking price - EBITDA multiple - Listing age - Acquisition score Each result should link directly back to the original advert\. --- # 10\. Acquisition scoring system I would like the architecture to allow an acquisition scoring model to be added\. For example, the system could eventually calculate scores based on: - EBITDA - Asking price - Asking price / EBITDA - Turnover - Freehold property - Net asset backing - Seller motivation - Retirement - Ill health - Management independence - Recurring revenue - Seller finance - Price reduction - Time on market - Number of employees - Sector - Location This could ultimately produce: **A\+ / A / B / C / D** acquisition ratings\. The scoring formula must be configurable rather than hard\-coded so that I can change the weighting later\. --- # 11\. Alerts The system should ideally be capable of sending email notifications when: - A new business matching my criteria appears - A price is reduced - EBITDA above a specified threshold appears - A freehold business appears - A retiring owner is identified - An advert mentions ill health - A business matches an A\+ acquisition score - A previously unavailable business becomes active again --- # 12\. Technology I am open to your recommendations\. Potential technologies could include: - Python - Scrapy - Playwright - Selenium where required - PostgreSQL - Supabase - AWS - Azure - Google Cloud - Power BI - Streamlit - React/Next\.js - FastAPI - Pandas - OpenAI/LLM APIs where genuinely useful I am **not committed to any particular technology stack**\. Please recommend the most reliable and cost\-effective architecture\. --- # 13\. Important scraping requirements The websites may have different structures and some may use JavaScript, pagination, rate limiting or anti\-bot measures\. You must therefore be experienced in building **production\-quality scrapers**, not just basic HTML parsers\. The system needs: - Robust error handling - Retry logic - Rate limiting - Logging - Monitoring - Failure notifications - Pagination handling - JavaScript rendering where necessary - Deduplication - Data validation - Database backups - Ability to resume after failure - Modular architecture Please design the system responsibly and respect applicable website terms, robots rules and access restrictions\. Do not use techniques intended to bypass authentication, CAPTCHAs or other access controls\. --- # 14\. Deliverables I expect the successful freelancer to deliver: ### Phase 1 - Technical architecture - Database structure - Scraper for all 5 initial sources ### Phase 2 - Full data extraction - Data cleaning - Deduplication - Structured fields - Change tracking ### Phase 3 - Excel database - Automated daily updates ### Phase 4 - Searchable dashboard - Filtering - Sorting - Business detail pages ### Phase 5 - Acquisition scoring - Alerts - Documentation - Deployment ### Final handover I need: - Full source code - Database - Scraper configuration - Dashboard - Excel output - Deployment instructions - API documentation where applicable - Maintenance instructions - All credentials/accounts to be under my ownership I do **not** want a solution where the freelancer retains ownership of the code or infrastructure\. --- # What I am looking for You should have demonstrable experience with: - Python - Web scraping - Scrapy/Playwright/Selenium - Data engineering - PostgreSQL/SQL - APIs - Excel automation - Dashboard development - Cloud deployment - Data deduplication - NLP/AI data extraction Experience scraping **business\-for\-sale websites, property websites, financial websites, marketplaces or large directories** would be a major advantage\. # Please include in your proposal 1. Examples of similar scraping systems you have built\. 2. Which scraping technology you recommend and why\. 3. Your proposed database architecture\. 4. Your proposed dashboard technology\. 5. How you would handle duplicate businesses appearing on multiple websites\. 6. How you would detect changes to listings\. 7. How you would extract structured information from unstructured descriptions\. 8. How you would deal with JavaScript\-rendered pages\. 9. How you would make the system reliable enough to run every day\. 10. Estimated development time\. 11. Fixed\-price quote or estimated project cost\. 12. Ongoing hosting/API costs you expect\. 13. Whether you can provide ongoing maintenance\. **Please do not apply if your experience is limited to simple one\-page scraping or copying website data into CSV files\.** I am looking for someone who can build this as a **professional, scalable acquisition intelligence platform** that can eventually monitor many more sources and tens of thousands of business listings\.