Automated Event Data Collection System

# Event Data Automation Developer – API & Web Scraping We are looking for an experienced developer to build an **automated event data collection system**. The system will automatically collect event information from selected websites and APIs, clean and standardize the data, remove duplicates, and deliver the information in a predefined format. The main goal is to **eliminate manual event research and create a reliable daily data collection process**. ### What we need The system should: * Automatically collect event data from selected websites and APIs * Run on a daily schedule * Extract information such as: * Event name * Date and time * Venue * City * Address * Category * Description * Event URL * Image URL * Ticket information * Price, when available * Clean and standardize the collected data * Convert different source formats into **one consistent data structure** * Detect and remove duplicate events * Store the data in a structured database * Provide the cleaned data through an API or another agreed format * Monitor failed sources and data collection errors * Make it easy to add new websites and APIs in the future ### API & Web Scraping This is **not simply a web scraping project**. We want to use: **API → whenever an official API is available** **Web scraping → when an API is not available** **Data normalization → for all sources** **Deduplication → essential** **Scheduled automation → essential** We already have access to the **Ticketmaster API** and are working on integrating additional event data sources. The system should be designed so that adding new sources does not require rebuilding the entire infrastructure. ### Initial Cities The initial system will cover several major cities, including: * Ankara * Istanbul * Izmir * London * Paris * Munich The architecture should be scalable to additional cities and countries. ### Ideal Developer Experience with the following is preferred: * Python * REST APIs * Web scraping * Data extraction and parsing * Database design * Data cleaning and normalization * Duplicate detection * Scheduled jobs / automation * JSON * Cloud-based systems * Building data pipelines Experience working with **event, ticketing, entertainment, travel, or location-based data** would be a strong advantage. ### First Phase We would like to start with a **working prototype using a small number of event sources**. The prototype should demonstrate: 1. Automatic daily data collection 2. Data cleaning and normalization 3. Duplicate detection 4. Structured database storage 5. Output in our required format 6. Basic error monitoring 7. Easy integration of additional sources If the first phase is successful, we expect to continue with the developer to expand the system to additional sources and cities. ### Please include in your proposal Please provide: * Examples of similar API integration, web scraping, or automated data pipeline projects you have built * The technologies you would recommend for this project * Your experience with scheduled/automated data collection * Your approach to duplicate event detection Most importantly, please answer this question: **How would you design a system that collects event data from 10–30 different sources every day while keeping the final data structure consistent, reliable, and free of duplicate events?**