Python & AI Developer (m/w/d)

About the Role We are looking for an experienced Python & AI Developer to transform our existing AI solution into a robust, scalable, and fully automated production pipeline. We already have a working prototype (currently implemented as a Claude Skill) that successfully identifies topic-specific articles on individual publisher websites. The goal is to evolve this prototype into a production-ready architecture capable of processing thousands of domains reliably and cost-effectively. You will have full ownership of the technical design and are free to choose the architecture and technology stack. Your Responsibilities Develop a scalable Python application for automated website analysis. Build a fully unattended pipeline that can be manually triggered. Process 5,000–6,000 domains per month within a defined processing window. Crawl and analyze publisher websites. Identify topic-specific articles based on predefined rules. Extract relevant structured information from matching articles. Automatically export the results to Google Sheets or other spreadsheet formats. Optimize the solution for performance, reliability, scalability, and API costs. Project Goals The completed solution should meet the following requirements: Reliably process 5,000–6,000 domains per month. Run fully automatically after being manually triggered. Achieve a minimum 75% success rate on domains where our labeled test dataset confirms that a valid article exists. Keep total monthly tool and API costs below approximately €1,000 at the target processing volume. Include robust error handling, logging, and recovery mechanisms. Existing Logic Our current prototype already performs the following tasks for individual domains: Discover relevant article URLs. Retrieve and analyze webpage content. Validate content against predefined business rules. Extract structured information. Export the results into a spreadsheet. Your task is to preserve and improve this logic while rebuilding it as a scalable, production-ready infrastructure. Your Profile Strong proficiency in Python. Experience with web crawling, web scraping, and data extraction. Experience designing and building scalable data pipelines. Experience working with LLMs, AI agents, or AI workflows (e.g., Claude, OpenAI, or similar models). Experience integrating third-party APIs and optimizing operational costs. Familiarity with the Google Sheets API or comparable export solutions. Ability to work independently and design scalable software architectures. Nice to Have Experience with any of the following is a plus: Async Python Playwright, Selenium, or Scrapy BeautifulSoup or lxml Docker Cloud platforms (AWS, GCP, or Azure) Queue systems such as Redis or RabbitMQ LangGraph, LangChain, or similar AI orchestration frameworks What We Expect We care more about the outcome than the specific technology stack. The solution should be reliable, scalable, cost-efficient, and easy to maintain while automatically processing several thousand domains each month. You will have significant freedom in choosing the architecture and implementation, provided the final solution meets our objectives for accuracy, scalability, reliability, and cost efficiency.