Python Developer Needed for Data Pipeline Enhancement

We’re DomainsBot, a domain data and analytics company. We run a production pipeline that crawls millions of websites and pulls data from them. Our problem: many modern sites build their content with JavaScript (React, Vue, Angular, other single-page apps), and our crawler doesn’t render it, so those pages come back empty or incomplete. We’re looking for a Python or Go developer with hands-on headless browser experience to design and build JavaScript rendering into the pipeline, reliably and at scale. What you’ll do Look at how our crawler works today and propose the right rendering approach (e.g., Playwright, Puppeteer, chromedp, headless Chrome, or a rendering service) Build a way to tell which pages need rendering, so we only pay for a browser when the page needs one Make rendering run at high volume: concurrency, timeouts, memory and CPU limits, retries, and recovering from browser crashes Handle the hard cases: lazy-loaded content, redirects, cookie/consent walls, infinite scroll, bot detection Feed the rendered output into our existing pipeline without breaking current jobs Measure the result (coverage gained, cost and speed per page) and document the solution What we’re looking for Strong production experience in Python and/or Go Proven experience running headless browsers at scale, not just scraping a few pages A solid understanding of how browsers load and run pages (DOM, network requests, JS execution, when a page counts as “loaded”) Experience with containerized workloads (Docker) and managing resources on cloud infrastructure, ideally AWS Clear written English and the ability to work on your own Nice to have Experience with large-scale crawlers or domain/web data Familiarity with dealing with anti-bot measures and handling them responsibly Experience with queues/orchestration (SQS, Kafka, Celery, n8n or similar) To apply, please include A short description of a project where you ran headless browser rendering at scale: volume, stack, and the biggest problem you solved How you’d decide whether a page needs JS rendering or plain HTML is enough Links to code samples or a GitHub profile Your availability (hours per week) and time zone Please start your proposal with the word “render” so we know you read the full post.