Overview: We are looking for Strong Middle-Senior Pytjon developer to join the project of UK-based IT company developing software for enterprise-level insurance companies. A part of software line is Python-based web framework, used to extract data from various car rental websites.
We utilize Botasaurus, a Python-based scraping framework. Each provider’s website has a unique structure, so our scraping logic is driven by tailored configurations per provider. Because web interfaces change frequently, our scrapers require ongoing maintenance to ensure robustness against site updates and layout changes.
Technologies we use: * AWS — ECS for container orchestration, Lambda functions as a serverless workflow layer, and S3 for storing screenshots. * PostgreSQL — for managing incoming requests and storing scraped data. * Python — the core language for scraper development, leveraging libraries for HTTP requests, DOM parsing, and automation. * JavaScript — injected into provider pages to manipulate DOM elements, fill out complex forms, simulate user interactions, block specific requests and bypass client-side restrictions. * JSON — configuration and data interchange format enabling flexible scraper behavior per provider. * Docker — containerizing scraper instances to isolate environments and scale horizontally.
Skills and Experience we are looking for: * 5+ years of commercial experience as Python engineer, with at least 1-2 years of experience with web scraping and browser automation. * Strong understanding of HTML, CSS, and JavaScript — able to accurately identify and interact with page elements using selectors (tags, classes, IDs, custom attributes). * Web automation and testing expertise — programmatically controlling browsers or HTTP sessions to fill forms, trigger clicks, wait for dynamic content, and handle navigation. * Human behavior simulation — implementing realistic delays, randomized mouse movements, and interaction patterns to evade bot detection and appear as legitimate users (Botasaurus has these features built-in but is a plus to know how to do these from scratch). * Problem-solving to overcome anti-bot measures — handling pop-ups, captchas, redirects, and other site-specific obstacles through clever workarounds and adaptive logic. * Experience with headless browser automation tools (e.g., Selenium, Playwright) for scraping JavaScript-heavy websites. * Familiarity with proxy management — to reduce blocking risks when using AWS IPs. * Robust error handling and logging — to detect failures early, retry intelligently, and maintain scraper reliability in production. * Data validation and cleaning skills — verifying extracted data accuracy and consistency before storage or further processing. Being able to identify wrong data that needs to be cleaned if it was already stored by a scraper which broke but still collecting data in production.
