Senior Software Engineer, Backend (Python)
GovTech company, United States · remote
- Sole engineer designing and operating web scraping systems that process millions of pages a day, including Airbnb, VRBO and Booking, for a compliance product used by US state and local agencies.
- Designed the ingestion and normalisation pipelines for an internal AI assistant, using tool-based and agent-to-agent orchestration patterns.
- Architected the background job and queue systems that carry high-concurrency workloads across several scraping pipelines, with retry logic, failure handling and observability.
- Company-wide subject-matter expert for crawling and browser automation, working with Data Science, Product, QA and Support.