Job Description
We are looking for a Web Crawling Engineer to design, develop, and maintain scalable web crawling and data extraction solutions. The role involves collecting structured and unstructured data from public web sources while ensuring data quality, compliance, and performance at scale.
Key Responsibilities
Design, develop, and maintain web crawlers and scrapers for large-scale data collection.
Extract, normalize, and validate data from diverse web sources (HTML, JSON, APIs, dynamic sites).
Monitor crawler performance, data accuracy, and coverage.
Troubleshoot crawling failures and optimize speed, reliability, and scalability.
Collaborate with data, ML, and product teams to define data requirements.
Qualifications
Proven experience in web crawling, scraping, or data extraction.
Strong programming skills in Python.
Basic knowledge of data privacy and compliance.
Experience with distributed crawling systems.
Experience handling dynamic websites and APIs.
Familiarity with HTTP, HTML, DOM, cookies, headers, and proxies.
Understanding of scalability, scheduling, and fault tolerance.
Show more
Show less