Research Engineer - Web Crawlers

• High-velocity: Rapid experimentation, lean autonomous teams, and minimal bureaucracy.
• Impact not job titles: We don’t have job titles. Instead, it’s about the impact you have. No task is above or beneath you.
• AI first: We use AI to move faster with higher-quality results. We do this across the whole company—from engineering to growth to operations.
• Excellence everywhere: Everything we do should match the quality of our AI models.
• Global team: We prioritize your talent, not your location.
• Building and operating large-scale, distributed web crawlers that discover, fetch, and extract data across billions of pages reliably and efficiently.
• Solving hard crawling problems such as content extraction from messy HTML, deduplication at web scale, freshness and recrawl strategies, and politeness and rate-limit handling.
• Designing targeted crawling pipelines that find high-value data sources, including audio, video, and multilingual content, and turn them into clean training-ready datasets.
• Creating tooling and infrastructure that lets researchers request, monitor, and explore newly crawled web data quickly and reliably.
• Hands-on experience building and scaling web crawlers or scraping systems, ideally in support of machine learning training data.
• Strong engineering skills in distributed systems at scale (e.g., Kubernetes, queue-based architectures, or custom pipelines processing billions of documents).
• The capacity to autonomously evaluate the quality, coverage, and compliance of crawled data, and to build the tooling to measure it.
• This role is remote and can be executed globally. If you prefer, you can work from our offices in London, New York, San Francisco, and Warsaw.
• We are an equal opportunity employer and do not discriminate on the basis of race, religion, national origin, gender, sexual orientation, age, veteran status, disability or other legally protected statuses.

None flagged.