Web Crawler — System Design Interview

Phase 1: Clarify, Scope & Constraints

Interviewer: "Design a web crawler. Where do you start?"

Candidate: "Web crawlers serve different purposes — search engine indexing, data mining, link checking, archival (Internet Archive), or price monitoring. The architecture differs significantly based on the goal. What's the primary use case here?"

Interviewer: "Search engine indexing, like Googlebot. We want to crawl and index the entire public web on a continuous basis."

Candidate: "That's the most comprehensive case. Let me define the core user journeys — though for a crawler, 'users' are internal engineering teams and the search index pipeline:

1. Seed and discover: Start from seed URLs, download pages, extract new URLs, add them to the crawl queue.
2. Recrawl: Pages change over time. High-value pages (news sites) recrawl daily; static pages recrawl monthly.
3. Respect crawl policies: Honor robots.txt exclusion rules and Crawl-Delay directives from website owners.
4. Deduplicate: Don't re-crawl the same URL repeatedly; detect and skip near-duplicate content.

What makes this system hard:

1. Scale: The web has hundreds of billions of URLs. Crawling even 10B pages requires massive parallelism.
2. Politeness: We must not overwhelm any individual web server — rate limiting per domain is essential to avoid being blocked or causing harm.
3. URL frontier management: The queue of URLs to crawl is enormous (billions) and must be prioritized (important pages first) and deduplicated.
4. Content deduplication: 30% of web content is near-duplicate (mirrors, syndicated content). Crawling duplicates wastes resources.

WHY: Politeness and URL deduplication are the two most distinguishing…