News Aggregator (Google News)

Phase 1 — Clarify & Scope

Interviewer: Design a news aggregator — think Google News or Apple News. What do you want to clarify?

Candidate: Key questions:

1. Are we crawling the open web for news articles, or relying on publisher RSS/Atom feeds?
2. What's the primary feature: a personalized news feed, trending stories, or topic-based browsing?
3. How do we handle duplicate stories (10 sources reporting the same event)?
4. What's the freshness requirement — how quickly should breaking news appear?
5. Scale: users, publishers, articles per day?

Interviewer: Both web crawling and RSS/Atom feeds. All three features: personalized feed, trending, and topic browsing. Deduplicate coverage of the same story (cluster related articles). Breaking news: < 5 minutes from publication to appearing in feeds. 100 million users, 50,000 news sources, 10 million articles per day.

Candidate: Let me size this:

$\text{Article ingestion rate} = \frac{10\text{M articles/day}}{86{,}400} \approx 116 \text{ articles/sec}$

$\text{Storage per article} \approx 5\text{ KB (text only) → 50 GB/day → 18 TB/year}$

$\text{Feed reads} = 100\text{M users} \times 5 \text{ feed opens/day} / 86{,}400 \approx 5{,}787 \text{ reads/sec}$

Interviewer: What are the core requirements?

Candidate:

• Ingest articles from 50K publishers via RSS and web crawl
• Deduplicate: cluster articles covering the same story into a "story cluster"
• Classify articles: topic (politics, sports, tech), entity extraction (people, companies)
• Personalized feed: rank stories by relevance to user's interests and reading history
• Trending: global and per-topic trending stories
• Breaking…