Define use cases and constraints
Use cases (in scope)
- Service crawls a list of URLs and generates a reverse index of words to pages, plus static titles and snippets.
- User enters a search term and sees relevant pages with titles and snippets (sketch components only).
- Service has high availability.
Out of scope
Search analytics, personalized results, page rank. Do not use existing systems like Solr or Nutch.
Constraints and assumptions
- Traffic is not evenly distributed.
- Anonymous users only; results must be fast.
- The crawler must not loop forever (a cycle in the link graph causes this).
- 1 billion links, refreshed ~weekly (more often for popular sites) → 4 billion links/month.
- Average page 500 KB; 100 billion searches/month.
Usage
- 2 PB of page content/month, 72 PB in 3 years.
- 1,600 writes/s, 40,000 searches/s.
Sketch the high-level design
- Query entry
The client reaches the web server and Query API.
- Crawl queue
The crawl queue schedules work for the crawler service.
- Documents
The document service and NoSQL store retain fetched content or metadata.
- Index
The reverse index maps search terms to documents for the query path.
Crawl scheduling, document storage, and search indexing have separate jobs.
How to read this model
A crawl-job feeder beside document and reverse-index archives. The object is a teaching analogy. Inspecting a part highlights its role; it does not execute or simulate the software. Cabinet trays are records, workpieces are data, and latches are checks. Unless an explanation states a quantity, compartments and sample workpieces do not denote data counts. Part dimensions are not a quantitative scale.
Define the core components
Crawl a list of URLs
Use a crawled_links table for processed links and their page signatures, and a links_to_crawl set (Redis sorted set ranks pages by popularity).
The Crawler Service loops:
- Takes the top-ranked link.
- Checks
crawled_linksfor a similar signature → if similar, reduces the link's priority (breaks cycles) and continues. - Else, crawls it: queues jobs to the Reverse Index Service and Document Service, generates the signature, removes it from
links_to_crawl, and inserts it intocrawled_links.
Freshness
Add a timestamp to each crawl result. Refresh after a default period (say one week), more often for popular sites. Optionally honor Robots.txt.
Search
- Query API parses the query (remove markup, tokenize, fix typos, normalize).
- Reverse Index Service finds and ranks matching documents.
- Document Service returns titles and snippets.
Scale the design
- Entry
DNS, a load balancer, and web servers precede the Query API.
- Crawl work
Crawl queues feed the crawler service independently of query demand.
- Index and documents
Reverse-index and document services use NoSQL storage for their data paths.
- Hot queries
Memory cache serves suitable repeated queries while underlying volume scales through the data services.
Queues absorb crawl bursts; shards handle document and index volume.
How to read this model
An expanded crawl feeder beside cached queries and index cabinets. The object is a teaching analogy. Inspecting a part highlights its role; it does not execute or simulate the software. Cabinet trays are records, workpieces are data, and latches are checks. Unless an explanation states a quantity, compartments and sample workpieces do not denote data counts. Part dimensions are not a quantitative scale.
- Popular queries served from a memory cache (Redis/Memcached) to spare the reverse index and document services.
- Reverse Index and Document services need heavy sharding and federation.
- The crawler keeps its own DNS lookup, uses connection pooling. Uses UDP for DNS resolution. Standard HTTP crawling uses HTTP/1.1 or HTTP/2 over TCP. HTTP/3 (QUIC) runs over UDP but is not yet standard for crawlers.
- Crawling is bandwidth-heavy: ensure enough bandwidth.