Design brief
Design a crawler targeting ten million public-page fetches daily with 100 KB average responses: about 116 requests per second and 1 TB per day, with a fivefold peak. Require URL deduplication, robots-policy checks, fair host scheduling, bounded retries, and recovery after worker failure. Define coverage and revisit priorities rather than treating every discovered URL as equally valuable. State per-origin politeness and response-size budgets.
Draw and explain
In Excalidraw, label discovery, URL identity, frontier ownership, robots state, fetch workers, content storage, and the external-network boundary. Trace discovery through admission, dispatch, completion, and link extraction. Show host eligibility, retry state, lease ownership, and stored-content metadata. Explain how the scheduler shares fleet capacity without allowing one host's backlog to consume its entire allowance.
Stress the design
One host returns 429 while another remains healthy. Trace retry timing and fair progress for both. Separately, a worker pauses after fetching, its lease expires, and a replacement starts. Draw how a late completion interacts with current frontier ownership, including any unavoidable repeated network request.
Review your design
Compare exact deduplication with probabilistic filtering when missed pages matter. Explain URL normalization risks, freshness versus coverage, and ownership handover during rebalancing. Check changing robots policy, redirect resolution, private-address protection, and retry exhaustion. Choose signals for per-host pressure, frontier age, duplicate fetches, and useful content yield.
Capture assumptions, sizing, and the tradeoffs behind your diagram.
Review is your own assessment of the design.