Define use cases and constraints
Use cases (in scope)
- User connects a financial account.
- Service extracts transactions (daily update), categorizes them (with manual override), and analyzes monthly spending by category.
- Service recommends a budget, allows manual budgets, and notifies when near/over budget.
- Service has high availability.
Out of scope
Additional logging and analytics.
Constraints and assumptions
- Traffic is not evenly distributed; automatic daily updates apply only to users active in the past 30 days.
- Adding/removing accounts is rare; budget notifications need not be instant.
- 10 million users, 100 million budget items, 50,000 sellers, 30 million accounts.
- 5 billion transactions/month, 500 million reads/month → 10:1 write:read (write-heavy).
Usage
- ~50 bytes per transaction → 250 GB/month, 9 TB in 3 years.
- 2,000 transactions/s, 200 reads/s.
Sketch the high-level design
- Entry
The client reaches a web server and the accounts, transaction, category, and notification services.
- Jobs
A queue separates slow account-extraction jobs from immediate requests.
- Storage
SQL stores structured data; object storage holds raw transaction logs.
- Notifications
Notification processing communicates job or account outcomes.
Raw transaction logs and structured application state have different storage jobs.
How to read this model
A service console beside an asynchronous job magazine and financial archive. The object is a teaching analogy. Inspecting a part highlights its role; it does not execute or simulate the software. Cabinet trays are records, workpieces are data, and latches are checks. Unless an explanation states a quantity, compartments and sample workpieces do not denote data counts. Part dimensions are not a quantitative scale.
Define the core components
Connect an account
Accounts API writes to the accounts table (with id, created_at, last_update, account_url, account_login, account_password_encrypted, user_id). Index id, user_id, created_at.
Extract and categorize transactions
- Accounts API places a job on a queue (SQS/RabbitMQ): extraction is slow, so do it asynchronously.
- Transaction Extraction Service pulls from the queue, stores raw logs in an Object Store, categorizes each transaction, updates the
transactionsandmonthly_spendingtables, and notifies the user.
Budget recommendation
Use a generic budget template by income tier (e.g. housing 40%, food 20%, gas 10%). Store only user overrides, not all 100 million budget items. Run MapReduce over raw transaction files to produce the monthly aggregate and reduce database load.
Scale the design
- Entry
DNS, CDN, and a load balancer precede web servers.
- Account work
Accounts, transactions, category, and notification services use asynchronous queue-backed work for slow bank operations.
- Reads
A Read API, memory cache, and SQL read replicas serve suitable reads.
- Durable data
The SQL write primary and object storage hold structured writes and raw transaction material.
Read scaling does not remove the need to coordinate account-extraction work.
How to read this model
An edge console beside a worker magazine and replicated archive. The object is a teaching analogy. Inspecting a part highlights its role; it does not execute or simulate the software. Cabinet trays are records, workpieces are data, and latches are checks. Unless an explanation states a quantity, compartments and sample workpieces do not denote data counts. Part dimensions are not a quantitative scale.
- Add a load balancer, web servers, and master-slave replicas.
- 200 reads/s → serve hot data from a memory cache; SQL read replicas handle misses.
- 2,000 writes/s → heavy; consider sharding, federation, and moving data to NoSQL.
- The
monthly_spendingaggregate has far fewer rows than the raw transactions.
Additional talking points
- Cache placement and update policy: see cache.
- Message queues and back pressure: see asynchronism.