Your server is struggling. Traffic has grown and pages are slow. You have exactly two options, and choosing between them shapes everything else about your system.
You can get a bigger machine. Or you can get more machines.
That is the entire topic. But the consequences of each choice run deep, and the second one comes with a hidden requirement that trips up most beginners.
Option one: a bigger machine
Vertical scaling means upgrading the server you already have — more CPU cores, more RAM, faster disks. Your app does not change at all. You reboot onto stronger hardware and carry on.
Think of a restaurant with one chef who cannot keep up. Vertical scaling is sending that chef to train, giving them a better stove and sharper knives. Same one chef, now much more capable.
Why people love it: it is almost effortless. No code changes, no new concepts. Your database keeps working exactly as before because there is still only one of it.
Why it eventually fails: you run into a wall, and the wall has three parts.
- There is a biggest machine. Hardware has a maximum. Once you rent the largest instance your cloud provider sells, there is no next step.
- Price grows faster than power. Doubling CPU rarely doubles the cost — it often triples or quadruples it. The last 20% of performance is wildly expensive.
- It is still one machine. This is the serious one. However powerful it is, if it reboots, your entire product is offline. One machine is always a single point of failure.
Option two: more machines
Horizontal scaling means adding more servers and splitting traffic across them. Instead of one chef who is twice as fast, you hire a second chef.
Diagram, as text
- Server 2 CPU, 4GB leads to Server 32 CPU, 256GB
- Load Balancer leads to Server
- Load Balancer leads to Server
- Load Balancer leads to Server
Why it wins long-term:
- No ceiling. Need more capacity? Add another server. Cloud providers will happily rent you thousands.
- Failure stops being fatal. With five servers, losing one costs you 20% of capacity. Users barely notice. This is the real prize — not speed, but survival.
- Cheaper per unit. Ten ordinary machines usually cost less than one enormous one.
Why it is harder: you now need a load balancer to distribute requests, your servers must coordinate, deployments touch many machines, and debugging means asking "which server did that happen on?"
The hidden requirement nobody warns you about
Here is where beginners get hurt. Horizontal scaling only works if your servers are stateless.
Stateless means a server keeps no memory of previous requests. Every request arrives carrying everything needed to handle it, and any server can answer it identically.
Why does this matter? Picture a login system that stores your session in the server's memory:
- You log in. The load balancer sends you to Server A, which remembers "this user is logged in".
- You click a link. The load balancer sends you to Server B.
- Server B has never heard of you. You are thrown back to the login page.
The user experiences random logouts. The cause is that state lives inside one machine, so requests are no longer interchangeable.
The real fix is to move state out of the server into somewhere all servers can reach:
- The load balancer spreads users across three servers.
- Every server reads and writes the same shared session store instead of keeping state locally.
Describe this diagram
Components
- users (Client)
- lb (Load balancer)
- app (Service)
- sessions (Redis)
Connections
- users sends to lb
- lb sends to app
- app sends to sessions
Now every server can answer any request, because the truth lives in one shared place rather than in local memory. This is why you will hear "make your services stateless" repeated endlessly — it is the precondition for scaling out at all.
This is the arithmetic behind that warning. The slider is the share of the work that can genuinely run in parallel; the gap between the two lines is money spent on machines that are waiting.
Notice that even at 95% parallel the ceiling is twentyfold, no matter how many machines you buy. The serial remainder — the lock, the single writer, the one coordinator — sets the limit, so that is where the work goes.
Which do you actually choose?
In practice, not one or the other. You use both, at different times.
| Vertical (scale up) | Horizontal (scale out) | |
|---|---|---|
| Effort | Very low — no code changes | High — needs a load balancer and stateless design |
| Ceiling | Hard limit at the biggest machine | Effectively unlimited |
| Failure | One machine down means total outage | Lose one of many, system survives |
| Cost curve | Rises steeply at the top end | Roughly linear |
| Best for | Databases, early-stage apps, quick relief | Web and application servers, long-term growth |
Why databases are the awkward case
Application servers are easy to scale out — make them stateless, add more. Databases are the opposite: their entire job is holding state.
You cannot simply run five copies of a database and expect them to agree. If two copies accept a write to the same row at the same moment, which one is right? This problem is genuinely hard, and it is why databases are usually scaled vertically for far longer, and why techniques such as replication and sharding exist as separate lessons later.
For now, hold this asymmetry in your head: stateless things scale out easily, stateful things do not.
Check your understanding
Common mistakes
Scaling out before making things stateless. Adding servers to a stateful app produces exactly the random-logout bug above. Statelessness comes first.
Scaling the wrong layer. Teams add app servers while the database is the real bottleneck. Now ten servers queue for the same overloaded database, and you have spent money to make nothing faster. Always find the bottleneck before adding capacity.
Treating "scalable" as a property you can add later. Some designs cannot be scaled out without a rewrite. Storing uploaded files on a server's local disk is the classic one — the moment you add a second server, half your users get "file not found".
Practice
A single-server application is at 85% CPU during peak. The team has budget for either a machine with twice the cores, or three more identical machines behind a load balancer. What questions decide which is right, and when is the 'lazy' answer the correct one?
Write your own answer first, on paper or in a note. Then compare.
Show a hint
One of these options buys time, the other buys headroom. Ask what the app currently keeps in memory.
Show a strong answer
Three questions decide it:
- Is the app stateless? If sessions, uploads or caches live in that server's memory or disk, horizontal scaling produces random logouts and missing files. Statelessness is a prerequisite, not a follow-up.
- How far away is the ceiling? Vertical scaling has a hard wall — the largest machine you can buy — and price rises faster than capability near the top. If doubling cores buys eighteen months, that may be exactly the right eighteen months.
- What is the availability requirement? One machine is one power supply. Three machines survive one failing; the bigger machine does not. If uptime matters, horizontal is the only option that addresses it at all.
When the lazy answer is right: an early-stage product with an uncertain future, a stateful app, and no availability commitment. Upgrading the box is one afternoon; making the app stateless and adding a load balancer, health checks and shared session storage is weeks. Spending those weeks before you know the product survives is a poor trade. Buy the bigger machine, and spend the time it buys removing the state.
Your team scaled the web tier from 2 to 20 servers and peak latency got worse, not better. Give two plausible explanations and how you would tell them apart.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
More clients is not always a gift to the things downstream.
Show a strong answer
Two explanations, both common:
- Connection-pool exhaustion downstream. Each server opens its own pool to the database. Two servers × 20 connections is 40; twenty servers is 400, which may exceed what the database will accept or can schedule. The database spends its time context-switching rather than answering, so every query slows. How to tell: look at database connection count and time spent waiting on connection acquisition, not CPU.
- A shared bottleneck that was previously hidden. A rate-limited third-party API, a single-leader write path, or a lock in a shared cache was comfortable at 2× concurrency and is now contended at 20×. How to tell: measure a single request's latency with the fleet idle. If one request is still fast, the code is fine and the problem is queueing for a shared resource.
The distinguishing move in both cases is the same: measure the queue, not the server. CPU dashboards on the web tier will look healthy in both scenarios, because the servers are waiting, not working.
Key takeaways
- Vertical scaling buys a bigger machine: trivially easy, but has a hard ceiling and leaves you with one point of failure.
- Horizontal scaling adds more machines: no real ceiling and survives failure, but requires a load balancer and coordination.
- Horizontal scaling requires stateless servers. State must move to a shared store, or users get random, baffling behaviour.
- Fault tolerance, not raw speed, is the main prize of scaling out. Losing one of five servers is an inconvenience; losing your only server is an outage.
- Scale up first, out later — and always confirm which layer is actually the bottleneck before adding anything.
Next we look at how to measure whether any of this is working, using the two numbers that describe every system's performance.
Further reading
- Neil Gunther, Guerrilla Capacity Planning (Springer, 2007) — the Universal Scalability Law, which models why adding machines stops helping.
- Gene Amdahl, "Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities" (AFIPS 1967) — the original statement of Amdahl's law.
- John Ousterhout, "Scaling to a Million Users" (Stanford CS course notes) — vertical and horizontal trade-offs from first principles.
- Adam Wiggins, The Twelve-Factor App (2011) — in particular the "Processes" factor, on why statelessness is the precondition for scaling out.
- Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), chapter 1 — scalability as a describable property rather than an adjective.