The Backend Fails I Never Thought About Until Now
I've been bouncing back and forth between theory and actually shipping stuff for a while now. Learn a concept, go build something with it, get stuck, go learn more, repeat. After finishing 2-3 projects and getting close to the end of the Backend from First Principles playlist on YouTube (great playlist btw), I decided it was finally time to sit down and finish it properly — and also share notes this time instead of watching and nodding along like I understood everything. (Not always, though)
This part of the playlist was all about the stuff that only shows up once real users start hitting your app and the internet decides to be the internet: errors you didn't plan for, servers that need to restart mid-transaction, processes that die without cleaning up after themselves. None of this shows up when you're building your fifth CRUD API on localhost. It shows up in production, usually at the worst time.
Here's what stuck with me.
Not all errors are the same
I used to lump "errors" into one bucket in my head. Turns out backend engineers split them into categories, and each one needs a different response:
Logic errors — the scary ones, honestly. The app doesn't crash, nothing looks wrong, but the business result is just... wrong. Like applying a discount twice. Usually comes from misunderstanding requirements or missing an edge case, not from bad syntax.
Database errors — can't reach the DB, connection pool's exhausted, or you just violated a unique/foreign key constraint.
External service errors — anything you don't control: AWS S3, an email provider, an auth service. Hit them too often and you'll eat a 429, which is why exponential backoff exists.
Input validation errors — the easy ones. Bad data from the user, catch it early, throw a 400, move on.
Configuration errors — missing or wrong env variables when you move between dev, staging, and prod. The note that stuck with me here: it's way better for your server to refuse to start than to boot up and randomly throw 500s later because it's missing an API key.
Splitting errors like this changes how you think about handling them. A validation error and a database constraint violation are not the same problem wearing different masks — they need different responses and different urgency.
Catching problems before they become problems
The best error handling doesn't happen in a catch block. It happens before the error causes damage.
Health checks were the part that reframed things for me. I always assumed a health check just meant "does the server respond with 200 OK." Apparently that's barely a health check at all — you actually want to check functionality: run a test query against the DB, maybe send a test email, actually confirm the things your app depends on are alive.
Same idea with monitoring. Tracking error rates is obvious, but performance metrics matter more than I expected — degrading performance is often the early warning sign before a full system failure, way before anything technically "errors out."
Recoverable vs non-recoverable, and bubbling up
This is the distinction I keep coming back to: some errors you retry (network timeout, DB pool limit — handle automatically with retries and backoff, but don't hammer an already-stressed system), and some you contain (non-recoverable errors get graceful degradation — disable the non-essential feature, keep the core app alive).
And instead of writing try/catch everywhere, low-level errors get caught and bubbled up to higher-level processes that actually have the business context to decide what to do with them. Which leads to the pattern that felt like the biggest "that's smart" moment of this whole section —
The Global Error Handler
Instead of scattering try/catch logic across your codebase, you bubble every error up to a single centralized middleware sitting at the end of the request cycle. That middleware inspects what came in and decides the correct HTTP response:
Validation error → 400 Bad Request
Unique constraint DB error → 409-ish territory
"No rows" DB error → 404 Not Found
The advantage isn't just less repeated code — it's that if a developer forgets to handle some edge case, the system still catches it gracefully instead of leaking a stack trace to the user. That's the part I liked. It's not just DRY for the sake of DRY, it's a safety net for the stuff you didn't think of. I actually used this in a couple of my recent projects — not the cleanest implementation, but I could see exactly why it's necessary.
Security lives in your error messages too
This is the section that made me realize how many small habits I've probably been getting wrong without knowing it. Error messages and logs are a primary target for anyone trying to poke at your backend.
A few rules that are now marked into my brain:
Never leak the raw database error back to the user. Generic message, always.
Prevent enumeration attacks — if someone types a wrong email into a login form, never say "email doesn't exist." Say something generic like "invalid username or password." Otherwise you've just handed an attacker a way to check which emails are registered.
Sanitize your logs. No API keys, passwords, card numbers, or emails sitting around in log files.
Graceful shutdown: the part I never thought about
This was genuinely new territory for me. Picture an e-commerce app — a user is mid-transaction, and the server needs to restart for a deployment. If the server just dies abruptly, that transaction could get lost or the DB could end up corrupted. Graceful shutdown is the fix: ask the backend to finish what it's doing, clean up, and then close.
Every backend process runs on the OS like any other process — starts, executes, terminates. When the OS wants it to stop, it talks to it through signals, and your app registers handlers that listen for specific ones:
SIGTERM — the polite request. Sent by process managers, deployment tools, Kubernetes, PM2, systemd. Something like "Please finish up."
SIGINT — also polite, but user-initiated. This is just Ctrl+C on a running terminal process.
SIGKILL — the nuclear option. Instant, cannot be caught or ignored by your handlers. If your app ignores SIGTERM/SIGINT for too long, the OS eventually forces a SIGKILL anyway.
Most production systems set a timeout — 30 to 60 seconds — and if your app hasn't finished within that window, it gets force-stopped regardless.
The actual shutdown happens in two steps:
Connection draining — stop accepting new requests, but let in-flight ones finish and return their response to the client.
Resource cleanup — commit or roll back any open DB transactions, close connections to the pool, release file system access, stop background jobs. And resources get cleaned up in the reverse order they were acquired, which makes sense once you think about it but wasn't something I'd have guessed on my own.
The good news: modern frameworks mostly give you this out of the box. You rarely need to hand-roll the signal handling yourself. But knowing what's happening underneath makes the framework docs make a lot more sense.
That's this batch of notes. Next up — scaling and performance, which honestly deserves its own post.
If you've dealt with a nasty production incident that came down to one of these — a missed SIGKILL, a leaked DB error, an unhandled edge case — I'd genuinely like to hear it. Drop it in the comments.
