CranL High Availability
A hardware failure took part of our platform offline. Here’s what we’re building so it doesn’t happen again.
Hi, we’re the CranL team. This is one of the first posts we’re publishing here, and we decided the most honest place to start is with something that went wrong.
Last week, a hardware failure in our infrastructure affected 10-15% of the services we host and caused several hours of downtime. Below is what broke, how we responded, and the part we care about most: the architectural change we’re rolling out so that a dead server stops being an outage.
What happened
Our monitoring flagged lost connectivity on a small number of servers. There was no graceful degradation and no warning signs in the preceding metrics. The machines simply stopped responding.
The first hour went into diagnosis. Remote access was gone, so we were working through out-of-band management and datacenter remote hands rather than a terminal. Most of that hour went into narrowing the fault down to something a technician could physically act on.
We identified faulty RAM on the primary affected machine. Remote hands swapped the modules and we brought the servers back up.
The fault persisted. The RAM was genuinely bad, but it wasn’t the whole story. Further diagnosis pointed to the mainboard.
Remote hands replaced the board. After the wait for parts and the physical work, the affected servers came back up and have been stable since.
Separately, the other machines caught up in this incident failed for a different reason: power supply units. Two unrelated hardware faults in the same window.
One clarification worth making, because we’d rather be precise than dramatic: the largest of the affected machines was not our largest server. This was not a failure at the core of our platform.
What we took away from it
We held a post-incident review once the dust settled.
The conclusion wasn’t that we’d been careless with hardware. Hardware fails. RAM goes bad, boards go bad, PSUs go bad. That happens at every provider, at every scale, and no procurement policy makes it stop.
The real finding was this: a single component failure should never be able to take a customer’s service offline. The hardware fault wasn’t the incident. The lack of anywhere for those workloads to go was the incident.
So we stopped asking how to avoid bad hardware and started asking how to make bad hardware irrelevant.
Standby instances
We looked at the existing options first. Most high-availability tooling assumes an orchestration layer, a load-balancer tier, or a shared-storage setup that didn’t map cleanly onto how our platform is built today. So we designed something that fits our stack.
We call it a standby instance. If you know the pattern as warm standby or active-passive failover, that’s exactly the family it belongs to. We’re not claiming to have invented failover. What’s new is that we’re building it in at the platform level, so you get it without architecting for it yourself.
A standby instance is a full replica of your running service, but instead of sitting cold on disk as a backup archive, it is live and running, on separate hardware, with no DNS pointing at it. It’s warm, it’s current, and it’s invisible until it’s needed.
How it works
Say you’re running three services on CranL: a frontend, a backend, and a database.
Each one has a standby replica running on a different physical host. Under normal conditions your traffic never touches them. They exist purely to be ready.
If the hardware underneath your primary instances fails, we shift the network configuration to the standbys. In our current implementation that cutover takes under two minutes, which is the time needed to make the routing change on our side.
From the outside, most users won’t notice anything beyond a brief spike in latency. In many cases, neither will your monitoring.
What this doesn’t do
We’d rather set expectations correctly than oversell this.
We are not promising 100% uptime. Anyone who does is either rounding generously or hasn’t had a bad week yet. What we’re promising is that hardware failure stops being a several-hour event and becomes a several-minute one.
Cutover is currently operator-triggered. Detection is automated, but the failover decision still passes through a human. Automating it is on the roadmap, and we’d rather ship the manual version now than promise the automatic one later.
DNS caching is real. Some clients and resolvers will hold a stale record past our TTL. We’ve tuned TTLs down accordingly, but a small tail of traffic may take longer to converge than the two-minute figure above.
Database standbys carry replication lag. For most workloads it’s negligible. For write-heavy ones it isn’t nothing, and we’ll publish per-tier specifics as we roll this out.
What else is changing
The standby work is the headline, but a few less exciting things came out of the review and matter just as much.
We’re keeping on-site spares for the components that failed here, so a swap becomes a same-hour job rather than a same-day one. We’re replacing best-effort ticketing with a defined, time-bound escalation process with our datacenter partner. We’re adding hardware-level telemetry so degrading components surface before they fail outright. And there’s more to change. We will publish them as soon as they are ready to publish. So we can publish real chanes.
Finally
To everyone whose service was affected: we’re sorry. Downtime you didn’t cause and couldn’t fix is genuinely frustrating, and the hours you spent waiting on us is not the experience we want to be known for.
Standby instances start rolling out shortly. If you want to talk through what this means for your specific setup, get in touch and we’ll walk you through it.
We’ll keep writing these, the good launches and the bad weeks both.
The CranL Team