My next project (check part 1 and part 2 of this series) was taking a proper look at the AWS bill. The total was not alarming for a company our size, but the breakdown was harder to defend, because most of it was compute that was not doing anything. Dev clusters running at full size overnight, every night, while everyone was asleep.
Part 1 covered the IaC foundation (we also worked on security hardening, but there is no post for that). Because of the foundational work, by the time we got to cost, we had enough visibility to see what we were paying for and why.
A cloud bill is a profiler that runs continuously. It shows you the unused cloud resources you’re paying for. This is different from technical debt: an idle node or a forgotten VPC endpoint means you pay for “technical infra debt” each and every month.
Mapping spend to usage
Before cutting anything we mapped every significant cost against actual usage. Not whether a service was running, which tells you very little, but how much it actually spent handling traffic or doing work.
The control plane workloads came out reasonably utilized, not perfect, but still defensible. The problem was the dev data plane clusters, the isolated Kubernetes clusters we run per environment. They are expensive because they are real clusters, proper EKS with real nodes and the full topology, and they were running around the clock.
Karpenter's pre-allocation settings were more generous than they needed to be. We had set them conservatively early on, when we did not yet trust Karpenter's scale up speed and did not want engineers waiting on nodes. That was reasonable at the time, but it meant permanently paying for a buffer of spare nodes against a delay that had stopped being a problem months earlier.
VPC endpoints were another surprise. They had been added here and there, but the total was higher than we had expected. We trimmed the count to the ones we actually use.
Scaling dev clusters down overnight
The obvious lever, once we said it out loud, was to stop running the dev data plane clusters when nobody is working. All dev node groups now scale down at 23:00 and come back up at 08:00, Sunday through Thursday.
The concept is straightforward and the sequencing is not, because Karpenter manages its own node lifecycle. If you drain the managed node groups and stop there, Karpenter will not necessarily clean up the instances it provisioned, so you end up with nodes that are drained but still running and still billing. You have to terminate the Karpenter managed instances explicitly first and scale down the managed node groups after. Get the order wrong and you either strand workloads or the scale down never finishes.
Stateful components need care as well. Most of our dev workloads are stateless enough that this is not much of an issue, but getting it wrong even in dev means destroying the state an engineer was relying on for the next morning. So the order is first graceful drain, then explicit Karpenter cleanup, and managed node group scale down last.
Dev clusters now run 75 hours a week rather than 168, which took out a little over half of dev compute hours, all of it time we had been paying for and not using.
Karpenter consolidation and spot instances
We also tuned Karpenter's consolidation thresholds, which had been set high for the same cautious reasons as the pre-allocation.
There is a real trade-off in that setting. Consolidate too eagerly and Karpenter drains and repacks nodes more often than it needs to, interrupting pods and making the environment feel unsettled. Consolidate too rarely and you pay for underused nodes. We found a middle setting that brought the average node count down without anyone reporting instability, although it took a few weeks of observation to be confident, since it is not something you can validate in a day.
We moved dev almost entirely onto spot instances at the same time, and dev now runs at roughly 90% spot. The usual objection is what happens when a node gets reclaimed mid-work, and in our experience, with the autoscaler configured properly, workloads reschedule within seconds and engineers rarely notice. Spot runs 60 to 80% cheaper than on demand for comparable instance types, and dev work is the interruptible, non critical kind that spot was designed for. Having run workloads across AWS, GCP and Azure, my sense is that reclaim rates vary enough between clouds that it is worth checking rather than assuming, but for dev on EKS it was an easy decision.
Why we bought the commitments last
Everything in this post so far is about using less. The other half of a cloud bill is paying less for what you do use, which on AWS means Savings Plans and Reserved Instances. Both are worth doing, and the order matters more than either one.
Commitments do not apply to spot, and they do not care whether your clusters are switched off overnight. You promise to spend a certain amount per hour for one or three years, and the discount is what you get in return for that certainty. Buy the commitment first and then spend a quarter cutting your usage, and you have locked in hours you no longer intend to consume. The coverage sits unused, and the mistake is easy to miss, because the bill did still go down.
So we did the usage work first, left the new pattern alone long enough to see where our steady state floor actually settled rather than where we had assumed it would, and only then committed against the part that genuinely runs all the time. A commitment is priced against your current usage, which means it is only worth buying once your current usage is the shape you intend to keep.
The test we didn't mean to build
There was also a benefit we had not planned for. Because the clusters cold start every morning, the schedule doubles as a recurring test of whether the system can cold start at all. Services that break on a cold start usually have a hidden assumption inside them: something cached in memory that should not be, a warm cache taken for granted, a dependency assumed to be ready at boot.
What we accidentally built is a chaos experiment nobody has to schedule, justify or remember, running every morning against every dev environment and funded by the thing it is attached to. Chaos engineering programmes usually die because they compete with a roadmap. This one cannot be deprioritised, because switching it off costs money and somebody would notice.
Two cold start bugs surfaced. One service was caching configuration in memory at startup and never refreshing it, which was invisible while clusters stayed up for weeks and broke on the first cold start, serving stale config until someone restarted it by hand. The other was a startup ordering assumption, where a service expected a dependency to already be healthy when it booted. That held in a long running cluster and failed on a fresh morning start where both came up together. Neither bug was caused by the cost work, and neither would have been found without it.
Both would have found us eventually, but the difference is that we discovered this early.
What broke
The schedule caught engineers out in week one. We had assumed the shutdown window would be self-evident and it was not. Someone lost a late debugging session and someone else had a long test killed part way through. Nobody was upset, but nobody was pleased either. We added a Slack notification 30 minutes before the scale down fires, telling people to finish up or save their state, which we should have shipped alongside the schedule rather than after it.
Two cold start bugs surfaced. One service was caching configuration in memory at startup and never refreshing it, which was invisible while clusters stayed up for weeks and broke on the first cold start, serving stale config until someone restarted it by hand. The other was a startup ordering assumption, where a service expected a dependency to already be healthy when it booted. That held in a long running cluster and failed on a fresh morning start where both came up together. Neither bug was caused by the cost work, and neither would have been found without it.
Where this goes next
A fixed cron is a blunt instrument. It does not know whether someone is running a long integration test at 22:50, whether a deployment is mid-flight, or whether the on-call engineer is in the middle of something. The next version checks for active deployments, long running jobs and recent commit activity before firing, defers by a window if it finds any, and then rechecks. Engineers will also be able to extend the window themselves through a Slack command that grants a time bounded keep alive, logged and auto expiring, without needing anyone from infra.
Where we landed
The work turned out to be worth more than the savings. We now know which workloads are bursty and which are steady, and which components cost more than their apparent complexity would suggest, which is useful when making architectural decisions. The schedule also gave everyone a definite answer about when dev resources exist: up from 08:00 to 23:00, Sunday through Thursday, rather than probably running.
None of this was clever. Idle compute on a schedule is usually the largest single opportunity, autoscaler settings have more cost impact than they look like they should, and VPC endpoints accumulate quietly. The bill will tell you most of it if you sit down and read it.
Next up: part 4, the release system. How we unified versioning and deployment orchestration across four repos and around 40 components into one coherent system, and why we think immutable release manifests are the right abstraction.