I joined Impala earlier this year as its Head of DevOps. On my first day, the team was just myself, there was no automation to speak of, and the product was already running in front of paying customers. This four part series covers what I did about that.
The application worked fine. The infrastructure worked, in the sense that there was an AWS console, a lot of tribal knowledge, and a few long flat Terraform files that Claude had written, the kind of code where you avoid changing anything because you don't know what else you'll break and no test will tell you. Having some IaC was better than having none, but not by much.
Before Impala I spent six years at Transmit Security, mostly on multi-cloud systems for large enterprise customers. That is where I learned what flat Terraform and tribal knowledge cost you once scale arrives, which is most of why I didn't want to say "we'll figure it out later."
IaC for a full BYOC model
Impala had grown fast by then. It is a B2B AI platform with an unusual deployment model: full BYOC, which for us means more than taking a customer's credentials and dropping a workload into their account.
Inside each customer's own AWS account, we provision a complete isolated cloud environment, VPC, EKS cluster and more platform services. We had dozens of components, a growing list of customer environments, and an expectation that standing up a new one should be fast, frictionless, and optimized.
So we started from scratch, though not with the infrastructure itself so much as with the way we managed it.
The argument of this post is as follows: in BYOC, a module's input variables behave like a public API. Break one and every customer breaks at once. Do it right and standing up a whole environment becomes something anyone on the team can do.
We nearly bought our way out of this
Before writing any new code we looked at the managed platforms: Terraform Cloud, env0, Spacelift, a couple of others. We tried to talk ourselves into choosing one of them, because the pitch is a good one: less work for us, a decent UI, and audit logs we would not have to build ourselves.
But the math got ugly quickly. Every customer gets their own AWS account, VPC and EKS cluster, fully isolated. That means is an independent deployment per customer, plus dev, staging, production and our own management environment. Managed platform pricing usually scales with workspaces or runs, and multiplied across our environment count it added up faster than we wanted.
What surprised me is how early this happens, because the number that matters is not how many environments you have but how many each new customer adds. A flat count means a flat bill and buying vs building almost always wins. A count that tracks your customer list means paying per unit of your own success.
Flexibility was the other concern. These platforms have opinions, and when your topology is unusual you can end up working around the provider instead of with it. Other teams we spoke to said the same.
I had made a similar call before, running environments across AWS, GCP and Azure at once. Managed platforms have a real gravitational pull, and for most teams they are probably the right answer, but our environment count and our topology put us outside that group.
So we went with Terraform and Terragrunt, self hosted, with pipelines running on infrastructure we control. That was more upfront work than buying something would have been, but it was worth it.
Terragrunt specifically, because what we needed was not more Terraform but a way to say the same thing once: a dependency graph between stacks, one backend definition instead of one per environment, and per environment inputs that are genuinely just inputs.
Two layers: a library and a stack
The architecture has two layers, and keeping them genuinely separate matters more than anything else about it.
The module library is generic building blocks: VPCs, EKS clusters, node pools, IAM role patterns, certificate management, DNS, database connectivity. Some we wrote ourselves. Others are community modules that we evaluated, wrapped in a thin compatibility layer and versioned to fit our conventions. The rule for this layer is that modules know nothing about our environments. They take inputs and produce infrastructure, and they make no assumptions about what they are being used to build.
The rule survives because breaking it is visible: a module that mentions Impala, or a customer, or an environment name is in the wrong layer, and that shows up in a diff. Today it holds because we look for it in review. It should be a grep in CI, and the only reason it is not is that we have not needed it yet. Rules enforced by attention have a known failure mode.
The stacks are environment specific compositions of those modules, each one a declarative description of a particular kind of infrastructure. Two matter most. The control plane stack builds Impala's shared platform: our own services, the networking between our environments, and the GitOps engine that manages deployments. The data plane stack is the BYOC one.
Modules know nothing about environments, stacks compose modules, and every customer deployment is a versioned artifact applied to a new account with that customer's variables.
It is worth saying how little a customer environment is, because the interesting engineering lives behind that description rather than in it. A new customer is a directory with one variable file: an identifier, an account, a region, the CIDR range their VPC will occupy, a Kubernetes version, the GPU families they need, and which of our control plane regions their tunnel terminates in. That file is the public API. Adding a field is cheap. Renaming one is a migration across every customer we have.
What’s behind it is heavy enough to need its own post, and that is part 2: an airgapped VPC with no route to the internet, a tunnel to our control plane, and a package the customer runs in their own account rather than one we run for them. The rest of this post is the layer underneath that, which is what made it possible to ship at all.
We are careful about which inputs a module exposes for the same reason. Rename one and every environment calling that module breaks, which in BYOC means hundreds of customer environments at the same time, none of which are ours to fix from the inside. So we add inputs carefully, don't remove them, and don't rename without a migration path.
Modules as an API contract
Turning flat monolithic Terraform into composable versioned modules made everything else possible. It is also what lets you reason about the infrastructure.
Keep the inputs minimal and stable, version them properly, and you can instantiate an environment from a handful of variable files.
Underneath that is a common software engineering idea: separating what an environment is from how it is implemented. A good module lets you change the implementation, a newer Kubernetes version, a different node type, a different networking approach, without any caller editing their variable file.
We built modules for whatever got reused: VPCs, EKS clusters, node group configuration, IAM patterns, GitOps setup, runner infrastructure. Each has a version. A change bumps the version, and consumers pin to a version and upgrade when they choose.
Calling something an API is easy. What makes it one is writing down what you will and will not do to it. Ours are four:
- A new input with a default is a minor version.
- Changing a default is a major version.
- Removing or renaming an input is a major version, and the old name keeps working alongside the new one for a release.
- A module never widens what it manages without a major version, because "we also handle DNS now" is a breaking change dressed as a feature.
The default-change rule is where this departs from how semver is usually applied. In a library, changing a default is minor: the signature is compatible, so callers still build. In infrastructure the thing you ship is not a program, it is a plan. Change a default volume size or instance family and every caller who never set that value gets a resource change, sometimes a replacement, out of a bump they read as safe. Their code is identical and their infrastructure is not. So the test for a breaking change cannot be whether the caller still parses, it has to be whether the caller's plan comes out empty. You are versioning outcomes, not signatures.
The scope rule is the same idea wearing a disguise. A module that starts managing DNS as well as clusters can do it with new inputs that all have defaults, which by the rules above looks like a minor. Any caller already managing those records now has two things claiming one resource, and finds out at apply time.
The deprecation rule is the one that costs something. Terraform gives you very little help deprecating an input, so honouring it means accepting both names inside the module and failing loudly if somebody sets both, which is uglier than deleting a line. It is also the difference between a customer upgrade being a decision and being an incident.
Infrastructure changes go through PR review
The old workflow for an infra change was to pull the repo, run apply from your laptop, and tell your colleagues what you did. CI/CD was something that happened to application code.
Now PRs trigger automated validation, and on merge the pipeline packages the changed stacks and publishes versioned artifacts. Per stack version bumping gives every stack a traceable history.
Opening a PR builds a full package of the changed stacks, versioned as the next stable version with the branch name appended. That is a deployable artifact you can point at a real BYOC environment before the PR merges. Validate end to end, then merge. Merging to main produces the clean stable version. Infra changes go through the same review as application code, which for us has turned out to matter at least as much.
Karpenter and the GPU floor problem
We run AI workloads, which need GPUs, which cost an order of magnitude more than standard compute.
With managed node groups you set a minimum node count. There's a catch here: set it to zero and every GPU workload waits for a fresh node to provision. Set it to one and you pay for an idle GPU node around the clock.
Karpenter handles this better, because GPU node pools can sit at a zero floor. When a workload requests GPU resources, Karpenter provisions the right instance type, the workload runs, and once the node is idle past the consolidation window Karpenter drains and terminates it. It is a fairly ordinary use of Karpenter, and it reduced our monthly bill considerably.
What is not ordinary is what a zero floor costs the person waiting. A GPU node is not a small instance and its cold start is a real wait, so a zero floor spends somebody's minutes to save the company's dollars. This can make sense for batch and training, but there are additional considerations. Part 3 of this series is largely about us determining the right costs.
Where we landed
EKS version upgrades are now a variable bump in a config file, reviewed in a PR, applied through CI. Provisioning an environment takes under 30 minutes.
Two caveats on the 30 minutes: most of it is AWS taking its time over a cluster and its add ons. It also starts after the account exists and its requirements are met, and what is still slow is human: settling on a CIDR range that does not collide with the customer's network, getting a quota raised, waiting on a DNS delegation.
The other thing worth saying is that properly versioned infrastructure stacks turned out to be a prerequisite for the release system we built later. At my previous role, the hardest part of release management was that infrastructure versioning and service versioning were completely disconnected. You could never get a trustworthy snapshot of what was deployed, because the Terraform state and the microservice versions lived in separate worlds with nothing linking them. Getting the IaC layer right first meant infrastructure was already first class when we came to build releases at Impala.
That is what the rest of this series is about.