Here is a test you can run on your own platform, and it costs nothing.
Walk over to whoever knows the most about production and ask them what version is running. Then time the answer.
If it comes back in five seconds with a file you can open, you are operating a system. If it comes back as "I think it's 1.1.0, let me check a couple of things," you are not operating it. You are running it and hoping it’s done right, which usually holds right up until a customer reports a bug you cannot reproduce.
When I joined Impala we were still guessing, and this post is about how we stopped.
What we needed
We needed "this customer is running Impala v1.1.0" to mean something precise, rather than that their service is at one version, their Terraform stack at another and their charts at whatever GitOps synced last.
Breaking changes also need an upgrade floor, so clusters below version X cannot receive version Y until they migrate, and enforcing that requires a version concept every component participates in. Rollback has to be deterministic, which means a known state is a verified set of artifacts rather than the set you believe was deployed.
Doing this took a week of design and a few weeks of implementation. On a mature platform it is a multi-quarter migration, which is why it’s better to do this early.
The architecture
The system is hub and spoke across several repos. A release hub owns the release manifests, the environment state and the deploy automation, and the other repos, application services, Helm charts and Terraform, are the spokes that build artifacts and notify the hub. Coupling runs one way: source repos know about the hub, not the reverse.
Impala is BYOC: a control plane we run, and a data plane that runs inside each customer's own cloud account. The core primitive is an immutable release manifest, one versioned file pinning every artifact for both planes under a single semver: Terraform packages, Docker image tags and Helm chart versions for the control plane, and the same three for the data plane. Anything that runs on our clusters or a customer's is in it.
Most systems let current state be emergent, the sum of whatever was deployed last. Making it a document you can read without cluster access is the point, and immutability is what makes it worth reading. An editable manifest means a version number is no longer a reliable pointer to a state of the world, so you cannot look up what ran during an incident three months ago and trust the answer.
Environment state files track what is actually deployed to dev, staging and production. State files are mutable and manifests are not: the manifest says what a release contains, the state file says what is deployed, and between them you can answer what is running without touching a cluster. Only the deploy automation commits state files.
A JSON Schema and a semantic validation layer run on every PR that touches the release machinery. The semantic layer catches what a schema cannot, such as a manifest version lower than its predecessor, which sounds too obvious to check and has caught real regressions.
What a version number tells you
A dev release is X.Y.Z-dev.N (e.g. 1.1.1-dev.4). Staging mints X.Y.Z-rc.N (e.g. 1.1.1-rc.1) from it, production strips the suffix to X.Y.Z (e.g. 1.1.1). Same content, restamped, so 1.1.1-rc.1 tells you it cleared dev and is sitting in staging, and 1.1.1 tells you it is in production, without opening anything.
Promotion to production copies the candidate's pins verbatim, so production ships what staging validated rather than the same versions recomposed. Production is also the one deploy that does not fire automatically, and that took a deliberate removal: the workflow used to trigger on the clean stable tag, so merging a release manifest shipped it to production. Previews, promotions and dev deploys still run themselves. Production waits for a human to name the version. Each release also tags all repos at their current HEAD, which is what makes a hotfix possible: check out impala-v1.1.1 in whichever repo holds the bug and you are building off the exact tree production is running.
Preview manifests
The hard design problem was isolation: two engineers with PRs in flight against different components, each needing to test against dev without clobbering the other.
When a PR is pushed to any source repo, CI builds the changed components at a preview-tagged-version and notifies the hub, which upserts a preview manifest: a complete, valid manifest where changed components carry preview tags and everything else inherits from the latest stable baseline.
The ticket ID holds it together. A PR touching application services and a PR touching Helm charts for the same ticket update the same preview manifest, so anything that needs to know what is in a preview reads a single file. Each ticket gets an isolated snapshot of the world with that change applied, which is what made parallel work possible.
That is the clean version. Before it we spent a while on one question: when a stable release lands, should every live preview move with it? First we rebased every live preview automatically, with a freeze-baseline label to opt out, so your preview changed under you mid-test and a failure might be yours or might be something swept in ten minutes earlier. Now previews freeze by default and you opt in per PR with an auto-refresh-preview label. If automation is going to change something an engineer is actively working on, they should have to ask for it rather than remember to turn it off.
The promote flow
When a PR merges, the promote flow reads the preview manifest, works out what changed, composes a new release manifest and opens a PR. Auto-merge takes it from there, the merge triggers tagging across all repos, and the tag triggers the dev deploy.
The first version read all component versions fresh from the registry on each promotion. One merged change to a data plane service produced a release that also bumped Terraform stacks, control plane images and any chart that had drifted in the registry since the last release, so a one-line-bug-fix arrived as a twenty-component-diff nobody could read.
The fix was to treat the manifest rather than the registry as the source of truth. The flow reads the state file to find which release is deployed and uses that release's manifest as its baseline, so only components in the preview diff change. The registry records what was built and the state files record what was deployed, and those two diverge constantly. Reading the live cluster is an escape hatch for a drifted environment, not the normal path.
The stale preview is a related problem. A developer branches early in the week, a minor version is cut mid-week, and promoting their preview would mint a version below what is deployed. That now fails loudly, with an error naming both streams, the version that would have been minted and the command to fix it. For automated systems the error message is the documentation, because nobody is watching when it fires.
A manifest is a promise that the artifacts exist
A pin asserts that an artifact exists and can be pulled, and the manifest cannot check itself. Because it is immutable a wrong pin is permanent: you cannot fix it, only mint a new version and hope nothing deploys the broken one. That is immutability's bill, and it means every check has to move ahead of the merge.
So a gate walks every image and chart in a manifest and confirms it exists. At promotion time it fails fast, before anything is written and before the release PR opens. At deploy time it heals instead, copying the artifact into the environment's registries in every region a data plane runs in, because by then the manifest is correct and the artifact is merely late.
The manifest as a product interface
A customer upgrade is one call now. A version goes in, the API service fetches that manifest, checks the upgrade floor, confirms every image exists, resolves each tag and rolls out the upgrade. Production releases also write the version into the collection behind the dashboard's upgrade banner, so a customer clicking upgrade and an engineer debugging at 2am are reading the same file. We built it for our own automation and it became a product interface.
What broke
The stale release PR collision. A monitor that merged release PRs automatically once picked a five-day-old PR over the current one and rolled dev back to an old version. We recovered by minting a new version on top, and now close stale release PRs explicitly before opening a new one.
The chart version override. Our reverse-manifest-tool read control plane chart versions correctly from live GitOps state, but a subset of apps used a different source type and the lookup fell through silently. A later composition step then overwrote the correctly resolved versions with stale ones from the registry, so the baseline was right and the composition reset it. We found it in a routine diff review, which is the argument for posting manifest diffs to Slack on every deploy. Fixing it needed a reconciliation release composed directly from live cluster state, the one exception we have made to never hand-authoring a manifest.
The GitOps revision mismatch. Our deploy workflow passed a release tag from the hub repo as the git revision for the Helm charts repo, where that tag does not exist, and all ten control plane applications went into unknown sync status at once. The revision should always have been main, which we knew and missed.
The guard that compared a release to itself. A check meant to stop a deploy moving a component backwards was anchored on the newest minted manifest, so a regressed manifest passed by comparing against itself. Anchoring on the deployed state instead, flooring each component at the higher of what is running and the latest stable, closed it.
What changed
The manifest turned into a communication tool, which we had not designed for. The diff between two releases reads as a plain list of what moved, and it has become how the team talks about deploys: product asks what is in a release, customer success asks what version a customer is on, and an engineer asks what changed before an alert fired. Someone who does not know what a Helm chart is can read a diff and see that the API service version changed.
Using the ticket ID as the namespace for previews, branches, PRs, notifications and storage paths worked better than expected, since anyone can follow a change from PR to deployment by searching one identifier.
The static clusters list has prevented several near-incidents. An explicit list of the clusters automation may touch means a new cluster is not upgraded until someone adds it, and upgrading everything by default is the wrong default for infrastructure a customer owns.
Most of the value is legibility rather than automation. The manifest, the state files and the diff on every deploy are not incident response tools so much as what makes incident response possible, and they are much easier to build in advance.
Where things stand
The release system is live and customers are being onboarded through it. The manifest is the version of record, deploys are deterministic, and when something breaks we can read the release file and know what the world looked like at the time. Rolling back to a version is trustworthy because its artifacts cannot have changed.
Most of the hard parts were not technical. The old shortcuts were load-bearing, so removing them meant first building something that could carry the weight.
The goal was never faster deploys, it was boring ones: deploys nobody calls a meeting about, that happen and produce a Slack notification and nothing else. The measure worth watching is how long ago the last deploy was something anyone had to pay attention to.
Links to all parts of "Building DevOps at Impala from the ground up":
IaC You Can Reason About
What Full BYOC Actually Takes
Right-Sizing The Cloud Bill