Part 1 was about a versioned module library, stacks composed from it, and an input surface stable enough to treat as an API. This post is what that layer builds, and the fact that with full BYOC you are not the only person with write access to your own infrastructure. The customer owns the account. This one fact decides where state lives, who runs Terraform, and what you are allowed to change.

Most vendors use "bring your own cloud" to mean: give us credentials and we will deploy a workload into your account. Our definition of BYOC is driven entirely by our technical requirements and our customers' security needs. Here is what it actually looks like in practice.

Real BYOC in action

A customer data plane is a fully isolated environment inside the customer's own AWS account, and the Terraform stack covers all of it. There is no partial handoff where they manage the networking and we manage the app.

The VPC is airgapped, with no internet gateway and no route to the public internet. Every AWS service the workloads need, such as container registries, object storage and secrets management, is reached through private VPC endpoints that stay on the AWS backbone. Nothing crosses the public internet to pull an image or read a config file.

On top of that sits an EKS cluster configured for these workloads: the right instance types, RBAC scoped so our automation has the access it needs and nothing more, add ons managed and versioned. Then the platform services, data stores and integrations the customer's workloads depend on.

The piece that makes the airgapped model usable is a network connector: a secure tunnel between the isolated customer environment and our control plane, so the two can communicate without either side touching the public internet. Customer data stays in their account and we can still operate the environment. Getting both at once was the most difficult part of the design.

The stack ships as a versioned artifact. Provisioning a new customer means running that artifact against their account with their variables. Updating means running a newer version. Every environment is pinned to a specific version, so we know what is deployed and can rebuild it from scratch.

The bootstrap paradox

An airgap is a chicken and egg problem that a normal environment does not present. Nothing inside the VPC can reach anything until the private path exists, and the private path is itself Terraform. So the apply is two phased: the first phase talks only to AWS control plane APIs and builds the network, the cluster and the tunnel, and the second runs through the tunnel and installs everything that has to pull an artifact to start.

But this isn’t about the ordering, it is the artifacts. There is no Docker Hub in here, no public Helm chart, no install script from a vendor's CDN. Every third party element we depend on has to exist as something we mirror and version ourselves, which means the airgap is not really a networking decision, it is a supply chain decision, and it arrives as work every time a component publishes a release.

One detail is worth mentioning. A registry pull is two services, not one: the registry hands you a manifest and the layers come out of object storage. An environment missing the object storage path authenticates perfectly, lists tags perfectly, and then hangs on the first byte of the first layer.

Where the state lives

State describes an environment, which means it describes a customer's network, and it has to live somewhere. The easy answer is one bucket in our account with a key per customer, which makes operations simpler and the blast radius worse.

So state lives in the customer's own account, next to the infrastructure it describes and next to the person running the apply. The cost is operational: a backend and a permissions model per account, and a recovery story that has to work in an account we cannot casually look inside. What we get is that nothing about a customer's environment leaves their account, including the file describing it.

Two things are worth noting: state is not a secret store but it behaves like one. And locking no longer needs a separate DynamoDB table on recent Terraform versions, which removes one resource per environment from a design that multiplies everything per environment.

The customer runs the apply

We are not the ones running Terraform in a customer's account. The customer is. The account is theirs, the resources are theirs, the state is theirs, and the apply is theirs. We ship a versioned package and a document, and somebody else types the command.

Which is what turns an input surface into a product surface. Internal configuration gets migrated by whoever wrote it. A product surface is consumed by engineers on somebody else's roadmap, with their own change windows and their own view about when they will take an upgrade. Part 1's versioning rules stop being good practice at that point and become the only reason any of this works.

It also produces the rule we care about most: the package is not to be edited. Not forked, not patched locally, not adjusted slightly to make one thing work. If a customer needs behaviour the stack does not have, we add a flag, disabled by default, and version it, so it exists for everybody. A local patch is the one failure a version bump cannot fix, because from the moment their copy differs from ours we can no longer tell them what a new version will do, and being able to say that is what matters.

Flags are the pressure valve and they are not free. Every flag is a combination we have implicitly agreed to support, and the paths through the stack grow faster than the flags do. We accept it because the alternative is a branch per customer, and a branch per customer is the opposite of a product.

Drift, in this arrangement, is not something we scan for. If somebody changes a resource by hand in their own account, their next plan tells them, in their account, and what reaches us is a support conversation rather than an alert. The people who can change the infrastructure are the people who find out.

Working through documents and not code

The same logic applies to everything in that account we do not control. Service quotas are per account and the account is theirs, so a GPU instance limit is a conversation rather than a config change. Organisational policies can forbid a resource we need. Encryption and logging requirements arrive as constraints rather than choices. So the stack takes what it needs as explicit inputs rather than assuming a clean account, and what the account itself must provide is written down as a document the customer works through before anything is applied.

A document is a weaker mechanism than a check. It has not bitten us yet, but this may happen, if a customer skims the document. Turning that document into code is the most clearly worthwhile thing here that we have not done.

Where we landed

A new customer environment is a directory with one variable file, applied by the customer, into an account we cannot see into, producing an isolated cloud we can nonetheless operate through a tunnel. Provisioning takes under 30 minutes of machine time, and the parts that are still slow are the human ones: agreeing on a CIDR range that does not collide with their network, getting a quota raised, waiting on a DNS delegation.

The thing I would tell anyone starting this is that the hard problems are not the ones that sound hard. The airgap is a solved problem with a long checklist. The tunnel took real design work and then just worked. What keeps costing us thought is that somebody else owns the account, so every decision about defaults, deprecations and flags is a decision about somebody else's change window. That is a product problem wearing infrastructure clothes, and it is the reason part 1 of this blog series focused on version numbers.