Portability is not deployability

Most software teams develop in environments they mostly control. You pick the cloud account, the networking model, the identity provider, the storage services, the observability stack, the release process. When something breaks, you can usually go look at the thing that broke and immediately change it.

Customer-managed deployments take most of that away. Your application has to run against infrastructure that some other organization selected, configured, and operates. Your access is limited. Services that show up in an architecture diagram may not actually be reachable from inside the cluster. And the people responsible for the environment are often several contractual layers away from the people who understand how it really works.

The natural instinct is to invest in portability. You cut your reliance on proprietary cloud services, you run important capabilities inside the deployment instead of leaning on external SaaS, and you put storage, identity, and model providers behind interfaces you can swap. For locked-down environments, you make sure the whole thing can run with no path to the public internet.

All of that is necessary. It also only covers the part of the system you control.

Portability is mostly a property of your application. Deployability depends on something else: whether the target environment actually behaves the way you were told it would, whether your team can diagnose it when it doesn’t, and whether anyone owns the failures that fall into the gap between the application and the infrastructure.

I learned that distinction the hard way, leading a deployment into a restricted customer environment. We’d invested heavily in making our application portable, and then the environment handed us a set of problems that application architecture alone was never going to solve.

We had prepared for portability

From fairly early on, we’d been deliberate about minimizing our dependence on any particular environment.

Our application ran on Kubernetes. We hosted our own observability stack instead of relying on an external SaaS. Most of the system could run with no public internet access, and the handful of cloud-specific dependencies sat behind interfaces with alternatives ready to go. We still used managed object storage and a hosted language-model service in our primary environment, but storage could be swapped for a local S3-compatible service, and our model gateway let the rest of the app work with different providers.

Before the customer deployment, we stood up the full cluster in a commercial environment on another cloud provider. We swapped out the remaining provider-specific integrations, fixed the configuration differences we expected, and had the application running in pretty short order.

This was real preparation. We’d moved the system between clouds, stripped out the most obvious assumptions about our home environment, and shown that its major dependencies were at least architecturally replaceable.

It also gave us more confidence than it should have.

The first failure looked like ordinary configuration work

One of the first problems showed up in our monitoring stack.

A service needed to authenticate before it could reach customer-provided blob storage, and the hostname for the authentication endpoint just wouldn’t resolve. The components came from third-party Helm charts whose defaults quietly assumed a commercial cloud environment, and the customer was running in a separate cloud partition with different service domains. After some trial and error, we found the overrides to point those charts at the right endpoints.

This was tedious, but it was exactly the category of problem we’d expected. A chart maintained by someone else contained an assumption about its environment; we were deploying somewhere less common; we found the assumption and supplied the right configuration. Routine.

The correct hostname still didn’t resolve.

A basic nslookup from inside the cluster failed before any of our application code even had a chance to run. That was enough to tell us the immediate problem wasn’t ours — and not nearly enough to get anything working.

We spent the next several days trying to understand why DNS was broken. The cluster looked healthy from the outside. The expected Kubernetes services were there. The DNS pods were running. The network config looked plausible. Nothing in the management interface so much as hinted that the cluster couldn’t perform one of the most basic operations a cluster is supposed to perform. And the behavior was inconsistent, which is always the fun kind: moving a workload sometimes changed the result, and a test that passed from one pod failed from another.

Through node isolation, pod rescheduling, and a slightly unreasonable amount of trial and error, we finally found the pattern. Pods could talk to other pods on the same node, but traffic between nodes was being dropped. The DNS failure was only a symptom — when an application pod sent its lookup to a DNS pod on another node, the request just vanished into the cluster network. Put both pods on the same node and the lookup worked fine.

This was not a failure mode we’d planned to test ourselves. In our defense, “does the network work” sits so far below the waterline of things you normally check that testing for it feels a little like verifying gravity before you get out of bed. You learn. It’s normal to verify that pods in a cluster can access a database sitting outside it, that an identity provider issues the claims you expect, or that a restricted cloud uses different service endpoints. It’s even a plausible thing that you might need to double-check that application pods are able to connect to each other — especially as an isolation step in debugging some more obvious issue. It is a lot less normal to ask whether a managed Kubernetes cluster will let a pod on one node talk to a pod on a different node, specifically. Inter-node networking is part of the platform’s basic contract. Without it, a “cluster” is mostly a group of computers that happen to share a control plane.

For the immediate engagement, we could work around it by scheduling everything onto one sufficiently large node. Our hardware needs for this specific engagement were modest, and engineers already ran most of the system on their laptops during development. The compute wasn’t a core issue. But a single node isn’t highly available, and the broken cluster network wasn’t the only thing that failed us. Some customer-provided cloud services were inaccessible or only half-configured — services we’d planned to use because we’d been told they’d be there. Our application had alternatives, but several of them hadn’t been exercised recently enough to trust under deadline pressure.

The deployment was eventually abandoned. We finished the engagement by having the customer approve access to an environment we ran ourselves.

The software worked. The deployment didn’t.

Ownership is broader than fault

Once a standard DNS lookup failed, we could say with confidence that the immediate problem sat outside our application. That mattered internally — some people reasonably worried we’d designed the product around public cloud services that couldn’t function in a restricted environment, which would’ve been a serious architectural failure. It wasn’t what happened. The environment was failing to provide capabilities it was supposed to provide.

Being right about that did not make the engagement succeed.

The customer wanted the system running. The distinction between an application bug, a cluster-network failure, and an unavailable managed service mattered to us, because each one needed a different fix — but none of those categories changed what the customer saw, which was a product that didn’t work.

We also couldn’t lean on the nominal environment owners to fix it. They had access and some familiarity with the platform, but not the depth to diagnose an unusual networking failure, and the support process never produced an effective escalation once the initial team ran out of ideas. Our own access was constrained, too. Exactly one customer-side operator could run commands, so debugging turned into several engineers and support staff sitting on a video call while one person typed into PowerShell. If you have never watched four highly paid engineers take turns spelling a kubectl command out loud, one flag at a time, over a laggy screen share, I can’t fully convey the specific flavor of despair. It’s a good motivator for doing better.

The operator was diligent and patient. The feedback loop was still miserable. Every iteration meant forming a hypothesis, translating it into commands, dictating those commands, waiting for them to run, reading the output through a screen share, and then deciding what to try next. A small typo could cost another round. Calls ran for hours and produced remarkably little new information.

We’d expected limited access. We hadn’t expected to do deep cluster-network diagnosis through a straw.

The deployment involved several organizations, and none of them had unambiguous ownership of readiness. We owned the application. Other parties owned the environment. In practice, everyone owned a slice of the work and nobody owned making the whole system function — which is how you end up with a system that doesn’t.

This changed how I think about responsibility for customer deployments. Fault still matters: we should be precise when a customer environment is broken, and we shouldn’t quietly accept the blame for every infrastructure defect. But once you’ve committed to a deployment, naming the responsible component is only part of the job. The useful question is what will actually get the customer to a working system.

An early-stage company can build a real advantage here by continuing to work after the clean contractual boundary has stopped being useful. Sometimes that means diagnosing infrastructure you didn’t build, helping another vendor put together an escalation, or showing up with a replacement for a service you were promised. It can’t mean cleaning up every external problem forever. It does mean treating customer success as the objective, rather than treating a defensible explanation of failure as the objective.

We contributed to the failure too

It’d be easy to tell this as a pure story about broken infrastructure. That version leaves out the more useful lessons.

We delayed the deployment because we were still finishing product work for the engagement. We thought our ability to push updated container images might close earlier than it did, so we prioritized last-minute application changes over validating the environment. The feature work felt concrete and urgent — we knew what was left to build and could watch progress as we shipped it. The environmental risk was harder to reason about. We’d already run the cluster in another cloud, and the target was supposed to provide managed Kubernetes; we expected some configuration work, not several days of platform archaeology. Starting late cost us the one resource we needed most once things went sideways: time.

I was also less involved than I should have been. I’d handed the deployment to an experienced engineer and assumed the infrastructure support process would handle problems in the environment. Neither assumption was crazy on its own, but together they left too much ambiguity about who would drive the work once the normal process fell apart.

And we’d confused architectural substitutability with operational readiness. Our storage interface could support a local service, but we didn’t have that service packaged and tested as a drop-in. Our model gateway let us switch providers, but we didn’t have a self-contained model deployment ready to push if the customer endpoint was down. An abstraction makes a fallback possible. It doesn’t make the fallback ready. The architecture gave us options; we just hadn’t done the work to make those options cheap under pressure.

Readiness has to be demonstrated from inside the environment

The biggest change after this deployment wasn’t another portability layer. It was a different standard for believing an environment was actually ready.

Before, we accepted statements like:

• The Kubernetes cluster is operational.
• Blob storage is available.
• The environment can authenticate to the required cloud services.
• The container registry is reachable.
• The provided model endpoint is ready to use.

These weren’t irrational things to believe. They came from the organizations responsible for supplying the services, and several of the resources were right there in management portals and architecture diagrams. But not one of them had been demonstrated from the execution context our application would actually run in.

We now treat statements like those as hypotheses.

The first workload we deploy into a new environment is a diagnostic pod. It tests the infrastructure in layers, so a failure tells us exactly which boundary broke. For blob storage, it doesn’t stop at resolving a hostname — it checks network reachability, authenticates, makes a real API call, writes an object, reads it back, and deletes it. For a model endpoint, it sends a representative request and checks that a usable response comes back. It tests DNS across nodes, service-to-service communication, and the other capabilities the application will lean on.

The goal isn’t to confirm that the environment resembles its architecture diagram. It’s to confirm that the application can perform the operations it needs. That distinction seems obvious written down. It was much easier to miss when each prerequisite had its own owner and every owner reported that their part was done.

A model for deployment readiness

I now split deployment readiness into three related concerns.

Application portability is whether the software can run across different infrastructure choices without a fundamental redesign.

Environmental compatibility is whether the target environment actually provides the networking, identity, storage, compute, and other capabilities the application expects.

Operational deployability is whether the deployment team can validate those capabilities, isolate failures, make changes, escalate problems, and recover when an assumption turns out to be wrong.

We’d invested heavily in the first, prepared for some predictable differences in the second, and left far too much of the third implicit. The customer, of course, experiences all three as one system. A failed DNS lookup, a dead model endpoint, and a bug in our backend all produce the same immediate result: the product doesn’t work. We need to tell them apart internally. We can’t expect that distinction to carry the deployment for us.

That model changed our process in four concrete ways.

Start validating the environment early

A deployment plan should have explicit milestones for gaining access, validating infrastructure, installing the application, connecting data, and testing the user workflow. Environment validation can’t wait until the product work is done — it has to start early enough that an unexpected infrastructure problem is expensive rather than fatal. In our case, starting sooner wouldn’t have fixed the broken cluster network, but it would’ve given us time to escalate, build workarounds, and decide whether the target environment was even viable.

Test capabilities, not resources

A resource existing in a portal is weak evidence that your application can use it. Tests should run from the same network and workload context as the product, and they should exercise the complete operation instead of stopping at the first layer that happens to work. Resolving a hostname doesn’t prove storage works. Opening a TCP connection doesn’t prove authentication works. Getting a token doesn’t prove the application can write and read back an object. The test has to reach the behavior the product actually depends on.

Decide fallback paths in advance

For each capability that could block a deployment, we now try to decide up front whether it’s:

1. A hard prerequisite the environment must provide.
2. Something the application can run without, in a degraded mode.
3. Something we’ll bring a local replacement for.

If customer-provided blob storage is unusable, we want a tested local object store ready to deploy. If a hosted model endpoint is down, we want a packaged model that can run inside the cluster. We’re still doing some of the engineering to make these fallbacks routine, but the intended operating model is at least explicit now.

There are limits. Shipping a replacement for every managed service carries real maintenance and security costs, and some infrastructure failures just can’t be absorbed by the application team. You shouldn’t try to build an entire cloud platform inside every customer environment on the off chance the supplied one is broken. The important step is deciding the boundary ahead of time: what you can replace, what you can temporarily live without, and what has to block the deployment.

Assign a deployment owner

Someone needs to own the deployment outcome across the application and infrastructure boundary. That person may not have the access or authority to fix every component, but they should own the plan, track the readiness milestones, surface blocked work, drive escalation, and make sure ambiguous responsibility doesn’t quietly turn into nobody doing anything.

This whole experience made me a lot more sympathetic to vendors that hand customers exhaustive readiness requirements. Those documents can be burdensome, and they often feel like the vendor shoving integration work onto you. They’re also usually scar tissue from deployments where basic assumptions turned out to be false. I used to read those documents as bureaucratic self-defense. Now I read them as a list of things that have personally hurt someone. The better version pairs precise requirements with a team that’s genuinely willing to help you meet them. A requirements document should clarify the work, not become an excuse to stop the moment reality diverges from it.

Own the deployment outcome

After this, I adopted a more aggressive operating assumption: act as though we’re on our own until the environment proves otherwise.

That doesn’t mean treating customers or other vendors as adversaries. Most failures like this aren’t deliberate. Infrastructure promises pass through contracts, architecture diagrams, status meetings, and several layers of organizations, and by the time a promise reaches the deployment team, the person repeating it may have no direct evidence behind it. Good intentions are common. A functioning cluster network is still better.

An ownership mindset means assuming the environment may contain failures nobody else has found, that the people officially responsible may not know how to fix them, and that your own preparation has to account for both. It also means being honest about your own contribution. We started too late, hadn’t exercised every fallback, and let readiness ownership stay nebulous. I didn’t engage deeply enough until the deployment was already in trouble. None of those choices caused the broken cluster network — but they made the failure much harder to recover from.

You can’t control every part of a customer environment, and there will be deployments where the underlying problems are just beyond what you can reasonably fix. You can still start earlier, test the real capabilities, show up with options, and take responsibility for driving the system toward something that works.

Portability makes it possible to run the software somewhere else. Deployability is the work of making it run where the customer actually is.