RPC · notes on building things

Productionizing research is a boundary problem

Phillip C. · ·19 min read

A research prototype and a production system can share most of their code, but they’re ultimately built to answer two different questions.

The prototype asks:

Can we make this work?

The production system asks:

Can other people depend on this continuing to work?

Those two questions pull toward two very different sets of engineering decisions.

A prototype should make it cheap to try ideas. It can assume a knowledgeable operator, a controlled environment, a familiar dataset, and the occasional bit of manual intervention. It might only need to run successfully a handful of times before the researcher changes it again.

A production system has to keep working when the inputs are surprising, the infrastructure is flaky, the original author is on vacation, and several other systems depend on its behavior — and it has to stay understandable while both the product and the underlying research keep moving.

The natural mistake is to treat the research prototype as an immature version of the production system: add an HTTP endpoint, drop it in a container, deploy it somewhere, and call the result “v0.” Then you begin a slow process of “cleaning it up” — adding logging and metrics, improving error handling, making the code look more like the rest of your production services.

Sometimes that’s the right approach for a demo or an MVP. It’s rarely a good long-term architecture.

Productionizing research isn’t mainly a code-cleanup project. It’s the work of designing a boundary between two environments that need to optimize for different things.

Research code is optimized for learning

Application engineers are often alarmed the first time they open research code.

They find thirty imports, two thousand lines, and every function in the system defined in one place. Variables named x2, tmp, and best_result_new. Important constants living as unexplained numbers in the middle of an expression. Configuration as a pile of mutable globals near the top of the file. Commented-out versions of the last three experiments, a hard-coded path to someone’s home directory, and a dependency on whatever happened to be installed the last time the thing worked. Sometimes it only runs after another script has generated a file in the right place. Sometimes it only runs if a notebook’s cells are executed in the right order. Failure handling consists of printing an exception, changing one of the magic numbers, and trying again. I want to be clear that I’m describing this with affection. The mistake isn’t writing code like this — the mistake is confusing that code for something it was never trying to be.

This code trips nearly every instinct an application engineer has spent years developing. We see implicit state, hidden dependencies, unclear interfaces, and an incident report waiting to happen, and our first impulse is to stop everything and clean it up before anyone builds on top of it.

That impulse is understandable, and it’s often premature. The code may be perfectly well adapted to its actual purpose. A variable hasn’t been given a durable name because the underlying concept might not be durable yet. A constant hasn’t been extracted because the researcher expects to throw it out tomorrow. Three approaches coexist in one giant file because comparing them is the work, not an unfortunate stop on the way to a clean abstraction.

Production engineering rewards explicit configuration, reproducibility, clear interfaces, and predictable failure modes. Research code is usually optimized for a different thing entirely: shrinking the time between having an idea and finding out whether it was wrong. A researcher might need to change the model, the preprocessing, the input representation, and the evaluation criteria several times in a single week. Building a stable abstraction around each of those decisions would slow the work down without producing anything useful. The point of the code is to help the researcher figure out which decisions are even worth keeping.

Application engineers have the opposite problem. Once a capability becomes part of a product, the uncertainty moves outward. The system may have to handle requests from many users, take data from external systems that misbehave, run jobs that take minutes or hours, support cancellation, or blow past available resources. Results may need to be stored, inspected, compared, and traced back to the exact version of the system that produced them. And the product creates expectations: other engineers start building against its outputs, customers start relying on its behavior, operators need to understand why it failed, and security and deployment constraints that were irrelevant during experimentation suddenly show up.

None of those production requirements mean the prototype was badly built. They mean it’s crossed into a different problem domain.

”Put an API around it” is not an architecture

The first production version of a research capability often looks like this:

  1. Take the researcher’s Python code.
  2. Add a web framework.
  3. Create an endpoint that calls the main function.
  4. Build a container.
  5. Deploy it as a service.

This is a perfectly reasonable way to test an integration. It proves the product can invoke the capability and do something useful with the response. The problem comes when the integration prototype quietly becomes the permanent design.

I ran into this on a product that incorporated an operations-research model. The product had a fairly conventional service architecture. A single-page web app sent GraphQL requests to an application backend; that backend gathered data from several sources, arranged it into the form the model expected, and sent a gRPC request to a separate model server; the model server ran the optimization and returned its results back up the same path to the UI.

One thing we got right early was keeping the model server mostly stateless. Each request carried everything needed for one optimization run, and the application backend — not the model — was responsible for gathering data from the rest of the product. That meant the researcher who built the OR model didn’t need to understand our databases, our auth system, our GraphQL API, or the various sources each input came from. His code could focus on transforming the supplied data, running the optimization, and producing a result. It was a useful boundary, and it kept a lot of product orchestration out of the research implementation.

But the service was still far more tightly coupled to the model than we understood at the time.

As the model evolved, the researcher would discover that it needed different inputs, or that a useful result required an extra output field. Each change rippled through almost every layer of the product:

  1. The gRPC request or response schema had to change.
  2. The model server had to implement the new schema.
  3. The application backend had to assemble or interpret the new data.
  4. The GraphQL API had to expose it.
  5. The UI had to render it.

A change that might take an hour to explore in a script could take days to see end to end in the product. That created a genuinely nasty feedback loop: the researcher often couldn’t tell whether an updated model produced the result the product needed until the engineering team had finished changes across several services and the UI.

At the same time, the researcher and the application engineers were trying to work in the same repository. In principle this sounded collaborative. In practice it made everyone slower. The researcher didn’t share the engineers’ familiarity with the arcane intricacies of Git or the conventions of a production service repo, so keeping branches in sync, resolving merge conflicts, and making a change without disturbing unrelated application code all became obstacles to experimentation. Watching a brilliant researcher lose an afternoon to a rebase is one of the more quietly demoralizing things in this line of work. Git is a great system with many strong points. Being intuitive to use is not one of them. And “just learn our branching conventions” is a strange toll to charge someone for the privilege of doing science. The engineers, meanwhile, were trying to keep a deployable server working while the behavior at its center changed constantly. Testing a model change often meant coordinating the model server, the application backend, the supporting data services, and the frontend — and when something broke, it could be genuinely hard to tell whether the problem was in the model, the input data, the integration, or the UI.

We’d successfully built an MVP and established a baseline for integration. We hadn’t given the researcher an independent feedback loop.

Researchers need a complete loop of their own

We eventually moved the research work into a separate prototype repository.

In that environment, the researcher could change inputs and outputs freely, build representative test scenarios, and write visualization tools specifically for understanding the optimization results. He no longer had to wait for the production UI to support every new concept before deciding whether the model behaved correctly.

That reordered the work. Before, we’d tried to evolve the model and the product integration together: the researcher would propose a change, the engineering team would propagate it through the service interfaces, and only then could anyone see whether the result was useful in the actual workflow. With the prototype repo, the researcher could first establish that the new behavior worked on representative inputs and made sense in his own visualization tools. Production work started after the capability had taken a clearer shape.

The engineering team could then focus on a more bounded question:

Given an operation that works in the prototype, how should the product supply its inputs, execute it reliably, and represent its outputs?

That didn’t make the integration automatic. The data available in the product rarely existed in exactly the form the prototype used. Outputs still had to become product concepts and understandable user interactions. We still had to decide which model details belonged in the product and which should stay internal. The immediate fix was for the engineering team to adapt production data into the form the model expected; the better long-term answer is to shrink the distance between the two environments.

The ultimate target here is a closed-loop “wet test”. The wet test is what lives in-between a “smoke test” and a “pressure test”, though it starts to slightly torture this terminology we’re borrowing from the plumbing industry. The researcher still controls the experiment, but can run it against realistic data supplied through the same integration services the product uses. That doesn’t mean handing every notebook an unaudited pipe to production. It might mean access-controlled read-only services, sanitized snapshots, or a development environment populated through production-shaped data interfaces. The important part is that the researcher can test assumptions about real data without waiting for the entire production application to get involved.

This is its own hard platform problem — access control, tenant isolation, reproducibility, cost, and the risk of accidentally coupling experiments to unstable production state all have to be handled deliberately, and it deserves a deeper treatment than I can give it here. But the direction matters. The goal isn’t to leave researchers experimenting against toy fixtures forever. It’s to give them a loop that’s both independent of the product UI and increasingly representative of what the production system will actually face.

Even before we’d fully built that environment, the separate prototype repository did something valuable: it divided two different kinds of uncertainty. The researcher could answer whether the model produced the intended behavior. The engineering team could answer whether the product could provide and consume that behavior reliably. Neither side needed the other in the room for every intermediate experiment.

A researcher needs a complete evaluation loop, not just a function that production code can eventually call. That loop might include scripts, notebooks, test fixtures, diagnostic output, or purpose-built visualizations. When the production UI is the only place a researcher can see whether a change worked, the research loop is already coupled to the product — and you’re going to feel it.

Build a boundary, not a rewrite

When I approach this now, I find it useful to think in terms of three systems rather than one.

The first is the research environment, optimized for experimentation and for understanding results. Researchers should be able to change important assumptions, build representative inputs, and visualize new outputs without negotiating every internal refactor with the product team.

The second is the production service, responsible for the guarantees the rest of the product depends on: authentication, validation, concurrency, resource management, persistence, retries, observability, and stable external behavior.

The third is the translation layer between them, which turns a changing research capability into a bounded production operation.

The translation layer might be an adapter, a job runner, a command-line contract, a packaging format, or a narrow internal API. Its exact form matters less than what it makes explicit:

The research implementation should generally own domain-specific behavior: transforming domain inputs, running inference or optimization, producing domain outputs, and reporting on the quality of the result. The production system should generally own operational behavior: authentication, data gathering, scheduling, persistence, retries, concurrency, observability, and the external API contract. The line isn’t always clean — domain knowledge can affect retry behavior, resource requirements, and how you interpret a failure — but drawing it forces the team to talk about those cases on purpose.

Without a clear boundary, operational concerns tend to pile up inside the research code. A script that once loaded a model and processed a dataset gradually learns to query databases, poll queues, refresh credentials, update job records, and upload results — eventually it’s a production service wearing a trench coat. The opposite failure happens too: the product starts encoding the current internal structure of the model into every API layer.

Our stateless model service avoided the first problem. The application backend assembled the data before making the gRPC request, so the researcher never had to pull from product databases or coordinate with other services. We did less well on the second. New model concepts kept becoming new gRPC fields, then new GraphQL fields, then new UI elements, before we were sure they represented durable product concepts. Keeping orchestration out of the model was necessary. It wasn’t sufficient. The researcher also needed somewhere to explore the meaning of new inputs and outputs before they hardened into production contracts.

Make the operation explicit, even while it changes

A production interface doesn’t need to freeze the underlying research. Often it can’t freeze the product-level operation either.

Early on, inputs and outputs are legitimately fluid. The team is still learning what information the capability needs, which outputs are meaningful, and what operation the product should expose at all. The goal isn’t to lock those decisions down prematurely. It’s to make the current operation explicit enough to integrate, and to make later changes intentional instead of letting them leak accidentally through the system.

In our case, the durable operation wasn’t “invoke this particular solver with these internal data structures.” It was closer to:

Given a complete planning scenario, produce a proposed plan along with enough information to evaluate it.

The objective function, the solver implementation, the intermediate representations, the diagnostic outputs — all of that could change behind that operation. The product only had to absorb a change when it altered what a user could ask the system to do, or how the result needed to be interpreted.

Even that operation wasn’t immutable. New product requirements could reveal that the model needed additional inputs, or that users needed a new class of output. When that happened, the right move was sometimes to change the contract — the important thing was to acknowledge the change at the boundary: version it, make it additive where practical, or coordinate a deliberate breaking change. It shouldn’t just emerge because some internal model structure quietly grew another field.

This is one reason I prefer fairly coarse boundaries around rapidly changing capabilities. A big collection of highly specific endpoints can make a system look well designed while it’s really just encoding today’s research implementation into the product architecture. A narrower contract leaves more room behind it. The contract still has to be precise — “run the AI” is not an interface — but its precision should describe the operation the product currently needs, not every stage of today’s algorithm.

The separate prototype repository gave the researcher somewhere to explore outputs before the engineering team had to decide whether they belonged in that contract. Some outputs were useful for diagnosing or evaluating the model but didn’t need to appear in the product at all. Others eventually became core product concepts. It was much easier to tell them apart after the researcher had shown how a result behaved and how he expected it to be read.

Sometimes the repository is part of the boundary

For us, the organizational and technical boundary eventually became a repository boundary as well.

The prototype repository was optimized for research iteration. It held the researcher’s test harnesses, exploratory code, visualizations, and rapidly changing data representations. The production repository was optimized for dependable operation — the service API, production packaging, validation, observability, deployment configuration, and integration with the rest of the product. When a capability was ready for integration, application engineers lifted the relevant implementation into the production repository and adapted it to the service boundary.

LLM coding agents made this a lot cheaper. They were useful for finding the parts of the prototype that needed to move, updating imports and data structures, and handling some of the repetitive adaptation the production repo required. They did not make the handoff automatic. Engineers still had to understand which behavior was essential, preserve domain-specific assumptions, review the transferred code, and test it against the production contract. The agent reduced the mechanical cost of moving code. It didn’t decide what the correct boundary should be — that part’s still on us.

Separate repositories created real costs, too. Code could drift. A fix in one repo might never make it to the other. The prototype and production implementations could start behaving differently. We accepted those costs because the alternative had gotten more expensive: every research experiment had to coexist with the constraints of a production service, and every production change risked disrupting the researcher’s working environment.

The two repositories were authoritative for different things. The prototype repo was authoritative for the emerging capability and the researcher’s current understanding of its behavior. The production repo was authoritative for what the deployed product promised its consumers. This is a different thing from independently rewriting the model from a description — we started with a working capability, adapted it to production constraints, and tried hard to preserve the behavior that had made it worth integrating in the first place.

A shared suite of representative scenarios would have made all of this much easier. These don’t need to test every possible input or demand byte-for-byte identical output; for an optimization or probabilistic system, that’s often unrealistic or actively counterproductive. Instead, each scenario can capture the input data, the important constraints, the expected invariants, and the diagnostics or visualizations needed to judge the result. A small set of these “universal scenarios” can then run in several places:

That gives you a common behavioral reference across environments. When a transfer changes the result, the teams have something concrete to inspect, instead of arguing about whether two loosely described runs were “basically the same.” The repositories don’t need identical structure. Their relationship needs to be explicit and testable. Duplication isn’t automatically a failure; unmanaged divergence is.

The handoff depends on trust

Repository structure and service boundaries only start working once the teams trust the handoff.

Researchers need to believe that if a capability works against representative data in their environment, the engineering team can carry it into the production workflow without changing its essential behavior. Engineers need to believe that the prototype’s inputs, outputs, scenarios, and visualizations are a complete enough description of what the product has to support. That trust doesn’t come from declaring ownership boundaries in a doc. It comes from completing the handoff successfully, over and over.

Researchers demonstrate a capability in the prototype. Engineers reproduce it through the production data path. When the results differ, both groups work through whether the cause is data preparation, implementation drift, or product presentation. Over time, the prototype visualization becomes more than a research convenience — it becomes the reference for what the model’s output is supposed to mean, and a shared scenario suite makes that reference repeatable. Once researchers trust that a result they can see there will show up faithfully in the product, they stop needing to sit in on every GraphQL schema change and frontend detail. Once engineers trust the prototype as a behavioral reference, they stop needing to pull researchers into every production concern.

The boundary also makes ownership clearer:

Clear ownership makes failures easier to classify, and an infrastructure failure, an integration defect, and a model-quality regression really are different problems even when they produce the same visible outcome. Telling them apart depends on keeping enough execution context to know which implementation ran, which model or solver configuration it used, what input it actually received, and whether the result can be reproduced. A log line that says the request succeeded is not enough.

A good technical boundary reduces coordination. A trusted one lets each side work on its own.

A practical sequence

The exact implementation varies, but these days I generally favor a sequence like this.

First, demonstrate the smallest end-to-end integration. It’s fine for this version to be fragile — its whole job is to reveal the real shape of the problem.

Next, make sure the researcher has a complete evaluation loop outside the production application: representative inputs, useful diagnostics, and whatever visualizations they need to judge whether the capability is working.

Then make the current product-level operation explicit. Specify its inputs, outputs, failure states, execution model, and version information, while accepting that all of it will keep changing. Treat the contract as deliberate and versioned, not permanent.

Put the research capability behind an adapter or another translation layer, and keep operational responsibilities — credentials, persistence, retries, scheduling, observability — out of the algorithm wherever you can.

Define how working capability code moves from research into production. Whether the teams share a repo or use separate ones, the transfer should have clear ownership and an agreed reference for correctness.

Build a small suite of representative scenarios that can follow the capability across environments. Prefer meaningful invariants and evaluation criteria over brittle exact-output assertions when the underlying system doesn’t guarantee an exact result.

Capture enough execution metadata to investigate and reproduce results before failures make the need obvious.

As the supporting infrastructure matures, give the researcher environment controlled access to increasingly realistic data — sanitized snapshots, or production-shaped data integration services — so the researcher can run closed-loop tests without making the whole application part of every experiment.

Finally, improve or replace the internal implementation in response to measured constraints. A rewrite should follow a demonstrated problem with performance, reliability, security, deployment, or maintainability. It shouldn’t be an initiation ritual you perform on research code before you’ll let it into production.

This sequence is less satisfying than designing the final architecture up front. It’s also a lot more likely to produce an architecture that’s actually good.

The boundary is where the capability becomes a product

It’s tempting to think of the model, the algorithm, or the research breakthrough as the valuable part of an AI product, and everything around it as supporting infrastructure. From a user’s perspective, though, the surrounding system is what decides whether the capability is usable at all.

The product is the ability to submit the right information, have the work execute under real constraints, understand its state, get back a meaningful result, and trust that the behavior will stay dependable as the underlying research changes. The research capability is essential. It isn’t sufficient.

The job of productionization isn’t to force research code to look like ordinary application code. It’s to build a boundary where experimentation can keep going on one side and dependable product behavior can emerge on the other. That boundary might be an API. It might be an adapter, a package format, a job contract, a suite of shared scenarios, or a deliberate transfer between repositories. Whatever form it takes, it has to do more than move code: it has to preserve the researcher’s ability to learn, give engineers something stable enough to operate, and build enough trust that both groups can work independently.

The prototype proves that something is possible. The production system makes it something other people can rely on.