Onefold

A team ships a feature on one model. It works. Two months later a cheaper model comes out that looks better on exactly their kind of workload, and someone says the obvious thing: we should try it.

Trying it means rewriting the request, because the system prompt is a message in one API and a top-level field in the other. Then the stream parser, because the events have different names and different shapes. Then the tool definitions, because one nests the schema under a function object and the other does not. Then error handling, because the failure body is a different shape again. None of this is hard. All of it is a day.

So the experiment gets scheduled, then bumped, and the feature stays on the model it launched with — not because it is the right model but because it was the first one. The cost of asking the question is higher than the value anybody assigns to the answer.

Onefold exists to make the question cost nothing: change the model string, send the same request, read the same response.

Providers diverge in four places, and it is worth being precise about them because the whole design follows from the list. The message shape: where the system turn goes, whether content is a string or typed blocks, whether consecutive turns from one role are legal. The parameter names: the same idea called different things, sometimes a scalar here and an array there. The stream protocol: both server-sent events, different event names, different delta semantics. And the error body: the same failure reported at a different path in a different envelope.

Each difference is trivial in isolation. The problem is that there are four of them per provider, and the combination is what a client has to learn.

• Compatible: An existing OpenAI client works by changing one string. No SDK to install, no wrapper to learn, no fork of anybody’s library.
• Streaming: Translated frame by frame. The first token reaches the caller as soon as it reaches the gateway, because buffering a response to translate it would undo the reason people stream.
• Extensible: A provider that reuses existing translation styles is configuration, not code. The engine should not grow a branch per vendor.
• Attributable: Every request that reaches a provider produces one durable record of what it cost and how long it took.
• Off the critical path: Nothing in the accounting or analytics path may delay, fail, or otherwise be visible to the caller.

• A client library per provider: The default answer, and it pushes the problem onto every app. Each one learns every dialect, upgrades on its own schedule, and nobody can see total spend.
• A proxy with a module per provider: Centralises it, which is right, but every provider is still hand-written code. The modules start similar and drift apart, and the fifth one is as much work as the first.
• Lowest-common-denominator API: Expose only what every provider shares. Easy to build and quietly useless — the reason to switch models is usually the thing that is not shared.

The public surface is two routes: list the models, and run a chat completion. Everything else is internal. A request authenticates a bearer key to an organisation and a balance, reads one catalog row to learn where the model lives and what it costs, picks the spec for that provider’s dialect, and translates.

The catalog row is doing double duty, and that is deliberate. It is the routing table — upstream id, base URL, dialect — and the price list in the same record, so the request that decides where to go has already loaded what it will cost.

A provider is a declarative object. It names the auth headers it needs, with a placeholder where the secret goes. It names a message style and a stream style, by string, rather than carrying the code for either. It maps each OpenAI parameter to a provider field with a small rule — rename it, wrap a scalar into an array, fill it from the catalog row when the provider requires it and the client omitted it, or hand the value to a named translator when the shape has to change rather than the name. And it names the dot paths where the provider’s error message and type live.

The engine reads that and does the work. The whole Anthropic spec is nineteen lines and contains no logic at all.

The honest boundary: this is only configuration for providers whose behaviour is already expressible in existing styles. A genuinely new protocol needs a new style, which is real code in the engine. The claim is that the second provider on a known protocol is nearly free, not that any provider is.

The temptation with a gateway is to write two response paths — one that forwards a stream and one that returns a whole body — and then spend a year fixing the places where they disagree. Onefold has one.

Provider bytes become server-sent event frames, frames become dialect-neutral events, and there are only six kinds of those: some text, a tool call opened, a fragment of that call’s arguments, a usage report, a finish reason, an error. Both response modes consume that same sequence. Streaming re-emits each event as an OpenAI chunk; non-streaming consumes the sequence to the end and assembles one body from it.

Non-streaming is therefore not a second implementation. It is the streaming path, read to completion before anything is written. The two can not disagree about what the model said, because they are reading the same events.

Tool calls are where dialects stop being cosmetic. Anthropic indexes every content block in a response against one counter — text blocks and tool blocks share it. OpenAI numbers tool calls densely from zero and does not count text at all. A response that goes text, tool, text, tool produces Anthropic blocks one and three, which the client expects to see as tool calls zero and one.

So the translator carries a map from block index to tool index, built as the response arrives. That map is meaningless outside a single response, which is why a stream style is a factory rather than a function: the engine constructs a fresh translator per request, and any state a dialect needs lives in the closure rather than in a module.

The second half is that tool arguments stream as partial JSON. Not an object, not valid JSON on its own — a fragment. The gateway forwards fragments verbatim and lets the client concatenate them, because the alternative is buffering until the object closes, and a gateway that waits for a whole tool call before saying anything has quietly stopped streaming.

Every request that reaches a provider produces one record: which model, which provider, status, finish reason, token counts both ways, latency, whether it streamed, which key and organisation, and what it cost. That record is written after the response has been handed to the caller, on the runtime’s post-response hook, and a failure to write it is a log line rather than an error.

It lands in a Postgres staging table first and is drained into ClickHouse once a minute. The hop exists for an unglamorous reason — the Worker runtime cannot reach ClickHouse directly — but it bought something real. The analytics store is append-only, queried by aggregation over time ranges, and has no business sharing a transactional database with the catalog. The staging table makes the boundary explicit and gives the drain somewhere to retry from.

Retries are safe because the destination table collapses rows with the same id on merge, so a batch replayed after a partial failure produces one row rather than two. Idempotency is a property of the schema instead of a discipline the drain script has to maintain.

Token order is preserved and nothing is held: a byte that arrives from a provider leaves the gateway as soon as it has been translated. Streaming and non-streaming responses describe the same completion, because they are built from the same events. Every provider error surfaces in one shape regardless of what the provider sent. And no analytics failure is ever visible to a caller, because the caller has already been served by the time the write is attempted.

An OpenAI-shaped surface is a ceiling. Anything a provider offers that has no OpenAI equivalent is unreachable through the compatible route. The abstraction that makes switching cheap is the same one that hides the reason you might want a particular model.

A failure mid-stream cannot be retried. Once the first token is out, the response has been committed. Failing over to a second provider would mean either buffering — which defeats the point — or emitting a contradiction. Today the stream ends and the client sees a truncated response.

Catalog reads sit on the request path. Routing needs a database round trip through a connection pooler. It is fast and it is a dependency: if Postgres is unreachable, the gateway cannot route, when it could plausibly serve a cached routing table and degrade instead.

Specs drift silently. A provider that changes a field name breaks one dialect at runtime, not at build time. Config-as-data moves the failure from the compiler to production, which is the real cost of the flexibility.

Almost everything good here came from finding the narrow waist. Six event types describe any completion well enough that both response modes can be built on them. One declarative object describes a provider well enough that the fifth one is configuration. One catalog row answers both where to send this and what it costs.

The parts that stayed hard are the parts where no narrow waist exists — tool call indices, partial JSON, a stream that cannot be un-sent. That is the useful line, and it is worth knowing which side of it a problem is on before deciding how much abstraction to spend on it.