We Shipped a Stealth Model in 24 Hours Without Leaking the Provider

The error messages leak the provider. The message IDs match their API schema. You have 24 hours.

That was the situation when OpenAI offered OpenRouter the chance to trial an unreleased model. The clock started ticking from "go live," and every API response was a potential fingerprint.

The threat model

You're routing requests to a model that doesn't officially exist yet. Every error shape, every message ID format, every tokenization pattern is a clue. The AI Twitter crowd reverse-engineers APIs for fun. They've already mapped the tool calling signatures of every major provider.

Your job: buy enough time for a two-week trial without being the team that leaked the source.

The scramble

The OpenRouter team had about 24 hours from approval to go-live. Not 24 hours to build something new. 24 hours to audit everything that could fingerprint the upstream provider.

"We did that in about 24 hours from being given the go live to how do we make sure we don't leak it ourselves and ruin this opportunity. It was a hectic 24 hours with a lot of double and triple checking."

Tovan, Office Hours S01E02

The checklist looked something like this:

1. Error shapes: Different providers return errors in different formats. A 429 from OpenAI looks different from a 429 from Anthropic. They needed to normalize or obfuscate those responses.

2. Message IDs: API responses often include IDs that follow provider-specific patterns. "chatcmpl-" is a dead giveaway.

3. Tokenization patterns: The way a model chunks text into tokens varies by architecture. Sophisticated users can infer the model family from how it handles edge cases.

4. Tool calling shapes: The structure of function calls and tool use responses differs between providers. Another fingerprint.

"We obfuscated the error shapes. We changed the IDs in the messages. We did a couple of things here and there just to make sure."

Tovan, Office Hours S01E02

The catch

It worked. Sort of.

The obfuscation held for the casual observer. But the "nerdier people in the AI scene" still figured it out.

"Of course, the nerdier people in the AI scene would be able to tell the differences between tokenization and tool calling shapes and all these kinds of things that ended up popping up on Twitter that we knew were likely to happen."

Tovan, Office Hours S01E02

The team knew this would happen. The goal was never perfect secrecy. It was operational security for a controlled trial: prevent accidental leaks, discourage casual identification, and accept that determined reverse-engineers would figure it out anyway.

For a two-week trial window, "good enough" was the right bar.

The naming game

A side note: the model names that emerged from this period, Quazar Alpha and Optimus Alpha, were brainstormed internally. The Quazar name allegedly had GPT-4.5 contributing to the brainstorm. Sometimes you ship a stealth model and let another model name it.

Why this matters for your work

This isn't just an OpenRouter story. It's a pattern for any team shipping something before you're ready to talk about it publicly.

The lesson: assume sophisticated users will reverse-engineer your dependencies. Your operational security goal isn't "nobody finds out." It's "we control the timeline of the announcement."

That means:

• Audit your error responses for upstream fingerprints
• Normalize or obfuscate ID formats that leak provider info
• Accept that tokenization and behavioral patterns will get discovered; plan for that timeline
• Define "good enough" for your window, not "perfect" for all time

If you're shipping integrations with third-party APIs you'd rather not advertise yet, start with the error shapes. That's where most accidental leaks happen.

How Roo Code handles multi-provider API complexity

When you work with AI coding agents that support multiple providers, the operational complexity multiplies. Roo Code's BYOK (Bring Your Own Key) architecture means you connect directly to your chosen provider - OpenAI, Anthropic, Google, or others through OpenRouter - without an intermediary layer that could expose provider fingerprints.

This matters for teams building internal tools or prototyping with pre-release models. Because Roo Code closes the loop by running commands, executing tests, and iterating on results locally, your API interactions stay between you and your provider. There's no middleware normalizing your requests or responses in ways that could leak implementation details.

For developers who need to swap between providers without changing their workflow, Roo Code's model-agnostic design lets you iterate on the same task across different backends while maintaining consistent behavior in your development environment.

Stealth shipping: old approach vs. new approach