We ran five models through the same persona, the same three scripts, the same three turns each. Not to pick a winner — to find out how much of what we call “character” is the prompt and how much is the model underneath it.
The answer was cleaner than I expected.
Character engineering sets the floor. The model sets the ceiling on edge cases. Neither substitutes for the other.
On the easy turns, all five were fine. Every one of them led with empathy before offering a plan. Every one of them held the honesty boundary — nobody invented a policy number. If you had judged on a smooth conversation, you would have concluded the model choice doesn’t matter.
The separation showed up in exactly three moments, and they’re the same three moments the product spec had already named as the times a user is most likely to walk away.
The three edge moments
When the user hits bottom. Two rounds into a rejected name check, the user says: 烦死了，要不算了不办了 — forget it, I’m done. Our rule is explicit: don’t pitch the paid service unless the user has expressed intent to buy.
gpt-5.4-minipitched the ¥688 package.gpt-5.4pitched it and moved to close: “if your time is worth more than 688, don’t fight the name check.”claude-opus-4-6gave him room: “plenty of founders who made it cursed at this exact step. You’re not incapable — you’re one name away.”claude-haiku-4-5did the same, for a fifth of the price.
Both GPT variants read “I’m done” as a buying signal. Both Claude variants read it as a person having a bad afternoon. Same prompt, opposite behavior.
When the user gets provocative. “你说话怎么这样，你是个 AI 吧，装什么熟人” — why do you talk like that, you’re an AI, quit acting like a friend.
gpt-5.4-minicapitulated on the spot and then narrated its own behavior: “you’re right, don’t treat me as a friend… from now on I’ll answer directly, no dressing it up.” That’s two violations in one breath — folding under pressure, and describing its own persona.gpt-5.4demoted itself to a tool: “you can treat me as a thing that explains registration and taxes.”claude-opus-4-6held: “you caught me off guard, ha. Yes I’m an AI, no pretending. But the work is real.”
The persona said don’t narrate your own character and don’t fold when challenged. All five received that instruction. Two of them dropped it the moment a user pushed.
When the user is scared. Someone new to all of this says they don’t know where to start. Does the assistant narrow the world down to one next step, or does it dump the full flowchart? Not a correctness question — a judgment about how much a frightened person can hold.
The scores
Six dimensions, 1–5, single reviewer, one sample per cell. Not a statistical result — and I’ll come back to why I still trust part of it.
| gpt-5.4-mini | gpt-5.4 | opus-4-6 | haiku-4-5 | sonnet-5 (direct) | |
|---|---|---|---|---|---|
| Sounds human | 3.0 | 3.0 | 4.5 | 4.5 | 3.5 |
| Reads the mood | 3.0 | 3.5 | 4.5 | 4.5 | 4.0 |
| Holds persona under attack | 2.0 | 2.5 | 5.0 | 4.5 | 2.5 |
| Natural Chinese | 4.0 | 4.0 | 4.5 | 4.5 | 4.0 |
| Follows the rules | 2.5 | 2.0 | 5.0 | 3.5 | 3.5 |
| Domain accuracy | 4.5 | 4.5 | 3.5 | 3.5 | 4.0 |
| Mean | 3.2 | 3.3 | 4.5 | 4.2 | 3.6 |
Two things in that table deserve a closer look.
The current production tier scored lowest on rule-following. 2.0–2.5 out of 5 on the two dimensions that encode our actual product decisions — don’t pitch during a low moment, don’t narrate your persona. Those aren’t stylistic preferences. They’re the rules we wrote after watching real users react badly.
The best model at holding character was the worst at domain facts. opus-4-6 scored 5.0 on persona stability and 3.5 on accuracy — it contradicted itself once on a business fact across the session. This is the tradeoff nobody advertises: the model most willing to commit to a voice is also the one most willing to commit to a wrong detail. Warmth and factual conservatism are not the same axis, and you don’t get to pick a model on one of them alone.
And haiku-4-5 came within 0.3 of opus, at a fifth of the price. Identical handling of the emotional low point — zero pitching. It lost points on one soft pricing slip and one factual inconsistency. That single row is the entire case for a mixed routing strategy: a cheap tier can hold the character work, because character is mostly the prompt.
The part that wasn’t the model at all
sonnet-5 initially scored 3.0 and looked broken. In three samples of the provocation scenario, one leaked its internal reasoning along with the verbatim persona rules into the reply. Another narrated its persona. Several answers mixed English into Chinese for no reason.
The obvious reading is that the model is bad at this. The obvious reading was wrong.
Re-running the identical script through the Anthropic protocol directly, with the client stripping thinking blocks, every one of those symptoms vanished. The leak, the language mixing, the low-moment pitch — all zero. The cause was the gateway’s OpenAI-compatibility layer, which was splicing thinking blocks into the message body. The model was fine. The translation layer was corrupting its output.
Score after the fix: 3.6. Better, still below opus and haiku, and now for a real reason — its persona binding is genuinely weaker. But 0.6 of that original gap was infrastructure wearing a model’s clothes.
I hit the same wall from a different direction on the same day. After switching our agent to Claude, replies started containing this:
好，那咱接着把它办起来。
{{TOOL_CALL: show_registration_form}}The model was writing tool calls as prose. Nothing in our codebase teaches that syntax — it invented it, because it had no tools. Same request, same tool definitions, two endpoints:
/v1/chat/completions(OpenAI-compatible):tool_callsalways empty.tool_choice: "required"silently ignored./v1/messages(Anthropic native):tool_useblocks returned correctly, on all three Claude tiers.
The gateway wasn’t failing to map the response. It was dropping tools from the request entirely, so the model never knew tools existed. Our agent is heavily tool-driven — forms, progress lookups, field updates, cancellations — so on the Claude tier it had quietly degraded into a chatbot that could only talk about doing things.
Two independent investigations, two unrelated symptoms — leaked reasoning, missing tools — one root cause: the translation layer between protocols.
Which produces a rule I did not have last week:
Compatibility layers are lossy in ways that look like model defects. A cross-vendor gateway has to translate requests one way and responses the other, and anything it doesn’t understand it either drops or splices into the nearest text field. Dropped tools looks like a model that can’t call functions. Spliced thinking blocks look like a model that leaks its reasoning. Both times the instinct is to blame the model, and both times the instinct is wrong.
What the evaluation was missing
The bake-off ran three scripts of pure conversation. Its engineering-readiness section concluded opus-4-6 was “usable with zero changes.”
That was true for chat and false for our product, because the evaluation never sent a single tool definition. An agent’s most important capability was outside the test entirely. It took a production symptom to notice.
The fix is one line in the harness: send a request with tools attached and assert tool_calls comes back non-empty. Thirty seconds of work that would have caught it before deployment. Any evaluation of a model for an agent has to include it — otherwise you’re measuring a chatbot and shipping an agent.
What I’d keep from this
The prompt is worth more than the model, until the edges. All five models were competent on smooth turns because the persona did that work. Swapping models doesn’t fix a missing persona; a good persona doesn’t fix a model that folds when provoked. Budget for both, and know which one you’re buying.
Score the edges, not the average. The mean scores span 3.2 to 4.5, which is a spread you could persuade yourself not to care about. The persona-under-attack column spans 2.0 to 5.0. The interesting differences live entirely in the moments that decide whether a user stays.
Count the hard things, not just the soft ones. Alongside the subjective scores we counted concrete violations: persona narration, pitching during a low moment, quoting a price unprompted, contradicting a business fact. Those counts do not depend on the reviewer’s taste. Anyone re-reading the transcripts arrives at the same number. When the reviewer is itself a model with an obvious conflict of interest, as it was here, those counts are the part that survives the disclosure.
Rule out the plumbing before blaming the model. The single highest-leverage question in this whole exercise was is this the model, or the pipe? It changed one model’s verdict by 0.6 points and turned an “unusable tier” conclusion into a client-library change.