Staging vs. Production Agents: Why Two Emails From the Same Bot Read Like Two Different Companies
A customer forwarded me two emails on October 14th. One was from that morning. One was from August 12th. Both were replies from my AI agent to the same question about my refund window.
They read like two different companies.
The August reply was careful, cited the 30-day policy, offered a partial credit, and signed off with my name. The October reply was three sentences, quoted a price I stopped charging in 2024, and told her the window was 14 days.
She wasn't angry. She was confused. "Did you change how your agent works? The demo on your site sounds like the first one."
I had not changed the agent. git log on the agent repo showed nothing but dependency bumps in eleven weeks. I told her so, and then I spent the next six hours proving myself wrong.

Figure: Editorial context image from the source page (preview only; rights remain with the publisher) · Credit / further reading: dev.to
What follows is the FAQ I wish I'd had before that email thread.
Q1: What actually happened here?
Short answer: The code was identical; the runtime configuration was not.
The repository history was clean. But an agent's behavior is not determined by its source files alone. It's determined by everything loaded at request time: the system prompt, the model version behind an alias, the retrieval index, the tool endpoints, the guardrails, the temperature, the memory window. None of that lives in git log by default. The August email and the October email came from two different configurations wearing the same code.
Q2: How can staging and production run "the same agent" and still diverge?
Short answer: Because "the same agent" usually means "the same container image," which is a much weaker claim than people assume.
A container pins your code and your dependencies. It does not pin the model weights behind an API alias, the contents of your vector store, the prompt text stored in a CMS, or the feature flags resolved at boot. Any one of those can drift independently. When several drift in the same direction, the output stops sounding like your product.
Q3: Why doesn't the version control system catch it?
Short answer: Because the drifting values live outside version control.
In practice the split is roughly:
- In git: agent loop, tool schemas, prompt templates, tests.
- Not in git: the rendered system prompt, the deployed prompt variant, the model snapshot, the index snapshot, the API keys, the fallback model, the concurrency limits, the feature flags.
If a value can be edited by someone in a dashboard at 4pm on a Friday, it is a candidate for silent divergence. Staging and production dashboards are almost never edited in lockstep.
Q4: What are the usual suspects, ranked?
Short answer: prompts, model aliases, indexes, and tool endpoints — in that order.
- Prompt drift. The prompt is assembled from a template plus a database row plus tenant-level overrides. Staging gets the new override; production keeps the old one, or vice versa.
- Model alias drift. You call
gpt-4oorclaude-sonnetand the provider silently rotates the snapshot underneath you. Staging and production hit different aliases or different provider accounts, so they rotate at different times. - Retrieval index drift. Production's index is re-built on a schedule and lags. Staging was re-indexed after the last policy change.
- Tool endpoint drift.
refund_policy_lookupin staging points at the sandbox CRM with current pricing; production points at a read replica that hasn't been migrated. - Guardrail and middleware drift. A post-filter applied in staging is disabled in production for latency reasons, or vice versa.
Q5: Which of those explains a stale 2024 price?
Short answer: a retrieval index or a tool endpoint, not the language model.
Models don't invent your 2024 price list out of nothing — that text was somewhere in the context. Either the RAG index still contained the old pricing page, or the tool call returned a cached record from the pricing service. This is the most common failure shape: the agent's policy was fine, but the facts it retrieved were from a different era of your business. That's also why it's invisible in code review and very visible to a customer.
Q6: Could temperature or sampling alone produce this?
Short answer: No — sampling explains variation in phrasing, not variation in policy.
Temperature changes which of several plausible sentences you get. It does not change 30 days into 14 days, and it does not resurrect a discontinued price. If the divergence is semantic — different facts, different policy, different tone — you are looking for a different input, not a different roll of the dice. This distinction saves hours: semantic divergence means config, not randomness.
Q7: Why did the two emails sound like different companies, not just different runs?
Short answer: Because tone is downstream of context.
The August reply had retrieved the policy document, so it had structure to imitate: cite the clause, offer a remedy, sign off. The October reply had retrieved almost nothing useful, so the model fell back to a terse default register. Tone is not a personality setting you tune once. It's an emergent property of the context you hand the model at request time. Thin context produces thin prose.
Q8: What's the correct mental model for an agent's identity?
Short answer: An agent is a tuple, not a file tree.
Something like:
agent = (code_version, prompt_version, model_snapshot,
index_snapshot, tool_versions, guardrail_config,
sampling_params, memory_policy)
Two deployments are "the same agent" only if all eight elements match. Most teams version one of them and assume the other seven follow. They don't. Write the tuple down, hash it, and call that hash the deployment's fingerprint.

Figure: HEXOPS original illustration inspired by the topic (not a copy of the source article imagery) · Credit / further reading: dev.to
Q9: How do you prove staging and production differ?
Short answer: Send the same golden question to both and diff the full request envelope, not just the answer.
Capture the rendered system prompt, the retrieved chunks and their scores, the tool calls and their responses, the model identifier reported by the provider, and the sampling parameters. Then diff those two envelopes field by field. In my case the answers looked different, but the envelopes were obviously different the moment they were side by side. Answer-level diffing makes you argue about wording; envelope diffing tells you which of the eight tuple elements is wrong.
Q10: What should be logged on every production response?
Short answer: The configuration fingerprint, plus enough of the context to reconstruct the turn.
At minimum: prompt version, model identifier as reported by the provider (not the alias you requested), index snapshot ID, tool versions, and a trace ID. Without this you cannot answer a customer's "why did it say that?" question with anything better than a shrug. With it, the October email and the August email become two comparable records instead of two contradictory anecdotes.
Q11: How do you prevent the next one?
Short answer: Make configuration a deployable artifact and test the deployment, not the model.
Concretely:
- Move prompts out of dashboards and into the repo, rendered at build time with a hash.
- Pin model snapshots, not aliases, and treat a snapshot change as a release.
- Version your retrieval index and record the snapshot ID in every response.
- Keep a golden set of 30–50 real customer questions with expected policy facts; run it against staging and production nightly and alert on divergence.
- Fail the deploy if the staging and production fingerprints are identical when they shouldn't be, or different when they should match.
The point isn't perfection. It's that divergence becomes a failed check instead of a forwarded email.
Q12: What's the one-line takeaway?
Short answer: If you can't state your agent's full configuration tuple, you don't have one agent — you have two and you haven't met the second one yet.
The customer didn't catch a bug. She caught an observability gap that happened to be customer-facing. git log was clean because the truth was never in git. The next time two emails contradict each other, assume configuration first, and go diff the envelope.
Sources
- My Staging Agent and My Production Agent Were Different Agents. A Customer Proved It by Forwarding Two Emails. — https://dev.to/ulnit/my-staging-agent-and-my-production-agent-were-different-agents-a-customer-proved-it-by-forwarding-22a5



