Tool-calling agents have moved from demos into revenue paths. That shift changes the question incident reviews ask. It is no longer "what did the model say?" but "why did it reach for that tool, and would it do it again?" A recent InfoQ session from Hannes Hapke on Dataiku's open-source Kiji Inspector makes the case for opening that black box directly at the decision tokens, using sparse autoencoders (SAEs), causal ablation, and activation capture through patched vLLM engines. You do not have to adopt that exact stack to benefit from the discipline behind it. This checklist turns the idea into gates your team can actually pass or fail.
Why tool-selection opacity is an ops problem
A chat answer that reads oddly is embarrassing. A tool call that reads oddly is an incident. Tool selection is where an agent spends money, mutates records, sends email, and touches third-party systems. When the selection is wrong, the natural-language reasoning trace is usually the least trustworthy artifact in the room — it is generated after the fact, optimized for plausibility, and often contradicts the actual token-level decision. Explainability here is not an academic nicety; it is the difference between a fix and a guess.

Figure: Editorial context image from the source page (preview only; rights remain with the publisher) · Credit / further reading: www.infoq.com
The 12-point agent explainability checklist
1. Decision tokens are captured separately from user-facing text
Pass: for every tool call you can retrieve the exact token span that emitted the tool name, plus the top-k alternatives and their probabilities at that step. Fail: your only record is the friendly "I'll look that up for you" message.
2. Chain-of-thought is treated as narrative, not evidence
Pass: reasoning traces are stored as a hint, and no postmortem is allowed to close on them alone. Fail: your runbook says "read the reasoning trace and see what it was thinking."
3. You maintain a labeled regression set of tool-choice cases
Pass: at least 200 curated prompts per critical agent, each annotated with the expected tool and one or more forbidden tools, running in CI on every prompt or model change. Fail: evals score answer quality only, and tool routing is untested.
4. Attribution is causal, not correlational
Pass: you ablate — suppress a candidate feature, direction, or neuron set and confirm the tool choice flips or degrades by a measurable margin. Effect sizes are reported across seeds with confidence intervals, and null results are recorded too. Fail: attention heatmaps and saliency charts are presented as explanations with no intervention behind them.
5. Sparse autoencoders (or an equivalent decomposition) are trained on your own traffic and version-pinned
Pass: the dictionary version, training corpus hash, sparsity penalty, and reconstruction loss are logged; a documented drift threshold triggers retraining. Fail: a one-off notebook from three quarters ago that nobody can reproduce.
6. Activation capture lives in the serving path with a bounded cost
Pass: the inference engine (for example a patched vLLM with hooks) captures activations only at designated layers and only for a configurable sample of requests. The p99 latency budget impact is written down and tested. Fail: explanations can only be produced offline, on a different model build, weeks later.
7. Features map to a reviewed concept inventory
Pass: every feature used in an explanation carries a human-written label, two or three examples, and a named owner. Unlabeled features are flagged rather than silently displayed. Fail: "feature 12043 activated strongly" is what you hand to a reviewer.
8. You alert on explanation drift, not just output drift
Pass: monitors track tool-mix distribution, activation frequency of your top features, and ablation effect sizes over time. Thresholds are tied to incident severity and someone is paged. Fail: the dashboards exist, but nothing has ever fired.
9. Guardrails are derived from explanations
Pass: when a feature reliably precedes unsafe tool selection, it becomes a runtime block, a router rule, or a forced confirmation step — and a regression test proves the guardrail holds. Fail: explanations are archived for compliance and never change behavior.
10. Cost and authority boundaries are explicit per tool
Pass: every tool has a documented cost profile, blast radius, and required approval tier. The agent cannot exceed a tier without a human gate. Fail: the answer to "who authorized this?" is "the agent decided."

Figure: HEXOPS original illustration inspired by the topic (not a copy of the source article imagery) · Credit / further reading: www.infoq.com
11. There is a reviewer workflow with a clock on it
Pass: low-confidence or high-impact tool calls route to a queue with a defined SLA and a structured decision record. Reviewer disagreements feed back into the labeled set from item 3. Fail: escalation means pinging someone in Slack and hoping.
12. Every production trace can be replayed
Pass: a stored trace can be re-run against a pinned model build, SAE version, and feature set, and the tool decision reproduces within a stated tolerance. Fail: "could not reproduce" is the most common line in your postmortems.
How to score it
Treat items 1, 4, 6, and 12 as blocking. A failure on any of them means you cannot diagnose a bad tool call at all — pause expansion of autonomy until they pass. Items 3, 8, and 9 are operational maturity: failing them will not stop you shipping, but it will make your first serious incident expensive. Items 2, 5, 7, 10, and 11 are governance and workflow; they are the cheapest to fix and the easiest to deprioritize forever, which is exactly why they belong on a checklist.
A reasonable target for a team already in production is seven of twelve passing within one quarter, with all four blocking items green. If you are pre-launch, get items 1, 10, and 12 done before the first external user touches a tool that writes anything.
Key takeaways
- Tool selection is the highest-consequence decision an agent makes, and the natural-language trace is the weakest evidence about it.
- Decision-token capture, causal ablation, and version-pinned decomposition turn explanations from storytelling into measurement.
- Explainability earns its budget only when it changes runtime behavior — guardrails, routing, approvals — or shortens incident resolution.
- Replayability is the floor. If you cannot reproduce a tool call, you cannot fix it.
Next steps
- Pull one week of tool-call logs and check how many have retrievable decision tokens. That number is your real starting point.
- Build the 200-case labeled tool-choice set before investing in instrumentation — it tells you what to instrument.
- Run your first ablation on a known bad tool call and write up the effect size, including if it is zero.
- Define the top three explanation-drift monitors and wire at least one to paging.
- Convert one recurring unsafe pattern into a runtime guardrail with a regression test.



