AI agents that troubleshoot Kubernetes are easy to demo and hard to trust. This FAQ condenses the practical lessons from a series of local-model experiments in which a language model was wired to real cluster tooling and asked to explain incidents — and what those results mean for teams building AIOps that can be defended in a postmortem.
The fundamentals: what tool calling actually gives you
The short version: tool calling gives a model reach, not judgment. Everything else in this article follows from that distinction.

Figure: Editorial context image from the source page (preview only; rights remain with the publisher) · Credit / further reading: dev.to
What is LLM tool calling in an AIOps context?
It is a structured handshake. The model returns a request naming a function and its arguments — list pods in a namespace, fetch recent events, read a deployment's rollout history — and a runtime outside the model executes it and returns the result. The model never holds cluster credentials and never runs anything itself. It proposes; your platform disposes. That separation is the single most important architectural property of a safe agent, because it keeps every action auditable and rate-limitable.
Why isn't tool calling sufficient for production troubleshooting?
Because access to evidence is not the same as discipline about evidence. A model can successfully call kubectl describe pod, receive a wall of output, and then write a narrative that blends what the output actually said with what the model expected to see. In a demo that reads as competence. In an incident it is the exact failure mode you cannot afford: a confident root cause that no one can trace back to a specific observation. Tool calling solves the plumbing problem and leaves the epistemology problem untouched.
What is deterministic AIOps?
Deterministic AIOps means the reasoning path is constrained rather than free-form. Instead of asking a model to investigate freely and narrate the result, you encode the investigation itself: a fixed sequence of evidence-gathering steps, explicit pass/fail checks at each step, and a rule that a conclusion may only reference evidence already collected and validated. The model's job narrows from "figure out what happened" to "interpret this specific evidence set and propose the next check." Same model, radically smaller blast radius.
Why Kubernetes specifically?
Kubernetes is an unusually good stress test because its failure modes are numerous, layered, and superficially similar. A CrashLoopBackOff can come from a bad image, a missing secret, a failing readiness probe, a resource limit, or a node-level problem — and the surface symptom looks identical. That ambiguity is precisely where a model's tendency to pattern-match toward a familiar story becomes dangerous, and where a checklist beats a hunch.
How do you separate observation from assumption in practice?
Force the two into different fields. Require the agent to emit a structured object in which every claim carries an evidence pointer — a specific command output, a log line, an event timestamp — and treat any claim without a pointer as an assumption rather than a finding. Then validate: re-run the referenced query and confirm the cited text exists. It is mundane engineering, and it is the difference between an assistant and a liability.

Figure: HEXOPS original illustration inspired by the topic (not a copy of the source article imagery) · Credit / further reading: dev.to
What does an evidence validation layer cost you?
Latency and complexity, both modest. The benefit is that hallucinated citations fail loudly at validation time instead of silently at 3 a.m. In the experiments, the validation step was where most incorrect conclusions died — not because the model was bad, but because unanchored claims could not survive a check that asked "show me."
Should the model ever execute remediation directly?
Treat remediation as a separate, explicitly gated capability. Read-only diagnosis, human-approved writes. If a team does allow the agent to act, the action set should be enumerated, scoped to a namespace or label selector, reversible where possible, and logged with the evidence that justified it. The failure you are protecting against is not malice — it is a plausible-sounding wrong hypothesis applied quickly.
Can smaller local models do this work?
Often yes, because the hard part moved. Once the investigation is a defined sequence and the output format is constrained, the model is doing summarization and next-step selection rather than open-ended detective work. Smaller models handle that competently and keep sensitive cluster context on infrastructure you control — a meaningful advantage for regulated teams. The tradeoff is more brittle adherence to format, which the validation layer catches anyway.
How should you evaluate an AIOps agent?
Not on whether its explanations sound right. Build a set of past incidents with known root causes, replay the evidence, and score two things separately: did the agent reach the correct conclusion, and did it cite evidence that actually supports it? The second number is the one that predicts production behaviour, and it is almost always lower than teams expect on the first run.
Does this replace runbooks or on-call engineers?
No, it compresses the boring middle of an incident. The agent is good at parallel evidence gathering, summarizing noisy logs, and proposing the next check. Humans still own priority calls, customer communication, and irreversible decisions. Framing it as a fast junior responder with perfect recall and zero judgment is closer to reality than framing it as an autonomous operator.
What is the one metric that matters most?
Time to a defensible hypothesis. Not time to any hypothesis — time to one you can justify with evidence a reviewer can inspect. That metric rewards both speed and discipline, and it does not reward the model for sounding confident.
Where should a team start?
Pick one narrow, high-frequency failure class — image pull errors, readiness probe failures, resource-limit OOM kills. Write the deterministic investigation path for it by hand, as a checklist a new engineer could follow. Then let the model execute that path and summarize. Expand only after the evidence validation holds up under replay.
Key takeaways
- Tool calling grants access to evidence; it does not grant standards for judging it.
- Deterministic AIOps constrains the reasoning path so conclusions are traceable, not merely plausible.
- Every claim should carry an evidence pointer, and every pointer should be re-validated before a human reads it.
- Keep diagnosis read-only by default; gate remediation behind explicit, scoped, logged approval.
- Evaluate agents on citation accuracy across replayed incidents, not on how convincing their narratives sound.
Next steps
Document one failure class as a hand-written diagnostic checklist this week. Instrument your agent to emit structured claims with evidence pointers. Build a replay set of five past incidents with known root causes and measure citation accuracy before you measure anything else.
Sources
- From LLM Tool Calling to Deterministic AIOps: Lessons from Kubernetes Troubleshooting — Dev.to



