AI Agents
Most property teams shipping LLM summarization in 2026 are not blocked by model quality. They are blocked by the second and third execution of the same job. A summary that was correct once turns into a duplicate record, a stale score explanation attached to the wrong applicant, or a candidate file that contradicts itself after a burst of provider timeouts.
The pattern is predictable. A summarization endpoint gets wired up in a day, the demo looks excellent, and then the retry logic — usually the least-reviewed file in the repo — quietly decides what the business believes about a tenant, a ticket, or an applicant.
This checklist is written for engineering leads and growth operators who own a summarization pipeline in a property-management context: applicant screening against a rubric, maintenance-ticket triage, lease abstraction, or vendor comparison. Each item has an explicit pass/fail test. Work through them in order; the early items constrain the later ones.
The Governing Rule: Route by Failure Cost, Not Token Price
Figure: Editorial context image from the source page (preview only; rights remain with the publisher) · Credit / further reading: dev.to
Before any technical item, settle the routing philosophy. Teams routinely start with the cheapest chat model because the per-token price is easy to justify upward. That is the wrong ordering.
Rank your summarization jobs by what a wrong output costs. A routine maintenance summary that is slightly clumsy costs a few seconds of a human's attention. A mis-scored applicant summary that becomes the justification for a rejection costs considerably more, and is hard to unwind once it has been read. Lease-clause summaries sit somewhere in between, depending on whether they feed a legal review step.
Once jobs are ranked, the architecture becomes obvious: a cost-efficient chat model behind a retrying, idempotent worker for the routine tier, and an escalation path to a stronger model or a human reviewer for anything ambiguous. The escalation path is not a failure mode — it is the design.
The Checklist
1. The summarization contract exists as a versioned schema
Your worker should depend on a small, explicit contract rather than a provider SDK shape. Input fields, output fields, allowed enumerations, and maximum summary length should be defined once and versioned.
Pass: A validator rejects any model response that does not match the schema, and the schema version is recorded alongside every stored summary.
Fail: The worker reads response.choices[0].message.content directly and hopes the JSON parses.
2. Every write carries a stable idempotency key
The key should be derived from business identity, not from request time or a random UUID — for example the applicant ID plus rubric version plus summarization type. Retries must produce the same key.
Pass: Replaying the same logical job a hundred times produces exactly one stored record, and the storage layer enforces this rather than relying on application checks.
Fail: The key includes a timestamp, so retries create near-duplicate rows that a human later has to reconcile.
3. Retries are bounded, jittered, and classified
Not all errors deserve a retry. Rate limits and transient upstream failures do; schema violations and content-policy refusals do not. Retrying a deterministic failure just burns budget and delays the escalation.
Pass: The worker distinguishes retryable from terminal errors, uses exponential backoff with jitter, caps total attempts, and records the terminal reason.
Fail: A blanket for loop with a fixed sleep, or a retry that fires on validation errors.
4. Deadlines propagate through the whole call chain
A summarization that arrives after the decision it was meant to inform is worse than no summarization, because it creates the appearance of diligence without the substance.
Pass: The worker receives an absolute deadline from the caller, passes a derived timeout to the model call, and abandons the job cleanly when the budget is exhausted.
Fail: A default SDK timeout that is longer than the human review window it is meant to support.
5. Ambiguity has a numeric trigger
Define, in the prompt and in the code, what "ambiguous" means. A confidence band, a missing-evidence flag, or a rubric score that lands between two thresholds — pick one and make it machine-readable.
Pass: The response schema includes an explicit escalation signal, and the router honours it without a second human judgment call.
Fail: Escalation happens when an engineer notices something odd in a dashboard.
6. Summaries cite the evidence they used
A summary that asserts "strong maintenance history" without pointing at the fields that support it is not auditable and will not survive a dispute.
Pass: Every substantive claim in the output maps to a referenced input field or document span, and the reference is stored.
Fail: Free-text prose with no provenance, which forces a reviewer to re-read the entire source file.
7. Retrieval indexing happens only after a durable write
This is the boundary most teams get wrong. Embedding a summary into a vector store before the transactional write commits creates a class of documents that exist in search but not in the system of record.
Figure: HEXOPS original illustration inspired by the topic (not a copy of the source article imagery) · Credit / further reading: dev.to
Pass: The index update is a downstream consumer of a committed write event, and it is itself idempotent so replays do not duplicate chunks.
Fail: The worker calls the embedding API and the database in the same function body with no ordering guarantee.
8. Prompts and rubrics are pinned and diffable
The rubric that scored a candidate six months ago must be reconstructable, or the score is meaningless.
Pass: Prompt templates and rubric definitions live in version control, are referenced by hash in stored outputs, and changes require review.
Fail: Prompts edited in a dashboard with no history, so nobody can explain a historical decision.
9. There is a fixed evaluation set and a drift signal
Routine summaries drift quietly. Model updates, prompt edits, and input distribution shifts all move quality without raising errors.
Pass: A labelled evaluation set runs on every prompt or model change, and a sampled human review of production output runs on a schedule.
Fail: Quality is judged by whoever last looked at the output.
10. Cost per accepted summary is a tracked metric
Token price is not the number that matters. Retries, escalations, and human review time all belong in the same numerator.
Pass: You can state the fully loaded cost of one accepted summary for each job tier, and compare it against the human baseline it replaced.
Fail: You can quote a per-million-token rate and nothing else.
Key takeaways
- Failure cost, not token price, should decide which model handles which job — and an escalation path to a stronger model or a human is part of the design, not a defect.
- Idempotency keys derived from business identity are the single highest-leverage safeguard in a retrying summarization worker; everything downstream depends on them.
- Retries must be classified. Retrying schema violations and policy refusals wastes budget and hides real problems.
- Indexing a summary into retrieval before the durable write commits is the most common structural bug in this class of pipeline.
- Prompts and rubrics are part of your audit trail. If they are not versioned, your historical scores are not defensible.
Next steps
- Grade your current pipeline against the ten items above and mark each pass or fail — resist partial credit.
- Fix items 2, 3, and 7 first; they cause the damage that is hardest to reverse.
- Write down the escalation threshold for ambiguity before your next model change, not after.
- Instrument cost per accepted summary and put it next to the human baseline on the same dashboard.
- Schedule a quarterly replay test: run a batch of historical jobs through the current pipeline and diff the outputs against what was stored.
