Version the whole system
An LLM release includes model and endpoint aliases, prompts, tools, policies, retrieval data and index, embedding and reranker models, code, dependencies, evaluation data, and feature flags. Store a deployable manifest rather than a loose model name.
Define compatibility checks for context limits, tool schemas, output formats, safety controls, and indexes. Provider aliases can change behavior without a source-code commit, so monitor resolved versions where available.
Create multidimensional release gates
Evaluate task success, claim support, refusal, tool correctness, security attacks, latency, reliability, and cost. Promote only after deterministic tests, fixed evaluations, and accountable review. Canary using comparable slices and stop quickly on severe failures.
Track cost per successful task, not merely tokens per call. Include retries, judge calls, retrieval, tool usage, telemetry, and human correction.
Prepare for failure
Create runbooks for provider outage, rate limiting, prompt regression, retrieval corruption, data exposure, unsafe tool action, runaway loops, cost spike, and telemetry loss. Define fallback, degradation, kill switch, credential revocation, rollback, and communications.
Run game days with synthetic incidents. Recovery must verify semantic and security behavior, not only return HTTP status to green.
Close the lifecycle
Monitor changes, incidents, complaints, costs, and evidence expiry. Retire unused prompts, models, indexes, tools, identities, and data. Confirm deletion and billing rather than assuming a deployment was removed completely.
Decision framework
| Area | Guidance |
|---|---|
| Failure | Prepared response |
| Provider outage | Bounded retry, fallback or explicit unavailability, and queue control |
| Quality regression | Stop canary, rollback manifest, run fixed probes |
| Unauthorized tool action | Kill switch, credential revocation, evidence preservation, impact review |
| Cost spike | Budgets, loop limits, rate control, alert, and traffic containment |
Practical checklist
- Maintain a complete release manifest
- Gate task, safety, reliability, and cost
- Test provider and tool failure modes
- Implement hard step and spending budgets
- Exercise rollback with compatible artifacts
- Verify retirement and billing cleanup
First-party sources
Source status last checked 2026-09-11. Links can change after publication.