What “production-ready” actually means for AI

“It works on my laptop” has a cousin in AI: “it scored well on the test set.” Both describe something that might be true and is almost never enough.

When a model or LLM application starts influencing real decisions — approving a claim, routing a customer, summarising a case file — the question shifts from is it accurate? to will it stay accurate, safe and accountable every day it runs?

Before we call any system production-ready, we check six properties.

1. It is reproducible

Can you rebuild exactly this model, from exactly this data, with exactly this code? If not, you can’t debug it, audit it or roll it back. Versioned data, a model registry and pinned environments are the minimum.

2. It is evaluated against reality

An evaluation suite built from real cases — including the awkward ones — runs before every release. For LLM applications that means golden question sets, grounding checks and human-calibrated scoring, not a handful of prompts someone tried in a demo.

3. It is monitored

Inputs drift. Upstream systems change. User behaviour shifts. We track data quality, drift, performance, latency and cost against agreed service levels — with alerts that reach a named human.

4. It fails safely

Every system will meet an input it can’t handle. Production systems know when they’re uncertain and fall back gracefully: a rule, a human reviewer, a polite refusal.

5. It is governed

There is a model card, a risk assessment and an owner. Under POPIA section 71, people have rights around decisions based solely on automated processing — so we design in meaningful human review where it matters.

6. It is operable

Someone knows how to run it at 2 a.m. Runbooks, on-call ownership and incident playbooks exist before launch, not after the first outage.

---

None of this is exotic. It is the same discipline that keeps payment systems and power grids running, applied to AI. That discipline is what we build at Data Rhythm.