Testing AI features is different — here's my setup
You can't assert on a model's exact output. You can assert on structure, invariants, and failure behaviour. That's where tests go.
By Nzubechukwu Cyprian · Pharmacy student at UNN, full-stack developer
Traditional tests assert exact outputs. AI outputs are non-deterministic, so teams either skip testing or test by vibes. Both are wrong. There's a middle path I use on every AI feature: test the contract, not the sentence.
The three layers
Schema tests. If the feature promises JSON with keys A, B, C, then every run must produce exactly that. Invalid output is a test failure regardless of how wise the text was.
Invariant tests. Some properties must always hold. A summariser's output must be shorter than the input. An extractor's prices must parse as numbers. A classifier's labels must come from the allowed set. These are assertable, and they catch most real damage.
Failure tests. Feed garbage input. Kill the network. Return a malformed response from the model and check that the fallback path works. In production, the failure path runs more often than anyone's demo assumes.
What I deliberately don't test
Exact wording. Tone, partially. Anything that would make every prompt tweak break the suite. The model is allowed to vary; the contract is not.
The reframe: you're not testing the model, you're testing your system's promises. The model is weather. Your contract is architecture.
Keep reading
- AI
The model is not the product
Everyone debates which model is smartest. Almost nobody talks about the ninety other things standing between a prompt and a shipped feature.
- AI
My AI workflow is boring, and that's the point
No autonomous agents rewriting my codebase. No prompt wizards. Just a disciplined loop between a draft, a review, and a test suite.
- AI
Prompting is just requirements gathering with worse vocabulary
The teams that get good results from AI aren't better at talking to models. They're better at saying what they want.