A working prototype is genuinely valuable: it proves the idea is possible. What it does not prove is that the idea survives real inputs, real latency budgets, real cost per call, and a model provider changing something under you. Those are the four things that turn a demo into a feature, and none of them show up in a notebook.
Decide whether it should be a model at all
Plenty of problems that get framed as AI problems are better solved by a rule, a database query, or fixing how data gets entered in the first place. This is worth establishing before anyone commits to inference costs, because a model is the most expensive and least predictable way to implement logic you could have written down.
When the answer genuinely is a model, that conclusion is now defensible rather than assumed.
Agree how you will measure it before you build it
Without an evaluation set, "is it working?" is answered by whoever demoed it most recently. Build a held-out set of real examples with expected outputs, and decide what score is good enough to ship. This is unglamorous and it is what lets you change a prompt or swap a model later without guessing whether you made things worse.
- A held-out set drawn from real inputs, not invented ones
- An agreed scoring method, automated where possible
- A threshold that means "ship" and one that means "stop"
Version prompts like code
Prompts are logic. When they live in a string in a file with no history, a one-word change becomes an untraceable behaviour change. Version them, tie each version to an evaluation run, and record which version produced which output in your logs.
Budget for cost, latency and failure
Every call has a price and a delay, and both scale with usage in a way that a prototype never reveals. Set explicit budgets, then decide what happens when the provider is slow or down - a cached response, a smaller model, a graceful message. Deciding that in advance is the difference between degraded and broken.
Then monitor after launch. Model behaviour drifts, input distributions shift, and the failure mode is quiet: the feature keeps returning confident answers that are gradually less correct.
The prototype answers "can this work?". Everything above answers "will this keep working, at a price we can predict, when we are not watching?" - which is the question the business is actually asking.