I Was Measuring AI Cost. I Should Have Been Measuring the Return.

Before I built software, I spent most of my career selling it. Technology sales, revenue leadership, quota. That work installs a reflex you never really lose: you learn the difference between activity and results, because the difference gets inspected in front of you every quarter. If a sales team's cost per call went down, nobody in the room mistook that for a good quarter. The questions came anyway. What happened to meetings? Pipeline? Conversion? Wins? Revenue? Margin? Did the customers stay?

Cost per call is an activity number. Useful for managing the machine. Never confused with the outcome.

Then I founded LYFYE, started building AI systems in earnest, and caught myself doing exactly the thing I would have flagged in someone else's pipeline review. I was watching token costs and model-call prices like they were the story.

In my defense, those are the numbers you can see. They sit on a pricing page. They show up labeled in a dashboard. They were the numbers I could compare, track, and report, so they were the numbers I watched. And for a while, managing them felt like managing the cost of AI.

Building broke that belief in pieces, and the order the pieces arrived in taught me as much as the lessons did.

The first piece was a benchmark we ran on our own generation system. The question was specific: can a model produce five working screens of an application in one pass, inside a 32,000-token output budget? It could not. Output quality fell as the generation grew. We got code that looked complete and would not run, which I now consider the most expensive kind of failure, because someone has to discover it.

Then we made a change intended to help, reordering the generation sequence, and the same job got materially worse: generated output dropped from about 46,600 characters to about 33,300. A reasonable-sounding improvement, measurably harmful, and the only reason we knew was that we were measuring.

What fixed it was not a cheaper model or a smarter prompt. It was architecture. We split the work so each generation call produces one module inside roughly a 16,000-token budget, with the split decided by plain arithmetic rather than by another model call. Same system, same models, passing cleanly with room to spare.

Here is what I noticed about the cost of that whole episode. The number I remember is not the token bill for the failed runs. I honestly could not tell you that number without looking it up. The cost I remember is engineering days: diagnosing output that appeared finished and was not, then redesigning the system so it could not fail that way again. Nothing on any pricing page had a line for that.

The lessons kept arriving from directions the token math does not cover. Providers time out, return errors, and occasionally return output that parses as almost right. Handling that stopped being an edge case for us and became part of the system itself: bounded retries, an explicit end state for every failure mode we had met, and a hard rule that truncated or malformed output gets refused rather than shipped. Every failed attempt adds some combination of model spend, latency, retry cost, diagnosis, and review. Even when the retry itself is automated, somebody eventually owns the result. All of it lands in the cost of whatever the workflow finally produces.

Human review is the part I most underestimated, and the example that humbled me most recently was not even a model call. We ran an automated cleanup across every page of our website copy. The mechanical work finished in one pass, everything compiled, every check was green. It still took a person reading every changed sentence, and rewriting eighty of them by hand, before the writing was something we would publish. The automation did the task. A person made the result trustworthy. Those review hours are not overhead on the workflow. They are the workflow.

And the bill that actually made me wince had nothing to do with AI at all. A serverless database, configured to wake on activity, was waking constantly. There was nothing intelligent about it, and it taught the same lesson from another direction: a cost you do not gate before it happens is a cost you meet on the invoice. We now check budgets before a model call runs, not after, and the check fails closed. If the system cannot confirm there is budget, the call does not happen.

Somewhere in the middle of all this, the sales reflex finally kicked back in, and I started sorting what we were measuring into the categories I would have used in my old job.

Token spend is an activity cost. So is cost per call. Activity is not outcome. I knew that years ago about dials and meetings. It took building to relearn it about model calls.

An output is not a trusted result. An output is what the model produced. Plenty of it does not survive validation. Some survives validation and still is not something anyone should act on. The result I care about most is the one a specific person will use without redoing the work. Everything between an output and that result, the validation, the review minutes, the failure handling, the occasional redesign, is real cost that belongs to the result.

We even relearned this on the measurement side. We built a small internal tool that scans public conversations for ones we might genuinely contribute to. It works, and on an early run it surfaced keyword matches that looked commercially interesting. A human read of the actual threads showed the strongest matches were not buyers at all. The tool surfaced candidates. Human review determined whether the apparent intent was real. We wrote the lesson down so we would stop rediscovering it.

But the step that changed my behavior most came after all of that, and it is the one I would put on a whiteboard if I only got one: a trusted result can still be a bad investment.

Trust makes a result usable. It does not make it worth producing. Reliability answers whether we can use the result. ROI answers whether we should keep paying to produce it. That second question is the one from my old life. What did this workflow actually create? Did it save real time, net of the review? Did it cut errors? Move conversion? Increase throughput? Earn revenue, or reduce operating cost? Sometimes the honest answer is that it produced learning, which I treat as a legitimate return, with a budget and an end date. And sometimes the honest answer is that it produced nothing defensible, which is not a reason to lower the bar. It is a signal to stop.

We have acted on that signal. This year we froze further development on work we knew exactly how to continue. The prototype was real and the next steps were clear, and I could not point to commercial evidence that the next phase of engineering was justified yet. So instead of quietly continuing, we wrote down the evidence that restarts the work, specific counts of qualified visits, tool starts and completions, movement toward the product, at least one qualified conversation, and we stopped. The finished work is preserved. The engineering waits for the evidence.

People with sales backgrounds will recognize what that is. It is qualification. You never work every deal you could work. You work the ones where the evidence says the effort will return something. Building is the same discipline pointed at your own roadmap, and it is harder, because the person you have to qualify honestly is yourself. Technical feasibility turns out to be a fact about your product. Commercial justification is a fact about the world. Only one tells you whether the market will pay for what you built.

So the questions I ask before committing engineering to an AI workflow have stopped sounding like engineering questions. What result are we trying to create, and who decides whether it is usable? What does the complete workflow cost to produce it, review included? What happens when the output is wrong, and what does that cost? What measurable value did the result create? Does the evidence justify more investment, or does it justify stopping?

I want to be straight about what I do not have. We have not instrumented review minutes across everything we run, so I cannot quote a precise all-in cost per trusted result for each workflow; that measurement discipline is work we still owe ourselves. I do not know whether better models will shrink the review burden or just move it to subtler failures; my experience so far says it moves. And our restart thresholds are considered bets, not laws. The direction I am confident in. The specifics I hold loosely.

What building AI actually gave me was not a new metric. It gave me back an old discipline I already trusted, applied somewhere I had stopped applying it. Businesses do not buy model calls. They buy results reliable enough to act on, and those results are only worth buying when the return beats the full cost of producing them.

If I were still on the selling side of the table, that is exactly the question I would want my buyer asking me. Now I ask it of myself, about my own systems, every time: what did it cost us to produce a result we trust, what did that result return, and does the answer justify doing more of it?