The demo that never gets checked
You have probably seen the demonstration. Someone types "show me last month's revenue" into a chat box, the AI writes a query, a chart appears, and everyone is impressed.
What the demonstration never includes is somebody verifying the number against the books.
When you do verify, on real business data, the failure rate is high enough to matter. Not because the technology is fake, but because writing a query and understanding a business are different skills, and only one of them can be inferred from a database schema.
Why your database lies about itself
Take any serious accounting or ERP system. Its structure reflects double-entry bookkeeping, not plain English, and that mismatch is where the errors live.
Ask a naive AI for last month's revenue and it will find something that looks like a sales invoice table, add up a column that looks like a total, and hand you a figure. That figure will not account for credit notes that reverse part of the sale. It will not exclude invoices still sitting in draft. In many systems, properly reconciled revenue lives in the general ledger rather than the invoice table at all.
Stock valuation is stored somewhere non-obvious too. And gross margin usually is not stored anywhere - it has to be reconstructed from cost of goods sold. None of this is exotic. It is simply how the system works, and a model that has never seen your configuration cannot know it. So it does the reasonable thing available to it: it guesses, fluently.
Why a confident wrong number is worse than none
Software that fails loudly costs you time. An error message, a blank screen, a call to support - irritating, but contained.
Software that fails quietly costs you decisions. You price against the wrong margin. You reorder against the wrong movement figure. You conclude a branch is doing well when it is not. And because the output looked exactly like every correct output, nothing prompts you to check.
This is why "the AI is right most of the time" is not a reassuring statistic in reporting. If you cannot tell which answers are in the wrong minority, an eighty-percent accuracy rate is not a tool - it is a trap with good manners.
The pattern that actually works
The fix is unglamorous and it works. Instead of letting the AI invent its own calculations, you give it a curated catalog of business measures - and a human defines each one properly, once.
- Revenue and gross margin, by item and category, over any period.
- Stock turnover and days of cover remaining.
- Dead and slow stock, with capital tied up.
- Best and worst movers, ranked on profit.
- Supplier price variance for the same item.
- Receivables aging and current till position.
Catalog first, and mean it
A catalog only protects you if the tool is actually required to use it. The rule that matters is that the AI must check the defined measures before anything else, and may only write its own query when it can state that no defined measure answers the question.
When it does improvise, three guardrails should apply. The query must be read-only, so nothing can be changed. It should be fenced with a row limit, a timeout and a hard scope to your company's data. And it should show its working - the query it ran and the assumptions it made - so you or your support partner can sanity-check it.
That last point costs the vendor something. It openly admits that this answer is less certain than the others. It is the right trade, because the alternative is a system where every answer looks equally confident and some of them are not.
How to test a tool before you trust it
You do not need technical skill to run a sensible evaluation. You need a figure you already know cold.
- Pick a month you have closed and verified. Ask the tool for its revenue and compare.
- Ask the same question two different ways and check the answers match each other.
- Ask for a figure involving cost - gross margin is the classic - since that is where naive tools break.
- Ask something deliberately awkward and see whether it admits it cannot answer.
- Ask it to show its working, and see whether that option exists at all.
- Repeat the spot-checks monthly for a season, not just during the trial.
Where Upeosoft stands on this
We build business systems for Kenyan companies, and we add AI to them where it earns its place. Our position on reporting is firm: an AI must reason over defined, verified business measures, not improvise against a live database.
That is slower to build. It means somebody sits down and works out exactly how gross margin should be computed in your setup, and writes it down, and tests it. It is also the only version of this that we are willing to let a business make pricing and purchasing decisions on.
If you are evaluating an AI reporting tool, or you have one and something about the numbers feels off, talk to us. Half an hour of checking is cheaper than a quarter of decisions built on a confident wrong figure.
