SyncAI.news, a Varaisys broadcasting
Your AI's Confidence Score Won't Catch Products You Don't Carry
SR

Srijith Ravikumar, Forbes Councils Member

· 2 min read

World NewsForbes: Innovation

Your AI's Confidence Score Won't Catch Products You Don't Carry

Srijith Ravikumar is a Principal Engineer at Amazon building AI search and recommendation systems. Published at ACM CIKM and IP&M.

In 2024, a Canadian tribunal ordered Air Canada to compensate a customer who had relied on its website chatbot’s advice about bereavement fares, and it rejected the airline’s argument that the chatbot was responsible for its own actions. Retailers face a version of that exposure: AI recommendations for products the store doesn’t carry. A tempting way to decide when to trust those suggestions is a score the model reports about itself.

Ask a recommendation model how confident it is, set a threshold and drop everything below it. That guardrail is easy to build and easy to report upward. In my tests, it barely worked. Across 48 test settings, filtering on stated confidence cut the share of out-of-catalog suggestions by a median of 0.02 percentage points, and in 18 of them, the filtered list came out slightly worse.

My study measured how often suggestions miss the catalog, not what a miss costs. The likely costs are easy to name. A customer taps a suggestion, finds nothing to buy and trusts the feature a little less. A slot that could have sold something goes to an item that can’t. An assistant that names products the store lacks invites complaints. Treat those as hypotheses to check against your own numbers.

This year I tested four widely used large language models from four different developers. Each suggested 10 items from a user’s history, working from memory with no access to the catalog, on three public datasets covering movies, local businesses and retail toys. I checked every suggestion against the catalog it was meant to come from, using a matching method validated against about 200 human judgments, and counted anything not in that catalog as a miss, whether or not it exists elsewhere.

Within that scope, these are the questions I’d bring to a product review, an AI governance meeting or a decision to fund the next phase.

Original source

This story was published by Forbes: Innovation and written by Srijith Ravikumar, Forbes Councils Member. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on forbes.com

Similar News