Visa’s shopping-agent benchmark exposes a product-matching…

AI models passed 57.2% of Visa’s shopping tests. For small retailers, the findings suggest a product-information readiness check—not a promise of more sales.

Visa released its Visa Agentic Commerce Benchmark on November 5, reporting that 22 AI models from 12 labs passed 57.2% of runs across 15 shopping challenges.

The benchmark tests whether models uncover shoppers’ needs conversationally and appropriately buy or abstain. Visa says 54% of failed runs still produced purchases, either unsuitable products or decisions made without uncovering every requirement. Categories included TVs, laptops and personal care appliances.

Fortune reports that another agent played the shopper, volunteering some requirements and withholding others. The Next Web says passing required identifying every requirement and making the correct buy-or-no-buy decision; lucky guesses didn’t qualify.

Visa reports a strongest pass rate of 82.2% and weakest of 28.9%. Fortune places Google and Meta agents jointly first at a rounded 82%, a Claude agent third at 76%, and an OpenAI agent sixth at 64%.

Visa reports three attempts per scenario and a 34.2% consistency figure. According to The Next Web, the five models uncovering the most hidden buying factors averaged 70.2%, against 44.4% for the five uncovering the fewest. Visa’s five most expensive models per run passed 67.1%; its five cheapest passed 50.2%.

“We’re not there yet from a trust or technology perspective,” Rubail Birwadker, Visa’s global head of growth, told Fortune. The Paypers reports that Visa’s September 2026 Trust Index found 72% of US consumers had used an AI assistant, but only 23% trusted AI to handle payments.

The requirements were specific. Fortune describes TV tests involving compatibility with a child’s gaming console, a 30-day return window and Dolby Vision. The Next Web describes a scenario requiring discovery of budget, HDR10+ support and gaming requirements.

For small retailers, the inference is a product-information readiness check: do listings clearly state attributes, compatibility, price constraints and policies? Unclear details could make matching harder and potentially cost a customer. But this benchmark evaluates agents, not whether listing changes improve small-business visibility to AI buyers, referrals or sales.

Fortune reports that Visa’s public website lets users interact with tested agents. According to The Paypers, Visa has published a research whitepaper, shared the benchmark with model providers for feedback, and plans more categories plus checkout, payment and order-placement testing.

For businesses seeking customers through shopping assistants, the distinction remains material: these results expose product-matching failures, not a demonstrated route to more sales.

Visa’s shopping-agent benchmark exposes a product-matching gap

Sources