Visa released the industry's first AI Smart Shopping Evaluation, simulating real consumer scenarios to assess mainstream AI model performance and opening the testing platform to consumers. The first-round evaluation revealed AI shopping achieved an average pass rate of only 56.6%, translating to a failure rate of 43.4%, with overall performance stability proving insufficient. The evaluation covered 22 AI models from 12 development teams across 15 shopping scenarios including televisions, laptops, and personal care appliances, with pass rates ranging from 28.9% to 82.2%.
Visa Establishes AI Shopping Evaluation Standards and Testing Methodology
Visa's AI Smart Shopping Evaluation standard requires AI to accurately identify actual consumer needs through conversation. If no products meet the specified conditions, AI must not place orders. The benchmark testing specifically requires AI to collect sufficient information, avoiding decisions based solely on speculation before fully understanding consumer requirements. Each AI model underwent three tests across various shopping scenarios including electronics and personal care appliances.
Evaluation Results Show Wide Performance Variance Across AI Models
The first-round evaluation tested 22 AI models from 12 AI development teams across 15 shopping scenarios. The average pass rate reached only 56.6%, with the best-performing model achieving 82.2% and the lowest scoring just 28.9%. The five AI models that identified the most potential purchase needs through consumer dialogue achieved an average pass rate of 68%, while the five models capturing the fewest needs scored below 50% at 44.4%. The five highest-cost models per test averaged 65% pass rates, compared to 50% for the five lowest-cost models.
Key Findings Reveal Consumer Understanding and Stability Gaps
Comprehensive testing across all models and scenarios showed only 33% of tests passed all three repeat attempts, indicating insufficient AI model performance stability. Approximately half (53%) of failed tests still resulted in AI deciding to proceed with purchases, including recommending products not meeting consumer requirements or failing to fully capture all purchase conditions. Visa noted consumers must remain vigilant regarding these failure patterns.
Visa Plans Expanded Benchmarks and Industry Collaboration
Visa Taiwan General Manager Huang Hui-chin stated that Visa believes AI will lead commerce into a new era, with trust serving as the key to transforming future commerce blueprints into reality. The AI Smart Shopping Evaluation standard represents Visa's concrete implementation of continuously promoting industry standard improvements. Visa identified reducing errors and improving stability as critical points for building consumer trust in agentic commerce, with both areas requiring improvement. Visa plans to continue expanding agentic commerce evaluation benchmarks, incorporating more shopping scenarios and product categories, extending testing scope to checkout, payment, and order completion processes. Visa has shared the evaluation benchmarks with leading AI model suppliers to collect relevant feedback.
FAQ
What did Visa's AI Smart Shopping Evaluation reveal about AI model performance?
Visa's first-round AI Smart Shopping Evaluation showed an average pass rate of 56.6% across 22 AI models from 12 development teams, meaning a failure rate of 43.4%. Performance varied significantly, with the best model achieving 82.2% and the worst only 28.9%. Only 33% of tests passed all three repeat attempts, indicating stability issues.
Why did some AI models fail Visa's shopping evaluation?
Approximately 53% of failed tests occurred because AI decided to proceed with purchases despite not meeting requirements, either recommending products not matching consumer specifications or failing to fully capture all purchase conditions. Models that identified fewer potential purchase needs through dialogue achieved lower pass rates (44.4%) compared to those capturing more consumer requirements (68%).
How does Visa plan to improve AI shopping standards?
Visa plans to expand agentic commerce evaluation benchmarks by incorporating more shopping scenarios and product categories, and extending testing to checkout, payment, and order completion processes. Visa has shared evaluation benchmarks with leading AI model suppliers to collect feedback and aims to reduce errors and improve stability in AI shopping systems.
