Merchants Rethink Online Shopping Metrics

Merchants adopting AI-driven shopping tools may need to evaluate different performance indicators and timing. TikTok product designer Zhengyang Yang notes traditional ecommerce analytics often rely on the idea that customers locate items through page comparisons and direct searches, yet AI alters that process entirely.

Yang wrote in an email exchange that a shopper who performed five searches and visited 10 product pages might now describe their requirements once, compare three recommendations, and proceed directly to a purchase. From the shopper’s perspective, less interaction is progress, but in an analytics dashboard, it might look like decline.

Measuring Engagement

An AI-assisted session could generate lower search volume, fewer product-page visits, and shorter sessions even when the shopper is having a better experience. Ecommerce marketing teams have often treated activity as a proxy for interest, with more product views suggesting consideration and more search activity indicating interest in a catalog.

However, the same numbers can mean something else. Five queries could mean the search experience is poor, while ten product-page visits could indicate incomplete product information. A long session could represent confusion, rather than interest.

On-Site AI Assistants

When a merchant launches an embedded AI assistant, the first metrics usually measure feature use rather than the shopping experience. Typical examples include AI conversations, messages per session, time spent interacting, and repeat usage. These metrics show whether shoppers use the AI assistant, but they do not show that it works.

Yang noted that a shopper exchanging 10 messages with an assistant might indicate strong interaction, but it could also mean the assistant repeatedly misunderstood her needs. The idea that less interaction is better may seem counterintuitive, but an embedded shopping assistant often aims for the opposite.

Evaluating AI Assistant Success

Feature use matters. If shoppers ignore an assistant, the merchant has a problem.

Merchants can measure behavior. The metric to track may depend on the shopping task.

Outcomes are difficult to summarize in a single dashboard number.

Measuring Decision Quality

A better way to measure an on-site AI assistant, according to Yang, is to track decision quality. This includes whether the AI tool helps shoppers identify relevant products efficiently, compares products accurately, and turns product recommendations into conversions.

Repeated searches may suggest that the first results missed the shopper’s intent, while repeatedly reformulating a request could show the same thing. Controlled experiments can help too, by comparing AI-assisted and conventional shopping experiences for similar purchase intentions.

Yang recommended examining product returns, customer satisfaction surveys, and repeat purchasing behavior when evaluating an AI shopping assistant. The theory is that an assistant that helps shoppers buy faster may still perform poorly if those purchases lead to relatively more returns or disappointed customers.