What the History of Web Analytics Can Help Us Predict About LLM User Analytics

I. Introduction: The Evolution of Web Analytics and the Rise of LLM Analytics

Back in 2015 I was involved in one of Europe’s largest Adobe Analytics implementation projects. By that time web analytics had already been around for more than 20 years. Thefirst web analytics tools, such as WebTrends, had been introduced in the mid-1990s, followed by Omniture and Urchin, the precursors to Adobe and Google Analytics. Despite decades of progress, many businesses still hadn’t fully embraced the importance ofinvesting in tools to truly understand their websites — one of their most important sales andmarketing channels.

Now, we’re at a similar crossroads with conversational interfaces powered by large language models (LLMs). These interfaces are creating new opportunities for businesses to engagewith users, but they also present a challenge: just like early websites, they’re a “black box”when it comes to understanding user interactions. To thrive in this new landscape, we can learn from the past — applying the lessons of web analytics to the rapidly evolving world ofLLM user analytics.

II. The Early Days of Web Analytics: Learning to See the Data

In the early days of the World Wide Web, websites were a mystery. Businesses knew users were visiting, but they had little understanding of what those visitors were actually doing. Thefirst web analytics tools—such as hit counters and basic traffic trackers — focused on simple metrics like page views and site visits. These tools gave businesses a glimpse into their audience’s behaviour, but they didn’t reveal much about why users were visiting, what they were looking for, or how effective the site was at meeting their needs.

As websites became more sophisticated, businesses needed deeper insights into user behaviour. Advanced analytics tools like Adobe Analytics and Google Analytics began to track user journeys, conversions, and engagement metrics. These platforms moved beyond surface-level metrics to help businesses understand the “why” behind user actions, allowingfor optimization of content and user experience.

III. The Black Box of Conversational Interfaces Today

Today, we find ourselves in a similar situation with conversational AI. Chatbots, conversational interfaces and AI agents powered by LLMs are becoming integral to how businesses engage with customers, but the interactions they generate remain largely opaque. We can see that users are engaging with these systems, but understanding the intricacies of these conversations — what users are asking, how they feel, and whether theirneeds are being met — continues to be a challenge.

This is where LLM user analytics comes in. Just as web analytics unlocked the mysteries ofearly websites, LLM analytics will help businesses decode the conversations taking place intheir AI-driven interfaces. By analysing the content, flow, and outcomes of these interactions and measuring implicit and explicit user feedback, businesses can optimise their conversational AI systems to deliver better experiences and drive greater value.

IV. Parallels Between Web and LLM Analytics: Lessons We Can Apply

One of the key lessons from web analytics is the need to move beyond basic metrics. In theearly days, page views and hit counters provided a starting point, but they didn’t tell the fullstory. Similarly, simple metrics like interaction count or session length aren’t enough to understand conversational interfaces. Just as web analytics evolved to track user journeys, LLM analytics needs to map conversational paths, analyse user sentiment and understand intent at a deeper level.

Another important lesson is the role of tools and platforms in driving analytics maturity. The introduction of sophisticated web analytics platforms like Adobe and Google Analytics transformed how businesses approached online optimisation. Today, new platforms like Context AI and Nebuly are emerging to provide similar capabilities for LLMs — allowing businesses to track, measure, and improve their conversational AI systems.

V. The Importance of Investing Early: How Early Adopters Benefit

History shows us that businesses that invested early in web analytics gained a significant advantage. By understanding user behaviour and optimising their sites accordingly, these early adopters were able to attract more visitors, increase conversions, and outpace their competitors. The same will hold true for LLM user analytics. Companies that embrace these tools now will be able to deliver better conversational experiences, improve customer satisfaction faster and drive tangible business results.

VI. Future Trends in LLM User Analytics: Predictions Based on Web Analytics

Looking ahead, we can expect LLM user analytics to follow a similar trajectory to web analytics in several key ways. As conversational AI continues to evolve, personalisation will become increasingly important, allowing businesses to use real-time insights to tailor interactions to individual users. Just as web analytics empowered companies to understand user journeys and behaviours on their websites, LLM user analytics will allow businesses to optimise conversations for better user experiences.

A significant shift will occur once we move from descriptive analytics — which simply explains what is happening in conversational experiences — toward interpretive, predictive, and prescriptive analytics. In the early days of web analytics, businesses relied on basic metrics to understand their website traffic. Over time, the focus shifted to more advanced insights, such as predicting user behaviours, identifying what would likely happen next, and prescribing specific actions to improve outcomes. The same evolution is now beginning with conversational interfaces, where we will move from analysing conversations retrospectively to using AI-powered tools that help us predict user needs and prescribe actions to optimise interactions in (near) real time.

However, insights — whether from web analytics or conversational interfaces — only provide value if they are followed by action. Generating insights alone incur costs for a business without delivering any measurable benefit. In the world of web analytics it took some time but eventually businesses learned that experimentation was key to realising the value of these insights. The rise of A/B testing and other methodologies enabled businesses to test hypotheses, compare outcomes, and continuously optimise their websites and apps for better results.

The same principle is likely to apply to LLM user analytics. As businesses gather more insights into their conversational AI interactions, the value will come from using thos einsights that drive continuous optimisation and inform the next best action. Without action, there is no value, and without value, there is no benefit to the business. Continuous experimentation, powered by insights, will be essential to driving improvements in conversational experiences, ultimately delivering greater satisfaction for users and measurable benefits for the business.

VII. Conclusion: From Web Analytics to LLM Analytics — The Journey Continues

The evolution of web analytics provides a valuable playbook for navigating the rise of LLM user analytics. Just as businesses once unlocked the power of user behaviour on websites and apps, they now face the opportunity to do the same with conversational interfaces. However, the key to success lies not just in gathering insights, but in taking action.

The real value of analytics has always been in its ability to inform decisions and drive continuous optimisation. For businesses ready to embrace this next frontier, the path forwardis clear: analyse, experiment, optimise, and let insights guide your next best action for your conversational AI. Those who act on these lessons will be well-positioned to thrive in the future of AI-driven interactions.