Reaching the Summit of Web Performance with otto.de

It continues with the possibility to parameterise scripts within each job and the ability to set the execution time of measurement jobs flexibly using crontab expressions.

Last but certainly not least OpenSpeedMonitor provides the ability to configure required network connection profiles for measurement.

A key part of the software, and an area where we have actually enriched the functionality of the WebPagetest tool itself, is support for multi-step measurements that follow a complete customer journey, measuring and logging each individual page view separately. That might be an ordinary feature for some commercially available tools such as Keynote or Gomez. But for the vast number of people who use WebPagetest to measure site performance, this is a breakthrough improvement over the status quo.

For one thing, the agent infrastructure can be used far more efficiently if an entire customer journey is taken and measured in each process. In addition, and far more importantly for practice, is that the First and Repeated View measurements of WebPagetest do not reflect the customer‘s reality because they are unable to properly depict the real caching effects in the browser. The First View gives an excessively negative picture of load times, since the measurement is taken with an empty browser cache. In reality, this is a rare situation. Starting with the second page view in the customer journey, the circumstances are very different, because the browser cache is not empty any longer. All commonly used artefacts can be loaded from the Browser cache now.

In contrast, the Repeated View presents an overly optimistic picture, because an identical page is viewed for a second time unchanged, something that web users will rarely do. Analysis of www.otto.de access statistics shows that these two cases account for little more than 5% of the page views of OTTO customers. Thus in about 95% of all customer page views, the real performance lies somewhere in between. By modeling a typical customer journey in our measurements we can depict reality far better.

One unique outstanding feature of the OpenSpeedMonitor is, that it calculates business metrics such as the Customer Satisfaction Index (CSI) on the basis of the measured page load times, essentially converting each page load time to a percentage expression of customer satisfaction. In other words, we measure the percentage to which OTTO customers would have been satisfied with the page load time.

Oliver: Can you give us some details on the customer satisfaction index?

Nils and Uwe: Everyone knows that page performance has an impact on customer satisfaction. However, using page load times to draw reliable conclusions on customer satisfaction is anything but straightforward. The few simple yardsticks on the market – such as “a one-second deterioration in load time causes a 7% loss of sales revenue” – might be readily understandable, but fall way short of reflecting reality. Unfortunately, the correlation is non-linear. The performance expectations of customers are contingent on the pages themselves. Expectations increase as greater use is made of www.otto.de by customers. And the situation is evolving. Today, OTTO customers are generally more impatient than they were two years ago.

OTTO has carried out extensive studies on this question and collected substantial data on the correlation between page load times and customer satisfaction. This investigation has recently been repeated in order to obtain fresher data on customer expectations. As a result, we now have conversion tables that can tell us the customer satisfaction percentage for any given page load time.