Understanding Web Performance Test Results Part 2
Guest Blogger: Arthur Zey
Last time, I discussed a few important considerations when evaluating performance test results from multiple CDNs: backbone versus last-mile testing and why the selection of performance monitoring company matters. In this installment, I’m focusing on two related issues: testing the performance of dynamic versus cacheable content and running full-page versus single-object tests.
Testing Dynamic Content vs. Cacheable Content
When an HTTP request comes in to a CDN for a particular object, there’s no way for the CDN to know in advance whether the request can be served from cache or whether the object needs to be generated anew by the origin on account of its dynamic nature.
All CDNs have their own method of performing this differentiation, but it often requires coordination with the customer to optimize the configuration. What’s important during testing is that both CDNs use the same rules. If one caches dynamic objects that would never be cached in production, this can dramatically skew results in their favor–this is, of course, outright fraudulent. But discrepancies can happen innocently, as well: if two CDNs don’t cache an object for the same length of time or they don’t agree on which objects are cached in a full-page test, this can lead to distortions in the test results. Make sure your CDNs are using the same caching rules.
And if you have a rich, dynamic site, requiring your origin server to generate content on every request, do not be satisfied with doing a mere comparison of cacheable performance (or even a full-page test where the dynamic base HTML page is overshadowed by the cacheable objects referenced by the page). Only a few CDNs offer middle-mile acceleration for dynamic delivery, and among them, they rarely have comparable performance, especially in the more difficult emerging markets. Demand that the CDNs prove the performance of their dynamic delivery offering.
Full-Page vs. Single-Object Testing
One of the easiest ways to remove noise in test results is to do single-object testing. Of course, full-page testing best approximates user experience, but it also blends together many different factors that can be hard to separate.
Like the problem with last-mile testing, full-page testing often introduces statistical noise that no CDN can control: the delivery of third-party objects, and they can dominate test results, as we’ve seen on several occasions. Third-party objects are those that are called from domain names that are usually not owned by the customer and therefore cannot be accelerated through the CDN in production. Examples include analytics, user tracking services, and off-site content.
Along those lines, sometimes the HTML of the page being tested has references to objects that would be delivered by the CDN in production, but those references specify an absolute path instead of a relative path. The absolute path causes the object to be treated as if it were a third-party object, since the path’s domain name during a trial is the customer’s origin domain and not servable by the CDN. Some CDNs rewrite the HTML during such tests to avoid this problem, so make sure that both CDNs are doing the same thing so it is a true comparison.
Another factor to be aware of is if both full-page tests are taking measurements for the same objects. Make sure to inspect a listing of all the objects measured for both CDNs. Otherwise, the comparison is invalid.
If all that is not reason enough to prefer single-object testing, one of the most important benefits is that single-object testing allows you to zero-in on the caching and dynamic performance separately for each CDN, whereas a full-page test shows blended performance. Knowing how CDNs stack up against one another for each of their solutions is very valuable in making an ultimate selection.
That’s all for now. Next time, I’ll cover a few obscure issues that don’t get a lot of attention, but can nevertheless make a significant impact in your analysis: DNS resolution time, the byte size of delivered objects, and disproportionate averaging of monitoring agents.
Sharon BellCDNetworks, Director of Marketing, Americas & EMEA
