Understanding Web Performance Test Results Part 3
Guest Blogger: Arthur Zey
Last time, I wrote about two of the major decisions you will confront in crafting a testing strategy: testing cacheable versus non-cacheable content and using backbone or last mile testing. In this installment, I cover a few miscellaneous “gotcha”s to be aware of, both before and after running your tests: the effect of the DNS resolution time on total performance, the effect of different compression algorithms on the content being tested, and avoiding stacking the deck with testing agents that don’t proportionately reflect the regions you care about.
DNS Resolution Time
Oftentimes, during a trial, CDNs will configure an alternative domain name for you to use that shows how your site would run through their service. For example, it may be of the form www.site.com.CDNsuffix.net. Or sometimes, they will ask you to CNAME a subdomain off your main domain to that alternative domain name. For instance,cdn.site.com IN CNAME www.site.com.CDNsuffix.net.
Make sure that both CDNs do the same thing. If you compare one CDN by CNAMEing a subdomain and another by directly accessing the alternative domain name, then the DNS resolution times in the performance test results will be skewed. Unless you’re already using a cloud-based DNS solution, CDNs almost always have faster DNS resolution around the globe than any particular customer might have in their datacenter or provided by their ISP. If one CDN is using that CNAMEd subdomain, it will be penalized by the poorer performance of the first DNS lookup of cdn.site.com, whereas the CDN accessed with the alternative domain name directly will have the benefit of all DNS resolution happening on their own fast servers that are authoritative for CDNsuffix.net.
Also, be sure that you use similar test patterns for each trial. If you are comparing your existing DNS resolution on your production site to an alternative site, and only your production site has users, your production DNS entry will be cached, while the other site’s DNS entry may fall out of cache. This can mislead you into thinking one site’s configuration is remarkably faster than the other’s. Be sure to account for this effect if you have different use patterns for each scenario.
Different servers compress differently. It’s not uncommon for the origin and different CDNs to deliver a different number of total bytes. This isn’t even necessarily a function of more efficient compression protocols–some of it is random luck.
If you can, test single objects that are already compressed, such as images or videos. Avoid testing highly compressible objects, like plain text files, as that will magnify any potential compression differentials. Alternatively, depending on the capabilities of the testing company, you may be able to send a directive to the server (through an HTTP header) to disable compression, thereby giving you a fair byte-for-byte comparison.
The most important thing is to simply look at the total number of bytes transferred on any performance report and factor that into your overall evaluation, even if you have to approximate.
Every CDN wants to put their best foot forward, so naturally, they will want to present you with results where they include many more data points from regions where they perform better.
Even if a CDN simply includes all the agents available with a particular testing company, since the results are often presented as a simple average across all agents, those results are often skewed toward where there are more agents. On the one hand, that’s usually where there may be more global end users; on the other hand, this hides any difference in performance between two CDNs in emerging and challenging markets, which may be your specific need.
It’s not easy to do weighted averaging, nor is it always easy to generate reports for every region of interest. So make sure that you’re clear with the CDNs which regions you want to test in, and be as involved as possible in selecting the agents you want to test on.
CDNs can provide a dramatic benefit to your organization, but selecting among them is no trivial matter. Performance analysis in particular is a narrow field requiring specialized knowledge! You don’t need to be an expert in Internet technologies to make sense of performance test results, but you do need to be aware of the issues in my posts to make sure that you’re able to ask the right questions and objectively assess the relative value of different CDNs to your business.