Results of the All.html Test

The much awaited results of the all.html redesign test are here!

What were we trying to do?: (You can skip to the Results section if you already know about this ;))
In September of last year, I started looking at the current all.html page to see if there were ways to optimize the page. We had a high bounce rate (40%) and I felt that the page was visually confusing and that there was a better way to display the information listed.

I then asked The Royal Order (an amazing design firm) to create three designs to address the problem of trying to display 60+ Firefox localizations and their platforms to a worldwide community of people with different needs (ie, language spoken, ability to read English, various levels of tech-savvy, country of download, etc).

These were the designs we received:

All is the current all.html page. (This picture is of the older page because we had already taken the all.html page out of circulation by publication).
All-1 was our most "designed" of the pages that included a map. ((This picture is of the psd because we had already taken the all-1.html page out of circulation by publication).
All-2 was in the middle, with some geographical images and expandable lists.(This picture is of psd because we had already taken the all-2.html page out of circulation by publication).
All-3 was the most like the current all.html with some changes to some details like # of columns.

After looking at them, we were unable to decide on face value which one should be the new replacement. We all had plenty of opinions, but nothing based on verifiable data. Consequently, we decided that we would test all three designs (and the current design) to see which one performed the best.

The Test:
We set up a week long test that ran from March 21-29, 2009. We rotated the pages evenly so each page showed for 15 minutes of every hour for the entire period. We defined success or "best performance" as:
1) Best User experience on the page comparing to all.html
2) Best download rate on the page comparing to all.html
(You can read more details about the test/process here and here.)


The Results:

The results are organized by version of all#.html and broken up into page views and downloads. The conversion rate was found by dividing download clicks by page views. The baseline we used for all comparison was the all.html page. The baseline, therefore, is that 52.7% of people who hit that page downloaded a version of Firefox. Better Performance would then be defined as anything above the average of 52.7%.

All-1.html performed much worse than expected, with 32.4% of people downloading Firefox after hitting that page. All-2.html, performed at 56.5% which was on par with my expectations.
Lastly, All-3.html was comparatively stellar, getting about 60.9% of the people who hit that page to download. This basically translates to an 8% improvement over our current all.html page.

Now, before we get TOO excited about these results, resident Metrics Guru, Ken Kovash, asked me to do some further digging. Basically, he was concerned because "download clicks" represent a snap shot of a site visitor’s experience at a single moment in time and it doesn’t factor in what actions a person takes after that moment (e.g., a download occurring when that person subsequently navigates away to other pages within Mozilla.com). A Firefox user could hit the all.html page, then move on to the mozilla.com homepage, and then click download. In this example, the action of a download is not counted in our numbers and results above.

To figure this out, I looked at a funnel report in our web analytics system for all-3.html and all.html to try and figure out where people were going after the all.html pages to download (if not from the actual page) and to integrate these numbers into our results above.

All.html

First column you see below is total number of page views.
Second column is the total of downloads from the all.html download buttons (all of them).
Third columns is the total of downloads from the entry page (mozilla.com)
Fourth column is the total of downloads from download buttons that are not on one of our top pages (eg customize).
Fifth column is total after subtracting colunms 2, 3, 4.
Sixth column is the number on the total downloads/page views which results in an average of 67.6% conversion rate.

All-3.html

First column you see below is total number of page views.
Second column is the total of downloads from the all.html download buttons (all of them).
Third columns is the total of downloads from the entry page (mozilla.com)
Fourth column is the total of downloads from download buttons that are not on one of our top pages (eg customize).
Fifth column is total after subtracting columns 2, 3, 4.
Sixth column is the number on the total downloads/page views which results in an average of 68.4% conversion rate.
Seventh column is the difference in conversion rates, with the average difference between all-3 and all.html being .89% (in favor of all-3.html).

Synopsis
All in all (pun intended) all-3.html was the best performer of the four pages. The 8% difference in user experience and .89% improvement in download rate (comparing to all.html) shows that even a small change in a page's design can have an effect on download numbers. And even though 8% improvement in ease of use or a .89% improvement in downloads may seem small, considering the number of people who see this site every day (70-90K page views a day) the effort and hardwork that went into this redesign and test were well worth it.

All-3.html has now taken the place of the old page and I'm looking forward to tweaking and testing some details and re-checking its' performance in the next few months.

Thanks to everyone who has put in so much hardwork these past two quarters: The Royal Order, Steven Garrity (Silver Orange), Wil Clouser, Jeremy Orem, Stephen Donner, Krupa Raj, Ken Kovash, Seth Bindernagel, Mike Morgan and John Slater--couldn't have finished this project without you!