Thursday, April 2, 2009

Results of the All.html Test

The much awaited results of the all.html redesign test are here!

What were we trying to do?: (You can skip to the Results section if you already know about this ;))
In September of last year, I started looking at the current all.html page to see if there were ways to optimize the page. We had a high bounce rate (40%) and I felt that the page was visually confusing and that there was a better way to display the information listed.

I then asked The Royal Order (an amazing design firm) to create three designs to address the problem of trying to display 60+ Firefox localizations and their platforms to a worldwide community of people with different needs (ie, language spoken, ability to read English, various levels of tech-savvy, country of download, etc).

These were the designs we received:

All is the current all.html page.
(This picture is of the older page because we had already taken the all.html page out of circulation by publication).
All-1 was our most "designed" of the pages that included a map. (
(This picture is of the psd because we had already taken the all-1.html page out of circulation by publication).
All-2 was in the middle, with some geographical images and expandable lists.
(This picture is of psd because we had already taken the all-2.html page out of circulation by publication).
All-3 was the most like the current all.html with some changes to some details like # of columns.

After looking at them, we were unable to decide on face value which one should be the new replacement. We all had plenty of opinions, but nothing based on verifiable data. Consequently, we decided that we would test all three designs (and the current design) to see which one performed the best.

The Test:
We set up a week long test that ran from March 21-29, 2009. We rotated the pages evenly so each page showed for 15 minutes of every hour for the entire period. We defined success or "best performance" as:
1) Best User experience on the page comparing to all.html
2) Best download rate on the page comparing to all.html
(You can read more details about the test/process here and here.)


The Results:


The results are organized by version of all#.html and broken up into page views and downloads. The conversion rate was found by dividing download clicks by page views. The baseline we used for all comparison was the all.html page. The baseline, therefore, is that 52.7% of people who hit that page downloaded a version of Firefox. Better Performance would then be defined as anything above the average of 52.7%.

All-1.html performed much worse than expected, with 32.4% of people downloading Firefox after hitting that page. All-2.html, performed at 56.5% which was on par with my expectations.
Lastly, All-3.html was comparatively stellar, getting about 60.9% of the people who hit that page to download. This basically translates to an
8% improvement over our current all.html page.

Now, before we get
TOO excited about these results, resident Metrics Guru, Ken Kovash, asked me to do some further digging. Basically, he was concerned because "download clicks" represent a snap shot of a site visitor’s experience at a single moment in time and it doesn’t factor in what actions a person takes after that moment (e.g., a download occurring when that person subsequently navigates away to other pages within Mozilla.com). A Firefox user could hit the all.html page, then move on to the mozilla.com homepage, and then click download. In this example, the action of a download is not counted in our numbers and results above.

To figure this out, I looked at a funnel report in our web analytics system for all-3.html and all.html to try and figure out where people were going after the all.html pages to download (if not from the actual page) and to integrate these numbers into our results above.

All.html


First column you see below is total number of page views.
Second column is the total of downloads from the all.html download buttons (all of them).
Third columns is the total of downloads from the entry page (mozilla.com)
Fourth column is the total of downloads from download buttons that are not on one of our top pages (eg customize).
Fifth column is total after subtracting colunms 2, 3, 4.
Sixth column is the number on the total downloads/page views which results in an average of 67.6% conversion rate.

All-3.html

First column you see below is total number of page views.
Second column is the total of downloads from the all.html download buttons (all of them).
Third columns is the total of downloads from the entry page (mozilla.com)
Fourth column is the total of downloads from download buttons that are not on one of our top pages (eg customize).
Fifth column is total after subtracting columns 2, 3, 4.
Sixth column is the number on the total downloads/page views which results in an average of 68.4% conversion rate.
Seventh column is the difference in conversion rates, with the average difference between all-3 and all.html being .89% (in favor of all-3.html).

Synopsis
All in all (pun intended) all-3.html was the best performer of the four pages. The 8% difference in user experience and .89% improvement in download rate (comparing to all.html) shows that even a small change in a page's design can have an effect on download numbers. And even though 8% improvement in ease of use or a .89% improvement in downloads may seem small, considering the number of people who see this site every day (70-90K page views a day) the effort and hardwork that went into this redesign and test were well worth it.

All-3.html has now taken the place of the old page and I'm looking forward to tweaking and testing some details and re-checking its' performance in the next few months.

Thanks to everyone who has put in so much hardwork these past two quarters: The Royal Order, Steven Garrity (Silver Orange), Wil Clouser, Jeremy Orem, Stephen Donner, Krupa Raj, Ken Kovash, Seth Bindernagel, Mike Morgan and John Slater--couldn't have finished this project without you!

65 comments:

Stephen said...

I'm up in your projects, launching your web pages :-)

But seriously, though, looking forward to working with you on the partnership stuff; can you send it my way when you get a chance? Thanks!

Congrats on shipping!

Tomer said...

I think the best thing to say is "We told you!", as most of the commenters said in the bugs. maybe next time a download page redesign survey would be enough. :)

Some thoughts:
a. How many of these users actually got the map page, than navigated to another page and when they came back to the download page they got another design? It might be possible that some people that was unable to locate the download in the first two designs found how to complete the download process only for another design, and the rise in the download count for the 3rd design was due to bad download rates on the 2nd design...

b. When I investigated the download rates on our local download page, it come to my attention that some people are clicking twice on the download link, fast enough to treat it as a double-click, while the analytic software we use (Google Analytics) treat it as two separated clicks. Checking what are their current browser told me that most of them are IE6 users who probably just heard about Mozilla Firefox.

We have two different Firefox download pages on our community sites in Israel, and while the download rates are much smaller, I would be happy to share with you some of our stats which may give some more knowledge about how the users are finding the download buttons useful -

A. mozilla.org.il/firefox - Our main Firefox page, with a download links for Windows/Linux/Mac versions. The page is fully-loaded with texts about the browser, and it is somewhat complicated to locate the download buttons even if you can read the texts. 53% of the non-Firefox users are actually downloading the browser, and 5% are clicking on the link to all.html, probably in order to download the browser in a different language (English).

B. firefox.co.il/download - Our second download page, which is less known but useful for people guessing they can locate Firefox if they'll type firefox and the country tld (.co.il) after (most clicks are direct). We don't have text on that page, and 75% are actually downloading the browser from that page (they can select which language they prefer from the dropdown language selection there, and we are not tracking their choices).

C. Some users are actually downloading the Hebrew localized version of Firefox from http://www.mozilla.com/he. I don't have access to downloads stats from that page, and I assume we have similar rates as our local sites.

Paul Kim said...

Great to see this go live, Laura. It was a tougher, much longer road than any of us anticipated, but I'm happy that you stuck with it and helped make this page better.

The Jack said...

Thanks ever so much, very useful article. Great information! Very smart, thanks for kindly sharing it with us. Very well done indeed. If you do not mind, please visit my article related to pandeglang district in Banten, Indonesia at Kenali dan Kunjungi Objek Wisata di Pandeglang or Kenali dan Kunjungi Objek Wisata di Pandeglang and information about district Banten Indonesia Banten dan Kabupaten Pandeglang

Daniela said...

Find your great Travel News and sing the songs at Free Song Lyric or you can watch the drama at Korea Drama Online one of great korea drama is A Love to Kill if you go to travel to Indonesia learn Learn Indonesia Language first! And find your home cari rumah or make a blog Belajar membuat Blog find your home again rumah dijual and again at jual rumah then if you want buy a new laptop see the Laptop Price List or you can buy a New Blackberry and then take care your Health & Jewerly.

Ben said...

Thanks for useful tips.
Web design company

bitheads said...

I don’t know If I said it already, but this so good stuff keep up the good work. I read a lot of blogs on a daily basis and for the most part Belajar seo blogspot just wanted to make a quick comment to say I’m glad I found your blog. Thanks.

Best regards
Seo Motivation Oes tsetnoc

deLi said...

sohbet odaları
sohbet
yonja
chat siteleri
forum siteleri
toplist ekle
sohbet
yonja
netlog
sohbet
kizlarla sohbet
sohbet
sohbet
dini sohbet
islami sohbet
sohbet
sohbet chat
mirc indir
cinsel sohbet
porno izle
camfrog indir
lida
kurumsalseo.com R10 lida fx15 pohudey zayıflama
sohbet
chat

nike said...

Nice work and thanks!
Running
Adidas currently manufactures several running shoesNike shoes, including the adiStar Control 5, the adiStar Ride
Cheap nike shoes
Discount nike shoes
the Supernova Sequence and the Supernova Cushion 7, among others.
Nike shox r4
nike shox shoes
Adidas also uses kangaroo leather to make their more expensive shoes.
Association football
One of the main focuses of Adidas is football kit and associated equipment.
puma cat
cheap sport shoes
Adidas also provides apparel and equipment for all teams in Major League Soccer. Adidas remain a major company in the supply of team kits for international football teams.
cheap nike shox
cheap nike max
Adidas also makes referee kits that are used in international competition and by many countries and leagues in the world. In the United States, referees wear the Adidas kits in MLS matches even though the primary referee supplier is Official Sports.
nike tn dollar
nike running shoes
The company has been an innovator in the area of footwear for the sport with notable examples including development of the Copa Mondial moulded boot on firm dry pitches for forty years.
nike air max tn
puma shoes
Adidas became renowned for advancing the "Predator" boot design.This design featured a ribbed rubber structure for the upper leather of the shoe, used to accent the movement of the ball.
discount puma shoes
puma mens shoes
The Predator also features the Craig Johnston invented "Traxion" sole. As the development and popularity of Football continued Adidas played a leading role in shaping the style of the play itself.
puma running shoes
puma shoes
FIFA, the sports governing body, commissioned specially designed footballs for use in its own World Cup tournaments to favour more attacking play.
ghd hair straighteners mk4
hair straightners
ghd iv styler hair straightener
ghd hair straightners
cheap ghd hair straighteners

nike said...

Clothing has always been a thing that has been given a great importance by human beings. It displays the attitude that the people exhibit.
ed hardy clothes
ed hardy shirts
Lots of brands have been existent in the clothing industry and a famous one among them is the Ed hardy clothing brand.
ed hardy jackets
ed hardy hoodies
The brand got its name from the famous American tattoo artist Ed Hardy. He was a very famous tattoo artist and has published many books on tattooing techniques.
ed hardy boots
ed hardy polo shirts
But his tattooing turned into a brand by the efforts of a company called Christian Audiger. This company was a very famous and very powerful company in the field of clothing.
ed hardy shoes
ed hardy jeans
They felt that it would be appropriate to create a brand called Ed hardy and use Hardy's art as the main selling point for the brand. This venture had turned out to be a very successful one and Ed hardy clothing is one of the most famous brands in the clothing industry.
ed hardy outerwear
ed hardy long sleeve shirts
The brand became very famous because it was worn by many famous celebrities like Madonna, Britney spears and also Sylvester Stallone.
ed hardy bags
ed hardy winter boots
The brand has clothes for men, women, and kids. They have also diversified their business by having a lot of accessories to support their clothing business.
ed hardy love kills slowly shirts
ed hardy love kills slowly shoes
ed hardy love kills slowly boots
for men they have a variety of products such as active wear, denim, outwear, swim trucks, sweaters, t-shirts, tops etc.
ed hardy trousers
ed hardy mens
the accessories include things such as belts, caps, scarves, shoes, socks, jeweler, sunglasses, ties and even wallets.
The art works is a unique combination of American and Japanese cultures. The unique selling proposition of the brand is the way in which the company has used the art works of Ed hardy.
ed hardy womens
ed hardy t shirts
ed hardy sunglasses

Arya Wiradana said...

thanks my dear.. this is great article for me. I recently came across your blog and have been reading along. I thought I would leave my first comment. I don't know what to say except that I have enjoyed reading.

Regards
Bali car rental

Bruce said...

Excellent Article. It'll surely help me out... thanks for the post

Web Design Company

james said...

Las Vegas Escorts wants to stop by for the last time. And just thanks you for every post you have done.

Money said...
This comment has been removed by the author.
Money said...
This comment has been removed by the author.
Money said...

make money
make money online
how to make money
make money at home
make money on the internet
making money
how to make money online
make money fast
make money from home

alva said...

The world's largest shoe industry portals, shoes, nets to provide a flood of Air Jordan Shoes of information, NFL Jerseys and puma chaussures hommes line trading and other information.

mobile phone

alva said...

You can find the latest and most Footwear tide of information, including NFL Jerseys, MBT shoes, nike tn, and other related content. 
                                 China Wholesale

mesut said...

Good an your my Lida - Biber hapı

Sheldon said...

To see the big run, Laura. This is a tough, longer time than we expected any road, but I am glad that you stuck with it and help to make this page better.

Thesis | Dissertation | Essay | Assignment

Huseyin said...

Thank you for this very useful information. sohbet odalari, chat sohbet

Sheldon said...

Excellent article. That it will help me ... thank you SEO Services Herbal Products Household Items Spy watch Camera Car Accessories

hayda said...

What a helpful post really will be coming back to this time and time again. mirc . chat .Thanks ..

Logo said...

You have raised problems,thanks for sharing all the solutions.

Website design | Logo design | Logo designs

Logo said...

Thanks @ Tomer for the breif, also have checked all urs links.Helps alot.
Logo Designs | Logo Design

Ankine said...

custom papers

This is really good and interesting piece of work. i liked it,and wanted to know more about it.

Sam.Pierce said...
This comment has been removed by the author.
Sam.Pierce said...

This is really good and interesting piece of work. i liked it,and wanted to know more about it.
Custom Logo Design

Mark said...

You have raised problems,thanks for sharing all the solutions.

Mark said...

I appreciate these type of instructions and guidelines for the beginner and sometimes it useful fr professionals as well.
Business Logo

7vpo said...

Although everyone Biber Hapı knew exactly what they want from these words can be attached to. If you Biber Hapı think you can participate in a good way.
Added an idea what it would have the power to make you guess. I recommend it Biber Hapı to you in any way.
As we always Fx15 say we're
right on this issue and will protect you from Fx15 various complaints by developing a
system that he will run. We are Fx15 always wishes us to continue to monitor, get
better.

Mobi said...

Dear thank you sharing you information.it was very useful for me.because i am presses in this section....happy blogging
home security cameras

e said...

Thanks for sharing such a useful information! This is a great post, thanks

koleksi gambar

immortal said...

actually i accept with your information.thank you sharing
@Tomer had been explained their situation.its also very impotent.
thank you very much.Sweet pick up lines

http://www.bmbcommunicationsinc.com said...

I impressed to visit your site.

aturservice said...

Impressive post. Keep it up!

Facebook Smileys
Car Parking Games

bob said...

Thanks for sharing such a informative and useful post.




Dial up Internet Services

bob said...

Thanks for sharing such a informative and useful stuff.

CHEAPSOCCERUNIFORM said...

Many thanks for your themed, a good post for me.
Now it will be very cold in winter, in order to keep warm yourself, moncler company has provide jackets for you, so it is better to own moncler yourselif as soon as possible. Eeveryone love fashion clothing, polo ralph lauren is very popular all over world, that is my dream to get one.We know Ray Ban by America soldier, all of them wear ray ban sunglasses when they walk on the road, it is fashionable for you. Most of people like to wear jeans, it is very modern when wearing true religion jeans in the street.

ramizdayoglu said...

The fungus is another B vitamin pantothenic acid, especially in times of stress the Biber Hapı adrenal glands helps protect fatig'den supporting the operation. Showing the great importance of this effect of zinc.

Cheapsocceruniforms said...

There are many brand from France, also including herve leger, and most of womens stars love wearing herve leger dress when they join in some important party. Now polo ralph lauren is very popular with youthful people, everyone want to get ralph lauren polo shirts, there are lots of online shop which are ralph lauren polo outlet, true religion jeans outlet, it will be convenient for us.

William said...

it as two separated clicks. Checking what are their current browser told me that most of them are IE6 users who probably just heard about Mozilla Firefox.
business logo design

Hafidz said...

Android Cellphones
SomeMap
Pointblank Lucu I'm up in your projects, launching your web pages :-)

But seriously, though, looking forward to working with you on the partnership stuff; can you send it my way when you get a chance? wallpaper lucu
model baju batik
superbowl sunday Thanks!

healthpharmacyrx1 said...

achat levitra
flagyl Online Bestellen
Buy cipro online
nolvadex Online Rezeptfrei
acheter cipro en france
viagra Online Bestellen
clomid Rezeptfrei
propecia achat
propecia Ohne Rezept Bestellen
kamagra Ohne Rezept Kaufen
achat xenical en ligne
lasix Rezeptfrei Bestellen
clomid Kaufen
Buy viagra soft
xenical Ohne Rezept Bestellen
augmentin Bestellen
achat cialis soft

youlacka said...

Hello to everyone who has blog.first I congratulate the blog owner.he is got excellent blog.That is really very good article. I am glad to know. Thanks for sharing !
kizlarla chatcetavrupa chatchat odalari
Mersin Chatsohbet odalari

Topluchat said...

Chat

Adrian said...

Great to see this go live, Laura. It was a tougher, much longer road than any of us anticipated,logo design_ | _company logo design

johndavid said...

What a Great Article it its really informative and innovative keep us posted with new updates. its was really valuable. shisha tobacco starbuzz shop thanks a lot. Thanks for sharing this information.starbuzz tobacco france

free google sms said...

I love your blog, thanks for the interesting info you choose.
free cna classes | CNA Classes Online , CNA Certification

Emma jacob said...

Sounding Out thanks is simply my little way of saying. Bravo for a fantastic resource. Take On my sweetest wishes for your incoming post.
Natural vitamins supplements

crook said...

Nice and awesome post. It was a tougher, much longer road than any of us anticipated and i am happy that you are touch with it

kris said...

I was just going to relax and read this morning then in the meantime I got your blog to read. Really good one.

business logo design

Blackberry Developers said...

thats really great work

Gral Yorumcu said...

Yemek Tarifleri Thanks for this post – I have struggled with this as I have NO comments yet on my blog and in some ways feel badly about that (until i read your posts about this) and yet I see there is more that I can do. thankss.

Company Logo Design said...

I would like to say that this blog really convinced me, you give me best information indeed! Thanks, very good post

eshaa said...

Nice post...
I have also find a great Free Classified Ads..That is geboon.com/
Its my habit to share good things to my friends...

iPhone Developers said...

They are really wonderful. I never heard of something like this.

logoonlinepros said...

hey i do love your post and this blog make me awesome..very useful and i like to share this into my friends just to read your post...well this blog sayin keep up your dream and lets make that together..thanks again for posting..what a great blog.

Banner Design

eshaa said...

I’m so glad that I took the time to look at this blog, because let me tell you. Not a lot of people know how to balance knowledge of a subject and content website Design perth.

website design perth

gokhan koclar said...

superb post :) thanx admin

http://fullpornoizzle.com

Movies Gallery 2011 said...

Thanks for the post. It was very interesting and meaningful.
Clone Script| Super Bowl 2012|

admin said...

hello to everyone who has blog.first I congratulate the blog owner.he is got excellent blog.actually I read all your articles from weary and your writing very attractive waiting to continue thanks aşk şiirleri aşk şiirleri, aşk sözleri, chatroulette chatroulette türkiye yemek tarifleri yemek tarifleri sohbet sohbet, güzel sözler chat kanalları chat kanalları, okey oyna okey oyna, aşk şiirleri aşk sözleri, aşk sözleri aşk sözleri, şiir, omegle omegle türkiye temsilcisi, omegle türkiye omegle türkiye, omegle omegle, omegle omegle, omegle türkiye omegle türkiye, omegle türkçe türkçe omegle, chatroulette chatroulette, netlog netlog turkchat turkchat, turkchat turkchat, turk sohbet, turkchat turk chat, online film izle film izle film izle full film izle, bedava film izle bedava film izle chat chat, sohbet sohbet, video indir video indir, almanya sohbet almanya sohbet, almanya chat almanya chat, canlı okey oyna canlı okey oyna, online okey oyna online okey oyna, muhabbet muhabbet , sevişme sahaneleri sevişme sahneleri , seks partner seks partner

Anabolic Steroids said...

Some thoughts:
a. How many of these users actually got the map page, than navigated to another page and when they came back to the download page they got another design? It might be possible that some people that was unable to locate the download in the first two designs found how to complete the download process only for another design, and the rise in the download count for the 3rd design was due to bad download rates on the 2nd design...
clenbuterol | steroids for sale | steroids

mobitily1 said...

What a Great Article it its really informative and innovative keep us posted with new updates.
its was really valuable.I must admit it, terrific job on this blog,
I’ll be sure to check back again real soon.

Mobile Application Development

Zain said...

online logo design
logo design canada