Thursday, April 2, 2009

Results of the All.html Test

The much awaited results of the all.html redesign test are here!

What were we trying to do?: (You can skip to the Results section if you already know about this ;))
In September of last year, I started looking at the current all.html page to see if there were ways to optimize the page. We had a high bounce rate (40%) and I felt that the page was visually confusing and that there was a better way to display the information listed.

I then asked The Royal Order (an amazing design firm) to create three designs to address the problem of trying to display 60+ Firefox localizations and their platforms to a worldwide community of people with different needs (ie, language spoken, ability to read English, various levels of tech-savvy, country of download, etc).

These were the designs we received:

All is the current all.html page.
(This picture is of the older page because we had already taken the all.html page out of circulation by publication).
All-1 was our most "designed" of the pages that included a map. (
(This picture is of the psd because we had already taken the all-1.html page out of circulation by publication).
All-2 was in the middle, with some geographical images and expandable lists.
(This picture is of psd because we had already taken the all-2.html page out of circulation by publication).
All-3 was the most like the current all.html with some changes to some details like # of columns.

After looking at them, we were unable to decide on face value which one should be the new replacement. We all had plenty of opinions, but nothing based on verifiable data. Consequently, we decided that we would test all three designs (and the current design) to see which one performed the best.

The Test:
We set up a week long test that ran from March 21-29, 2009. We rotated the pages evenly so each page showed for 15 minutes of every hour for the entire period. We defined success or "best performance" as:
1) Best User experience on the page comparing to all.html
2) Best download rate on the page comparing to all.html
(You can read more details about the test/process here and here.)


The Results:


The results are organized by version of all#.html and broken up into page views and downloads. The conversion rate was found by dividing download clicks by page views. The baseline we used for all comparison was the all.html page. The baseline, therefore, is that 52.7% of people who hit that page downloaded a version of Firefox. Better Performance would then be defined as anything above the average of 52.7%.

All-1.html performed much worse than expected, with 32.4% of people downloading Firefox after hitting that page. All-2.html, performed at 56.5% which was on par with my expectations.
Lastly, All-3.html was comparatively stellar, getting about 60.9% of the people who hit that page to download. This basically translates to an
8% improvement over our current all.html page.

Now, before we get
TOO excited about these results, resident Metrics Guru, Ken Kovash, asked me to do some further digging. Basically, he was concerned because "download clicks" represent a snap shot of a site visitor’s experience at a single moment in time and it doesn’t factor in what actions a person takes after that moment (e.g., a download occurring when that person subsequently navigates away to other pages within Mozilla.com). A Firefox user could hit the all.html page, then move on to the mozilla.com homepage, and then click download. In this example, the action of a download is not counted in our numbers and results above.

To figure this out, I looked at a funnel report in our web analytics system for all-3.html and all.html to try and figure out where people were going after the all.html pages to download (if not from the actual page) and to integrate these numbers into our results above.

All.html


First column you see below is total number of page views.
Second column is the total of downloads from the all.html download buttons (all of them).
Third columns is the total of downloads from the entry page (mozilla.com)
Fourth column is the total of downloads from download buttons that are not on one of our top pages (eg customize).
Fifth column is total after subtracting colunms 2, 3, 4.
Sixth column is the number on the total downloads/page views which results in an average of 67.6% conversion rate.

All-3.html

First column you see below is total number of page views.
Second column is the total of downloads from the all.html download buttons (all of them).
Third columns is the total of downloads from the entry page (mozilla.com)
Fourth column is the total of downloads from download buttons that are not on one of our top pages (eg customize).
Fifth column is total after subtracting columns 2, 3, 4.
Sixth column is the number on the total downloads/page views which results in an average of 68.4% conversion rate.
Seventh column is the difference in conversion rates, with the average difference between all-3 and all.html being .89% (in favor of all-3.html).

Synopsis
All in all (pun intended) all-3.html was the best performer of the four pages. The 8% difference in user experience and .89% improvement in download rate (comparing to all.html) shows that even a small change in a page's design can have an effect on download numbers. And even though 8% improvement in ease of use or a .89% improvement in downloads may seem small, considering the number of people who see this site every day (70-90K page views a day) the effort and hardwork that went into this redesign and test were well worth it.

All-3.html has now taken the place of the old page and I'm looking forward to tweaking and testing some details and re-checking its' performance in the next few months.

Thanks to everyone who has put in so much hardwork these past two quarters: The Royal Order, Steven Garrity (Silver Orange), Wil Clouser, Jeremy Orem, Stephen Donner, Krupa Raj, Ken Kovash, Seth Bindernagel, Mike Morgan and John Slater--couldn't have finished this project without you!

61 comments:

Unknown said...

I'm up in your projects, launching your web pages :-)

But seriously, though, looking forward to working with you on the partnership stuff; can you send it my way when you get a chance? Thanks!

Congrats on shipping!

Tomer Cohen said...

I think the best thing to say is "We told you!", as most of the commenters said in the bugs. maybe next time a download page redesign survey would be enough. :)

Some thoughts:
a. How many of these users actually got the map page, than navigated to another page and when they came back to the download page they got another design? It might be possible that some people that was unable to locate the download in the first two designs found how to complete the download process only for another design, and the rise in the download count for the 3rd design was due to bad download rates on the 2nd design...

b. When I investigated the download rates on our local download page, it come to my attention that some people are clicking twice on the download link, fast enough to treat it as a double-click, while the analytic software we use (Google Analytics) treat it as two separated clicks. Checking what are their current browser told me that most of them are IE6 users who probably just heard about Mozilla Firefox.

We have two different Firefox download pages on our community sites in Israel, and while the download rates are much smaller, I would be happy to share with you some of our stats which may give some more knowledge about how the users are finding the download buttons useful -

A. mozilla.org.il/firefox - Our main Firefox page, with a download links for Windows/Linux/Mac versions. The page is fully-loaded with texts about the browser, and it is somewhat complicated to locate the download buttons even if you can read the texts. 53% of the non-Firefox users are actually downloading the browser, and 5% are clicking on the link to all.html, probably in order to download the browser in a different language (English).

B. firefox.co.il/download - Our second download page, which is less known but useful for people guessing they can locate Firefox if they'll type firefox and the country tld (.co.il) after (most clicks are direct). We don't have text on that page, and 75% are actually downloading the browser from that page (they can select which language they prefer from the dropdown language selection there, and we are not tracking their choices).

C. Some users are actually downloading the Hebrew localized version of Firefox from http://www.mozilla.com/he. I don't have access to downloads stats from that page, and I assume we have similar rates as our local sites.

Paul Kim said...

Great to see this go live, Laura. It was a tougher, much longer road than any of us anticipated, but I'm happy that you stuck with it and helped make this page better.

Ben said...

Thanks for useful tips.
Web design company

deLi said...

sohbet odaları
sohbet
yonja
chat siteleri
forum siteleri
toplist ekle
sohbet
yonja
netlog
sohbet
kizlarla sohbet
sohbet
sohbet
dini sohbet
islami sohbet
sohbet
sohbet chat
mirc indir
cinsel sohbet
porno izle
camfrog indir
lida
kurumsalseo.com R10 lida fx15 pohudey zayıflama
sohbet
chat

Arya Wiradana said...

thanks my dear.. this is great article for me. I recently came across your blog and have been reading along. I thought I would leave my first comment. I don't know what to say except that I have enjoyed reading.

Regards
Bali car rental

Anonymous said...

Excellent Article. It'll surely help me out... thanks for the post

Web Design Company

Unknown said...

Las Vegas Escorts wants to stop by for the last time. And just thanks you for every post you have done.

Money said...
This comment has been removed by the author.
Money said...
This comment has been removed by the author.
Money said...

make money
make money online
how to make money
make money at home
make money on the internet
making money
how to make money online
make money fast
make money from home

mesut said...

Good an your my Lida - Biber hapı

Unknown said...

To see the big run, Laura. This is a tough, longer time than we expected any road, but I am glad that you stuck with it and help to make this page better.

Thesis | Dissertation | Essay | Assignment

Amber Melton said...

Excellent article. That it will help me ... thank you SEO Services Herbal Products Household Items Spy watch Camera Car Accessories

logo design - logoinn said...

You have raised problems,thanks for sharing all the solutions.

Website design | Logo design | Logo designs

logo design - logoinn said...

Thanks @ Tomer for the breif, also have checked all urs links.Helps alot.
Logo Designs | Logo Design

Unknown said...

custom papers

This is really good and interesting piece of work. i liked it,and wanted to know more about it.

Unknown said...
This comment has been removed by the author.
Unknown said...

This is really good and interesting piece of work. i liked it,and wanted to know more about it.
Custom Logo Design

Unknown said...

You have raised problems,thanks for sharing all the solutions.

Unknown said...

I appreciate these type of instructions and guidelines for the beginner and sometimes it useful fr professionals as well.
Business Logo

7vpo said...

Although everyone Biber Hapı knew exactly what they want from these words can be attached to. If you Biber Hapı think you can participate in a good way.
Added an idea what it would have the power to make you guess. I recommend it Biber Hapı to you in any way.
As we always Fx15 say we're
right on this issue and will protect you from Fx15 various complaints by developing a
system that he will run. We are Fx15 always wishes us to continue to monitor, get
better.

Mobi said...

Dear thank you sharing you information.it was very useful for me.because i am presses in this section....happy blogging
home security cameras

admin said...

Thanks for sharing such a useful information! This is a great post, thanks

koleksi gambar

Unknown said...

actually i accept with your information.thank you sharing
@Tomer had been explained their situation.its also very impotent.
thank you very much.Sweet pick up lines

http://www.bmbcommunicationsinc.com said...

I impressed to visit your site.

aturservice said...

Impressive post. Keep it up!

Facebook Smileys
Car Parking Games

Unknown said...

Thanks for sharing such a informative and useful post.




Dial up Internet Services

Unknown said...

Thanks for sharing such a informative and useful stuff.

ramizdayoglu said...

The fungus is another B vitamin pantothenic acid, especially in times of stress the Biber Hapı adrenal glands helps protect fatig'den supporting the operation. Showing the great importance of this effect of zinc.

Unknown said...

it as two separated clicks. Checking what are their current browser told me that most of them are IE6 users who probably just heard about Mozilla Firefox.
business logo design

Anonymous said...

Android Cellphones
SomeMap
Pointblank Lucu I'm up in your projects, launching your web pages :-)

But seriously, though, looking forward to working with you on the partnership stuff; can you send it my way when you get a chance? wallpaper lucu
model baju batik
superbowl sunday Thanks!

Blogspot said...

Chat

Adrian said...

Great to see this go live, Laura. It was a tougher, much longer road than any of us anticipated,logo design_ | _company logo design

johndavid said...

What a Great Article it its really informative and innovative keep us posted with new updates. its was really valuable. shisha tobacco starbuzz shop thanks a lot. Thanks for sharing this information.starbuzz tobacco france

free google sms said...

I love your blog, thanks for the interesting info you choose.
free cna classes | CNA Classes Online , CNA Certification

Emma jacob said...

Sounding Out thanks is simply my little way of saying. Bravo for a fantastic resource. Take On my sweetest wishes for your incoming post.
Natural vitamins supplements

Unknown said...

Nice and awesome post. It was a tougher, much longer road than any of us anticipated and i am happy that you are touch with it

kris said...

I was just going to relax and read this morning then in the meantime I got your blog to read. Really good one.

business logo design

Angelina said...

thats really great work

Company Logo Design said...

I would like to say that this blog really convinced me, you give me best information indeed! Thanks, very good post

Angelina said...

They are really wonderful. I never heard of something like this.

eshaa said...

I’m so glad that I took the time to look at this blog, because let me tell you. Not a lot of people know how to balance knowledge of a subject and content website Design perth.

website design perth

Unknown said...

superb post :) thanx admin

http://fullpornoizzle.com

AS said...

Some thoughts:
a. How many of these users actually got the map page, than navigated to another page and when they came back to the download page they got another design? It might be possible that some people that was unable to locate the download in the first two designs found how to complete the download process only for another design, and the rise in the download count for the 3rd design was due to bad download rates on the 2nd design...
clenbuterol | steroids for sale | steroids

tamsohbet said...

chat odaları
sohbet odaları
sohbet
chat

asa said...

Mobil Gabile Merhaba dostlar yeni tek makale ile yine karşınızdayım. Bu gün makalemizin hususu mobil sohbet sitesi siteleri ile ilgili olacaktır. Bunu ele almamızın emeli mobil sohbet sitesi sitelerinin sizlere sağlamış bulunduğu olanakları göz önüne sermek ve bunlardan sebep Mobil Gabile faydalanabileceğinizin yollarını göstermektir. mobil sohbet sitesi siteleri 90’lı senelerden Türkiye’ye sağlamak yapan ve girdiği günden bu yana defa dayanıklı tek kullanıcıları potansiyeline sahip olmuş seyrek sitelerden tek tanesidir.

asa said...

Dünya’nın dört tek yanından aramıza katılan bay, kadın kullanıcılar günün her zamanında eğlenceli ve keyifli tek şeklinde yeni arkadaşlarla tanışma imkânına sahip olmaktadır. Sizlerde kendinize ideal, bay ya da kadın yeni Gabile Chat arkadaşlarla tanışabileceğiniz; #sohbet, #samata, #eğlence, #İslamisohbet, #Kültür, #Sanat, #Felsefe gibi alaka alanlarınıza ya da size ideal odalarımız içinde tek araya gelebilirsiniz. Gabile Chat
Tamamen para ödemeden görev veren odalarımız içinde sizler de yeni arkadaşlıklara, dostluklara hem de ve hem de hayatınızın aşkına tek merhaba diyerek başlangıç yapabilirsiniz

kartal evden eve nakliyat said...

Kurumsal SEO
Üsküdar SEO
Levent SEO
Nice Servis
Nice Yetkili Servis

لمار للتسويق الالكتروني said...





Cleaning company in Dammam شركة مكافحة حشرات بالدمام حراج is one of the best companies present in the Kingdom of Saudi Arabia where it owns the best cleaning methods used from cleaning houses - apartments - villas - palaces - schools - شركة تنظيف بالدمام
government and private institutions and also available water tank cleaning service, شركة مكافحة حشرات بالدمام
provides the best house cleaning company in Dammam Lots of discounts on premium prices for the rest of Dammam cleaning companies .

Cleaning company in Dammam is concerned with the trained workers to use cleaning materials without any damage or damage, شركة تنظيف فلل بالدمام
even if something went wrong can solve the problem quickly without feeling any difference may occur, شركة مكافحة حشرات بالدمام حراج when we talk about the equipment and equipment and cleaning materials at cleaning company in Dammam will remind that افضل شركه تنظيف منازل بالدمام
we have the finest And best of all, we will also talk within the finest cleaning company in Dammam on the tasks of professional supervision and how to be and where and when and what is offered to the client.

افضل شركة تنظيف بيارات بالدمام
افضل شركة تنظيف شقق بالدمام

asa said...

Gay Sohbet Chat Yakınızdaki Insanları Keşfedin sohbet programı bu arada size en yakında tespit edilen azaları keşfetme gibi güzel tek özelliğe sahiptir. Bu özelliği seçenek ederek sizinle aynısı yerde yaşam sürdüren ve uzaklık olarak size defa yakında tespit edilen insanları derhal bulabilirsiniz. Gay Sohbet Chat

tatli said...

Yetişkin sohbet

hasmet said...

Sohbet Böyle sitelerin genel yapısal özellikleri, site çeşitlerine göre değişkenlik gösterir. Kimi tür siteler verdikleri görev yönünden yada içlerinde barındırdıkları yapı itibari ile değişiklik gösterir. Bu nedenlerden kaynaklı tam ve tek düze tek yapı göstermezler. Başka bir deyişle homojen tek yapı olmaz yada olamazlar. Homojen tek yapıda olmamasının birçok üstünlük ve dezavantajları Sohbet mevcuttur.

Unknown said...

mobil sesli
mobil sesli
mobil sesli
sesli mobil
sesli mobil
mobil sohbet
omegle mobil

hasmet said...

Sohbet
Sohbet Odaları
Sohbet Siteleri
Chat Siteleri
Mobil Sohbet
Bedava Sohbet
Geveze Sohbet
Mynet Sohbet
Mobil Sohbet Siteleri
Mobil Chat
Chat Odaları
Parasız Sohbet Siteleri
Sohbet Sitesi
Chat
Kaliteli Sohbet
Seviyeli Sohbet

mert said...

Hemen Okey Oyna kalıcı ve kalitel idostluklar kurun.

Güzel Sözler said...

It has been an article that has been described in the best way I have ever read. Thank you admin
Sevda Sözleri

mert said...

Bilhassa oyun oynamak, vakit geçirmek yerine harbiden defa iyi tek yöntemdir. Aktif tek şeklinde hem oyun oynayıp hatta kazanmak ve kaybetmenin heyecanını yaşayarak değişik tek boyutta eğlenebileceğiniz Okey plus oyunu, tüm isteklerinizi karşılıyor. Okey uzun süren partiler durumunda oynana tek oyun bulunduğu yerine birçok şahıs yerine oyunun Eklendi özelliğinin olması önemlidir. Sanal Okey Salonları Bu nedenden dolayı de tüm uygulamalarda okey oyna Eklendi et özelliği yer alır. Eklendi masada tespit edilen dört şahıs, arasında ortak olarak yapılabildiği gibi istediğinizde kalifiye şeklinde de Eklendi etme olanağı bulursunuz.

oktay said...

Son olarak elinizde kalan taşı ise yere atarak oyunu kazanmış olursunuz. Başlangıç üstünde her oyuncuya 14 taş verilmektedir. Yanlızca oyuna başlayan şahsa 15 taş verilir ve çok olan taş ile oyun dönmektedir. Yere açılan ve gösterge olarak tespit edilen taşın tek üst adedi okey olarak belirlenmektedir. Canlı Okey Minimum 3’erli taşlar durumunda küme oluşturulmaktadır. Okey 4 şahıs ile oynanan ve belirli sayıda tespit edilen taşların derli toplu tek biçimde dönmesi ile tek bireyin kazanması ile neticelenen oyun olarak bilinir. Belirli kaideler kapsamında oynanmasıyla birlikte her el değişik tek okey belirlenmektedir.

oktay said...

Kesif kullanıcıları kitlesi ile Gaziosmanpaşa okey salonlarında hemşehrileriniz ile tek araya gelebilecek tek düze geride bıraktığımız zamanlarınızı dostlarınızla ya da yeni okey oyuncuları eşliğinde okey oynayarak henüz keyifli hale getireceksiniz. Üyeliksiz Okey Salonları Ismarlama menüsünü kullanarak oyuncular birbirlerine ikramlarda bulunabilir, kalifiye sohbet ya da genel sohbet kısımları üstünden rahatlıkla başka kullanıcılar ile sohbet edebileceksiniz

oktay said...

üye olmadan eglenceli ve huzurlu adreslerde yer alarak Lezler sohbet sitesine katılabilirsiniz.