Friday, March 27, 2009

Looking at the Q1 2009 Survey Results--Time for a Step Back

After looking at last quarter's survey results, I started to get concerned about the validity of our high "customer satisfaction score". Although our responses have been rather consistent, the drop in satisfaction rate between Q3 2008 and Q1 2009 got me a little worried--mainly because it was a 4% drop. I started to re-examine the data and it appears as though there are many factors that should be considered when looking at the “drop” in “satisfaction”.


The first flag was what appears to be a drop in survey participation over the past few quarters. During the first installment of the survey we had 30,272 respondents. That number has dropped by nearly 50%, with 15,823 respondents in Q4 of 2008 and 15,682 respondents in Q1 2009.

Here are a few possible explanations for this “apparent” drop in participation over time:

1) The novelty of the first survey encouraged more participation.

-Even though we have 100 million people seeing the “What’s New” page with every update and we only allowed the survey to appear to 10% of that population, there is a self-selecting bias in any opt-in survey.
-The further we move away from Firefox 3 launch, the lower the enthusiasm for the product. I'm not saying that people have stopped loving Firefox 3, I'm just saying that the further we get away from the launch, the less likely people are to be willing to take a survey to tell us they love the product. (Its’ awesomeness, in essence, has become the status quo).

2) The first survey was released with 3.02 and 3.03 (released a day later), which may have inadvertently increased the number of people who saw the survey link, and consequently, may have artificially doubled our participation with the survey’s first release.

I think option #2 is really the only one that makes sense in this context. It’s easy to see how (self-selecting) individuals would have taken the survey twice and thus inflated our initial “customer satisfaction score”. And since the past two surveys have been consistent in participation and responses, I think that it may be worth completely throwing out the data from the first survey.


Other concerns about the survey came from looking at the responses to certain questions.

For example: "How long have you been using Firefox?"



About 50% of respondents to this question say they have used Firefox for more than two years. That response rate doesn't quite make sense to me if we are trying to get responses from a cross-section of all of our users. Those numbers would make more sense coming from our user-base or long term, loyal, customers. Other survey questions that were similarly concerning: "Have you installed an Add-On?" (50% said yes), "On average, how much time do you spend on the web?" (2-5 hrs a day). The responses to both of these questions again seem to indicate that a large number of the respondents to our survey were part of our core fan-base as opposed to part of a more general cross-section of users. (You can take a look at the Comscore data Dave Bottoms gathered last year if you want to know about the average Internet user.)


Another red flag was the number of people who started but did not complete the third installment of the survey. The survey released in December ’08 had a total of 37 “abandoned” surveys. The most recent installment of the survey was “abandoned” 5419 times. Why the huge jump in abandonment rate? I think it comes down to two changes we made to the survey:

1) We increased the number of questions from 8 to 10.

2) We reformatted the survey so that the questions were split equally on two pages.

I think both of these changes, though well intentioned, really took us away from our main goal of this program—to create a simple, short survey that would efficiently allow us to get a better picture of all of our customers.


To review, here are the takeaways:

1) Q32008 Results and “Satisfaction Score” are probably inflated and should be disregarded.

2) The surveys tend to be taken mostly by long-term, loyal Firefox users. We need to find a way to get responses from a more diverse cross-section of Firefox users.

3) We need to pay attention to the design of our survey to ensure we don’t scare users off with the number of questions/layout of the survey.

So, what next? I think our next steps are to step back and try and get a better understanding of what the data we have collected in the past nine months *really* means. That also means trying to figure out what we have previously called our "Customer Satisfaction Score" is really measuring. Is it a customer loyalty score? Is it an enthusiasm score? At this point, I'm not really sure, mainly because I don't know the extent of the biases in our data.

It would be great to hear your thoughts and suggestions: what can we do differently? How do we get a better sample of all types of Firefox users? How can we try to minimize bias?

Stay tuned!

20 comments:

The Jack said...

Thanks ever so much, very useful article. Great information! Very smart, thanks for kindly sharing it with us. Very well done indeed. If you do not mind, please visit my article related to pandeglang district in Banten, Indonesia at Kenali dan Kunjungi Objek Wisata di Pandeglang or Kenali dan Kunjungi Objek Wisata di Pandeglang and information about district Banten Indonesia Banten dan Kabupaten Pandeglang

Daniela said...

Interesting post. I have been wondering about this issue,so thanks for posting. I’ll likely be coming back to your blog. Keep up great writing.

Travel News | Free Song Lyric | A Love to Kill | Learn Indonesia Language | cari rumah | jual rumah | rumah dijual | Laptop Price List | New Blackberry | Health & Jewerly

nike said...

Clothing has always been a thing that has been given a great importance by human beings. It displays the attitude that the people exhibit.
ed hardy clothes
ed hardy shirts
Lots of brands have been existent in the clothing industry and a famous one among them is the Ed hardy clothing brand.
ed hardy jackets
ed hardy hoodies
The brand got its name from the famous American tattoo artist Ed Hardy. He was a very famous tattoo artist and has published many books on tattooing techniques.
ed hardy boots
ed hardy polo shirts
But his tattooing turned into a brand by the efforts of a company called Christian Audiger. This company was a very famous and very powerful company in the field of clothing.
ed hardy shoes
ed hardy jeans
They felt that it would be appropriate to create a brand called Ed hardy and use Hardy's art as the main selling point for the brand. This venture had turned out to be a very successful one and Ed hardy clothing is one of the most famous brands in the clothing industry.
ed hardy outerwear
ed hardy long sleeve shirts
The brand became very famous because it was worn by many famous celebrities like Madonna, Britney spears and also Sylvester Stallone.
ed hardy bags
ed hardy winter boots
The brand has clothes for men, women, and kids. They have also diversified their business by having a lot of accessories to support their clothing business.
ed hardy love kills slowly shirts
ed hardy love kills slowly shoes
ed hardy love kills slowly boots
for men they have a variety of products such as active wear, denim, outwear, swim trucks, sweaters, t-shirts, tops etc.
ed hardy trousers
ed hardy mens
the accessories include things such as belts, caps, scarves, shoes, socks, jeweler, sunglasses, ties and even wallets.
The art works is a unique combination of American and Japanese cultures. The unique selling proposition of the brand is the way in which the company has used the art works of Ed hardy.
ed hardy womens
ed hardy t shirts
ed hardy sunglasses

nike said...

Nice work and thanks!
Running
Adidas currently manufactures several running shoesNike shoes, including the adiStar Control 5, the adiStar Ride
Cheap nike shoes
Discount nike shoes
the Supernova Sequence and the Supernova Cushion 7, among others.
Nike shox r4
nike shox shoes
Adidas also uses kangaroo leather to make their more expensive shoes.
Association football
One of the main focuses of Adidas is football kit and associated equipment.
puma cat
cheap sport shoes
Adidas also provides apparel and equipment for all teams in Major League Soccer. Adidas remain a major company in the supply of team kits for international football teams.
cheap nike shox
cheap nike max
Adidas also makes referee kits that are used in international competition and by many countries and leagues in the world. In the United States, referees wear the Adidas kits in MLS matches even though the primary referee supplier is Official Sports.
nike tn dollar
nike running shoes
The company has been an innovator in the area of footwear for the sport with notable examples including development of the Copa Mondial moulded boot on firm dry pitches for forty years.
nike air max tn
puma shoes
Adidas became renowned for advancing the "Predator" boot design.This design featured a ribbed rubber structure for the upper leather of the shoe, used to accent the movement of the ball.
discount puma shoes
puma mens shoes
The Predator also features the Craig Johnston invented "Traxion" sole. As the development and popularity of Football continued Adidas played a leading role in shaping the style of the play itself.
puma running shoes
puma shoes
FIFA, the sports governing body, commissioned specially designed footballs for use in its own World Cup tournaments to favour more attacking play.
ghd hair straighteners mk4
hair straightners
ghd iv styler hair straightener
ghd hair straightners
cheap ghd hair straighteners

alva said...

You can find the latest and most Footwear tide of information, including NFL Jerseys, MBT shoes, nike tn, and other related content. 

alva said...

The world's largest shoe industry portals, shoes, nets to provide a flood of Air Jordan Shoes of information, NFL Jerseys and puma chaussures hommes line trading and other information.

CHEAPSOCCERUNIFORM said...

I would like to thank you for sharing your thoughts and time into the stuff you post!! Thumbs up
Now it will be very cold in winter, in order to keep warm yourself, moncler company has provide jackets for you, so it is better to own moncler jackets yourselif as soon as possible. Eeveryone love fashion clothing, polo ralph lauren is very popular all over world, that is my dream to get one.We know Ray Ban by America soldier, all of them wear ray ban sunglasses when they walk on the road, it is fashionable for you. Most of people like to wear jeans, it is very modern when wearing true religion jeans in the street.

Cheapsocceruniforms said...

There are many brand from France, also including herve leger, and most of womens stars love wearing herve leger dress when they join in some important party. Now polo ralph lauren is very popular with youthful people, everyone want to get ralph lauren polo shirts, there are lots of online shop which are ralph lauren polo outlet, true religion jeans outlet, it will be convenient for us.

Emma jacob said...

Hi… that was great stuff.. I really like this subject. Could you tell me more … I would love to explore.
Organic vitamins

eshaa said...

For this motive to provide every product to the customer, and to accomplish the all needs of the people we introduce an authentic and valuable E-marketing place where mushroom of beautiful and well equipped houses and accommodations are provided by the so worthy and trusty E-marketing place which is geboon.com .
see Free Classified Ads

Movies Gallery 2011 said...

Nice post. Great blog. Thanks for the share.
Bot and me| Images|

Admin said...

Thanks for the information. I’m really glad that you share your ideas about this. I must relay these ideas to my friends.

Create a Logo

Admin said...

Nice energy and vibe you put out there. You have so much to offer, I like the energy and the nice values you apply to your site. Real interesting and outstanding point of view.

affordable search engine optimization services

0Zero7 said...

Thanks i like well and good information in your blog very much , i come back most days to find new posts like this.
affordable web hosting

Alis Stafford said...

Thanks for your marvelous posting! I actually enjoyed reading it, you will be a great author.
CCTV Systems
CCTV Monitors

Admin said...
This comment has been removed by the author.
Alis Stafford said...

NiCe BlOg and NiCe PoSt
Sudoku

Moe Nawaz said...

Its really helpfull and informative Way of explanation. Please also consider visiting my site, I will be glad if I could be helpful for you.
Business Coach

liva said...

I really wanted to say thanks for this kind of post and I highly recommended it, thanks again for sharing such a wonderful post! Planes Wallpapers | Sports Wallpapers | Travel & World Wallpapers | Vectors & Designs Wallpapers

Saman said...

Thanks for the wonderful article, it is very appreciateable.
Phlebotomy
Phlebotomy Courses
Phlebotomy Courses