Taking a deeper look at the initial analysis

In my last post, I promised to look into a few questions a bit more deeply--so this post is about me making good on my promise. The big question had to do with question 3 of the survey.

3. Have you installed an add-on (including themes) to customize Firefox 3.0?:
Yes
No
Unsure
I do not know what an add-on is

The question I had was whether we could break these results down by usage so that we can understand why such a big number of users (~20%) are unsure of or unaware of what an add-on is.

Okay--bear with me--this table can be confusing but I think this may be the best way to show this information. The grey column holds the results without breaking down the data--ie, the results you'll find on the chart above this one. The "Light" column holds the results for all light users. 44% of light internet users (less than two hours a day) know what an add-on is. This is highlighted as orange because it is LOWER than the general result of 58.5%. 25.7% have not downloaded an add-on, and 13.6% are unsure, results that are higher than the general results in the grey column. These two percentages are highlighted in green because they are HIGHER than the general results. 16.5% do not know what an add on is, which is 6.5% higher than the general results, leading me to think a lot of users' awareness of add-ons comes from spending more time online with the browser.

"Moderate" internet users (2-5 hrs a day) follow the general results almost completely, which is why they were left in white. 67% of "heavy" users (+5 hrs a day) know what an add-on is, which is about 10% more than the general results. Only 16% of heavy users do not have add-ons and 7.5% of heavy users don't know what an add-on is--both much lower than the general results.

Once again, we can't know for sure why these results are the way they are, but I think its fair to suggest that the longer one spends time on the internet, the more likely one is to have downloaded or know what an add-on is. I think it is also fair to suggest that we may need to do a better job of improving the visibility of add-ons and their benefits for people who are only on the internet for less than two hours a day. We cannot rely on these people to find out about add-ons on their own, just by using the browser. We need to improve the accessibility and visibility of add-ons for light internet users in order to continue to grow that segment of our user base.

The next graph also looks at question 3, and compares it to how people found out about Firefox.

This next graphic is even more complicated, but works on the same premises. Green highlights means the respective number is HIGHER than general results, orange means the results were LOWER than general results. The numbers highlighted in blue are there because they may give us insight into add-on download behavior. The people who have add-ons are more likely to have heard about Firefox through an online blog or offline article. This makes sense because it is in these articles that there is more likely to be an explanation of what an add-on is, how to download them, or a description of the user's benefits because, well, there's space to. Online ads lack the space to make the case for add-ons, even though we (the Marketing team) have taken the initiative to try and promote them. Add-ons, as a concept, are hard to understand quickly, especially by people with less technology experience. Therefore, the more space we have to promote/explain add-ons, the more people are likely to download add-ons.

We can see this difference in the number of people who found out about Firefox through offline articles and have add-ons (73%) and those who found out through online banner ads and have add-ons (59%). It's interesting to compare these two categories because offline articles suggest a population still not comfortable with the internet as their recommendation source, but the category has a higher add-on download rate then the online banner ads category, a group you could assume would be more tech savvy. Compare this to online article rates (79%). They would arguably be the most tech savvy out of all the categories, so it makes sense that their add-on download rate is the highest. Peer recommendation is the lowest in the bunch--I'd expect people to focus on Firefox's speed and security before mentioning add-ons because speed and secuirty are tangible benefits, while add-ons aren't very easy to understand. This may be because IE has nothing very comparable to add-ons, thus current IE users and others don't really understand the concept.

Overall, looking at these numbers, it seems like the determinant of whether people know about add-ons and have downloaded them is the amount of space/time the source of information (recommending Firefox) likely has to explaining add-ons. Which then means, that we should be working on trying to make the concept of add-ons easier to understand and easier to EXPLAIN to others. The other take-away from this data is that the online/offline articles about Firefox seem to be doing an excellent job of communicating the benefits of Firefox, particularly about add-ons. Our PR team should be getting a huge pat on the back for doing an awesome job of getting the word out about Firefox.

Lastly, the next installment of the Survey is expected to be out in early December. I'll keep you posted.

Results for the Q3 Quarterly Customer Survey--First Look

After some thorough initial analysis, I am happy to share the preliminary/non-locale specific results of our first Firefox Quarterly Customer Survey. In general, it looks like this data confirms a lot of our "hunches" about our users and gives us more information about how we are doing when it comes to customer satisfaction--what some people view as the most important metric of this survey. At the same time though (surprise, surprise), there were some results we didn't expect. For this time around, I'm going to focus on our expected results--I'll go through the unexpected ones in the next posting. Before I get going, let me emphasize that although these results are interesting, we won't really have a good idea of how valid these results are until we've gone through a few rounds of quarterly testing. Also, I have not included question #4 for analysis ( Which of the following browsers have you used in the past week? : Internet Explorer, Safari, Firefox, Opera, Other) because we got conflicting answers from people, probably due to the wording of the question. We will reword the question for the next survey.

Here's a quick overview of the survey:
This survey was released on the 23rd of September and was available to 10% of FF3 updaters of both 3.02 and 3.03 in 10 locales, including: Japan, China, France, UK, Spain, Italy, Brazil, Poland, Germany and the United States. The basic goal of this survey was to create a standard the Marketing Team can use to periodically measure our users' awareness, usage habits and attitude towards the Firefox, a goal that I think we achieved.

Below you will find the six questions and some analysis. Others in the Marketing Team will also post some of their opinions, and in a future post, I will do more analysis of the responses to these questions by locale.

To begin, the six question survey was completed by 30, 272 people (the results are statistically significant)--a much higher number of responses than anyone expected. The distribution of respondents is here:

1. How did you find out about Firefox?:
Recommended by a friend or colleague
Offline newspaper or magazine article, review
Online banner ad
Search engine listing
Online article, blog, review
Other

If "other," please specify. Most popular answers were: "Cant remember how they started using it" or they've been using it for so long, they have no idea", "they've been using it since the beginning".

55% of responders found out about FF3 because of a peer recommendation, a finding that I expected in that I thought most people found out about Mozilla products by word of mouth. I did not expect that such a large majority of people would find out by word of mouth, imagining response rates closer to 40% and the other responses to be more evenly distributed. At the end of the day, seeing that the majority of people learn about Firefox through word of mouth is an excellent sign. For people to make the effort to recommend Firefox to another person means that they believe in our product enough and are satisfied enough to want to share it with others. As long as our satisfaction rates remain high, the rate of people finding out about Firefox from a peer will remain high--they are inextricably tied.

2. What is the primary reason you downloaded Firefox 3.0?:
Security
Customization
Performance
Open source
Recommendation
Other

If "other," please specify. Most popular answers were "Automatic update" or "All of the above".

This first graph shows that overall, performance seems to be the biggest reason people are choosing Firefox. Security, however, is not far behind--results we expected. I did not expect that customization to be so low down on the list, probably because I've always seen the ability to customize as the main difference between Firefox and the rest. At the same time, however, to have performance as the most popular reason people downloaded Firefox means that we're doing a good job--it means our product is doing what it's supposed to do and doing it well.

One reason customization may not have performed so well is that people don't know how to understand the term "customize"--i.e. they don't know that an add-on means customizing their browser. It also may be possible that people just don't know anything about add-ons, but that seems unlikely if you look at the results for question 3. I'll look into this more deeply for the next post.













3. Have you installed an add-on (including themes) to customize Firefox 3.0?:
Yes
No
Unsure
I do not know what an add-on is

The good news about this results to this question is that the majority of our users know what an add-on is and have installed one. At the same time, I think that for +20% of our users to not know what an add-on is seems a bit high. The next question to ask then is why? Is it an awareness issue? Is it an issue with the AMO site? Maybe people don't understand that installing plug-ins is the same thing as installing an add-on?

The next step here is to compare this data to our user profile information to try and figure out how a user's profile may explain this lack of awareness of add-ons--ie the people that don't know what add-ons are the ones who are on the Internet the least.

5. On average, how much time do you spend on the web?:
More than 8 hours per day
Between 5 and 8 hours per day
Between 2 and 5 hours per day
Between 1 and 2 hours per day
Less than 1 hour per day

This graph is pretty self-explanatory and the results are representative of what we expected. Most of our base is made up of moderate to heavy users, although there is a good number of light users. I think that the light internet user segment of the population is where the most of our growth needs to come from in the coming months, but its nice to see a good proportion of them responding to this survey. (see D Bottoms' Comscore data).














6. On a scale of 1-5, with 1 being 'not likely' and 5 being 'most likely', how likely are you to recommend Firefox to a friend?:

This last graph is the one that I think most people will end up thinking about and following the closest. To clarify, we got this result by combing the answers of people who responded with 1, 2 or 3 into the "unlikely" to recommend column and combining the answers of the people who responded with 4 or 5 in the "likely" to recommend column. Taken together, we have an 87% likely to recommend rating which can also be understood as an 87% customer satisfaction rating. This is an incredible number, one that few other products reach. As such, we will continue to monitor this rating and to see how it continues to trend over the coming quarters.

Be on the look-out for more posts in the next few days. I'll be posting a complete pdf version soon.