I'm sure you've wanted to know the results from the survey we recently ran. We're incredibly grateful to all 242 of our respondents and your replies are extremely valuable to us.
The purpose of this survey was to gain insights allowing us to better suit the needs of you, our customers. It's important to me to share with you our results, and how we plan to use these insights to move forwards.
An interesting problem in designing your survey is your methodology. The survey had a mixed use of qualitative and quantitative questions. In this instance this means the difference between open-ended form answers (qualitative as the respondent can answer whatever they like) or something like a radio button list (quantitative because the respondent can select from a pre-defined list, which we can then extract direct statistics from).
When soliciting feedback, try to take any observer bias out of the equation (in this case my bias). This is important. To do this I left many key questions as free text responses. This way the respondent is not guided by my suggestions. While quantitative data is far easier to analyse, setting the categories ourselves would introduce this bias. It's better to leave these options open ended, see what responses we are likely to receive, and then use that information to optimize the design of further surveys.
The best example of this would be in our question regarding competitors to BugHerd (more on this later). If I had simply listed businesses the businesses we thought were our competitors in a series of check boxes, the data would be easy to analyze but inaccurate By allowing open answers, we received data very different from our expectations, which is invaluable for us in positioning and developing the product.
To overcome the difficulty of analysis we manually categorized the qualitative responses, which were then graphed. These categories can then be used in future surveys (with an understanding they are based on actual responses not my opinion) for statistical validation.
Now, onto the data.
Q. What is your primary role within the company?
We know that the majority of our users are developers of some description. What is interesting to us is the number of users who we would classify as managers. This would be users who, in their primary role, are no longer technical but tasked with looking after a team or organisational performance.
This is an extremely valuable piece of information to us when it comes to how we position the product and understanding how BugHerd finds its way into an organisation.
For example, the pattern we see is a developer encounters BugHerd and then recommends it internally to a manager/project manager. These people become the eventual account holder. Moving forward we need to find ways to remove as much friction as possible from this process.
This is quite different to a startup or freelancer environment where the person who discovers Bugherd is, in many cases, the same person who pays.
Finally, while we think that BugHerd has a potential to help Marketing or QA teams, it is interesting that only a very small proportion of our respondents come from these fields. This is likely as a result of the positioning we have employed to date.
Q. Of the people you use BugHerd with, who gets the most benefit?
What I find interesting here is the relative importance of non-technical roles like project managers and clients. This is particularly informative from a marketing perspective as it is not a segment that we have actively had messaging to support. It is also important in terms of considering what features to work on. Developers, designers and managers all want and appreciate the app differently, so it's often a balancing act deciding what to work on next.
Q. What is the primary benefit you receive from BugHerd?
Here we ran into one of the problems of our survey method. This was a free text response allowing us to generate high value insight into the language of our users. However, it does make it hard it visualize and there is always going to be error in re-categorisation of data (which I have had to do to make this graph). To give you an example of how I have done this:
Ease of Use:
"The interface. It is easy to use and clear to understand."
"Simplicity and clarification of communicating bugs."
"Time - it takes less time to report, find and resolve bugs"
"I absolutely love Bugherd. It has substantially decreased the time and effort it takes documenting issues during quality assurance testing"
"Simple communication from non-technical clients that still has all the details we need to address an issue."
"overview of open issues during development and prioritising during delivering a website"
"issue reporting within the website context"
"Pins that point directly to the visible bugs."
"The ability to facilitate communication between departments about what needs to be done on the websites"
"Efficient communication for remote workers"
The most interesting part to me regards efficiency and collaboration. I feel these are a result of other things (cause vs. effect)
Its great to know that things we have actively focused (Bugherd’s ease of use, workflow management & visual bug capture) are helping teams become more efficient & collaborate more effectively (achieved) which will positively impact the bottom line of our customers.
If you consider that, even on our highest plan for $180 p/m, Bugherd can effectively pay for itself by saving a team 2+ hours a month.
The next step will be to turn this question into a multiple choice and validate the categorization allowing for correlation between this and other data.
Q. If you could no longer use BugHerd, what alternative would you choose?
This was an extremely interesting response for us. We have traditionally defined our competitive set as being the full-featured bug trackers (Jira, Redmine etc.)
However, the top 7 responses (almost 60% of total) did not include any of these full-featured competitors.
It's important to note that this was an unprompted question (i.e. we did not list competitor names). If we did list competitor names we would likely have biased results based on what our assumption of the competitive set is.
Given that, what it does tell us? BugHerd is doing something that hasn't been done before, or is replacing some fairly prehistoric methods for accomplishing the same thing (screenshots and email anyone?). This is fantastic.
Q. Do you use Bugherd with external clients?
Wow! Exactly 50%!
This is not even as simple as freelancers & agencies vs. the rest. We had previously discovered that some agencies have been happy to use us internally but reluctant to use us with clients. This is often a cultural thing but something we wanted to explore in more detail below.
Q. How disappointed would you be if you could no longer use BugHerd?
People using BugHerd with external clients are more likely to be "very disappointed" if Bugherd no longer existed. This suggests the value that we are providing for teams using BugHerd externally seems to be greater.
What do we do with this information? It helps us focus our marketing and communication specifically. We have previously focused equally between internal teams and agency types. While this is still important we can see through the numbers that attempting to acquire more customers who use us with external clients will likely result in lower churn and greater recommendation due to the additional value they perceive.
Importantly this question is also a very good measure of customer satisfaction that we can actively measure improvement against over time.
Note: If you want to know more about the purpose behind this question then have a read of Sean Ellis’s blog.
Q. Have you recommended BugHerd to anyone?
The same pattern can be seen here. Of respondents, those people using it with external clients are significantly more likely to have recommend BugHerd to someone else.
Again this starts to tell us that much of the products value is in the way that it manages feedback between technical and non-technical teams. While we have always known intuitively this rings true it is good to get some data that supports our assumptions.
Q. If you where to recommend BugHerd, how would you describe it?
As a little addendum its important to understand what language people use when talking about BugHerd, Asking a question gives us extremely valuable information for landing page messaging and keyword strategy.
Note: I don't like word clouds very much but it seemed the best way to illustrate the point.
We learnt a huge amount through this process and a couple of key themes emerged.
- Ease of use, efficiency & workflow management are top tree benefits you get from BugHerd (and if BugHerd saves you time it pays for itself)
- Our competitive set is not what we thought.
- We provide more value to those of you using us with external clients
What do we do with this? It is most valuable in heaping us better manage our customer acquisition. The more we know about who our customers are, how they use BugHerd and what benefit they receive the better. This then fuels our marketing strategy though a combination of better targeting, ad messaging, landing page conversion and customer onboarding.
It took me about 1/2 a day to put the survey together using a simple Google Form and then a bit more to do the analysis However if you consider what we have learned, it's time extremely well spent.
Next up we will talk separately about the question "How can we improve BugHerd".