Implementing Startup Metrics for Pirates – Part 2: Conversion Funnel
Written by David Mytton — Subscribe now.
Back in October 2009 I wrote about using MixPanel for implementing startup metrics for pirates. This focused on measuring feature usage rather than actual conversion to paying customers, which is of course the ultimate goal. Since then I have played with a number of different tools and considerably enhanced our funnel. This post will look at these things.
The funnel for our server monitoring service, Server Density, looks like this:
- The user visits our website.
- They sign up and log in
- They add a server to their account
- They download and install the monitoring agent
- The user then either upgrades to paid, downgrades to free or lets their account expire
This is split up differently (e.g. signing up and logging in as separate actions) depending on which tool we’re using.
Although we’re using MixPanel to track usage of features within the app, Google Analytics is actually an excellent tool for tracking most of this funnel. We can track the user from landing on the site to completing the upgrade form, and every step inbetween except for installing the agent (since this is not a page view).
The problem with MixPanel was that the funnel structure was too rigid. It has to be defined in Analytics too but it is a lot more flexible about where users can enter in on the funnel and it ties in with the general website analytics too. I find MixPanel much better at logging actions like the dashboard refresh I wrote about previously, even if logging lots of actions gets expensive due their pricing structure.
I also tried KISSmetrics and whilst it has an excellent UI and seems like a very flexible product, it didn’t seem to report website views properly, so we dropped it in favour of Analytics.
Google Website Optimizer
Making changes is great but you can never know what the actual effect is if you don’t measure. A/B testing is the way to do this and Google Website Optimizer does a great job for free. We use this for all our website tests.
Part of Dave McClure’s metrics is AARRR – Acquisition, Activation, Retention, Referral and Revenue. Using the data we have from our own customer system and the Google Analytics API, I was able to build a real time AARRR dashboard. Google Analytics is a useful tool but this is what I really use to measure the important metrics.
This tracks our funnel in the form:
F1 Website visit ⇒ F2 Signup ⇒ F3 Add server ⇒ F4 Receive data ⇒ F5.1 Trial upgrade OR F5.2 downgrade
From here I can see that in June we had a website signup rate of 3.8% but when a popular blog post appeared on Reddit, YC and a number of other websites, whilst the July traffic to the website increased 531.34%, signup rate actually decreased to 1.04%. Signups did increase but there was also a drop in the % of users completing the “Add server” step.
Funnel optimisations – signup form
When we were asking users for a subdomain for their account e.g. example.serverdensity.com we found a lot of people were entering full domains e.g. www.example.com which were then being converted to a URLformat e.g. wwwexamplecom, and not what the user was expecting. This was solved by adding a real time JS label that shows what your end result will be:
We ran an A/B test for a month and actually saw only a 0.47% improvement in signups. However, users were no longer creating “bad” account URLs so we rolled it out anyway.
Funnel optimisations – adding a server
Previously, on first login you would be presented with a check list of things to do next. This included things like adding a phone number and updating your timezone, but buried in there was the “Add server” link. In reality, the first thing we want you to do when you log in is to add your server.
Now on first login the only thing you see is the “Add server” form which is a single field – nothing else shows up on the dashboard. As soon as you have done this you are taken to the agent installation instructions screen to proceed to configure the agent on your server.
This change increased the number of users who add a server from 74.74% in Oct 09 to 86.09% in Nov 09.
Make things as simple as possible. We thought we were doing that originally but when you examine each step of the funnel in detail, especially with the outside input from ThoughtWorks, we were able to optimise each step even more.
However, we had to be able to measure these changes. Where possible you want to use a 3rd party metrics system like Google Analytics of MixPanel because building your own metrics system is difficult. We only do it for the aspects of the funnel we know most about – things like adding servers and receiving data.
Finally, measuring does not always mean looking at conversion for improvements. This is probably the most important but if you can reduce frustration or unexpected outputs as we did with the new form changes, that is a bonus too because it means happier users!
Part 3 – Revenue
All of this is leading up to one thing – revenue. I’m writing these posts as I build our tools and run tests, which means revenue is what I’m looking into right now. I’m still building the “Revenue” part of the AARRR dashboard and we’re running a number of tests that I want to write about in the next post. Unfortunately this means it’ll be at least a month or two but do stay tuned!
Update: Part 3 is now available here.