Customer Satisfaction and Achievement of Service Goals

Since we launched Retention Radar, a lot of people at SaaS companies have asked us what type of data they should put into a machine learning model in order to analyze their subscriber churn. The answer is very simple in theory, but there are a lot of challenges in practice. Let’s start with a simple reality check, without any statistics and data science: At the end of the day, subscribers to a service who think they are getting what they pay for are going to keep on paying (okay – only as long as they can – but voluntary churn is most churn in a typical SaaS company.)

Any subscriber who thinks they aren’t getting what they pay for is going to cancel, sooner or later (except those wonderful people who forgot they signed up, don’t use the service but keep on paying, but we’ll say more about that in a future post.) So that’s it: if you want to know who will churn, figure out who is and who is not getting value for their money. This is another way of saying that people are (mostly) rational and are just looking for a value for their money.

Goals A lot of the companies we talk to are measuring the number of logins, use of the site features, etc. And these are called “engagement metrics”. But are people really paying just to log in to your web site? Click on a few UI widgets and hover a mouse? Fun! Well, maybe not so much. Your website is delivering a service – the service is probably not a tangible good, and it is probably not explained simply by a click on a few website widgets and links. Most SaaS companies are offering to do something for their subscribers, or to help them do something. At Sparked we refer to the true aims of the subscribers using a service as the subscribers’ goals.

Subscriber goals in different SaaS services are very varied, but there always has to be one. Though they can be hard to define. In order to make a goal concrete and usable in analysis there are two main criteria:
  • A goal is something that is clearly valuable or desirable to the subscriber, either in and of itself or as a means to achieving another goal.
  • Achievement of the goal is Measurable or countable, at least in principle. The degree to which the achievement of goals is measurable in practice varies widely as we will see.
Let’s consider some goals for different types of services to make it clear, and for now we’re not going to worry about whether or the achievement of the goal is actually measurable.

  1. The goal of using a dating Web Site is to go on dates – with people you like. Okay, you can argue that some people want to hook-up and others want to find a spouse as their “true” goal. But anyone using the service who at least gets out there with people who they find a bit attractive are going to be satisfied with the service for doing its part (you can’t expect a website to close the deal for you!) A better objection to think about is that some people like to just interact on-line and rarely go out – that’s another goal for some people.
  2. The goals of subscribing to a financial management tool include using reports like balance sheets and cash flows to making financial decisions, and having ready access to reports for third parties like tax forms, etc. And note that we really mean using the financial reports, but setting them up is probably not really a goal for most people – we’d like them to be just there waiting for us when we need them (if only!)
  3. The goal of using a music/video streaming service is to listen to/watch music/videos that you enjoy. It could also include discovery: new music/videos that you didn’t know you would enjoy.
  4. Lastly, consider a traditional service that’s available on-line and off: a bank account. The goals of having a bank account include many things but at a minimum there is usually interest on deposits and writing checks to pay our bills.
In each example the goals are something that is pretty obviously desirable for the subscriber. It doesn’t matter so much why the goal is a goal for some people: it could be the emotional happiness, planning for the future, hedonistic pleasure, or even good old dollars and cents! They key is to recognize what the subscriber is expecting to receive for their hard earned dollars.
So how measurable are the goals we described above?

  1. In principle a dating service could know how many dates a person goes on, and even if they liked the people they went out with – you would just need every subscriber to fill out a detailed survey of their dating experiences. Okay, not so likely in practice. Survey response rates tend to be low.
  2. For a financial management tool achievement of the goals ought to be measurable, at least in a well-designed service. Here the challenge is mostly technical – a service should track detail beyond simple page visits to include things like amount of content in reports, whether it is printed/exported, sent to third parties, etc.
  3. For a streaming service it is easy enough to count how much someone is listening to / watching. To have perfect information about how much someone likes what they are getting requires complete and accurate survey responses – again, that’s not going to happen.
  4. For the bank account, interest and checks are definitely tracked in detail.

So what can be done if the subscriber’s goals are not measurable by the service? What can you use to model churn then? In that case you use whatever you can measure that is correlated with the goals – the more correlated, the better. That’s why using simple measures of logins or feature clicks can be used as the attributes in a churn model – because they are correlated, to some degree, with achieving the goals. Clearly, someone who doesn’t even login isn’t going to get anything out of the service. And to some degree, the more someone uses the features the more likely it is that they are achieving some of the goals. For the music/video streaming services, the amount of downloads captures the quantity aspect of the goal – the subscriber is receiving the content. So it’s clearly going to be correlated with the enjoyment of the service, which is the goal. What’s missing is the quality dimension, how much the subscriber really like the content.
Turning back to the examples above:

  1. A dating site might not know exactly how many dates a person actually goes on as a result of using the service, but it could definitely track how many connections are made through the service’s messaging feature. If it is allowed in the privacy policy, the service could even scan text messages for exchange of phone numbers or actual plans to go out. Also, a dating web site probably integrates with popular calendar apps and those events could be tracked as well. All of these actions can form events that go into a churn prediction model. These events will improve prediction because you can expect that a user who is exchanging more numbers and making more plans is going out on more dates, is happier with the service and is less likely to cancel it. (Until they find that special someone of course, but I’ll have more to say about that in a future post…)
  2. A financial management service should easily be able to keep track of how many reports a user creates and even prints or shares electronically. But the important distinction here is that it is not about just measuring how much a user is using the service at a gross level, but how much are they are doing the specific things that indicate accomplishing their goals. If it is permitted in the privacy policy, watching how much money a user manages would be worthwhile measure for a churn model as well: the more money someone is managing, the more value they are getting from the service.
  3. There really is no practical way a streaming service can know how much users like or don’t like every single thing they see and hear. (Since we don’t have users connected to a brain scan…) Probably the best option is to use the cancelling (skipping) of a song or video is an event in the churn model. That is the most consistent indicator of user displeasure available, other than thumbs up/down. As I mentioned above, thumbs up/down is more like a survey than an event since it is voluntary for the user, and many users don’t bother.

So why not just use logins or page visits or other simple events, and forget about the subscriber goals, which can be so troublesome to measure? Because if you use only counts of page visits you can’t distinguish between productive and un-productive users: Some users are going to login a lot, click on a lot of links and tools, but not get much done – and they are going to be frustrated and are likely to churn. There will be some power users who get loads done with least effort – they are likely to be the most satisfied customers, but not the most active in terms of logins and page visits. In fact, it is better to track both usage statistics and measures that more closely correlate with goal achievement in order to identify frustrated users.
So that’s it, in a nutshell:

  • The best data for estimating a churn model are metrics that capture the actual achievement of the goals of the service by the subscribers.
  • If that’s not possible then try to find something correlated with the goal.
Hopefully these examples will give you some better ideas for improving retention and churn metrics at your organization. Let me know if you know of an interesting case where a SaaS company can or can’t measure it’s subscribers’ goals, I’d like to hear what you think.

The following two tabs change content below.


Carl has a PhD in Computation and Neural Systems from the California Institute of Technology. Prior to his PhD Carl earned masters degrees in both Computer Science (NYU) and Neural Networks and Information Processing (King’s College London). After his PhD Carl spent seven years working in the finance industry at MSCI Inc. in the Fixed Income Research group where he developed statistical models of interest rate and credit risk for fixed income investors. Carl is excited to bring this wealth of experience to data science problems at Sparked.
0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *