Using predictive analytics for client retention/CRM?

[CLKeenan]

One thing I'm working on is developing some models for using my usage analytics to determine which clients are likely to not renew their subscription. Is anyone working on something similar? I know there are people here that run SaaS subscription based businesses (looking at you Dchuk).

In my case one client = many users, so the most basic test would be to segment the users into different buckets (key stakeholders/power users/other) and then set targets for usage for each bucket. Then receive notifications if a user group falls below the target for X weeks in a row so we can reach out to those users to see if there's something we can do to help. My users are not the most technically savy, so sometimes it's as simple as 'I forgot my password'.

Thoughts?

 

[nickCR]

When I did this in the past for a sales system (determining who to call and when) we used a points system. Essentially we would have triggers that assigned points. You could do something similar on the user base. Then depending on the total points users generate for a client can help you determine which clients are top priority and most likely to buy again

 

[amateursurgeon]

There's an entire field devoted to this - lookup "Cohort Analysis"

https://www.kissmetrics.com/features/cohort-analysis

 

[dchuk]

there's actually not much out there about this for SaaS businesses. Analytics like Totango kind of do it, but I think there's room to improve.

Ironically, I have a fully mocked up idea for such an analytics tool that I want to try and build out over the course of the next month or two to use privately. I am planning on using Decision Trees as the machine learning basis to start. Essentially, decision tree algorithms try and identify the metrics that have the highest correlation with some outcome and then split your dataset into different buckets based on that metric (a Yes/No branch). Then it finds the next highest correlating metric, and so on.

You end up with literally a Yes/No flowchart that gives you a confidence score for an outcome. Then you can take action like reach out to the user before they cancel and encouraging them to use X feature because most people who convert use that feature a lot.

I am tempted to bring the tool to the market itself but not sure yet, it's literally just some photoshop designs and pages and pages of notes.

 

[Kiopa_Matt]

I'm working on something, and will hopefully have it released within a couple days.

It segments your traffic extremely well, and allows users to drill down into their data very well. You can go as broad or in-depth as you want. For example, you could view full statistics on 55-64 year old females in Baton Rouge, Louisiana, USA who came to the site from Facebook, and were shown the split test XYZ.

For comparisons, right now it just compares the different segments of traffic, and notifies you of any discrepancies. For example, it may notify you're getting really good traffic from 18 - 24 males, but the conversions and earnings-per-visitor just aren't up to par with the rest of your traffic. Then with a couple clicks you can create a separate split test (or multiple) that gets shown only to 18 - 24 year old males, then the system will watch the conversion rate to see if it helps.

For another example, it may tell you you're getting loads of traffic from 25 - 34 year old females in Germany, but conversions are shit. That will tell you it might be time to have the site translated to German, and maybe create a separate marketing campaign geared towards European females 25 - 34.

What I still need though is a good ranking algorithm, but after playing with it myself, I've realized there's some other math geniuses around who are far more qualified than me, so I'll see what I can do to bring one of them on shortly. I need to be able to plug in say 20 different variables regarding a segment of traffic, and have it analyzed against existing data, and spit back out a nice ranking.

Then very soon there will be the option to anonymously share your data with a global pool. If you choose to do so, all info regarding your traffic will be anonymously fed into a central database. In return, you'll have access to that data. So instead of just viewing your own statistics for say Texas, USA, you'll be able to view data on a wide range of industries / markets for Texas.

 

[CLKeenan]

there's actually not much out there about this for SaaS businesses. Analytics like Totango kind of do it, but I think there's room to improve.

Ironically, I have a fully mocked up idea for such an analytics tool that I want to try and build out over the course of the next month or two to use privately. I am planning on using Decision Trees as the machine learning basis to start. Essentially, decision tree algorithms try and identify the metrics that have the highest correlation with some outcome and then split your dataset into different buckets based on that metric (a Yes/No branch). Then it finds the next highest correlating metric, and so on.

You end up with literally a Yes/No flowchart that gives you a confidence score for an outcome. Then you can take action like reach out to the user before they cancel and encouraging them to use X feature because most people who convert use that feature a lot.


I am tempted to bring the tool to the market itself but not sure yet, it's literally just some photoshop designs and pages and pages of notes.

I like that idea. Going to add to it a bit. Create profiles for users based on their role within the company and create a list of activities 'successful' users complete. Then reach out to users who have not completed those actions with a tips and tricks guide customized for them.

 

[dchuk]

I'm working on something, and will hopefully have it released within a couple days.

It segments your traffic extremely well, and allows users to drill down into their data very well. You can go as broad or in-depth as you want. For example, you could view full statistics on 55-64 year old females in Baton Rouge, Louisiana, USA who came to the site from Facebook, and were shown the split test XYZ.

For comparisons, right now it just compares the different segments of traffic, and notifies you of any discrepancies. For example, it may notify you're getting really good traffic from 18 - 24 males, but the conversions and earnings-per-visitor just aren't up to par with the rest of your traffic. Then with a couple clicks you can create a separate split test (or multiple) that gets shown only to 18 - 24 year old males, then the system will watch the conversion rate to see if it helps.

For another example, it may tell you you're getting loads of traffic from 25 - 34 year old females in Germany, but conversions are shit. That will tell you it might be time to have the site translated to German, and maybe create a separate marketing campaign geared towards European females 25 - 34.

What I still need though is a good ranking algorithm, but after playing with it myself, I've realized there's some other math geniuses around who are far more qualified than me, so I'll see what I can do to bring one of them on shortly. I need to be able to plug in say 20 different variables regarding a segment of traffic, and have it analyzed against existing data, and spit back out a nice ranking.

Then very soon there will be the option to anonymously share your data with a global pool. If you choose to do so, all info regarding your traffic will be anonymously fed into a central database. In return, you'll have access to that data. So instead of just viewing your own statistics for say Texas, USA, you'll be able to view data on a wide range of industries / markets for Texas.

where are you getting demographic data for traffic at that level of detail without using facebook/linkedin/whatever sources?

 

[Kiopa_Matt]

where are you getting demographic data for traffic at that level of detail without using facebook/linkedin/whatever sources?

Simple. We ask for it. That's optional though, and I guess if wanted, I did scrape a good chunk of the US Census Bureau's 2010 data, so for US traffic you can base it off the county they're in. If they're in a predominantly Latino community, married or not, rich or poor, etc. For example, you may want to pitch differently to people in Orange County versus Compton.

It's geared more towards conventional SMBs who have a brick & mortar presense though, and not really for MFA sites, or some MMO membership group. For example, take a typical ski & snowboard store in Denver who has a website. When a new visitor comes to their site, a small bar appears on the bottom saying "please allow us to better assist you", and asks for the age range, gender, and maybe skill level (beginner, intermediate, or advanced). Obviously, a 22yo advanced male will get pitched different products than a 48yo beginner female.

That's the premise behind this. With brick & mortar stores, you only get one shot at product placement & design of your store, but your sales staff know exactly who's walking through the door. With online stores, you can change the product placement & design for each person who comes in if wanted, but you have no idea who's coming in. This is developed to help bridge that gap. Whether or not it'll work, fucked if I know. Time will tell.

 

[mituozo]

I'm working on something, and will hopefully have it released within a couple days.

It segments your traffic extremely well, and allows users to drill down into their data very well. You can go as broad or in-depth as you want. For example, you could view full statistics on 55-64 year old females in Baton Rouge, Louisiana, USA who came to the site from Facebook, and were shown the split test XYZ.

For comparisons, right now it just compares the different segments of traffic, and notifies you of any discrepancies. For example, it may notify you're getting really good traffic from 18 - 24 males, but the conversions and earnings-per-visitor just aren't up to par with the rest of your traffic. Then with a couple clicks you can create a separate split test (or multiple) that gets shown only to 18 - 24 year old males, then the system will watch the conversion rate to see if it helps.

For another example, it may tell you you're getting loads of traffic from 25 - 34 year old females in Germany, but conversions are shit. That will tell you it might be time to have the site translated to German, and maybe create a separate marketing campaign geared towards European females 25 - 34.

What I still need though is a good ranking algorithm, but after playing with it myself, I've realized there's some other math geniuses around who are far more qualified than me, so I'll see what I can do to bring one of them on shortly. I need to be able to plug in say 20 different variables regarding a segment of traffic, and have it analyzed against existing data, and spit back out a nice ranking.

Then very soon there will be the option to anonymously share your data with a global pool. If you choose to do so, all info regarding your traffic will be anonymously fed into a central database. In return, you'll have access to that data. So instead of just viewing your own statistics for say Texas, USA, you'll be able to view data on a wide range of industries / markets for Texas.

Isn't that very different to what the OP is talking about though? That's all about converting to sales, this OP's post is about reducing cancellations by reaching out to people using your software in a way which correlates highly with cancellations.

I think there's a great start-up to be built doing this. The challenge is making it simple enough for people to use.

You could integrate it with the main CRM systems probably as a starter for ten. I've no idea what the competition is like though. I guess you'd want to focus on the developer market too, by making a really simple API people can integrate with their products.

All you'd need is an API which submits an event to your system, with a user ID, another one to submit a new user, and a final one to submit a cancellation. You could then track all the actions of a particular user, and determine which event flows correlate most highly with cancellations. Build that up over time and you can contrast event flows to get an idea of how likely it is that someone is going to cancel this month, in 6 months, a year, etc.

Fuck, now I want to build it.

 

[CCarter]

I'm working on something, and will hopefully have it released within a couple days.

It segments your traffic extremely well, and allows users to drill down into their data very well. You can go as broad or in-depth as you want. For example, you could view full statistics on 55-64 year old females in Baton Rouge, Louisiana, USA who came to the site from Facebook, and were shown the split test XYZ.

For comparisons, right now it just compares the different segments of traffic, and notifies you of any discrepancies. For example, it may notify you're getting really good traffic from 18 - 24 males, but the conversions and earnings-per-visitor just aren't up to par with the rest of your traffic. Then with a couple clicks you can create a separate split test (or multiple) that gets shown only to 18 - 24 year old males, then the system will watch the conversion rate to see if it helps.

For another example, it may tell you you're getting loads of traffic from 25 - 34 year old females in Germany, but conversions are shit. That will tell you it might be time to have the site translated to German, and maybe create a separate marketing campaign geared towards European females 25 - 34.

What I still need though is a good ranking algorithm, but after playing with it myself, I've realized there's some other math geniuses around who are far more qualified than me, so I'll see what I can do to bring one of them on shortly. I need to be able to plug in say 20 different variables regarding a segment of traffic, and have it analyzed against existing data, and spit back out a nice ranking.

Then very soon there will be the option to anonymously share your data with a global pool. If you choose to do so, all info regarding your traffic will be anonymously fed into a central database. In return, you'll have access to that data. So instead of just viewing your own statistics for say Texas, USA, you'll be able to view data on a wide range of industries / markets for Texas.

Simple. We ask for it. That's optional though, and I guess if wanted, I did scrape a good chunk of the US Census Bureau's 2010 data, so for US traffic you can base it off the county they're in. If they're in a predominantly Latino community, married or not, rich or poor, etc. For example, you may want to pitch differently to people in Orange County versus Compton.

It's geared more towards conventional SMBs who have a brick & mortar presense though, and not really for MFA sites, or some MMO membership group. For example, take a typical ski & snowboard store in Denver who has a website. When a new visitor comes to their site, a small bar appears on the bottom saying "please allow us to better assist you", and asks for the age range, gender, and maybe skill level (beginner, intermediate, or advanced). Obviously, a 22yo advanced male will get pitched different products than a 48yo beginner female.

That's the premise behind this. With brick & mortar stores, you only get one shot at product placement & design of your store, but your sales staff know exactly who's walking through the door. With online stores, you can change the product placement & design for each person who comes in if wanted, but you have no idea who's coming in. This is developed to help bridge that gap. Whether or not it'll work, fucked if I know. Time will tell.

Kiopa_Matt... Did you hack my skype account? Predictive A/B Split test was going to be my next post. I've been working on it for weeks. God-dammit.

Great posts though!!

Looks like my job is done here. Carry on...

 

[cardine]

where are you getting demographic data for traffic at that level of detail without using facebook/linkedin/whatever sources?

I can do this for (almost) any user who lands on my site (no other info needed). If interested feel free to PM me.

 

[dchuk]

Isn't that very different to what the OP is talking about though? That's all about converting to sales, this OP's post is about reducing cancellations by reaching out to people using your software in a way which correlates highly with cancellations.

I think there's a great start-up to be built doing this. The challenge is making it simple enough for people to use.

You could integrate it with the main CRM systems probably as a starter for ten. I've no idea what the competition is like though. I guess you'd want to focus on the developer market too, by making a really simple API people can integrate with their products.

All you'd need is an API which submits an event to your system, with a user ID, another one to submit a new user, and a final one to submit a cancellation. You could then track all the actions of a particular user, and determine which event flows correlate most highly with cancellations. Build that up over time and you can contrast event flows to get an idea of how likely it is that someone is going to cancel this month, in 6 months, a year, etc.

Fuck, now I want to build it.

This is what got me excited about the idea. There are lots of good funnel analytics solutions out there like KISSmetrics or Mixpanel but there's really nothing at all for Free Trial analytics (which is specifically what I want to build a tool for). Funnel analytics are technically "easy" as it's linear, you start at one point and move in a single direction during a signup path. Once in an app though, usage is non-linear and you need a different solution for tracking usage.

It's a narrow niche as it would only work for Free Trial subscription products, but trials are perfect for this because there's a defined entrance point (sign up), defined exit point (either conversion or cancellation) and then actions in between (in the case of serpIQ, using the discoveries tool or generating a pdf report, etc).

A user can define all of these events, then feed the system with events per user taking the action, and then you can generate a model of what a converting user looks like and then can use that as a template for new users.

The next stage of such a product would be automating marketing emails and suggestions for usage, like Intercom.io and getvero.com do.

I can do this for (almost) any user who lands on my site (no other info needed). If interested feel free to PM me.

will do

 

[guerilla]

Kissmetrics totally works for free trials.

 

[greenleaves]

Simple. We ask for it. That's optional though, and I guess if wanted, I did scrape a good chunk of the US Census Bureau's 2010 data, so for US traffic you can base it off the county they're in. If they're in a predominantly Latino community, married or not, rich or poor, etc. For example, you may want to pitch differently to people in Orange County versus Compton.

It's geared more towards conventional SMBs who have a brick & mortar presense though, and not really for MFA sites, or some MMO membership group. For example, take a typical ski & snowboard store in Denver who has a website. When a new visitor comes to their site, a small bar appears on the bottom saying "please allow us to better assist you", and asks for the age range, gender, and maybe skill level (beginner, intermediate, or advanced). Obviously, a 22yo advanced male will get pitched different products than a 48yo beginner female.

That's the premise behind this. With brick & mortar stores, you only get one shot at product placement & design of your store, but your sales staff know exactly who's walking through the door. With online stores, you can change the product placement & design for each person who comes in if wanted, but you have no idea who's coming in. This is developed to help bridge that gap. Whether or not it'll work, fucked if I know. Time will tell.

Holy fuck. Thanks, awesome post, I mean, REALLY AWESOME. You gave me an idea to implement something. Absolutely awesome. Thanks. +rep

 

[mituozo]

This is what got me excited about the idea. There are lots of good funnel analytics solutions out there like KISSmetrics or Mixpanel but there's really nothing at all for Free Trial analytics (which is specifically what I want to build a tool for). Funnel analytics are technically "easy" as it's linear, you start at one point and move in a single direction during a signup path. Once in an app though, usage is non-linear and you need a different solution for tracking usage.

It's a narrow niche as it would only work for Free Trial subscription products, but trials are perfect for this because there's a defined entrance point (sign up), defined exit point (either conversion or cancellation) and then actions in between (in the case of serpIQ, using the discoveries tool or generating a pdf report, etc).

A user can define all of these events, then feed the system with events per user taking the action, and then you can generate a model of what a converting user looks like and then can use that as a template for new users.

The next stage of such a product would be automating marketing emails and suggestions for usage, like Intercom.io and getvero.com do.



will do

Surely it has applications in paid solutions too? You can correlate events and flows with cancellation rates, and flag up in the company's CRM system that someone should be contacted and told more about X, Y or Z if they're using it in a way that correlates highly with cancellations. (I'm talking about enterprise products, things like that).

It could also be used to note flawed features, e.g. people using a particular feature lots which cancel more often than other users.

 

[dchuk]

Kissmetrics totally works for free trials.

It works for the signup funnel really well. And with cohort analysis you can do some simple stuff for when someone is actually in the trial, but they offer absolutely nothing in terms of machine learning/predictive analytics like we're discussing here.

Given enough user data, you can fairly well predict who will cancel during a trial, and if you know that ahead of time, you can increase support to those users in danger of canceling and help your bottom line. KISS doesn't have a solution for this at all.

Surely it has applications in paid solutions too? You can correlate events and flows with cancellation rates, and flag up in the company's CRM system that someone should be contacted and told more about X, Y or Z if they're using it in a way that correlates highly with cancellations. (I'm talking about enterprise products, things like that).

It could also be used to note flawed features, e.g. people using a particular feature lots which cancel more often than other users.

Sure, it could be used for that. The system I intend to build will be for just trials though as it's a neat, compartmentalized problem to solve via software. Not looking to create a be-all-end-all solution, just something to help after the user has left the signup funnel.

 

[guerilla]

It works for the signup funnel really well. And with cohort analysis you can do some simple stuff for when someone is actually in the trial

With cohort analysis, you can do anything unless you want someone to build an algorithm for you. Kiss captures the data, and you can query it back any way you want.

Maybe there is room for an algo in between the data and the user, but Kiss does everything you need it to do. It all comes down to how much work you want to put into setting it up.

Given enough user data, you can fairly well predict who will cancel during a trial, and if you know that ahead of time, you can increase support to those users in danger of canceling and help your bottom line. KISS doesn't have a solution for this at all.

The data is all in there if you set it up right.

 

[mgtarheels]

Kiopa_Matt... Did you hack my skype account? Predictive A/B Split test was going to be my next post. I've been working on it for weeks. God-dammit.

Great posts though!!

Looks like my job is done here. Carry on...

Genetify?

 

[Pb.com]

It works for the signup funnel really well. And with cohort analysis you can do some simple stuff for when someone is actually in the trial, but they offer absolutely nothing in terms of machine learning/predictive analytics like we're discussing here.

Given enough user data, you can fairly well predict who will cancel during a trial, and if you know that ahead of time, you can increase support to those users in danger of canceling and help your bottom line. KISS doesn't have a solution for this at all.



Sure, it could be used for that. The system I intend to build will be for just trials though as it's a neat, compartmentalized problem to solve via software. Not looking to create a be-all-end-all solution, just something to help after the user has left the signup funnel.

FYI, KM just hired 3 ML guys. I know because they tried hiring a friend who works in the finance department at a bank. To top it off, I don't think they are stopping there.

Not saying you can't be more agile and microtarget customers (they are going after maximum profits), I just thought you should be aware.

 

[dchuk]

FYI, KM just hired 3 ML guys. I know because they tried hiring a friend who works in the finance department at a bank. To top it off, I don't think they are stopping there.

Not saying you can't be more agile and microtarget customers (they are going after maximum profits), I just thought you should be aware.

they also just announced this: KISSmetrics Launches Power Reports

still not what I was describing but they're getting there