Customer Event Prediction in Online Subscription Products

At Intuit, we fall in love with the customer problem, not the solution, and are continuously looking for better and faster ways to serve them better. Technology has always been a huge enabler, aiding better customer experiences. What makes all of our products special is how successfully AI and data have been integrated to solve customer problems. 

Predicting customer events ahead of time is extremely important for Smart Sales, Marketing and Customer Service for subscription software such as QuickBooks Online (QBO). 

Some examples of customer events of concern include: 

• Upgrade to a higher SKU
• Cancellation of the subscription
• Connect to an add-on service
• First-time use of a new feature
• Abandonment of the task at hand

Predicting these events allows sales, marketing and customer service to operate more efficiently and provide personalized experiences to customers in an automated and scalable fashion. For example: 1) dynamically identifying customers who are at high risk of churn can help marketing do a proactive reach out to such customers, 2)  identifying customers better suited for a higher SKU can help sales target these customers, and 3) identifying a customer in real-time who is stuck in the product, and likely to abandon the task at hand, can be used to surface relevant help articles. 

These proactive reach-outs, both inside and outside of the software, can help the business increase customer engagement, resulting in more long-term-value (LTV) per customer, which is one of the most important business key performance indicators (KPIs) in subscription-based commerce.

Download our Mobile App

AI/ML can help to predict future customer events by learning from historical events, customer profiles and product usage data. However, training a machine learning model for predicting customer events is a challenging problem due to: a) the rarity of customer events, b) the majority of the data for customers is not fully baked at the time of training, and c) the temporal sensitivity due to the dynamic nature of the events.

Democratization of event prediction through Intuit’s Propensity Modeling as a Service (PMaaS)

Fig. 1: Components within the PMaaS Architecture

PMaaS is Intuit’s internal AI service to deploy large-scale event prediction models in production. Any authorized user within Intuit (e.g., product management, sales or marketing teams) can onboard a use case in PMaaS through ‘PMaaS API/CLI’.  Once onboarded, based on the provided requirements in the onboarding configuration, PMaaS automatically builds and validates multiple classification models and, finally, schedules weekly or monthly prediction generation to be integrated with the downstream sales and marketing systems.

To predict a  future customer event (e.g., customer churn or upgrade/downgrade of a subscription), first a reference time point, say ‘t’, has to be defined. For every eligible customer, the AI/ML models try to predict whether the concerned event will happen in the time period (t, t+H] for a given user. Here, H is a business-defined time period (such as 15 or 30 days), depending on the problem at hand. 

For multiple such reference time points in historical data, using an intelligent sampling approach (Bootstrapping/Sampling block in Fig. 1), the target Y(t) can be defined as a categorical variable, depending on whether or not the event happened in the time period (t, t+H]. Next, through the PMaaS Feature Pipeline, predictors, say X(t), can be computed for each reference time point. For example, customer profile information, subscription information, and product usage data such as transactions recorded, clickstream, etc., are usually considered predictors. Finally, (Y(t), X(t)) data for many combinations of customers and reference time points in history are used to build several state-of-the-art classification models (Model Train block in Fig. 1). Some of the best-performing models are then automatically deployed in production to generate the ‘batch’ of propensity outputs in a scheduled manner, e.g., weekly or monthly, for all the ‘eligible’ customers. The marketing and sales team can use these predictions to target the top customers (in terms of the predicted propensities) based on their bandwidth.

In the above-mentioned fashion, ML models within PMaaS can learn to predict time-to-an-event (denoted by T) by breaking the problem into a series of classification problems. For example, the probability that a customer will not cancel their subscription within the next 90 days, i.e.  P(T > 90), can be written as P(T > 90) = P(T > 30) P(T > 60 | T > 30) P(T > 90 | T > 60). Each of these probabilities can be estimated through a separate binary classification model, and the 30 days and 60 days define the candidate breakpoints or reference time points.

This representation reduces the time-to-event prediction into multiple disjoint classification problems. The number of such breakpoints/reference time points decides the bias-variance trade-off. With an increase in the number of breakpoints, the accuracy of the estimated propensities will increase (decrease in bias), whereas the size of the data set available for learning each classification model would diminish (increase in variance). The Bootstrapping block and rigorous validation in the model training block in PMaaS combine multiple models from different reference time points to optimally solve this bias-variance trade-off.

Extension of PMaaS for real-time event prediction for in-session intervention

Though batch predictions are good for targeting customers through marketing and sales channels, dynamic and scalable in-session interventions are often not served by batch predictions. For example, to assist a customer who is using an online product and facing some difficulties, an intelligent system needs to identify such customers in real-time, detect the type of help the user is looking for, and proactively provide in-session contextual help through a digital assistant or live expert. A batch prediction framework cannot help in such a scenario since it is not sensitive to in-session user activities. On the other hand, a complete real-time system that relies solely on user in-session clickstream data is often not enough to accurately identify top customers to target for the concerned event, e.g., churn, upgrade etc. 

To solve this problem, we have extended the PMaaS for real-time event prediction, as shown in the left figure. We have designed three components: (1) the PySpark Batch Processing is similar to PMaaS Feature Pipeline, which generates batch features and feeds to a Long Window Model (LWM) to get batch inferences – these inferences are stored in an online feature store (DynamoDB) to be used at the time of real-time inference; (2) Spark Stream Processing pipeline process the real-time Kafka events from user clickstream and stores it in an online feature store; (3) finally, at the time of real-time inference request, the stream processed clickstream is fitted to a Short Window Model (SWM) and the SWM output and LWM output is fitted to an Ensemble Model to get the final prediction.

The above approach has certain advantages, both in terms of accuracy and implementation:

1. The proposed mechanism for real-time event prediction pre-processes huge volumes of historical user behaviour data over the last few weeks in batch and thus reduce latency to incorporate long-term user-behaviour in real-time inference.
2. It processes in-session clickstream data in real-time, and hence, it is sensitive to dynamic user actions.
3. By segregating short-window and long-window feature processing, this mechanism reduces the feature space dimensionality. Combining the SWM and LWM scores through an ensemble layer improves the accuracy of targeting high-risk customers in real-time and in session.

The outcomes

At Intuit, by democratizing AI/ML-based customer event prediction through PMaaS, we have deployed and integrated several batch prediction pipelines for several events, such as subscription cancellation, feature usage, add-on, upgrade/downgrade of the subscription, etc. 

By using these AI predictions through the sales and marketing channels, we have seen a relative drop of 8% in customer churn rate – which led to a multi-million dollar revenue impact for Intuit. Finally, by extending PMaaS for real-time event prediction, we have been able to increase the scope of customer targeting to more contextual and scalable intervention methods, such as serving automated help articles through digital assistants, etc. We have seen a 1.6x lift in the prediction accuracy for detecting customers with a high risk of churns in the real-time system compared to batch systems, especially for brand-new customers who have only been using the product for a short time.

Co-author Credits: Shrutendra Harsola

BIO: Shrutendra is an applied machine learning professional with ~10 years of industry experience in building machine learning systems. He is currently working as a Staff Data Scientist / Tech Lead at Intuit AI team in Bangalore, focussed on developing ML/NLP solutions for QBO-Advanced. Prior to joining Intuit, he worked as an Applied Scientist at Microsoft BingAds team, where he worked on developing machine learning models for query reformulation, semantic ad matching and query-ad relevance. Academically, Shrutendra completed his bachelors from NIT Bhopal and Masters from IISc Bangalore. (https://www.linkedin.com/in/shrutendra/)