Watched Netflix and Prime in my free time – the “new” normal for everyone. Netflix and Prime have such engrossing content that keeps us glued to the screen all the time. There is a section on both of these platforms which displays the recommended content on the basis of the previous content that you have watched. These recommendations seem to be quite relevant to your watch history and the kind of content you would want to engage yourselves with. How this works in the background is by designing certain recommendation systems.
Recommendation systems are a set of algorithms which give you recommendations based on your history. The most common example where you must have seen recommendation systems in action would be Amazon where you would see similar products displayed under the particular product that you have chosen to explore. This kind of list is displayed using the recommendation systems which are running at the backend. This is an example of collaborative filtering which uses item to item based mapping which follows the psychology that if a user buys a product x he will buy product y. Recommendations systems were traditionally divided into collaborative filtering and content-based filtering.
Collaborative filtering is very widely used and it is based on the concept of people liking the products/ services in the past and continuing with the same trend in the future too. By using this methodology, we can create user/item mapping for giving recommendations on the basis of future actions. For example, a user rated comedy movies higher in comparison to the other genres in the past so the recommendation system will recommend the user comedy movies in the future.
These collaborative filtering techniques make use of distance-based similarity metrics to identify users who are similar to each other and then subsequently make recommendations based on the user – user mapping. The most common method that is used for finding similar users is by using the KNN approach. Collaborative filtering is currently being used by social networks to recommend friends/connections. Some of the common problems faced while implementing collaborative filtering is a cold start problem. The cold start problem arises when there is a new user in the system. The recommendations for the new user are generally made on the basis of the most popular items in the catalog.
Content based filtering makes use of the item features i.e. product features like product title and other attributes along with the user preferences to make a calculated recommendation. It is mainly described as a classification problem and the classifier is trained to recommend it on the basis of the user’s likes and dislikes. Data for the items is collected implicitly or explicitly and a user profile is built on the basis of that.
This user profile is used to make suggestions to the users. The recommendation engine improves with the help of user actions on the recommendations. The concepts used in content based filtering are of TF – IDF. TF refers to the term frequency which is the number of times the word appears in the document. IDF refers to Inverse Document Frequency which measures how important the term is.
- TF(t) = (Number of times term t appears in a document) / (Total number of terms in the document).
- IDF(t) = log_e(Total number of documents / Number of documents with term t in it).
The above formulas are used to calculate tf*idf metric. This is one of the approaches used in content based filtering.
These were the basic concepts used around recommendation systems. But with time there was a birth of hybrid recommender systems which made use of a lot of new techniques. The Netflix Prize was an ongoing competition which aimed at improving the accuracy of the recommendation system already pre existing within the Netflix system. This prize saw the rise of use of hybrid recommender systems. Hybrid Recommendations Systems are a combination of collaborative and content based recommendation systems. Netflix is a very big example of hybrid recommendation system as they make recommendations by :
- A comparison of watching and searching habits of users and finding similar users on that platform. This makes use of collaborative filtering.
- Providing such movies to the users which share common characteristics with the movies which have been rated highly by the user. This is more towards content based filtering.
Hybrid systems can make use of different techniques like a weighted performance of the combination of filtering applied or can use a switching technique where we could use the concept of voting.
The nature of recommendation systems has become very competitive. Terms like SVD (Singular Value Decomposition) and NMF(NonNegative Matrix Factorization) are being increasingly used. SVD is a dimensionality reduction technique and is a collaborative filtering technique. NMF is mainly used in cases where there is a high dimensionality of data and extraction and analysis can help extract meaningful features from the matrix.Earlier machine learning would suffice in catering to the different recommendation systems, but with the increasing number of subscribers and online users, deep learning and neural networks are mostly used to deal with the enormous data and make subsequent recommendations.
