A recommendation engine is a class of machine learning algorithm that suggests products, services, information to users based on analysis of data. Robust recommendation systems are the key differentiator in the operations of big companies like Netflix, Amazon, and Byte Dance (TikTok parent) etc.
Alok Menthe, Data Scientist at Ericsson, gave an informative talk on building ‘Custom recommendation engines for real-world problems’ at the Machine Learning Developers Summit (MLDS) 2021. “Whenever a niche business problem comes in, it has complicated intertwined ways of working. Standard ML techniques may be inadequate and might not serve the customer’s purpose. That is where the need for a custom-made engine comes in. We were also faced with such a problem with our service network unit at Ericsson,” he said.
Menthe said the unit wanted to implement a recommendation system to provide suggestions for assignment workflow — a model to delegate the incoming projects to the most appropriate team or ‘resource pool’
Sign up for your weekly dose of what's up in emerging technology.
Credit: Alok Menthe
There were three kinds of data available:
Download our Mobile App
Pool definition data: It relates to the composition of a particular resource pool–the number of people, their competence, and other metadata.
Historical demand data: This kind of data helps in establishing a relationship between the feature demand and a particular resource pool.
Transactional data: It is used for operational purposes.
Menthe said building a custom recommendation system in this context involves the following steps:
- Using clustering algorithms to generate clusters of resource pools based on similar meta-features
- Re-labelling the pool name in the historical project demand data with a corresponding pool cluster label
- Using the classification algorithm to classify project demand into one of the cluster pools
- Based on the real-time resource availability and qualification score, ranking the classifier output cluster
- Recommend the most suitable pool for the resource manager for task/project assignment
Credit: Alok Menthe
“After building our model, the most difficult part was feature engineering, which is imperative for building an efficient system. Among the two major modules –classification and clustering–we faced challenges with respect to the latter. We had only categorical information making it difficult to find distances within the objects. We went out of the box to see if we can do any special encoding for the data. We adopted data encoding techniques and frequency-based encoding in this regard,” said Menthe.
Clustering module: For this module, initially the team implemented K-modes and agglomerative. However, the results were far from perfect, prompting the team to consider the good-old K-means algorithm. For evaluation purposes, it was done manually with the help of subject matter experts.
The final model had 700 resource pools condensed to 15 pool clusters.
Classification module: For this module, three kinds of algorithm iterations were used–Random Forest, Artificial Neural Network, XGBoost. Classification accuracy was used as an evaluation metric. Finally, upon 50,00,000 training records, this module demonstrated an accuracy of 71 percent.
Results & Takeaways
Menthe said this recommendation model is monitored on a fortnightly basis by validating the suggested pools against the allocated pools for project demands:
The model has proved to be successful on three fronts:
- Fulfilment ratio: It is the number of demands that were satisfied against the total demands coming into the system. We aim to increase this ratio on a weekly basis.
- The time taken for the demand to enter the system and exit with successful routing. Currently, the time taken is 2-3 days, the goal is to reduce it to 1 day.
- First time right routing: It determines the instances in which the demand routed to a particular pool was correct in the first attempt itself.
Menthe summarised the three major takeaways from this project in his concluding remarks: the need to preserve business nuances in ML solutions; thinking beyond standard ML approaches; and understanding that there is no ‘silver bullet’ ML solution.