With more than 200 million users logging in every month, the platform has a hectic task of collecting, handling and serving the data accurately without any latency. Although there are many conventional on-demand services and state-of-the-art machine learning models, the engineers at professional social networking platform LinkedIn were flexible enough to combine these conventional strategies with their built in-house tools to drive better results.
Challenges With The Recommendation System
LinkedIn employs a technique called ‘online learning’, where the model is trained is to get smarter with every activity of the user. This helps in narrowing down on the targets — jobs or connections.
One challenge the engineering team at LinkedIn faces is maintaining consistency on both ends; job poster and the candidate. The algorithms have to learn respective preferences of both job seeker and poster and deliver a relevant search by unifying multiple aspects like skills required, roles preferred- leadership beginner or preferred job location.
In order to obtain proper feedback, the online learning recommendation system distinguishes between the interactions with the applicant and that of the recommended match, which helps in explicitly representing the channel in which a potential portfolio has been discovered.
“One of the engineering challenges here is that we need to optimise for two-way interest. We want candidates to find the messages they receive compelling and not miss out on any opportunities, and we want hirers to reach out to candidates who are qualified and might be interested,” notes the engineering team at LinkedIn in a blog post.
In order to achieve this, the team uses Open Candidate status as an indicator to present the candidates with opportunities where the prospect(qualifications/eligibility) of getting hired is relatively higher.
Profiling With AI
Before delivering the results, the algorithm associated with the hiring process would consider information from job postings, search queries, feedback of the candidate. This information is aggregated in real time to learn about a user’s preference for a specific opening.
This can also be interpreted in terms of fundamental mathematical equations as follows:
The weight of a term i of type t from rating r in sourcing channel s is determined by the following equation:
Where
S is set of all channels, R is set of all feedback types
T is set of term types
ci,t,s,r is the number of candidates
Whereas, the Personalization Feature for a candidate with term type t from rating r in sourcing channel s is determined by the following expression:
The above expression gives the results regarding customised search which is a combination of various profile attributes and corresponding weights.
These personalisation scores are then run through XGBoost model. This model then produces results which can then be manipulated for matches by checking the scores of specific attributes like skills or experience.
The results of an online learning technique show that there is a 49.61 % improvement in Normalised Discounted Cumulative Gain (NDCG) when compared to a model without online learning.
NDCG is a measure of ranking the information that is being received from search engines. This is used as a tool to measure the effectiveness of search engine algorithms.
Since search results vary in their length, comparing these results with their associated scores requires normalisation. This is done by sorting at the corpus level. And these values can then be averaged to get an overall idea of the performance of the search engine ranking.
Future Direction
With employment rates hitting an all-time low in many parts of the world, LinkedIn is determined to contribute with its homegrown technologies, which have paved ways to a faster and better job search. LinkedIn has proved from time to time that to tap the most out of AI, it is necessary to build service-specific tools and frameworks in addition to the existing models and other upcoming personalised online platforms can learn a thing or two from LinkedIn’s success.
