Why Are Senior Data Scientists Missing From Kaggle?

Kaggle has emerged as a popular platform for newly-minted data scientists and those that seek a simulated platform to put their skills to use, but the upper echelons of the data science community are largely missing from it. While winning Kaggle competitions has become an unspoken rite of passage to the next phase for amateur data scientists, the platform still offers an ideal testing ground for experienced professionals to practice new skills and discover latest tools.

Active participation demands a dogged persistence and a drive to learn, neither of which are lacking in data scientists across the spectrum. Then why are senior data science professionals walling off platforms like Kaggle? We find out.

Lack Of The Most Valuable Asset – Time

Almost all the data scientists we spoke to for this story were unanimous in stating that time could be the foremost reason why senior data scientists may be missing in action on Kaggle. 

“A senior data scientist is typically occupied with a lot of deliverables in their professional work, which may give them very little bandwidth to actively participate in Kaggle,” says Director of Data Science at Dell, Saurabh Jha. “Moreover, apart from work, most prefer to spend their time reading research papers, collaborating with AI researchers, conducting workshops or even speaking in AI conferences. Amid all of this, they may not have the time to work on Kaggle problems,” he adds.

This may be true, because for data scientists to truly get the most out of the platform, they need to commit a lot of time and energy into it – both of which may be in short supply among senior data scientists. Adds Prashant Kikani, data scientist at Bengaluru-based startup, Embibe:

“Unlike their younger counterparts, they may not be participating to win these competitions. Their focus, instead, is on learning something new. But without that drive to win, it may be tough to keep interest levels up,” he says. “That is not to say that senior data scientists do not rank highly on the platform – some do, but most may be preoccupied with other priorities in their personal and professional lives,” he adds.

ALSO READ: How To Initiate Your Kaggle Journey

Used As A Medium To Build A Strong Resume

Another possible reason for its unparalleled popularity among the younger breed of data scientists is its increased use in getting a job. Many companies use Kaggle performance as a metric to gauge a candidate’s expertise when evaluating them.

“Many of the participants in Kaggle are either just finishing their education or have less than 3-4 years of experience in the domain,” says Shefali Bedarkar, a machine learning professional at Mphasis. “Most are either looking to add a layer of experience to their portfolio, or strengthen their resume to switch jobs or increase their employability,” she adds.

But Shilpa Rao, Head – AI powered Strategic Intelligence and Sustainability at TCS adds a caveat:

“The way it’s structured, Kaggle competitions are conducive for data scientists who are just getting started, or are in the early stages of their career. However, it will add value to their resume only if they have a good rank,” she says. 

While it is true that these job-related incentives could be an important factor that drives participation, it may not be enough to keep senior data scientists interested.

“Most Grandmasters end up getting lucrative jobs as many companies prefer hiring them,” says Usha Rengaraju, who is incidentally India’s first female Grandmaster. “For an already established data scientist or someone who has his own company, that may not be the motivating factor to participate in Kaggle competitions,” she says.

Adds Saurabh, “Kaggle is also taken up by aspiring data scientists who want to get into this field quickly, and learn the skills of data munging, visualisation, modeling, and more. Another set of people who are drawn to these competitions are data scientists who have taken time off from professional responsibilities or are in between jobs. For instance, if an ML engineer wants to hone his computer vision skills, Kaggle offers the perfect opportunity to learn it by solving a related problem,” he adds.

Thrill Of Solving Real-World Problems

Problems that play out in the real-world are undeniably more challenging and complex than simulated ones. These competitions, although a great platform for amateur data scientists to imbibe good problem-solving skills, pales in comparison to the issues that transpire in an actual business setting.

“Kaggle competitions are essentially designed. This means that the data is cleaned into some usable form and success criteria are set,” says Shilpa. “For a senior data scientist, who has probably worked on real-life problems that involve discussion with stakeholders, extensive time in arriving at the right data, identifying additional sources to support real-life assumptions and then modelling, these competitions may not be as engaging,” she adds.

This manufactured problem, then, may not offer the same rush that solving a real challenge will bring. The sense of gratification in solving real-life problems for a company or a customer is much higher. Adds Navin Manaswi, founder of WoWExp Technologies:

“Solving real business problems may even be fun for data scientists, since it involves data preparation, machine learning, data visualisation, business analytics, project planning and deployment,” he says. “Kaggle, on the other hand, offers a small and watered down version of this business challenge. It mostly focuses on machine learning and data preparation, which generally interests data scientists in the initial stages of their career,” he adds.

ALSO READ: Winning A Kaggle Competition Is Not Enough – Learn How To Take The Learning Experience Forward

Shift To Managerial Roles & Change In Outlook

Active participation and accomplishing higher ranks on Kaggle requires very good coding and technical skills. However, while many senior data scientists continue to pick up more skills and regularly update themselves on latest tools and approaches to solve problems, a shift to a more managerial position would mean a shift in priorities as well.

“Most Kaggle competitions are graded by ‘accuracy’ like ROC, MSMS, Sensitivity, etc. Higher the accuracy, higher is the chance of winning. And to get there, participants need to apply complex data science algorithms,” says Shishir Gupta, Head of Data Science & Partnerships at NBFC Loan2Grow. 

According to him, accuracy of a model alone does not dictate success in the data science domain as professionals climb the seniority ladder. Instead, Shishir firmly believes that it gradually hinges on these five questions instead:

• How well are data scientists able to solve a given problem?
• Is the solution in line with current business logic?
• How easy and quick is it to deploy the model in their system?
• Are the results interpretable to a non-technical person?
• Can the same algorithm be used for another product or hierarchy with minimum change?

“To achieve the above parameters, it might not be necessary to apply the most complex algorithm which gives the best accuracy,” he says. “Rather, it may suffice to create a relatively simple algorithm to achieve the above goals, and take a slight hit on the final accuracy,” he adds.

Wrapping Up

Kaggle competitions require dedicated effort over a period. Although there are no restrictions with age, and there are quite a few senior Kagglers on the platform, time and motivation are the two important factors that drive active participation. Usha concludes:

“There are a lot of young Kagglers who have dropped out because of work overload and the inability to cope with rejection,” she says. “It takes a while to build the skills and to get medals. But persistence and consistency are very crucial,” she adds.