What Are the Biggest Challenges in Big Data Analytics?

Key takeaways

Many organisations are yet to fully embrace data-driven decision-making, hindering analytics adoption.
There is a significant shortage of professionals skilled in big data analytics.
eMudhra faces specific challenges in the analytics space, impacting its strategic initiatives.

As part of this month’s theme -- “Challenges in the analytics industry”, we bring an interview with Kaushik Srinivasan, Senior Vice President(SVP) of eMudhra, a popular tech company which provides data security solutions for digital documents across the nation. Srinivasan is also the Domain Coordinator for eGovernment at UN/CEFACT and facilitates projects that focus on the use of Blockchain, IoT for Trade Facilitation and cross-border recognition of electronic interactions. In his previous stints, he worked with top financial companies like Trafelet Delta Funds and UBS. He is also a CFA Charterholder and a member of New York Society of Security Analysts and Indian Association of Investment Professionals. In this conversation, Srinivasan talks about the challenges faced by eMudhra in the analytics space, and how these insights can have a positive impact. Analytics India Magazine: What are the 3 key challenges you face being in the analytics industry? Kaushik Srinivasan: Firstly, a lot of organizations have not shifted to a data-driven decision-making process where analytics becomes integral to their overall business strategy. This has inherently led to the lower adoption of analytics platforms at enterprises but this is changing gradually. The second challenge is an acute shortage of professionals who understand big data analytics.As an analytics platform vendor, it’s easier to convince a customer to buy an analytics product if they have a ready understanding of use cases but need platforms to solve their challenges of processing data at scale. The third challenge from a vendor perspective is the ever-changing platform landscape – technologies are getting outdated very fast. We were on Hadoop, already we have Spark which does distribute in-memory computing. So, a big challenge is how does one plan their tech stack in a way that it is modular. AIM: Talent shortage is often considered the biggest challenge. What are your thoughts on it? KS: This is absolutely true as college curriculum currently does not reflect the needs of the market. AI is going to explode and unless we start training people right from college, this shortage will continue to exist. AIM: What are the key challenges while setting up analytics and AI startup? KS: We started off in India, the biggest challenge in analytics is creating awareness and making end users understand that analytics can indeed drive decision making in creating better customer experiences or for instance reducing fraud. Analytics sales always have to be married with domain-specific use cases for sales to happen. Coming from technology backgrounds, we had to get domain experts onboard to figure out industry-specific problems and tackle them with analytics platforms and show tangible benefits for end-user adoption. This process took a lot of time. AIM: What according to you is the biggest challenge in carving an AI roadmap – budget, talent or senior management buy-in? KS: It’s a combination of all three as budget and senior management buy-in is invariably tied in. A lot of times, it’s hard to define tangible ROI on analytics projects before they start. Then it becomes a difficult call for senior management to allocate budgets for the adoption of analytics. Shortage of talent adds to this question — can you run meaningful analytics projects internally, what kind of training is required, can you find the right outsourcing partner. AIM: What are the biggest challenges in moving from pilot to production stage? KS: A good analytics platform is only as good as the input data that is fed into it. From pilot to production, multiple challenges exist. These include:

Inability to accurately predict data volumes and therefore size hardware appropriately for large volume data processing
Multiple legacy systems in production each having their own data stores requiring need for a separate data warehouse
Data accuracy and cleansing
Ability to quickly retrieve data from multiple sources

AIM: How has the analytics industry evolved over the years? KS: It has certainly matured as Business Intelligence is slowly transforming into Artificial Intelligence. The recent Google announcement of AI Chatbots powering real conversations with humans on the other side of the phone is a huge milestone and shows what’s possible with technology and its ability to solve complex real work problems. From a technology standpoint, superfast open-source processing platforms running on commodity hardware have emerged thereby opening up advanced analytics to anyone that wants to do it. AIM: How is the tech industry grappling with finding the requisite skills to beef up the bench strength? KS: We have seen that we able to train engineers over 2-3 months and get them ready for analytics engagements. In general, I think the talent pool is smart, with a little bit of training, they pick up new technologies very quickly and are able to solve real-world challenges. Over the recent interviews, we have seen a lot of self-motivated candidates who go the extra mile to learn analytics even though it’s not on the job as they realize they need to be ready for the wave. AIM: How does one decide what part of the IT budget should be allocated to emerging tech – ML applications, AI-based products? KS: I think at minimum 20% of budgets should be allocated to R&D efforts and analytics adoption. Over time this share will definitely grow higher as companies that adopt analytics will have a significant edge over those that don’t. AIM: What performance metrics are used by the C-suite to measure the business outcome – growth in revenue, improved business margins or other factors? KS: Analytics can help in multiple areas such as delivering better Customer Experience and therefore Revenue, Cost Optimization, Compliances, Risk and Fraud Management.

Acute Shortage Of Professionals Who Understand Big Data Analytics The Biggest Challenge: Kaushik Srinivasan Of eMudhra

Key takeaways

Stay ahead with AI insights.