How to make your First Big Data Initiative a Success? – A Vendor’s Perspective

Big data analytics offers revolutionary capabilities but achieving results requires a strategic vision—and a view into how it can create business value. Over the last few years, the term “big data” has evolved from a vague concept into a mainstream business strategy. But somewhere at the intersection of conceptual promise and real-world implementation, many business leaders, including CIOs, have faced a tough realization: putting the concept to work requires different tools, technologies and strategies than in the past.

Imagine you are a CIO who has identified a business use case for Big data analytics but is figuring out how to go about executing it. Tools, platforms, skill set and domain expertise all these weigh on your mind as you try to put together a team. Roping in external partner is an advised course whether for consulting or for full implementation. Reason is simple: your experienced partner knows best as to what works and where things can fail. So he can limit your downside risk and at the same time maximize your upside gain.

Valiance recently did a Big data analytics project with an enterprise communications firm whereby we were supposed to extract customer insights from terabytes of unstructured data they had. We were responsible for creating in house big data infrastructure to store and process data, data mining, creating machine learning algorithms and finally exposing result set through an API.


Sign up for your weekly dose of what's up in emerging technology.

Here are key learning’s we have had from this engagement and we recommend you to keep same in mind when engaging external help

  1. It’s best to clearly understand technical approach used by your partner for data mining or machine learning along with related limitations to avoid disappointment later. There are limitations to what one can achieve with all the best algorithms available at your hand. You should definitely spend some time in understanding how related technologies and algorithms work. Understanding where outcomes may not be favorable can help you manage stakeholder expectations.
  2. Big data analytics project demand infrastructure and often it’s difficult to exactly estimate capacity requirements in the start. Going for a fixed CapEx investment complicates the matter when your team needs additional capacity and that too variable. It’s best to consider using cloud infrastructure for your project if there can be uncertainties with processing workloads.
  3. Defining rational measures of success. Big data promises to be revolutionary technology with answers to all your problems. It may or may not be the case. It’s advised to define achievable and tangible measures of success rather than having irrational expectations. Big data isn’t answer to all your problems. Period!
  4. Collaborate with your partner throughout the exercise and not just towards the end. Big data analytics project demand more collaboration and attention than typical IT projects. Understand data processing cycle and related challenges. You will surely discover issues related to data quality. Participate in analytics discussions, you will understand how algorithms are used and may later help you in building your own teams.
  5. Learn to accept downside! Even though your partner must have tried their best, outcomes may not come. There could be multiple reasons for same starting with data. If data isn’t right and doesn’t support conclusions you need to accept it. Not even the best of algorithms or teams can help if your data isn’t worthwhile. You partner couldn’t have predicted this at the start.

We hope these insights help you achieve better outcomes in your big data initiatives.


More Great AIM Stories

Vikas Kamra
Vikas is co-founder & CEO of Data Analytics firm, Valiance Solutions. He spear heads business development efforts at Valiance. With Valiance he has consulted with Life Insurers, Retailers on various business problems with application of advanced analytics. Prior to Valiance, Vikas has worked as IT consultant with Investment banks like Merrill Lynch, Bank of America, Jefferies. He is an IIT Delhi Graduate and CFA level 2 qualified.

Our Upcoming Events

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan, 2023

Conference, in-person (Bangalore)
Rising 2023 | Women in Tech Conference
16-17th Mar, 2023

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
27-28th Apr, 2023

Conference, in-person (Bangalore)
MachineCon 2023
23rd Jun, 2023

Conference, in-person (Bangalore)
Cypher 2023
20-22nd Sep, 2023

3 Ways to Join our Community

Whatsapp group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our newsletter

Get the latest updates from AIM