How to make your First Big Data Initiative a Success? – A Vendor’s Perspective


Big data analytics offers revolutionary capabilities but achieving results requires a strategic vision—and a view into how it can create business value. Over the last few years, the term “big data” has evolved from a vague concept into a mainstream business strategy. But somewhere at the intersection of conceptual promise and real-world implementation, many business leaders, including CIOs, have faced a tough realization: putting the concept to work requires different tools, technologies and strategies than in the past.

Imagine you are a CIO who has identified a business use case for Big data analytics but is figuring out how to go about executing it. Tools, platforms, skill set and domain expertise all these weigh on your mind as you try to put together a team. Roping in external partner is an advised course whether for consulting or for full implementation. Reason is simple: your experienced partner knows best as to what works and where things can fail. So he can limit your downside risk and at the same time maximize your upside gain.


Sign up for your weekly dose of what's up in emerging technology.

Valiance recently did a Big data analytics project with an enterprise communications firm whereby we were supposed to extract customer insights from terabytes of unstructured data they had. We were responsible for creating in house big data infrastructure to store and process data, data mining, creating machine learning algorithms and finally exposing result set through an API.

Here are key learning’s we have had from this engagement and we recommend you to keep same in mind when engaging external help

  1. It’s best to clearly understand technical approach used by your partner for data mining or machine learning along with related limitations to avoid disappointment later. There are limitations to what one can achieve with all the best algorithms available at your hand. You should definitely spend some time in understanding how related technologies and algorithms work. Understanding where outcomes may not be favorable can help you manage stakeholder expectations.
  2. Big data analytics project demand infrastructure and often it’s difficult to exactly estimate capacity requirements in the start. Going for a fixed CapEx investment complicates the matter when your team needs additional capacity and that too variable. It’s best to consider using cloud infrastructure for your project if there can be uncertainties with processing workloads.
  3. Defining rational measures of success. Big data promises to be revolutionary technology with answers to all your problems. It may or may not be the case. It’s advised to define achievable and tangible measures of success rather than having irrational expectations. Big data isn’t answer to all your problems. Period!
  4. Collaborate with your partner throughout the exercise and not just towards the end. Big data analytics project demand more collaboration and attention than typical IT projects. Understand data processing cycle and related challenges. You will surely discover issues related to data quality. Participate in analytics discussions, you will understand how algorithms are used and may later help you in building your own teams.
  5. Learn to accept downside! Even though your partner must have tried their best, outcomes may not come. There could be multiple reasons for same starting with data. If data isn’t right and doesn’t support conclusions you need to accept it. Not even the best of algorithms or teams can help if your data isn’t worthwhile. You partner couldn’t have predicted this at the start.

We hope these insights help you achieve better outcomes in your big data initiatives.


More Great AIM Stories

Vikas Kamra
Vikas is co-founder & CEO of Data Analytics firm, Valiance Solutions. He spear heads business development efforts at Valiance. With Valiance he has consulted with Life Insurers, Retailers on various business problems with application of advanced analytics. Prior to Valiance, Vikas has worked as IT consultant with Investment banks like Merrill Lynch, Bank of America, Jefferies. He is an IIT Delhi Graduate and CFA level 2 qualified.

Our Upcoming Events

Conference, in-person (Bangalore)
MachineCon 2022
24th Jun

Conference, Virtual
Deep Learning DevCon 2022
30th Jul

Conference, in-person (Bangalore)
Cypher 2022
21-23rd Sep

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM
Sreejani Bhattacharyya
Why is edtech falling first?

With the lockdown being imposed due to the COVID-19 pandemic and schools being shut down, the edtech startups witnessed some of their best times during 2020 and 2021.