Active Hackathon

Predicting Consumer Behaviour: Big Research vs Big Data

Whether you are a manufacturer or a retailer, a mid-sized company, or someone with a small business that caters to various customers and clients in a niche market, understanding consumer behaviour is vital for your business to succeed.  Traditional market research or Big Research is the most often used tool for that.

With research, marketers aim to extract helpful data from consumers for predicting behaviour and future preferences, and to plan their brand accordingly.


Sign up for your weekly dose of what's up in emerging technology.

Research is a useful tool, yes, but it often becomes a crutch. Using research data to inform every aspect of a product, brand, or campaign may seem like a scientifically-sound idea, but it is very dangerous in practice.

Market research, and the information it uncovers, is inherently flawed. It’s flawed because what research asks respondents to do – to rationalize the irrational – is nearly impossible. The factors that affect what we do and why we do are mostly unknown to us and impossible to put into words or rate on a 10-point scale. Research asks us to post-rationalize our actions, assigning sensible reasons to our past decisions, whether they actually played a part in them or not.


And of course, there is the classic question of “How many responses do you really need?”  This simple question is a never-ending dilemma. A larger sample can yield more accurate results — but excessive responses can be an expensive affair. On the other hand, a very small sample size may not represent the universe.

In the mid-1980s, the Coca-Cola Company made a decision to introduce a new beverage product. The company had evidence that taste was the single most important cause of Coke’s decline in the market share in the late 1970s and early 1980s. A new product dubbed “New Coke” was developed that was sweeter than the original-formula Coke.

Almost 200,000 blind product taste tests were conducted in the United States, and more than one-half of the participants favoured New Coke over both the original formula and Pepsi. The new product was introduced and the original formula was withdrawn from the market. This turned out to be a big mistake! Eventually, the company reintroduced the original formula as Coke Classic and tried to market the two products simultaneously.

Ultimately, New Coke was withdrawn from the market.

What went wrong?

There was a flaw in the market research taste tests that were conducted: They assumed that taste was the deciding factor in consumer purchase behavior. Consumers were not told that only one product would be marketed. Thus, they were not asked whether they would give up the original formula for New Coke.

In recent times, the growth of information technology has transformed market research, with a growing number of analysts learning about consumer preferences and buying habits by mining massive sets of quantitative data and employing complex algorithms to uncover patterns and correlations that enable more effective marketing. Data mining often gives businesses enormous amounts of information about their customers’ behaviors and buying habits, enabling them to do effective marketing.

For example, through data mining, online retailers these days have features that tells a potential customer that people who like one particular product also like certain other items or recommend products to them based on their behaviour on the site.

More Great AIM Stories

Radhika Shah
Head - Business Development & Marketing at Infinite Analytics. Radhika completed her Masters in Management Studies in Marketing from University of Mumbai in 2002. Her last stint was at ING Investment Management involved with Brand & Communication strategy for Mutual Fund & PMS businesses. Prior to that she was at Contract Advertising handling a clutch of key brands spanning FMCG, Banking, Investment Management & Entertainment.

Our Upcoming Events

Conference, in-person (Bangalore)
Cypher 2022
21-23rd Sep

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
21st Apr, 2023

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM

Council Post: Enabling a Data-Driven culture within BFSI GCCs in India

Data is the key element across all the three tenets of engineering brilliance, customer-centricity and talent strategy and engagement and will continue to help us deliver on our transformation agenda. Our data-driven culture fosters continuous performance improvement to create differentiated experiences and enable growth.

Ouch, Cognizant

The company has reduced its full-year 2022 revenue growth guidance to 8.5% – 9.5% in constant currency from the 9-11% in the previous quarter