Why This Company Built By Kafka Creators is One To Watch Out For

Confluent is very interested in the Indian market and hopes to make huge strides here.
confluent
Listen to this story

In 2011, three LinkedIn engineers: Jay Kreps, Neha Narkhede, and Jun Rao, set out to solve the low-latency ingestion problem associated with real-time processing of large amounts of event data from the website. Getting data and subsequently moving it around was a Herculean task and the available options did not optimally solve the problem. 

The trio then came up with a system that offered a unified, high-throughput, and low-latency platform for handling real-time data feeds, called Kafka. It was incubated  in 2012 at the Apache Incubator and hence called Apache Kafka now. Ten years on, this stream processing platform now forms the skeleton of infrastructure at leading organisations such as Uber, British Gas, and of course – LinkedIn.

The three engineers behind one of the most important pieces of technology proceeded to launch their own company in 2014. Confluent is a complete event streaming platform and a fully managed Kafka service. Analytics India Magazine spoke to Srinivasulu Grandhi, vice president of engineering, Confluent, about the company, its biggest competition, and its goals.

THE BELAMY

Sign up for your weekly dose of what's up in emerging technology.

What has Confluent set out to achieve

“Our mission is to set data in motion. Look at the enterprises – a lot of the data here is locked up in various systems and silos. This is not a new problem, and to solve it, many organisations use the ETL processes. The problem with this process is that it operates in batch mode. Confluent, as a fully managed Kafka service provider, offers a real-time view of your enterprise data at the moment, which could then be used to power interesting use cases,” said Grandhi.

Grandhi tells us that one of the biggest promises of Confluent is that as a customer, one need not worry about running Kafka at all. “We deploy Kafka for you and offer unlimited storage and elasticity on the cloud. At the macro level, we have separated compute and storage,” Grandhi told AIM

Courtesy: Confluent

Additionally, up to 60% of Kafka committers are Confluent employees. “We continue to evolve Kafka and we are very invested; a lot of our code makes its way back to the community,” he said.

Confluent and Kafka

Kafka API has been around for ten years and after being adopted in so many ways, it has now become a gold standard for streaming data and processing, Grandhi told us. The challenge, however, is really the non-functional aspects. This is where Confluent gains brownie points – it is completely self-managed, offers high scalability, and low latency. “Bringing these features takes a lot of innovation and here is where we continue to invest a significant amount of resources.”

Courtesy: Confluent

Currently, a lot of innovation is happening at Confluent. For example, it has developed an alternative for Kafka Zookeeper — a software for metadata management – called kRaft. Recently, the company introduced Stream Designer, an easy-to-use visual designer to build, run, and monitor data pipelines.

When asked about Kafka’s user base and its effect on Confluent’s business, Grandhi said, “Kafka is huge – in Bangalore alone, there are close to 5000 Kafka developers. It is the foundational layer for most of the new-age architectures and building streaming pipelines.” He added that Kafka is much older compared to less-than-a-decade-old Confluent. This, however, hasn’t completely stopped companies from adopting the latter.

“We have many clients that have been managing their Kafka services for several years, but that has changed with Confluent entering the scene, and many more are expected to follow suit. It is mostly a mindset; I think people are used to a certain way of working, but a lot of companies, when they understand the value proposition, they sign up with us,” he said.

Confluent is very interested in the Indian market and hopes to make huge strides here. Some of its biggest clients – like Meesho and Swiggy – are based here. “We are investing in India for two reasons: one, we see India as a huge growth market for us, globally. India is going through this big digital revolution, so we are pretty bullish on this market. Second, India, especially Bangalore, where we opened our first office in July, has a huge talent pool. At this point, I would like to let you know that the connector product line of Confluent is built in India and now is being used globally. Our goal is to make India a significant part of our success story,” Grandhi signed off.

More Great AIM Stories

Shraddha Goled
I am a technology journalist with AIM. I write stories focused on the AI landscape in India and around the world with a special interest in analysing its long term impact on individuals and societies. Reach out to me at shraddha.goled@analyticsindiamag.com.

Our Upcoming Events

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan, 2023

Conference, in-person (Bangalore)
Rising 2023 | Women in Tech Conference
16-17th Mar, 2023

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
27-28th Apr, 2023

Conference, in-person (Bangalore)
MachineCon 2023
23rd Jun, 2023

Conference, in-person (Bangalore)
Cypher 2023
20-22nd Sep, 2023

3 Ways to Join our Community

Whatsapp group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our newsletter

Get the latest updates from AIM