|
Listen to this story
|
In 2011, three LinkedIn engineers: Jay Kreps, Neha Narkhede, and Jun Rao, set out to solve the low-latency ingestion problem associated with real-time processing of large amounts of event data from the website. Getting data and subsequently moving it around was a Herculean task and the available options did not optimally solve the problem.
The trio then came up with a system that offered a unified, high-throughput, and low-latency platform for handling real-time data feeds, called Kafka. It was incubated in 2012 at the Apache Incubator and hence called Apache Kafka now. Ten years on, this stream processing platform now forms the skeleton of infrastructure at leading organisations such as Uber, British Gas, and of course – LinkedIn.
The three engineers behind one of the most important pieces of technology proceeded to launch their own company in 2014. Confluent is a complete event streaming platform and a fully managed Kafka service. Analytics India Magazine spoke to Srinivasulu Grandhi, vice president of engineering, Confluent, about the company, its biggest competition, and its goals.
Subscribe to our Newsletter
Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.
What has Confluent set out to achieve
“Our mission is to set data in motion. Look at the enterprises – a lot of the data here is locked up in various systems and silos. This is not a new problem, and to solve it, many organisations use the ETL processes. The problem with this process is that it operates in batch mode. Confluent, as a fully managed Kafka service provider, offers a real-time view of your enterprise data at the moment, which could then be used to power interesting use cases,” said Grandhi.
Grandhi tells us that one of the biggest promises of Confluent is that as a customer, one need not worry about running Kafka at all. “We deploy Kafka for you and offer unlimited storage and elasticity on the cloud. At the macro level, we have separated compute and storage,” Grandhi told AIM.
Courtesy: Confluent
Additionally, up to 60% of Kafka committers are Confluent employees. “We continue to evolve Kafka and we are very invested; a lot of our code makes its way back to the community,” he said.
Confluent and Kafka
Kafka API has been around for ten years and after being adopted in so many ways, it has now become a gold standard for streaming data and processing, Grandhi told us. The challenge, however, is really the non-functional aspects. This is where Confluent gains brownie points – it is completely self-managed, offers high scalability, and low latency. “Bringing these features takes a lot of innovation and here is where we continue to invest a significant amount of resources.”
Courtesy: Confluent
Currently, a lot of innovation is happening at Confluent. For example, it has developed an alternative for Kafka Zookeeper — a software for metadata management – called kRaft. Recently, the company introduced Stream Designer, an easy-to-use visual designer to build, run, and monitor data pipelines.
When asked about Kafka’s user base and its effect on Confluent’s business, Grandhi said, “Kafka is huge – in Bangalore alone, there are close to 5000 Kafka developers. It is the foundational layer for most of the new-age architectures and building streaming pipelines.” He added that Kafka is much older compared to less-than-a-decade-old Confluent. This, however, hasn’t completely stopped companies from adopting the latter.
“We have many clients that have been managing their Kafka services for several years, but that has changed with Confluent entering the scene, and many more are expected to follow suit. It is mostly a mindset; I think people are used to a certain way of working, but a lot of companies, when they understand the value proposition, they sign up with us,” he said.
Confluent is very interested in the Indian market and hopes to make huge strides here. Some of its biggest clients – like Meesho and Swiggy – are based here. “We are investing in India for two reasons: one, we see India as a huge growth market for us, globally. India is going through this big digital revolution, so we are pretty bullish on this market. Second, India, especially Bangalore, where we opened our first office in July, has a huge talent pool. At this point, I would like to let you know that the connector product line of Confluent is built in India and now is being used globally. Our goal is to make India a significant part of our success story,” Grandhi signed off.
