5 min
After all, managing a huge volume of information in the form of billions of tweets and engagements is a no child’s play. And it takes ‘Apache Storms’ and ‘Herons’ for stream processing of data in real time and keeping pace with trending topics and conversations. So why knowing about it is of any importance now?
In what may be the biggest analytics related news coming from the web-scale crowd, Twitter Inc. open sourced its real-time stream-processing system, Heron. This is the second such step taken by the social media giant after 2011 when it open sourced Apache Storm for various takers in the field.
Heron, a home-grown analytics platform that was used by engineers to glean high level insights from tweets, was devised as a replacement for Apache Storm more than two years ago. It was born out of challenges to manage increased volume and diversity of data. It was an ambitious project that aimed to scale better, was easier to debug, had better performance, was easier to deploy and manage and worked in a shared multi-tenant cluster environment. Over the time it had proved its reliability, easy support and magnitude reduction of incidents.
The making of Heron was a no easy task as it involved challenges as to whether extend Storm, switch to another platform or develop a new system. Extending Storm would have required extensive redesigning and rewriting of its core components. Using an existing open-source solution would have had compatibility issues with Storm’s API. So the best option was to write a system from the scratch.
So, what made Twitter to release the code for Heron? The official statement by the company states that the decision is aimed at sharing their insights and knowledge and continuing to learn from and collaborate with the real-time streaming community.
With both Fortune 500 companies, like Microsoft and start-ups adorning Heron, it is one of their favourites for expanding set of real-time use cases, including ETL, model enhancement, anomaly/fraud detection, IoT/IoE applications, embedded systems, VR/AR, advertisement bidding, financial, security, and social media.
While the decision by Twitter to open source Heron is welcomed by the industry, there is not much gain that either the company or its investors have in store. Neither would it fetch Twitter more users nor have any major impact on investor’s business. It could be said that the decision majorly serves as a step towards a strengthened market presence. - Why Buying Specialist Engineering Firms Won't Fix Indian IT’s Growth Problem
- The Anthropic-Indian IT Partnership Is a Two-Way Bet
- Coforge Is Outgrowing Its Larger Rivals. Is AI the Reason?
- Are Boutique AI Firms India’s Next Big Thing?
- Indian IT’s New Workforces Come with a Margin Challenge
- ‘Owning of the IP Itself is Meaningless,’ Says Mphasis’ Ravi Vasantraj