Key takeaways
- The explosion of data from various applications and devices is creating operational challenges for organisations.
- Data types are diversifying, including structured and semi-structured formats from multiple sources.
- Current data processing systems remain complex and are in early stages of market adoption.
- Qubole offers a comprehensive data infrastructure that includes services for data collection, computation, and querying.
- The platform aims to enhance data processing capabilities for real-time and batch data needs.
With the proliferation of applications and end devices - web, mobile, sensors etc. - the last few years and the foreseeable future promise the continuation of the trend of explosive growth in the volume of data being collected by various organizations. At the same time there is an ever growing variety in the types of data - ranging from structured data originating in application databases, to more semi-structured content originating in social media, web properties such as wikipedia as well as internal applications such as email systems, company support boards etc. Additionally systems and software stacks (most notably Apache Hadoop) that are able to keep up with this growth - both in variety as well as in volume - are still complex to operate and far from perfect. They are still in the early stages of development and market adoption. As a result it comes as no surprise that many organizations struggle to keep up with operating, optimizing and making their data infrastructure work to serve their data processing needs.
Qubole Vision
The complete data infrastructure solution has many components. The main ones are as follows:
- Data Collection Service for both real time and bulk upload of data from different data sources such as applications, databases, web crawls etc.
- Batch Computation Service such as Hadoop/Hive to process this data and transform it from data to information.
- Real Time Computation Service to generate real time results on data streams and data captures for time sensitive and actionable reporting and monitoring.
- AdHoc Query Service to answer one of queries sometimes exactly and other times approximately in a short amount of time.
- Tools and Frameworks for job dependencies, data and query discovery, SLA and monitoring etc.
Spotted an error? Tell our editors. Read our editorial standards and corrections.




