BigQuery, a fully-managed, serverless data warehouse, recently turned 10. Celebrating a decade of data, it believes that a lot has changed. What earlier was big data is now essentially, just data, and is an embedded part of the business and IT teams. Working on making the world’s data look like one giant database, the company believes that it has come a long way and a lot closer to achieving that goal.
It has brought a new way of thinking about big data into data warehouse format that could now scale quickly at low cost. It also brought about a big shift in separating storage and processing. The method ten years ago was essentially just to throw compute resources at big data problems, so that users often ran out of room in their data warehouse, thus running out of querying ability.
On the occasion, Analytics India Magazine interacted with Debanjan Saha, GM & VP Engineering, Data Analytics, Google Cloud to understand how the importance of data warehouse changed over the years, challenges it faces, adoption in the data science community and more.
Analytics India Magazine: How has the importance of the Data warehouse increased over the years?
Debanjan Saha: As the world has become data-driven, data warehouses have become more pervasive. They have also evolved in scale, performance, and capabilities. Ten years ago in the data warehouse market, high scalability meant high cost. But solutions like BigQuery were developed to scale quickly and at low cost. With cloud data warehouses, you can quickly scale up and down to accommodate your changing business needs.
Today, with the right tools, data analysis helps customers make data-driven decisions while eliminating constraints of scale, performance and cost. For example, when we worked with HSBC to move their legacy data warehouse to BigQuery, we used custom-built tools and an automation-first approach which has allowed them to make huge leaps in data analytics capabilities and ensure high-fidelity data. ShareChat, India’s largest Indian language social media platform, uses BigQuery federation model to query operational data from within their enterprise data warehouse, that enables ad-hoc and business intelligence related analysis, all by data analysts, without any need for data processing or transforms before querying.
AIM: What are the major challenges you face while addressing customer demands?
DS: One thing we heard time and time again from customers is that they care about the predictability of their spend. While we’ve had flat-rate pricing in BigQuery for years, we still felt we had room to make additional pricing improvements. So, we recently introduced BigQuery Reservations which allows customers to seamlessly purchase flat-rate compute capacity in seconds. It also allows customers to programmatically and dynamically distribute BigQuery slots across workloads and teams for optimal performance. We also introduced Flex Slots, which is a new way for organizations to purchase BigQuery Reservations for short durations – as little as seconds at a time. Flex Slots let users nearly instantly respond to rapid demand for analytics and plan for major business events such as retail holidays and game launches. And, since BigQuery is serverless and cloud-native, customers are abstracted from system management.
AIM: What is the roadmap for BigQuery for the coming 10 years?
DS: What I can tell you is we will continue to anchor on three themes: Open, intelligent and flexible. We will continue to help our customers enjoy an open ecosystem of products, services and clouds, we will continue building intelligence in our products, allowing customers to take advantage of real-time and predictive insights and lastly, we will continue offering flexibility within our portfolio. All these innovations are to help our customers democratize advanced analytics and digitally transform their businesses.
AIM: How do you foresee the developments in the space of AI, ML and NLP change in the coming years?
DS: Today, AI is still too hard. In the coming years, we’ll need to simplify AI adoption by bringing it closer to the data and the end-users. For example, BigQuery ML is helping business analysts to build ML models without data science expertise using standard SQL. We’ll see more democratization of AI, ML and NLP which will transform our interactions with data to the level that we cannot fully imagine today. Tune in to Google Cloud’s upcoming virtual event, Next ‘20: OnAir, to find out more about how we’re innovating in this space.
AIM: How has the adoption of BigQuery been in the Indian developer community?
DS: BigQuery is particularly effective at the largest scales, and India already has twice the number of online users as the US, which is set to grow by nearly 100M by 2023. Companies and developers alike are building for today’s scale and tomorrow’s, which means BigQuery is at the forefront of these conversations. We expect the recent addition of Looker to our portfolio, which makes data application development even easier, to accelerate this trend even further.
