Data Warehouses in Age of Decentralised Architecture

In 2019, Transforming Data with Intelligence (TWDI), a data and analytics research company, came up with a report, ‘The Modernisation of the Data Warehouse’, detailing why there is a need to modernise data warehouses and strategies to modernise data warehouses.

Incidentally, this was the same year the data mesh concept started gaining ground after Zhamak Dehghani, director of Emerging Technologies at ThoughtWorks, published a paper, ‘How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh’. Now, unlike data warehouses that are centralised data architectures, data mesh is a highly decentralised distributed data architecture.

Dehghani conceived data mesh from the challenges inflicting the centralised approach. Firstly, most of the data is always generated by operational systems and applications like CRMs or ERPs. There are dedicated people for these domain-specific applications. The operational application team and the data platform team currently have little or no communication. Often, the former is not at all aware that "their" data is being sent to a data platform. Second, the data team itself lacks a solid understanding of how business applications operate. Thus, none of the groups is able to take ownership of the data, often leading to situations where data quality issues remain unaddressed or are addressed very late in the process. Moreover, the data engineering team working on the centralised platform struggles to produce usable data for consumption as they don’t understand how the data has been generated and what it actually means, given there is no context to the data — just tables with names, rows, columns, or files.

Does that mean with the introduction of something like data mesh, older data architecture becomes obsolete? Well, not really. There are several drivers— technological, strategic as well as business — for data warehouse modernisation. The TWDI report, too, cited several factors driving modernisation of data warehouses as shown below.

(Source: TWDI Pulse Report)

Modernising data warehouses is advantageous

Today, there are a lot of companies engaged in the business of data warehouse modernization like Google Cloud, Qlik, Astera, OneGLobe and others. There are several benefits of data warehouse modernization, for instance, it brings agility in the business, which is key in scaling up businesses and taking data-driven decisions. Modernising data warehouses facilitates easier integration of data in the centralised repository and streamlines the process of deriving insights from it.

Modernisation of data warehouses does away with the need to hire additional resources to query data, analyse it and generate reports. It can help businesses save costs associated with identifying data that is lost during ETL process or poor quality data due to want of validation while mapping data from source to warehouses.

Added to these, is the high speed and better performance that modernised data warehouses provide. They enable data to be accessed without compromising their quality and efficiency. They also support larger volumes of data that can give any organisation a holistic picture regarding where it stands.

Drawing inferences from recent developments

Notwithstanding the benefits offered by data warehouse modernisation, a question that arises here is that in the age of decentralised data architectures, are data warehouses and their modernisation still relevant? Well, discussions and debates could go on. However, the fact that data warehouses are still relevant can be inferred for several developments.

The future of data mesh that was conceived as an alternative to centralised data architectures is very uncertain. Gartner’s Hype Cycle for Data Management 2022 expects data mesh to become ‘obsolete before plateau’. Many industry experts are of the view that it is more of a hype doomed to be obsolete.

According to James Serra, big data and data warehousing solution architect at Microsoft,  a decentralised solution like data mesh is not for smaller companies. It is more suited for big companies having complex data models, high data volumes, and multiple data domains. Thus, a decentralised solution would be ‘an overkill’ for at least 90% of companies. 

Cloud-based data warehouses like Amazon’s Redshift or Google’s Big Query offer an added advantage – organisations do not need to incur upfront costs, but pay as their data volume expands.   In fact, cloud-based platforms and tools have become one of the several strategies for modernisation of data warehouses.

According to Sigmoid, a data engineering and AI-solutions company,  adoption of cloud data warehouses is growing at a CAGR of 15%. Cloud-based data warehouses make it easier to scale without having to compromise on performance. Companies can deploy the resources they actually need and scale capacity up or down as business needs change.