The data pipelines built a decade ago will definitely fall short of the current rise in data usage across the world. A data enterprise deals with multiple stakeholders on a wide range of use cases. It’s important to identify the key processes to make sure they’re aligned with the strategic goals. As organisations make a move from traditional to modern warehouses, many challenges surface. But why make a move at all?
Why Do Organisation Migrate
According to Google Cloud, some of the reasons why organisations plan to make a move from legacy systems are:
Lack of agility
AIM Daily XO
Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.
For example, the change in the landscape of digital payments is enormous. So are the technical challenges associated with such systems. Imagine a service that has to handle critical transactions; in countries like India, a successful payment system can easily draw in half a billion customer base. So, real-time insights and operation is central to such applications. And, legacy data warehouses will fall short of providing business agility.
To Cut Costs and Inefficiencies
Download our Mobile App
Traditional data warehouses usually function around pay for technology that includes associated hardware and licensing costs and ongoing systems engineering. This already sounds inefficient. At least, in the case of a bludgeoning data-driven economy. Organisations can’t rely on this pay on the go for every enhancement. With a rise in data, the costs and technical challenges arise.
Don’t Offer Intelligence
On the cloud, AI-based decision making is a reality. Cloud providers like GCP and AWS offer a variety of services for different use cases so that the users can build recommendation engines, chatbots, handle time series modelling and more on the go. Legacy warehouses do not facilitate predictive analytics. Machine learning is already changing the face of businesses, so organisations would like to have these services at their disposal.
So, when an organisation decides to change the way it deals with data, it suddenly has a handful of problems like infrastructure, dependencies, access control and more to deal with. Google, which has pioneered the art of building data pipelines for high-profile customers, has prepared a framework for warehouse migration. Here are a few tips to avoid pitfalls of migration:
Watch Out for Dependencies
By understanding the current technical landscape and classifying existing solutions to identify independent workloads, you can more easily separate upstream and downstream applications to further drill down into their dependency on specific use cases. It’s key that you are clear on what you are migrating. This includes identifying appropriate data sources with an understanding of data velocity, data regionality, and licensing, as well as identifying business intelligence (BI) systems with current reporting requirements and desired modernisations during the migration.
By discussing process options, you can uncover dependencies between existing components and data access and governance requirements, as well as the ability to split migration components.
Prepare the Personnel
“Identify and interview each functional group within the team by conducting workshops, hackathons, and brainstorming sessions.”
To make sure you’re getting input and buy-in for migration, start with aligning leadership and business owners. Then, explore the skills of the project team and end-users. You might identify and interview each functional group within the team by conducting workshops, hackathons, and brainstorming sessions.
For example, upgrading current systems might require employees to be re-trained and new additional licenses to be purchased. Quantifying these requirements, and associating them with costs, will allow you to make a pragmatic, fair assessment of the migration process. Google suggests that the staff should have time to be hands-on and start using the new system to learn by doing.
According to Google Cloud, the following thumb rules can come in handy to any organisation looking to make changes to their data warehouse:
- Identify data sources for up and downstream applications
- Identify datasets, tables and schemas relevant for use cases
- Outline ETL tools and frameworks
- Define data quality and data governance solutions
- Identify Identity and Access Management (IAM) solutions
- Outline BI and reporting tools
Read more here.
