There is No Need for a Databricks for the Postgres World

The database ecosystem has been going through a tectonic shift. Since last year, Postgres has been gaining popularity while MySQL is seeing a decline, with cloud providers prompt in building extensions within their offerings to provide Postgres services natively, connecting different databases.

The latest example is Databricks announcing that LakeFlow Connect is available for public preview for SQL Server, Salesforce, and Workday. This would allow the ingestion of data from different databases and enterprise apps. The platform is also native to Databricks’ Data Intelligence Platform, allowing both serverless compute and Unity Catalog governance.

“Ultimately, this means organisations can spend less time moving their data and more time getting value from it,” read the announcement blog, which listed the problems associated with connecting external sources like an increased extraction time and data preparation.

At the sideline of the Databricks’ Data+AI Summit 2024, CEO Ali Ghodsi said that two years ago, when he asked his customers if they need AI or the ease of accessing their data on Databricks, most of them chose the latter. But the company was not at all interested in it at the time.

“This was two years ago. Then we started our journey towards ‘we need to do this’,” said Ghodsi. This later led to the acquisition of Arcion for exactly the same purpose, and led to this announcement. “You don't need to cobble it [databases] together yourself or use something else. It will just work seamlessly. So this is a big strategic area for us,” said Ghodsi.

The Old is New Again

Pondering this, Harsh Singhal, senior engineering manager of machine learning at Adobe, said that despite being an older piece of technology, Postgres is experiencing renewed interest and relevance in modern applications.

“Postgres has a large number of extensions to provide access to other databases and apps. And if there isn’t one available, developing an extension for a specific purpose is a very lucrative business,” he said, giving the example of Citus, which offered an open source extension to Postgres for distributed databases, increasing performance and scale for application developers.

The company was acquired by Microsoft in 2019, which is comparable to the acquisition of Arcion by Databricks.

Singhal asks the question - “Is there a need for a Databricks like company for the Postgres ecosystem?”

Developing custom extensions for Postgres can be a profitable business. Databricks, a prominent data technology platform, is praised for its capabilities in the AI sector. However, the question is whether a similar company focused solely on the Postgres ecosystem might be equally or more beneficial, given Postgres' expanding feature set.

Not Really

There are companies such as Crunchy Bridge which are still offering Postgres services on different clouds. But as the offerings and capabilities are increasing in the ecosystem, it is very hard to catch up and survive as a business, with cloud providers also providing similar services. Take Databricks, for example, or even Google, Microsoft, and Oracle.

Eventually, Crunchy Bridge might also get acquired by big companies, marking an end to its entire business segment.

People say that the future of MySQL is Postgres, which is concerning as there were several companies that used to offer relational database capabilities. However, with the allure of MySQL dying, these companies are also in deep water. The same could happen with Postgres in the future, making it risky for anyone to venture into the field.

However, many MySQL applications still exist, and developers continue to use it. Although the industry trend is moving towards Postgres, transitioning from MySQL typically involves traditional migrations, which, while manageable for data, are quite challenging for applications.

Regardless, Postgres has become the future of data. But taking a step towards Postgres and making a business out of it is tricky, as announcements like LakeFlow and others can make business solely focused on offering Postgres a lucrative business.