Data storage is at the foundation of every digital transformation, cloud computing and data science application today. Any company starting its infrastructural data roadmap or advancing its digital transformation needs a secure, flexible and easy place to store its data.

REGISTER FOR FREE AI WORKSHOP >>

Snowflake and Databricks are at the forefront of the race to provide cloud computing services, and despite being competitors, they are different in several ways. Analytics India Magazine has created the ultimate comparison any company needs to decide on the best provider for their applications. 

What is Snowflake?

Snowflake is a cloud computing-based data warehouse company founded in 2012 by Benoit Dageville, Thierry Cruanes, and Marcin Zucowski. It is a fully managed service providing almost infinite scalability of concurrent workloads, allowing customers to integrate, load, analyse, and securely share their data. Snowflake’s popular offerings are Data Lakes, Data Engineering, Data Application Development, Data Science, and secure consumption of shared data. Additionally, it is known for its separate compute and storage facilities allowing customers to access only the single needed copy of the data with efficient performance.

Snowflake is known for its innovation in the data warehouse. This relational database is designed for analytical rather than transactional work and serves as a federated repository for all certain data sets. It is headed by a business executive, CEO Frank Slootman. 

What is Databricks?

Databricks is a cloud-based data platform powered by Apache Spark and found in 2013 by the original creators of Apache Spark, Delta Lake and MLflow. It has become a one-stop solution for the entire analytics team instead of giant vendors. Databricks’ unique offerings include Machine Learning Runtime, managed ML Flow, Collaborative Notebooks, Dataframes and Spark SQL libraries. Its unified analytics platform allows the team of Data Engineers, Data Analysts, Data Scientists and Machine Learning Engineers to work on a project together. The Data Engineers can build cutting edge data pipelines by realising data architectures like Lambda Architecture and Delta Architecture.

Databricks is known for its proprietary innovation, the Data Lake, where users can dump all of their data in any format and can still be used to generate insights. The company, led by an academician himself, CEO Ali Ghodsi, is tech-focused and engineering-led.

Data Ownership

Snowflake has decoupled processing and storage layers that can independently scale in the cloud. Additionally, the ownership is retained for both layers. Snowflake uses the Role-based Access Control (RBAC) method to secure data and machine resources access. 

Unlike the decoupled layers in Snowflake, the data processing and storage layers are fully decoupled in Databricks. Since its main objective is data application, users can leave their data wherever in any format, and Databricks will process it efficiently. 

Data Structure

Snowflake supports structured and unstructured data without the need for an ETL tool to organise it. The data is stored in database tables, logically structured as collections of columns and rows using micro-partitions and data clustering methods. Snowflake automatically transforms the data into its internal structured format upon uploading.

Databricks is compatible with all data types in their original format and even allows users to add structure to their unstructured data. The Databricks database is a collection of tables of structured data. The user can cache, filter, and perform any Apache Spark DataFrames on these tables. 

Scalability

Both Databricks and Snowflake offer strong scalability, but scaling up and down is easier with Snowflake. In Snowflake, processing and storage layers scale independently. This allows for in-time scaling without bothering the queries in the process. Additionally, it provides near-infinite scalability by isolating simultaneous workloads on dedicated resources.

Databricks auto-scales depending on the workload where it can scale down during circumstances of the platform being 100% idle for long enough. Then, it removes idle workers on under-utilised clusters.  

Security

Snowflake provides separate customer keys, including encryption at rest, role-based access control and Virtual Private Snowflake. The management of this key to protect customer data occurs automatically using AES-256 strong encryption. Additionally, it offers Time Travel and Fail-safe. Snowflake’s Time Travel features to preserve the original data state before it is updated with the period ranging from one day to 90 days.                        

Databricks provides protection through Delta Lake, which has a similar feature to Snowflake’s Time Travel. It also enables compliance with data laws, given Delta Lake’s additional transactional layer that provides structured data management on top of the data lake. This allows users to simplify the process and quickly locate and remove personal information. In addition, Databricks offers separate customer keys and RBAC for data clusters. Since Databricks runs on Spark and its object-level storage, the platform does not actually store any data, allowing it to address on-premises use cases.

Architecture

Snowflake’s architecture is a hybrid of traditional shared-disk and shared-nothing database architectures. It uses a central data repository for persisted data accessible from all compute nodes in the platform and provides a serverless solution based on ANSI SQL that separates storage and compute processing layers. Snowflake’s architecture is based on parallel processing that stores a portion of the data locally with every individual virtual warehouse. Snowflake uses micro partitions to organise and internally optimise data into a compressed columnar format to be kept in cloud storage. Its architecture consists of three layers; Database Storage, Query Processing and Cloud Services. Snowflake automatically manages various aspects like file size, compression, structure, metadata, statistics, and other data objects that can only be accessed through SQL queries. A SaaS solution, Snowflake manages the backend from user requests, infrastructure management, metadata, authentication, query parsing, access control, and optimisation. It runs it on three major clouds, AWS, GCP, and Azure.

Databricks’ architecture is built on Spark around single nodes that can be deployed in the cloud. It currently runs on AWS, GCP, & Azure like Snowflake. Databricks operate out of a control plane and a data plane. The control plane includes the backend services in Databricks’ AWS account, storing notebook commands and workspace configurations, and encrypting at rest. The data plane is where the data is processed. It further offers serverless computing for users to create serverless SQL endpoints managed entirely by Databricks, enabling instant compute. These resources are shared in a serverless data plane and allow users to connect to external data sources to ingest data outside the AWS account and external data streaming sources.

Use Cases

Both Databricks and Snowflake heavily support BI and SQL use cases. 

Snowflake offers JDBC and ODBC drivers that easily integrate with third-party applications. It is majorly known for its use-cases in BI and for companies opting for a simplistic platform for analysis, given that users dont have to manage the software. 

Meanwhile, Databricks has introduced an open-source Delta Lake that acts as an added layer of reliability on their Data Lake. Delta Lake allows customers to submit SQL queries with high-level performance. Databricks is generally known for their use-cases preventing minimal vendor lock-in, better suited for ML workloads, and supporting tech giants given their versatility and superior technology.