Using DDN Infinia as a Unified Data Layer for Databricks and Snowflake

In the era of cloud-native data platforms, organizations are increasingly adopting data lake architectures to unify their data management and analytics workloads. Tools like Databricks and Snowflake have become go-to solutions for handling large-scale data processing, analytics, and business intelligence (BI).

However, managing data across these platforms while maintaining performance, scalability, and governance can be challenging. This is where DDN Infinia, a high-performance, scalable storage solution, steps in as a unified data layer for external table data lakes, seamlessly integrating with Databricks, Snowflake, and similar cloud-native tools.

Here we explore how DDN Infinia can serve as a robust unified data layer, enabling efficient data management and interoperability for Databricks and Snowflake, with a focus on external table data lakes. We’ll cover the key benefits, technical setup, and practical use cases to demonstrate its value.

Why a Unified Data Layer?

Modern data architectures often combine data lakes and data warehouses to leverage the scalability of object storage and the structured querying capabilities of cloud-native platforms. However, this can lead to data silos, redundant storage, and governance challenges. A unified data layer addresses these issues by:

• Centralizing Data Storage: Providing a single source of truth for data accessed by multiple platforms.
• Enhancing Interoperability: Enabling tools like Databricks and Snowflake to read and write to the same data lake using open table formats (e.g., Delta Lake, Apache Iceberg).
• Optimizing Performance: Delivering high-throughput, low-latency access to data for analytics and machine learning (ML) workloads.
• Simplifying Governance: Supporting unified metadata management and access control across platforms.

DDN Infinia is uniquely positioned to meet these needs with its high-performance storage, cloud-native integration, and support for open table formats, making it an ideal choice for a unified data layer in a data lakehouse architecture.

DDN Infinia: A High-Performance Data Platform

DDN Infinia is a next-generation storage solution designed for data-intensive workloads, offering:

• Massive Scalability: Infinia can scale to petabytes of data with consistent performance, ideal for large-scale data lakes.
• High Throughput: Optimized for parallel data access, it supports demanding analytics and ML workloads.
• Cloud-Native Integration: Infinia integrates seamlessly with cloud object storage (e.g., AWS S3, Azure Data Lake Storage) and supports hybrid deployments.
• Open Table Format Support: Compatibility with Delta Lake and Apache Iceberg ensures interoperability with Databricks and Snowflake.
• Advanced Data Management: Features like data versioning, tiering, and encryption enhance governance and security.

By leveraging these capabilities, DDN Infinia serves as a unified data layer that bridges the gap between Databricks’ lakehouse capabilities and Snowflake’s data warehousing strengths.

Integrating DDN Infinia with Databricks and Snowflake

DDN Infinia can be configured as an external table data lake, allowing Databricks and Snowflake to access the same data stored in open table formats. Below, we outline the key steps to set up this integration and highlight how it works.

Step 1: Deploy DDN Infinia as the Data Lake Storage Layer

DDN Infinia is deployed as the primary storage layer for your data lake, typically on-premises or in a hybrid cloud environment. It integrates with cloud object storage (e.g., S3) to provide a scalable, high-performance backend. Key configurations include:

• Object Storage Backend: Configure Infinia to use S3-compatible storage for data persistence.
• Table Format Selection: Choose an open table format like Delta Lake (for Databricks) or Apache Iceberg (for Snowflake). DDN Infinia supports both, ensuring compatibility.
• Access Control: Set up role-based access control (RBAC) and encryption to secure data at rest and in transit.

Step 2: Configure Databricks for External Tables

Databricks can access data stored in DDN Infinia using Delta Lake or UniForm (Delta Lake with Iceberg metadata).

Here’s how to set it up:

• Mount DDN Infinia Storage: Use Databricks’ DBFS or direct S3 paths to mount the Infinia storage
  
  location.spark.conf.set(“fs.s3a.access.key”, “<access-key>”) spark.conf.set(“fs.s3a.secret.key”, “<secret-key>”) spark.conf.set(“fs.s3a.endpoint”, “<infinia-s3-endpoint>”)
• Create External Tables: Define external tables in Databricks pointing to Delta Lake tables stored in Infinia.
  
  CREATE EXTERNAL TABLE my_table USING DELTA LOCATION ‘s3a://<infinia-bucket>/path/to/delta-table’;
• Enable UniForm (Optional): For interoperability with Snowflake, enable Delta Lake
  UniForm to generate Iceberg metadata alongside Delta metadata.
  
  ALTER TABLE my_table SET TBLPROPERTIES (‘delta.universalFormat.enabledFormats’ = ‘iceberg’);

Databricks can now use its Spark-based compute engine to process data, leveraging Infinia’s high throughput for ETL, ML, and analytics workloads.

Step 3: Configure Snowflake for External Iceberg Tables

Snowflake supports Apache Iceberg tables stored in external data lakes, making it compatible with DDN Infinia. Here’s how to configure it:

• Create an External Stage: Define an S3 stage in Snowflake pointing to the Infinia storage location.
  
  CREATE OR REPLACE STAGE my_stage URL = ‘s3://<infinia-bucket>/path/to/iceberg-table’ CREDENTIALS = (AWS_KEY_ID = ‘<access-key>’ AWS_SECRET_KEY = ‘<secret-key>’);
• Set Up an Iceberg Catalog: Use Snowflake’s Polaris catalog or an external Iceberg REST catalog (e.g., Databricks Unity Catalog) to manage metadata.
  
  CREATE OR REPLACE EXTERNAL VOLUME my_volume STORAGE_LOCATIONS = ( ( NAME = ‘my_s3_location’ STORAGE_PROVIDER = ‘S3’ STORAGE_BASE_URL = ‘s3://<infinia-bucket>/’ STORAGE_AWS_ROLE_ARN = ‘<arn>’ ) );
• Create an Iceberg Table: Define an external Iceberg table in Snowflake.
  
  CREATE OR REPLACE ICEBERG TABLE my_iceberg_table EXTERNAL_VOLUME = ‘my_volume’ CATALOG = ‘polaris’ BASE_LOCATION = ‘path/to/iceberg-table’;

Snowflake can now query Iceberg tables stored in Infinia, leveraging its compute engine for BI and SQL-based analytics.

Step 4: Unified Governance with Unity Catalog or Polaris

To ensure consistent governance across Databricks and Snowflake, use a unified catalog like Databricks Unity Catalog or Snowflake Polaris.

For example:

• Unity Catalog: Configure Unity Catalog in Databricks to manage Delta Lake and Iceberg tables. Snowflake can access these tables via Unity Catalog’s Iceberg REST API.
• Polaris Catalog: Use Snowflake’s open-source Polaris catalog to manage Iceberg metadata, accessible by both Snowflake and Databricks.

Both catalogs support RBAC, auditing, and lineage, ensuring compliance and security across platforms.

Benefits of Using DDN Infinia as a Unified Data Layer

By deploying DDN Infinia as the unified data layer for Databricks and Snowflake, organizations can realize several benefits:

1. Single Copy of Data: Eliminate data duplication by storing a single copy of data in Infinia, accessible by both platforms via open table formats. This reduces storage costs and simplifies data management.
2. High Performance: Infinia’s high-throughput storage accelerates data ingestion, ETL, and query execution, benefiting both Databricks’ Spark-based workloads and Snowflake’s SQL queries.
3. Interoperability: Support for Delta Lake and Iceberg ensures seamless integration with Databricks, Snowflake, and other cloud-native tools like Starburst or BigQuery.
4. Cost Efficiency: By centralizing storage and leveraging Infinia’s scalability, organizations can optimize costs compared to maintaining separate storage for each platform.
5. Unified Governance: Integration with Unity Catalog or Polaris provides consistent metadata management, access control, and auditing across platforms.

Practical Use Cases

Here are a few real-world scenarios where DDN Infinia shines as a unified data layer:

Enterprise Data Lakehouse:

• Scenario: A retail company uses Databricks for ML-based demand forecasting and Snowflake for BI dashboards.
• Solution: Store raw and processed data in DDN Infinia using Delta Lake UniForm. Databricks processes data for ML models, while Snowflake queries the same data as Iceberg tables for BI reporting.
• Outcome: Reduced data duplication, faster query performance, and unified governance.

Hybrid Cloud Analytics:

• Scenario: A financial institution operates a hybrid cloud environment with on-premises data and cloud-based analytics.
• Solution: Deploy Infinia on-premises as the data lake, integrated with AWS S3. Databricks handles ETL and ML, while Snowflake powers cloud-based BI.
• Outcome: Seamless data access across environments with high performance and security.

Multi-Platform Data Sharing:

• Scenario: A healthcare provider needs to share data between Databricks, Snowflake, and Starburst for different analytics teams.
• Solution: Use Infinia as the central data lake with Iceberg tables, accessible by all platforms via a shared Polaris catalog.
• Outcome: Simplified data sharing, consistent governance, and reduced operational overhead.

Conclusion

DDN Infinia is a powerful solution for organizations looking to unify their data lakehouse architecture across Databricks, Snowflake, and other cloud-native tools. By serving as a high-performance, scalable data layer for external table data lakes, Infinia enables seamless interoperability, reduces data duplication, and simplifies governance.

Its support for open table formats like Delta Lake and Apache Iceberg ensures compatibility with modern data platforms, while its advanced storage capabilities deliver the performance needed for demanding analytics and ML workloads.

Whether you’re building an enterprise data lake, enabling hybrid cloud analytics, or sharing data across multiple platforms, DDN Infinia provides the foundation for a unified, efficient, and future-proof data strategy.