An Exploration of the Commercial Iceberg Catalog Ecosystem

Get Data Lakehouse Books:

• Apache Iceberg: The Definitive Guide

• Apache Polaris: The Defintive Guide

• Architecting an Apache Iceberg Lakehouse

• The Apache Iceberg Digest: Vol. 1

Lakehouse Community:

• Join the Data Lakehouse Community

• Data Lakehouse Blog Roll

• OSS Community Listings

• Dremio Lakehouse Developer Hub

Apache Iceberg has quickly become the table format of choice for building open, flexible, and high-performance data lakehouses. It solves long-standing issues around schema evolution, ACID transactions, and engine interoperability. Enabling a shared, governed data layer across diverse compute environments.

But while the table format itself is open and standardized, the catalog layer, the system responsible for tracking and exposing table metadata, is where key decisions begin to shape your architecture. How your organization selects and manages an Iceberg catalog can influence everything from query performance to write flexibility to vendor lock-in risk.

This blog explores the current landscape of commercial Iceberg catalogs, focusing on the emerging Iceberg REST Catalog (IRC) standard and how different vendors interpret and implement it. We’ll examine where catalogs prioritize cross-engine interoperability, where they embed proprietary optimization features, and how organizations can approach these trade-offs strategically.

You’ll also learn what options exist when native optimizations aren’t available, including how to design your own or consider a catalog-neutral optimization tool like Ryft.io (when using cloud object storage).

By the end, you’ll have a clear view of the commercial ecosystem, and a framework to help you choose a path that fits your technical goals while minimizing operational friction.

The Role of Iceberg REST Catalogs in the Modern Lakehouse

At the heart of every Apache Iceberg deployment is a catalog. It’s more than just a registry of tables, it’s the control plane for transactions, schema changes, and metadata access. And thanks to the Apache Iceberg REST Catalog (IRC) specification, catalogs no longer need to be tightly coupled to any single engine.

The IRC defines a standardized, HTTP-based API that lets query engines like Spark, Trino, Flink, Dremio, and others communicate with a catalog in a consistent way. That means developers can write data from one engine and read it from another, without worrying about format mismatches or metadata drift.

This decoupling brings three major benefits:

• Multi-language support: Since the interface is language-agnostic, you can interact with the catalog from tools written in Java, Python, Rust, or Go.

• Compute independence: Query and write operations don’t require the catalog to be embedded in the engine, everything runs through REST.

Adoption of the IRC spec is growing rapidly. Vendors like Dremio, Snowflake, Google, and Databricks now offer catalogs that expose some or all of the REST API. This trend signals a broader shift toward open metadata services, where engine choice is driven by workload needs, not infrastructure constraints.

But as we’ll see next, implementing the REST API is only part of the story. The real architectural decisions start when you consider how these catalogs handle optimization, write access, and cross-engine consistency.

Key Considerations When Choosing a Catalog

Picking a catalog shapes how your Iceberg lakehouse runs. The decision affects who can read and write data, how tables stay performant, and how easy it is to run multiple engines. Focus on facts. Match catalog capabilities to your operational needs.

Read-write interoperability.

Some catalogs expose the full Iceberg REST Catalog APIs so any compatible engine can read and write tables. Other offerings restrict external writes or recommend using specific engines for writes. These differences change how you design ingestion and cross-engine workflows.

Server-side performance features.

Catalogs vary in how much they manage table health for you. A few provide automated compaction, delete-file handling, and lifecycle management. Others leave those tasks to your teams and to open-source engines. If you want fewer operational jobs, prioritize a catalog with built-in performance management.

Vendor neutrality versus added convenience.

A catalog that automates maintenance reduces day-to-day work. It also increases dependency on that vendor’s maintenance model. If your priority is full independence across engines then you may prefer a catalog that implements the Iceberg REST spec faithfully so you can plan for external maintenance processes.

Costs and Compatibility
Some catalogs may be limited on which storage providers they can work with or may charge just for usage of the catalog even if you use external compute and this should be considered.

A short checklist to evaluate a candidate catalog

• Does it implement the Iceberg REST Catalog APIs for both reads and writes?

• Does it provide automatic table maintenance or only catalog services?

• What write restrictions or safety guards exist for external engines?

• Which clouds and storage systems does it support?

• Are there extra costs to using the catalog?

Use this checklist when you compare offerings. It helps reveal trade-offs between operational simplicity and multi-engine freedom.

Catalog Optimization: Native vs. Neutral Approaches

Once your Iceberg tables are in place, keeping them fast and cost-effective becomes a daily concern. File sizes grow unevenly, delete files stack up, and query times creep higher. This is where table optimization comes in — and where catalog differences start to matter.

Most commercial catalogs fall into two categories:

• Native Optimization Available

• Manual Optimization Required

Native Optimization Available

Vendors like Dremio, AWS Glue, and Databricks Unity Catalog offer built-in optimization features that automatically manage compaction, delete file cleanup, and snapshot pruning. These features are often tightly integrated into their orchestration layers or compute engines.

Benefits:

• No need to schedule Spark or Flink jobs manually

• Optimizations are triggered based on metadata activity

• Helps reduce cloud storage costs and improve query performance

Tradeoff:

• These features are often proprietary and non-transferable. If you move catalogs or engines, you may lose automation and need to build optimization pipelines.

Catalog-Neutral or Manual Optimization

Some catalogs, including open-source options like Apache Polaris, don’t come with built-in optimization. Instead, you have two options:

1. Run your own compaction pipelines using engines like Spark or Flink. Can also manually orchestrate Dremio’s OPTIMIZE and VACUUM commands with any catalog.

2. Use a catalog-neutral optimization service like Ryft.io, which works across any REST-compatible catalog, but currently only supports storage on AWS, Azure, or GCP. There is also the open source Apache Amoro which automates the use of Spark based optimizations.

This route offers maximum flexibility but requires:

• Engineering effort to configure and monitor compaction

• Knowledge of best practices for tuning optimization jobs

• A way to coordinate across engines to avoid conflicting writes

In short: if optimization is a feature you want off your plate, look for a catalog that handles it natively. If you prefer full control or need a more cloud-agnostic setup, neutral optimization tools or open workflows may serve you better.

What If Native Optimization Doesn’t Exist?

Not every catalog includes built-in optimization. If you’re using a minimal catalog, or one that prioritizes openness over orchestration, you’ll need to handle performance tuning another way. That’s not a dealbreaker, but it does require a decision.

Here are the two main paths forward when native optimization isn’t part of the package:

Option 1: Build Your Own Optimization Pipelines

Apache Iceberg is fully compatible with open engines like Apache Spark, Flink, and Dremio. Each of these supports table maintenance features such as:

• File compaction

• Manifest rewriting

• Snapshot expiration

You can schedule these jobs using tools like Airflow or dbt, or embed them directly into your data ingestion flows. This approach works in any environment, including on-prem, hybrid, and cloud.

Pros:

• Complete flexibility in how and when you optimize

• Can tailor jobs to match data patterns and storage costs

• Fully open and vendor-independent

Cons:

• Requires engineering effort to build, monitor, and tune jobs

• No centralized UI or automation unless you build one

Option 2: Use a Catalog-Neutral Optimization Vendor

Vendors like Ryft.io offer managed optimization services designed specifically for Iceberg. These tools run outside your query engines and handle compaction, cleanup, and layout improvements without relying on any one catalog or engine.

NOTE: Apache Amoro offers an open source optimization tool if looking for an open source option.

Key detail: Ryft currently only supports deployments that store data in AWS S3, Azure Data Lake, or Google Cloud Storage. If you’re using on-prem HDFS or other object stores, this may not be viable.

Pros:

• No need to manage optimization logic

• Works across multiple compute engines and catalogs

• Keeps optimization decoupled from platform lock-in

Cons:

• Limited to major cloud object storage unless using Apache Amoro

• Adds another vendor and billing model to your stack

When native optimization isn’t available, the best path depends on your team’s appetite for operational work. DIY gives you control. Neutral services give you speed. Either way, optimization remains a critical layer — whether you manage it yourself or let someone else handle it.

5. The Interoperability Spectrum

One of the key promises of Apache Iceberg is engine interoperability. The Iceberg REST Catalog API was designed so any compliant engine — whether it’s Spark, Flink, Trino, or Dremio — can access tables the same way. But in practice, not all catalogs offer equal levels of interoperability.

Some catalogs expose full read/write access to external engines using the REST API. Others allow only reads — or place restrictions on how writes must be performed. This creates a spectrum, where catalogs differ in how open or engine-specific they are.

Here’s how several major catalogs compare:

What This Means for You

If your architecture depends on multiple engines, the safest route is to choose a catalog that:

• Implements the full Iceberg REST spec

• Allows both reads and writes from all compliant engines

• Avoids redirecting writes through proprietary services or SDKs

This isn’t just about standards, it’s about reducing long-term friction. The more interoperable your catalog, the easier it is to plug in new tools, migrate workloads, or share datasets across teams without rewriting pipelines or triggering lock-in.

Architectural Patterns: Choosing the Right Iceberg Catalog for Your Stack

With a clear understanding of feature capabilities across commercial Iceberg catalogs, the next consideration is architectural alignment. How should teams select a catalog based on their engine stack, deployment model, and optimization philosophy?

Here, we explore common deployment patterns and their implications:

Single-Engine Simplicity

Use case: Organizations standardized on one compute engine seeking high performance and low operational overhead.

Benefits:

• Seamless integration between compute and catalog.

• Native optimization features (e.g., OPTIMIZE TABLE, Z-Ordering).

• Simplified access control and performance tuning.

Trade-offs:

• May impose file format restrictions (e.g., Parquet-only).

• Optimization tightly coupled to engine but if REST-Spec is ahered to you can still develop your own optimization pipelines.

Recommended Platforms: Dremio, Databricks Unity Catalog, AWS Glue (with managed compute).

Multi-Engine Interop (Spark + Trino + Flink)

Use case: Organizations with complex, multi-engine environments that require consistent metadata across tools and Clouds.

Benefits:

• Use the right engine for the right job (ETL, BI, ML).

• Maximize transactional openness via full IRC support.

Trade-offs:

• Optimization is either manual or vendor-dependent.

• Catalog-neutral solutions may lack server-side performance tuning.

Recommended Platforms: Dremio Enterprise Catalog, Snowflake Open Catalog. (Both based on Apache Polaris)

Streaming-First Architectures

Use case: Teams integrating real-time data from Kafka into the lakehouse for analytics or ML.

Benefits:

• Stream-native catalog (e.g., Confluent TableFlow) materializes Kafka topics into Iceberg tables.

• Seamless schema registration and time-travel.

Trade-offs:

• No schema evolution.

• Limited optimization control (rigid snapshot retention).

• Often designed for read-heavy use cases.

Recommended Platforms: Confluent TableFlow, integrated with external catalogs for downstream processing.

Cloud-Embedded Storage Catalogs

Use case: Teams deploying AI or analytics workloads in private/hybrid cloud environments.

Benefits:

• Built-in REST Catalog support directly within storage (MinIO AIStor).

• Simplifies deployment — no separate metadata layer deployment.

• High concurrency and transactional consistency at scale.

Trade-offs:

• Tightly bound to object storage vendor.

• No native table optimization

Recommended Platforms: MinIO AIStor (on-premise/private cloud), AWS S3 Tables (cloud-native equivalent).

Governance-Led Architectures

Use case: Enterprises prioritizing metadata lineage, compliance, and discovery.

Benefits:

• Centralized metadata layer for observability and access management.

• Easy discovery and tracking across teams and tools.

Trade-offs:

• No native write capabilities (metadata-only catalog).

• Optimization must be handled by external systems.

Recommended Platforms: DataHub Iceberg Catalog (OSS or Cloud) or using an external catalog (Dremio Catalog, Apache Polaris) connected into Datahub.

Each pattern has architectural trade-offs. Rather than seeking a perfect catalog, successful teams prioritize alignment with workflow needs: engine independence, optimization automation, governance, or real-time ingestion. In some cases, hybrid strategies, like dual catalogs or catalog, neutral optimization overlays — provide the best of both worlds.

Optimization Strategy Trade-offs: Native, Manual, or Vendor-Neutral

Once an organization selects a catalog, the next major architectural decision is how to maintain and optimize Iceberg tables. While the IRC standard guarantees transactional consistency, it says nothing about how tables should be optimized over time to preserve performance and control storage costs.

Three primary approaches emerge:

1. Native Optimization (Catalog-Integrated Automation)

Many commercial catalogs offer built-in optimization features tightly coupled with their own compute engines. These include operations such as:

• Compaction (file size tuning)

• Delete file rewriting

• Snapshot expiration

• Partition clustering

Platforms like Dremio, AWS Glue, and Databricks provide SQL-native or automated processes (e.g., OPTIMIZE TABLE, auto-compaction) that manage these operations behind the scenes.

Pros:

• Zero setup — optimization is automatic or declarative.

• Built-in cost and performance tuning.

• Reduces engineering overhead.

Cons:

• Usually catalog-bound.

• Often restricted to Parquet format.

• Switching catalogs later requires reengineering optimization logic.

2. Manual Optimization (Bring Your Own Engine)

Open-source Iceberg supports all required lifecycle management operations, compaction, snapshot cleanup, rewrite manifests, but leaves it up to users to implement these jobs using engines like Apache Spark, Flink, Apache Amoro or Trino.

Pros:

• Total freedom , no vendor lock-in.

• Can be integrated into any data pipeline or orchestration framework (Airflow, dbt, Dagster).

Cons:

• Requires custom development and scheduling.

• Monitoring and tuning are the user’s responsibility.

• Risk of misconfiguration or inconsistent maintenance across tables.

This model works well with catalogs like Apache Polaris, OneLake, or Snowflake Open Catalog, which support R/W operations but do not enforce optimization strategies.

3. Catalog-Neutral Optimization Vendors

A newer middle ground is emerging with vendors like Ryft.io, which offer catalog-agnostic optimization as a service. These platforms connect to your existing Iceberg tables, via any REST-compliant catalog, and run automated optimization jobs externally.

Pros:

• Centralized, automated optimization regardless of catalog.

• Maintains interoperability and neutrality.

• Works across major cloud storage (e.g., S3, ADLS, GCS).

Cons:

• Still a maturing category.

• Requires compatible storage (cloud object stores).

• Additional cost and integration complexity.

This is particularly valuable in multi-engine or multi-catalog environments where optimization cannot be centrally enforced but must still be automated and reliable.

Summary: The Optimization Dilemma

There is no one-size-fits-all solution:

Choosing an optimization strategy is not just about performance — it’s a decision about how much control you need, how much complexity you can absorb, and how much optionality you want to preserve in your architecture.

Architectural Patterns for Balancing Optimization and Interoperability

As organizations adopt Apache Iceberg REST Catalogs (IRC) to decouple compute from metadata, a recurring challenge emerges: how to balance open interoperability with the benefits of proprietary optimization. No single approach satisfies every use case. Instead, data architects are increasingly designing hybrid strategies that reflect the unique demands of their data workflows, regulatory environments, and performance SLAs.

1. Read-Only Catalogs Paired with External Optimization

Some catalogs, provide high-performance read access to Iceberg tables but restrict external writes via IRC. In these scenarios, organizations may:

• Maintain a separate write-optimized catalog (e.g., Apache Polaris, Nessie, or Glue) for ingestion, transformation and optimization.

• Expose tables to the read-optimized catalog after ingestion and optimization is complete.

• Schedule synchronization jobs to ensure both catalogs reference consistent metadata snapshots.

This dual-catalog approach preserves the performance of engines with these restrictions while maintaining external transactional control via a neutral or R/W-capable catalog.

Pros:

• Best of both worlds: performance + flexibility.

• Avoids modifying data in restrictive environments.

Cons:

• Adds metadata orchestration complexity.

• Difficult to manage at high scale without automation.

2. Embedded Catalogs for Self-Managed Environments

Solutions like MinIO AIStor and Dremio Enterprise Catalog take a radically different approach, embedding the IRC layer directly into the object store or lakehouse platform in Dremio’s case. This creates a streamlined deployment architecture for private cloud, hybrid, or air-gapped environments where full control is required.

• Enables transactional Iceberg workloads without deploying a separate metadata database.

• Suited for exascale, high-concurrency AI/ML pipelines.

• Can be used alongside external catalogs for metadata synchronization if needed.

This model is also increasingly relevant for regulated industries or enterprises seeking on-premise lakehouse designs with built-in metadata authority.

3. Virtualized Format Interop via Metadata Translation

Microsoft OneLake, using Apache XTable, pioneers a virtualized metadata model. Instead of writing new Iceberg tables, XTable projects Iceberg-compatible metadata from Delta Lake tables in OneLake.

• ✅ Enables external Iceberg engines to query Delta-based data with no duplication.

• 🔄 Metadata is derived dynamically, enabling near real-time interop.

• ⚠️ Complex Iceberg-native features may be unsupported due to reliance on Delta primitives.

This architecture is ideal for organizations deeply committed to Delta Lake but wanting to provide Iceberg-compatible access for federated analytics or open-source tools.

Architectural Takeaway: Mix and Match for Your Use Case

The modern Iceberg ecosystem isn’t about picking a single vendor. Instead, it’s about selecting interoperable components that align with your architecture’s performance, governance, and flexibility goals.

Designing an effective catalog strategy means embracing modularity, using REST interoperability as the glue while tailoring optimization and governance layers to the needs of your teams.

Conclusion: Choosing the Right Iceberg Catalog for Your Strategy

The Apache Iceberg REST Catalog ecosystem has matured into a diverse landscape of offerings — each with its own balance of interoperability, optimization capability, and vendor integration strategy. From hyperscalers to open-source initiatives, every catalog presents unique strengths and trade-offs.

At the heart of this evolution is a simple but profound architectural truth:

Compute and metadata must decouple — but performance, governance, and interoperability must still align.

Key Takeaways

• If performance and simplicity are your top priorities, a native-optimization platform like Dremio, Databricks, or AWS Glue offers seamless, powerful lifecycle management.

• If complete control and flexibility across tools and clouds matter more, choose a self-managed catalog like Apache Polaris and prepare to invest in your own optimization pipeline or use a neutral optimizer like Ryft.io (when on major cloud object storage) or use the OSS Apache Amoro.

• If you’re locked into an analytics platform like Snowflake or BigQuery, understand the implications of the differing level of Iceberg support on these platforms.

Among the platforms reviewed, Dremio strikes a rare balance: offering full Iceberg REST compatibility, native R/W support from any engine, and automated optimization, without locking users into its compute layer.

Unlike platforms that charge per API call or limit external writes, Dremio only charges for compute run through Dremio itself, meaning you can leverage external engines freely while still benefiting from the platform’s integrated catalog.

This model promotes interoperability and performance without compromise, aligning with the core principles of the Iceberg Lakehouse architecture: open metadata, multi-engine flexibility, and governed performance.

Final Thought

The Iceberg REST Catalog isn’t just an API spec, it’s the foundation for a new kind of lakehouse: open, transactional, and cloud-agnostic. Your choice of catalog defines how far you can scale without friction.

Choose wisely.