How Apache Iceberg 1.11.0 eliminates query bottlenecks and secures metadata

I have spent the last week digging into the Apache Iceberg 1.11.0 release that dropped on May 19. The changelog boasts over 1,000 commits from more than 200 contributors. It stabilizes the V3 specification and promises to fix query bottlenecks while locking down metadata. The hype cycle in Silicon Valley is already spinning this as a massive architectural milestone. But when you look closely at the shift from a passive metadata store to an active compute layer, you have to ask who actually benefits from this and what it breaks in the process.

Shifting the bottleneck with server-side scan planning

Historically, query engines like Spark and Trino choked on memory and network limits because they had to download massive manifest lists to plan data scans on the client side. Iceberg 1.11.0 introduces REST catalog server-side scan planning to solve this. The catalog now takes on the job of evaluating queries and streaming only the relevant file tasks back to the client. Databricks Unity Catalog is already using this to enforce cross-engine attribute-based access control for external engines like Starburst.

It sounds great on paper. But we are just moving the compute burden. The catalog used to be a lightweight metadata store. Now it has to handle heavy planning workloads. If you are running at high scale, your catalog is going to need serious provisioning. Who needs this? Mostly massive enterprises with highly fragmented compute engines. For smaller teams, this might just introduce a new infrastructure headache and a single point of failure.

Metadata security and envelope encryption

The other major selling point is native envelope encryption. Until now, structural metadata in object storage was dangerously exposed. Iceberg 1.11.0 fixes this with a three-tier key hierarchy using AES-GCM. A Table Master Key wraps Key-Encryption Keys stored in the metadata. Those keys then wrap unique per-file Data-Encryption Keys to encrypt both data files and manifest lists. It integrates natively with AWS KMS, Google Cloud KMS, and Azure Key Vault.

Total index protection is a strict requirement if you work in healthcare or finance. It also makes sense if you are building environments where autonomous AI agents need to query data without complex bespoke permissions. But encryption requires client engines that actually support these new standards. We are going to see temporary interoperability gaps as different tools struggle to catch up with the V3 specification. You might lock down your metadata only to find your favorite query engine can no longer read it.

Engine upgrades and the reality of adoption

Engine support got a significant bump in this release. Apache Spark 4.1 and Apache Flink 2.1 are now the default build targets. This brings in an asynchronous micro-batch planner for Spark Structured Streaming and a DynamicIcebergSink for Flink workloads.

I always question the immediate utility of these upgrades. Forcing enterprise data teams to migrate to Spark 4.1 just to get the most out of Iceberg 1.11.0 is a tall order. Most organizations I talk to in the Bay Area are still untangling legacy pipelines. They do not have the bandwidth to upgrade their entire compute layer overnight.

Iceberg 1.11.0 is pushing the open data lakehouse forward. But the transition from a passive catalog to an active governance layer is not a simple plug-and-play upgrade. You have to weigh the benefits of centralized access control and encrypted metadata against the very real risk of catalog compute bottlenecks and engine compatibility issues. Plan your architecture carefully before jumping on the V3 bandwagon.

References

• https://dev.to/alexmercedcoder/an-in-depth-overview-of-the-apache-iceberg-1110-release-4l1n
• https://iceberg.apache.org/blog/apache-iceberg-1.11.0-release/
• https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFA2yTa5Ux1z_G-2ahwR5n5996gY-aNU6_MT3zGjvO8J_buf0HzJYVG2TJxdM8X-jtcsO3ZFkFkc_ZZ7qxw1rC3ZHHNH20vABppb806CuMi5wxCaiF7dPKAi3xaMdzT35V5Ofj0K75ivtY0-2OEK3DxV2J-emhGMofss8zdVP1Hx1aBSbDAye8qj9k=
• https://github.com/apache/iceberg-python/issues/2775
• https://opensource.googleblog.com/2026/05/announcing-apache-iceberg-1110.html
• https://iceberglakehouse.com/iceberg/iceberg-spec-v3-file-encryption/
• https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQF2KrjHoPJnWVsByUvYhWeEcU48eAOFG0hYjNgWej1GhgLUv61mfCD5wnTfm6ScyE5XC5Z0o_5XdSiei-1MI9SUgyzc6FuuXjIDMET9WvC8mOpwEw0cBwuPSrnw06bXvJYtSB7LvpEV2OXTS_TQhrQTGSeUEA9mSx7PTbtcA6btC1GFnsf0syaTcCjgKJvLxA==
• https://dev.to/alexmercedcoder/the-state-of-apache-iceberg-catalogs-in-june-2026-265e
• https://datastrato.ai/blog/iceberg-rest-catalog-actually-explained/
• https://www.databricks.com/blog/unity-catalog-and-next-era-apache-icebergtm
• https://www.databricks.com/dataaisummit/session/supporting-fine-grained-access-control-server-side-scan-planning
• https://stepto.net/blog/iceberg-lakehouse-catalog-ai-ready-data-2026
• https://docs.aws.amazon.com/glue/latest/dg/connect-glue-iceberg-rest-ext.html
• https://www.dremio.com/blog/whats-new-in-apache-iceberg-1-11-0/
• https://dataverses.io/resources/blog/iceberg-1.11.0-release
• https://www.snowflake.com/en/blog/engineering/lakehouse-data-governance-interoperability/
• https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFRiYCShOYn4CPMgVO0mrq9nO4Xg84MnAKy_E7aQlxMN9GawFTYLC6ARmWQYfC2Ho20xoB_iaIClAHkc5Mkdwrdhov05iAOs0EYLbYGpmOMdBR304PAi1TOvGy1