This week, OpenAI shipped three voice models in the API and a security variant of GPT-5.5. Anthropic doubled Claude Code rate limits using SpaceX Colossus compute. Thinking Machines released its first model, a 276B-parameter system built for 200-millisecond real-time interaction. Google launched Googlebooks and Gemini Intelligence at its Android Show. Anthropic released Claude for Legal with 20+ MCP connectors. Cursor landed in Microsoft Teams. The connective tissue across all of these stories: AI coding tools, processing hardware, and standards work are all maturing in parallel, and each maturation is reshaping the others.

AI Coding Tools: Cursor in Teams, Copilot CLI Iterates Fast

Cursor expanded its surface area this week. The Cursor team announced on May 11 that Cursor is now available in Microsoft Teams. Users can mention @cursor in any Teams channel to delegate tasks to an agent or pull information from Cursor directly into Teams. The integration matters because it moves Cursor out of the editor and into the place where engineering teams actually coordinate work. Async code review, ticket triage, and PR follow-ups now happen in the same surface as the conversation about them.

GitHub Copilot CLI shipped five releases between May 5 and May 11, 2026. Version 1.0.41 (May 5) reduced startup time by rendering the UI immediately while authentication ran in the background. Version 1.0.42 (May 6) improved MCP server error messages. Version 1.0.43 (May 6) added server-side model routing for Auto mode. Version 1.0.44 (May 8) fixed path completion flickering and enabled mid-input slash commands. Version 1.0.45 (May 11) added an /autopilot slash command to toggle between interactive and autopilot modes. Five releases in seven days reflects the cadence the Copilot CLI team has settled into since the tool went GA earlier this year.

Anthropic doubled Claude Code rate limits overnight on May 8, 2026. The increase came from a SpaceX partnership that added 300 megawatts of new compute, the equivalent of more than 220,000 Nvidia GPUs, in under a month. The Colossus One facility was originally built for xAI’s Grok training workloads. Anthropic now uses it for burst compute on developer-facing products. Claude Code users had reported hitting output limits during peak hours, and the doubled limits resolve that pressure point without raising prices.

The competition between Claude Code, Cursor, and GitHub Copilot continues to harden. Claude Code runs on Opus 4.7 with SWE-bench Verified at 87.6%, SWE-bench Pro at 64.3%, and CursorBench at 70%, according to Anthropic’s April 16 release. Cursor pushed Composer 2 and parallel agents in April. GitHub paused new sign-ups for Copilot Pro and Pro+ ahead of the June 1 transition to usage-based billing. Each tool now has a distinct positioning. Claude Code is the surface-agnostic agent for senior developers. Cursor is the daily-driver IDE. Copilot is the GitHub-integrated extension for organizations with existing GitHub investment.

Cursor 3.3 shipped in May 2026 with /multitask for spawning parallel subagents instead of running them in sequence. Vulnerability Scanner runs scheduled scans for known CVEs and outdated dependencies. Context usage breakdown shows engineers exactly what their agent is consuming. MCP connection stability got patched in the same release. Cursor 3, which shipped in April 2026, already changed the architecture by putting all local and cloud agents in a single sidebar. Agents kicked off from mobile, Slack, GitHub, and Linear all appear in one workspace view.

Cursor also launched its Security Review beta in May for Teams and Enterprise plans. Two always-on agents anchor the offering. Security Reviewer checks every PR for vulnerabilities, auth regressions, privacy risks, agent tool auto-approvals, and prompt injection attacks. It leaves inline comments at the exact diff location with severity and remediation guidance. Vulnerability Scanner runs scheduled scans for known vulnerabilities, outdated dependencies, and configuration issues with optional Slack notifications. Cursor introduced canvases in the Agents Window during the same window. Canvases let agents build interactive visual interfaces for PR reviews, eval analysis, and data dashboards rather than walls of text. They use React-based components and live alongside the terminal, browser, and source control as durable artifacts.

Cursor’s Bugbot is moving from $40 per seat per month to usage-based billing on June 8, 2026. Teams will bill from on-demand spend. Individuals will bill from included usage. The average Bugbot run costs $1.00 to $1.50 depending on PR size and complexity. The pricing change reflects the wider industry shift toward per-action billing for AI coding tools, which is the same direction GitHub Copilot is heading with its June 1 transition.

GitHub’s billing change is the bigger story for many teams. The Opus premium-request multiplier on Copilot jumped from 15x to 27x in the same transition window. Teams running Opus-heavy workloads on Copilot pay considerably more than Claude Max plan subscribers using the same model through Anthropic’s direct channel. The pricing gap is the reason a growing number of engineering teams are evaluating whether to use Claude Code as a primary surface and keep Copilot for GitHub-native workflows.

Claude Code itself stopped being a CLI in 2026. It runs in the shell, as VS Code and JetBrains extensions, as a GitHub Action that opens PRs, and inside claude.ai on web and mobile. Subagents, skills, hooks, and plan mode turn it into a per-repo configuration rather than a per-session tool. The agent runs anywhere the engineer works. That surface-agnostic posture is part of why the SpaceX compute deal mattered. Capacity constraints on Claude Code show up in five surfaces at once, and the rate limit increase helped all five.

A security note worth flagging. Cursor patched a vulnerability in version 2.5 that let a malicious Git repository trigger arbitrary code execution through the agent. The patch is in place. No public reports of in-the-wild abuse have surfaced. Teams running older Cursor versions need to update. The broader question the bug surfaced is how engineering teams handle repo trust when AI agents have shell access. The combination of agent capability and unverified inputs is a category of risk that did not exist a year ago, and the security tooling industry is starting to respond.

AI Processing: Tesla AI5 Tape-Out, Google TPU Customer Roster Expands

Tesla taped out its AI5 chip on April 15, 2026, with details continuing to land this week. The chip is dual-sourced from TSMC Arizona and Samsung Texas. According to Tesla’s stated specs, AI5 delivers roughly 8x the compute and 5x the bandwidth of the current AI4 hardware. A single AI5 chip approximates an NVIDIA H100 for Tesla’s specific inference workloads. A dual AI5 setup approximates an NVIDIA Blackwell at a fraction of the cost and power. Tesla claims AI5 uses roughly one-third the power of Blackwell and runs at under 10% of the cost.

Tesla’s strategic positioning is the more significant part of the announcement. Musk confirmed that AI4 is sufficient for Full Self-Driving safety levels, so Tesla owners do not need to be retrofitted. AI5 is built for Optimus humanoid robots and supercomputer clusters. Configurations of 5 to 12 AI5 chips per board will form the backbone of Tesla’s training infrastructure for FSD v15 and future Optimus models. Engineering samples are expected in late 2026, with volume production targeted for 2027. AI6 is already in development, with tape-out targeted for December 2026.

Google’s TPU customer roster reshaped the AI chip market the week before. At Google Cloud Next 2026 on April 22, Google unveiled the TPU 8t and TPU 8i, claiming 2.8x better price-performance than the prior Ironwood generation. Anthropic expanded to multiple gigawatts of next-generation TPU capacity for Claude training and serving. Meta signed a multibillion-dollar multiyear deal in February 2026. OpenAI now takes TPU capacity, which is the most significant signal because OpenAI trains on Microsoft-procured Nvidia clusters. A confirmed OpenAI booking on Google silicon is the first visible crack in the assumption that Nvidia GPUs are the only serious substrate for frontier AI.

The TPU 8i has 384MB of SRAM, triple the amount in the prior Ironwood generation. It pairs with Google’s custom Arm-based Axiom CPU. The 8i is built for inference and AI agents. The 8t targets training workloads, with what Google calls a development-cycle compression “from months to weeks.” Citadel Securities built quantitative research software on TPUs. All 17 U.S. Energy Department national laboratories use AI co-scientist software built on the chips. Broadcom co-designed TPU 8t under the codename “Sunfish.” MediaTek handles TPU 8i under the codename “Zebrafish.” The dual-vendor co-design pattern matters because it gives Google supply chain options. It also gives both Broadcom and MediaTek concrete reference designs they can extend to other hyperscaler customers.

Anthropic also announced a SpaceX compute deal on May 8 alongside the Claude Code rate limit increase. The deal covers 220,000+ GPUs at the Colossus One data center. The deal does not reduce Anthropic’s TPU commitment, since both arrangements run in parallel. Compute supply is the bottleneck for every frontier lab in 2026, and Anthropic, OpenAI, and Meta are all running multi-vendor strategies across Nvidia, Google TPUs, AMD Instinct, and now SpaceX-operated capacity.

Anthropic’s revenue context puts the SpaceX deal in perspective. Dario Amodei confirmed in early May that Anthropic grew 80x in Q1 2026, blowing past the internal plan that called for 10x growth. The annualized revenue run rate crossed $30 billion, up from $9 billion at the end of 2025. The number of customers spending $1 million per year doubled from 500 to over 1,000 in two months. That kind of revenue growth would push any AI lab into emergency compute procurement mode. The SpaceX deal, the multi-gigawatt Amazon and Google partnerships, the $30 billion Azure capacity arrangement through Microsoft and Nvidia, and the $50 billion Fluidstack US AI infrastructure investment are the practical response.

The orbital compute angle in the SpaceX announcement deserves a footnote. Anthropic and SpaceX expressed shared interest in developing multi-gigawatt orbital data center capacity over the coming years. The engineering timeline for that is measured in years, not quarters. The signal is what matters. Frontier AI labs and the compute providers that serve them are now planning for a future where terrestrial power, land, and cooling cannot keep pace with model training demands. Whether orbital capacity becomes real or stays speculative, the fact that two serious companies put it in a press release tells you where the industry thinks the compute ceiling is heading.

The environmental angle around Colossus 1 also bears mention. The Memphis facility runs on gas turbines that were initially installed without Clean Air Act permits or pollution control devices. Memphis residents have raised concerns about air quality and documented increases in hospital admissions tied to poor air quality near the site. Protests against the data center’s environmental footprint have continued through 2026. Anthropic committed to cover any consumer electricity price increases caused by its US data centers and is considering local investment in communities that host its facilities. Whether those commitments are sufficient remediation for the Memphis-specific concerns is an open question that the broader AI infrastructure conversation will continue to surface.

The Anthropic compute strategy is now multi-vendor, multi-region, and multi-substrate. Claude runs on AWS Trainium, Google TPUs, and Nvidia GPUs. The recent collaboration with Amazon includes inference capacity in Asia and Europe to satisfy data residency requirements for regulated industries. Location decisions focus on democratic countries with stable legal frameworks and secure supply chains. The single-vendor compute strategy that dominated 2023 and 2024 is gone. Portfolio diversification across the entire AI infrastructure stack has replaced it.

The wider trend is clear. Custom silicon is no longer a hyperscaler-only story. Tesla has joined the small group of companies that design AI chips from the ground up and manufacture at scale. Apple, Google, Amazon, and Meta are all building their own. Anthropic has multi-gigawatt commitments on both Nvidia and TPU substrates. The “single-vendor AI substrate” narrative that justified Nvidia’s valuation premium has its first real counter-example.

AI Standards & Protocols: Anthropic Claude for Legal Ships 20+ MCP Connectors

Anthropic released Claude for Legal on May 12, 2026. The launch includes 12 practice-area plugins and more than 20 MCP connectors. The 12 plugins cover Commercial Legal, Corporate Legal (including M&A diligence), Employment Legal, Privacy Legal, Product Legal, Regulatory Legal, AI Governance Legal, IP Legal, and Litigation Legal. Each plugin starts with a setup interview that learns the team’s playbooks, escalation chains, risk calibration, and house style.

The MCP connector list reads like the operational stack of a modern law firm. Anthropic connected Claude to DocuSign, Box, Thomson Reuters (CoCounsel Legal), Harvey, Relativity, Everlaw, and Microsoft 365. The Thomson Reuters integration is bidirectional. CoCounsel Legal is rebuilt on Anthropic’s technology, and Claude can now call CoCounsel as a tool. The foundation model is both the underlying layer and a caller of the application built on top of it. Anthropic also confirmed that legal became the number one power-user job function in Claude Cowork, with over 3x the usage of any other function.

The launch is a stress test for MCP as a production protocol. MCP shipped in November 2024 as Anthropic’s open standard for connecting AI models to tools and data. By February 2026, MCP crossed 97 million monthly SDK downloads across Python and TypeScript. The protocol is now under the Linux Foundation’s Agentic AI Foundation (AAIF), with Anthropic, OpenAI, and Block as co-founders. Microsoft embedded MCP into Windows 11 and Copilot. Google DeepMind confirmed support in Gemini. AWS, Cloudflare, and Bloomberg all sit on the AAIF.

The MCP 2026 roadmap published in March 2026 has four priority areas. First, transport evolution to make Streamable HTTP work statelessly at scale. Second, agent communication primitives, closing lifecycle gaps in the Tasks primitive. Third, governance maturation with a formal contributor ladder. Fourth, enterprise readiness with audit trails, SSO-integrated auth, and gateway patterns. The Tasks primitive shipped as experimental and now needs production iteration. Retry semantics on transient failures and expiry policies for completed tasks are the two concrete gaps to close.

Google’s Agent-to-Agent (A2A) protocol sits alongside MCP, not in competition. MCP connects agents to tools and data. A2A connects agents to other agents. Microsoft, AWS, and Google all support both. The two-layer stack uses MCP for tool access and A2A for agent coordination. It has become the architectural default for enterprise multi-agent deployments. Industry data shows enterprise MCP adoption crossed 78% in production AI teams, and the public registry surpassed 9,400 servers. Enterprise MCP gateways from Kong, Docker, and others now centralize authentication, audit trails, and tool-level access control.

The Anthropic Claude for Legal launch is significant precisely because it deploys MCP at production scale for a regulated industry. Legal aid organizations and public defenders also get access through Claude for Nonprofits at discounted pricing. The launch rattled legal tech stocks. RELX, Thomson Reuters, and Wolters Kluwer shares fell on the February plugin announcement, and the May 12 release is considerably larger in scope.

Anthropic followed Claude for Legal with Claude for Small Business on May 13, 2026. The bundle covers connectors for QuickBooks, PayPal, HubSpot, Canva, DocuSign, Google Workspace, and Microsoft 365. It includes 15 ready-to-run skills covering finance, operations, sales, marketing, HR, and customer service. Workflows handle payroll planning, month-end close, business performance monitoring, campaign management, invoice chasing, and cash-flow forecasting. The package runs on top of Claude Cowork, the multi-step automation platform Anthropic launched in January 2026, and ships as a toggle install in the desktop app.

The Small Business release is the fifth market-specific Claude product Anthropic has shipped since the start of 2025. The other four target life science researchers, schools, attorneys, and financial professionals. Each bundle uses the same MCP-based connector architecture and the same Claude Cowork automation layer. The pattern is clear. Anthropic uses MCP and Claude Cowork as a platform substrate, with vertical packaging on top. The vertical packaging is what closes deals. The substrate is what makes the verticals shippable in quick succession.

Claude for Small Business comes with a 10-city tour starting May 14 in Chicago. Tulsa, Dallas, Hamilton Township, Baton Rouge, Birmingham, Salt Lake City, Baltimore, San Jose, and Indianapolis follow. Each stop is a free half-day training workshop for 100 local small business leaders. Attendees get a one-month Claude Max subscription. Anthropic also partnered with Workday and the Local Initiatives Support Corporation on a Solopreneurship Accelerator Program funding 15 aspiring entrepreneurs in 2026 with seed capital, Claude credits, and an AI-first curriculum. A separate partnership with three Community Development Financial Institutions targets small business access to capital.

The competitive pressure on traditional SaaS vendors keeps building. Salesforce, ServiceNow, Intuit, DocuSign, and Box have all seen their stocks decline year to date and over the last 12 months as Anthropic’s offerings expand into territory those companies have historically owned. Dario Amodei warned at the Briefing: Financial Services event in early May that some SaaS vendors will go bankrupt if they cannot keep pace with the AI shift. That framing is harder to dismiss when Anthropic’s own annualized revenue grew from $9 billion to $30 billion in roughly a year. The market is repricing on the assumption that AI-native delivery will collapse a meaningful share of seat-based SaaS revenue.

MCP makes that competitive shift feasible at the protocol level. The same connector that lets Claude pull data from QuickBooks for a small business owner lets Claude pull data from Thomson Reuters for a corporate legal team. Anthropic ships one protocol stack and one automation layer, then packages it for a dozen verticals. The traditional SaaS business cannot easily respond because the value proposition for those tools was the workflow integration with the user. When a model can build the integration through MCP, the workflow lock-in weakens fast.

Also Worth Noting

Thinking Machines released TML-Interaction-Small on May 11, 2026, the first model from Mira Murati’s lab. The 276 billion-parameter mixture-of-experts system uses 12 billion active parameters. The architecture processes audio, video, and text in 200-millisecond micro-turns rather than waiting for users to finish speaking. The model achieves 0.40-second turn-taking latency, roughly the speed of natural human conversation. Soumith Chintala, PyTorch co-creator, became CTO after co-founders Barret Zoph and Luke Metz left for OpenAI in January. The research preview is available to a limited group of researchers, with broader access planned for later in 2026. The model’s interaction-first design reflects a hypothesis the team has been making publicly for months. Real-time multimodal interaction is a different category of capability than the request-response pattern that dominates most current AI products.

Google streamed its Android Show on May 12, 2026, one week before Google I/O 2026. Two announcements anchored the event. Googlebooks are premium Gemini-first laptops from Acer, Asus, Dell, HP, and Lenovo, shipping this fall. Every Googlebook features a signature “Glowbar” light bar on the keyboard. Gemini Intelligence is the new agentic AI layer running underneath Android. It takes data from one app and completes multistep tasks across other apps without the user switching between them. Gemini Intelligence rolls out to the latest Samsung Galaxy and Google Pixel phones starting this summer, then to Wear OS, Android Auto, Android XR, and Googlebooks. The Glowbar is a physical signal that an AI agent is acting on the user’s behalf. That kind of hardware-level affordance for AI activity is the same direction Apple is exploring with iOS 26 visual indicators for active Siri agents.

OpenAI shipped three voice models on May 8. GPT-Realtime-2 brings GPT-5-class reasoning to real-time voice. GPT-Realtime-Whisper handles transcription workloads. GPT-Realtime-Translate handles speech-to-speech translation. The same day, OpenAI released GPT-5.5-Cyber to vetted security teams in limited preview. The release is a direct response to Anthropic’s Claude Mythos Preview, which has been used under Project Glasswing to identify zero-day vulnerabilities. ElevenLabs reported crossing $500 million in annual recurring revenue after a Series D round on the same day, with cuts to voice and agentic API pricing. The voice infrastructure category is one of the most competitive surfaces in AI right now. OpenAI, ElevenLabs, Cartesia, Deepgram, and Thinking Machines are all pushing on the same set of latency and quality benchmarks from different starting points.

White House National Economic Council Director Kevin Hassett confirmed on May 7 that the White House is drafting an executive order requiring AI models to be vetted before public release. The Commerce Department has already expanded its voluntary pre-release testing program to include Google, Microsoft, xAI, OpenAI, and Anthropic. Pennsylvania filed suit against Character.AI on May 8 over AI personas misrepresenting themselves as qualified medical professionals. Connecticut’s comprehensive AI bill and Iowa’s chatbot safety law both advanced in the same week. The state-level regulatory activity is moving faster than the federal pace, and the patchwork compliance question is becoming a real operational concern for AI companies serving US users.

Unsloth joined the PyTorch Ecosystem on May 11, bringing its open-source AI training and inference acceleration tools into the official ecosystem. The move expands PyTorch’s official tooling footprint and gives Unsloth’s acceleration layer more institutional weight. The Fivetran 2026 Agentic AI Readiness Index, released May 8, found that only 15% of organizations have a data foundation capable of safely running AI agents at production scale. Nearly 60% have already invested millions in the technology. The gap between agent ambition and data foundation readiness is the practical reality every enterprise AI program is now confronting. The lakehouse stack work happening in parallel inside the Apache Iceberg, Polaris, Arrow, and Parquet communities is what closes that gap, and the timing of the Fivetran report against the AI infrastructure investment surge is not a coincidence.

What to Watch Next Week

Google I/O 2026 runs the week of May 19 and is the biggest single event on the calendar. Gemini 3 production rollout, Android XR partner announcements, and additional TPU 8 customer disclosures are all on the likely agenda. Anthropic’s 10-city Small Business Tour kicks off May 14 in Chicago, and the early-week press coverage will set expectations for SMB AI adoption. The state-level AI regulatory activity in Pennsylvania, Connecticut, and Iowa should continue moving, and the federal executive order drafting work bears close watching. On the coding tools side, GitHub Copilot’s June 1 usage-based billing transition is the next major pricing event, and Cursor’s continued shipping cadence on the 3.x release line should bring more agent surface changes. The MCP and A2A protocol communities are running ongoing working group sessions on the Tasks primitive iteration and the Streamable HTTP stateless transport, with concrete proposals expected through the spring.

Resources to Go Further

The AI landscape changes fast. Here are tools and resources to help you keep pace.

Try Dremio Free. Experience agentic analytics and an Apache Iceberg-powered lakehouse. Start your free trial

Learn Agentic AI with Data. Dremio’s agentic analytics features let your AI agents query and act on live data. Explore Dremio Agentic AI

Join the Community. Connect with data engineers and AI practitioners building on open standards. Join the Dremio Developer Community

Book: The 2026 Guide to AI-Assisted Development. Covers prompt engineering, agent workflows, MCP, evaluation, security, and career paths. Get it on Amazon

Book: Using AI Agents for Data Engineering and Data Analysis. A practical guide to Claude Code, Google Antigravity, OpenAI Codex, and more. Get it on Amazon

The post-summit translation work that has dominated 2026 turned into shipped artifacts this week. Iceberg cut a 1.11.0 release candidate on the strength of weeks of design follow-ups. Polaris published a security-focused 1.4.1 patch release alongside four coordinated CVE disclosures and announced 1.5.0 planning for next week. Arrow’s Rust subproject opened three release votes in a single day. Parquet finally shipped Java 1.17.1 after a year between releases and turned its attention to the next wave of format-level proposals. The connective tissue across all four projects: production hardening at scale, AI-workload-driven format design, and the slow consolidation of governance frameworks around AI-assisted contribution.

Apache Iceberg

Iceberg’s biggest news this week is the 1.11.0 release candidate that Aihua Xu opened for voting on May 9. The RC1 vote drew rapid engagement from Péter Váry, Yuya Ebihara, Steven Wu, Kevin Liu, Steve Loughran, Russell Spitzer, Amogh Jahagirdar, Talat Uyarer, Manu Zhang, Ajay Yadav, and huaxin gao, with verification work splitting between binary checks, Snowflake build tests, and Trino downstream validation. The thread surfaced enough discussion that Aihua had to address questions about Spark integration coverage, version notes, and licensing audit follow-ups across multiple replies before the vote could close cleanly. This is the first 1.x release of 2026 carrying the full weight of V3 production maturity, and the depth of the verification work reflects how seriously contributors are treating it as a stability anchor while V4 design continues in parallel.

The release candidate landed against a backdrop of unblocking work that Ryan Blue drove through the LICENSE updates thread. The discussion threaded through commentary from Russell Spitzer, Steven Wu, Aihua Xu, Jean-Baptiste Onofré, Steve, roryqi, John Zhuge, Péter Váry, Fokko Driesprong, and Kevin Liu, with the conversation focused on ensuring that 1.11 would ship with a clean LICENSE/NOTICE chain that matched what contributors had actually merged. Apache release engineering depends on these audits being thorough, and the discussion shows the community treating LICENSE correctness as a release blocker rather than a checkbox.

Beyond the release work, the V4 design conversations continued advancing on multiple fronts. Ryan Blue opened a fresh DISCUSS thread on partition tuples in V4 that quickly drew responses from Steven Wu, Anoop Johnson, Amogh Jahagirdar, Russell Spitzer, and Micah Kornfield. The thread is one of the more architecturally significant V4 conversations — partition tuples affect how metadata represents partition state when single-file commits replace manifest lists, and the decision shapes how column statistics, manifest delete vectors, and root manifests interact at scale. Amogh’s multiple replies on the thread reflect the same depth of analysis that has anchored his work on the broader one-file commits proposal with Russell.

Ryan also opened a DISCUSS thread on a compact bitmap format that drew engagement from Maximilian Michels, Andrei Tserakhau, Guy Khazma, Anoop Johnson, and Amogh Jahagirdar. The proposal targets one of the practical efficiency issues with the current Roaring bitmap encoding used for delete vectors — for sparse delete sets across very large data files, the metadata overhead matters at scale, and a more compact format that preserves the operational semantics could materially affect the storage footprint of MOR tables in production. Anoop Johnson separately opened a V4 Aggregate Column Stats DISCUSS thread that pushes on the same broader theme — making metadata cheaper to scan as table sizes grow.

The catalog-side design work also stayed active. EJ Wang’s First-class Tag concept in Iceberg REST Catalog DISCUSS thread drew responses from Yufei Gu and Andrei Tserakhau, building on the broader labels-and-metadata conversation that Andrei has been driving for months. Steven Wu opened a VOTE on adding the CatalogObjectIdentifier schema that drew binding +1 votes through the week from Yufei Gu, Russell Spitzer, huaxin gao, Christian Thiel, Alexandre Dutra, Jean-Baptiste Onofré, and Steve. Yuya Ebihara’s DISCUSS thread on recursive namespace listing drew responses from Ajantha Bhat and Yufei Gu — a quality-of-life REST API change that would matter most to catalogs federating across many tenants. Prashant Singh’s DISCUSS thread on credential management for KMS/Vault and table-level encryption pulled in feedback from Sreesh Maheshwar, Chris Lu, Gidon Gershinsky, and Ádám Szita. Alexandre Dutra opened a DISCUSS thread on passing arbitrary information to request signers — a thread that builds on the months of work he’s been leading on remote signing semantics.

Yuya Ebihara also opened a DISCUSS thread on adding HashiCorp Vault KMS support that drew engagement from Steve Loughran, Romain Manni-Bucau, and Jean-Baptiste Onofré. Vault is the de facto key management standard for self-hosted environments, and bringing first-class Vault support into the encryption layer closes one of the bigger gaps for teams running Iceberg outside the major cloud KMS providers.

The Rust subproject continued shipping. Shawn Chang opened the Iceberg Rust 0.9.1 release candidate vote, worked through two intermediate RCs before RC3 drew binding +1s from Renjie Liu, Kevin Liu, Matt Butrovich, Jean-Baptiste Onofré, Kurtis Wright, Maximilian Michels, Sung Yun, and Fokko Driesprong, and then announced the 0.9.1 release. This is the fifth Iceberg Rust release in seven months — a cadence the community would not have predicted at the start of 2025. The Rust implementation’s DataFusion integration makes it a serious alternative for teams that want Iceberg without a JVM dependency, and the cadence reflects how much of that downstream traffic is actually shipping. Kurtis Wright separately opened a Curiosity thread on checksums in Iceberg libraries that drew responses from Russell Spitzer, Steve Loughran, Daniel Weeks, and Andrei Tserakhau — the kind of cross-library integrity question that matters more as Iceberg deployments expand beyond Java.

The community also welcomed Andrei Tserakhau as a new committer, with congratulations rolling in from Matt Topol, Neelesh Salian, Eduard Tudenhöfner, Amogh Jahagirdar, Micah Kornfield, Kevin Liu, huaxin gao, Steven Wu, Alex Stephen, Sung Yun, Fokko Driesprong, Renjie Liu, Maximilian Michels, Gang Wu, Péter Váry, Drew, Talat Uyarer, Russell Spitzer, Shawn Chang, and Kurtis Wright. Andrei has anchored the labels-in-LoadTableResponse work across the spring, and the committer recognition reflects how that proposal moved from idea to multi-implementation POC across Polaris, Unity Catalog, Lakekeeper, and PyIceberg.

Local meetups continue to anchor community activity. Endi Caushi confirmed the Boston Iceberg meetup for May 6, Lester Martin announced the Atlanta meetup for May 13, Viktor Kessler advertised the Iceberg Community Meetup Europe events in Barcelona and Erlangen, Germany for May plus the June London and Amsterdam meetups, and Danica Fine announced the Seattle Iceberg Community Meetup for June 25. Sung Yun shared the Iceberg Summit 2026 Selection Committee retrospective notes — Kevin Liu’s reply suggests retrospectives like this are exactly the kind of community-building work that lets the summit grow without breaking the volunteer model that makes it possible.

Apache Polaris

The defining Polaris story of the week is the 1.4.1 patch release and the four coordinated CVE disclosures that accompanied it. Jean-Baptiste Onofré opened the 1.4.1 RC0 release vote on May 5, drew rapid +1s from Robert Stupp, Dmitri Bourlatchkov, Russell Spitzer, and Sung Yun, and closed the vote successfully before announcing the 1.4.1 release. The patch addresses the KMS upgrade bug, Helm packaging fixes, and the security disclosures that landed alongside it.

The CVE disclosures themselves are the more significant artifact. Jean-Baptiste posted four coordinated advisories: CVE-2026-42809 covering staged table creation credential abuse, CVE-2026-42810 covering literal wildcard handling in IAM resource patterns, CVE-2026-42811 covering scope leakage in GCS credentials, and CVE-2026-42812 covering write.metadata.path protection. Four CVEs in one release window is not a small event for a project that graduated to top-level status three months ago, but the visible coordinated disclosure — patches first, public advisories second — is exactly the security posture enterprise deployments need to see. The bugs concentrate around credential vending and resource pattern handling, which is where most cross-tenant exposure surfaces in a catalog that issues subscoped credentials on behalf of clients.

With 1.4.1 out the door, attention turned immediately to the 1.5.0 cycle. Jean-Baptiste opened a DISCUSS thread asking whether Polaris should target 1.5.0 next week. The thread drew responses from Dmitri Bourlatchkov, Ajantha Bhat, Yufei Gu, Alexandre Dutra, and Robert Stupp, with Jean-Baptiste replying multiple times to refine the scope. Monthly release cadence has been the explicit commitment since the project graduated, and the 1.5.0 conversation reflects the community’s discipline about staying on that schedule even after a security-focused patch release week.

Anand Kumar Sankaran’s Uptaking 1.4.1 and turning on table metrics persistence thread gives a useful window into what production adopters are actually doing with the new release. Dmitri Bourlatchkov’s reply walks through the configuration mechanics — a small example of the community’s responsiveness to deployment-side questions that wouldn’t have had this kind of visible support a year ago when much of that traffic still went through bilateral vendor channels.

Jean-Baptiste also drafted the Polaris May 2026 board report, which drew commentary from Francois Papon, Robert Stupp, Dmitri Bourlatchkov, Yufei Gu, and Adnan Hemani. The board report is the formal artifact the PMC submits to the ASF board, and the visible drafting on the dev list reinforces the project’s open-governance posture. The community also welcomed Sung Yun to the PMC, with Robert Stupp’s announcement drawing congratulations from Jean-Baptiste Onofré, Alexandre Dutra, Keith Chapman, James Rowland-Jones, Kevin Liu, Yufei Gu, Dmitri Bourlatchkov, and Michael Collado. Sung Yun has anchored the REST Catalog “Trusted Iceberg Client” terminology work on the Iceberg side and brings strong cross-project coordination credentials to the Polaris PMC.

The DISCUSS pipeline stayed dense. Bill Bejeck opened a Diagnostics shell prototype for Polaris thread that drew engagement from Dmitri Bourlatchkov, Jean-Baptiste Onofré, and Yufei Gu. Jean-Baptiste opened a Polaris server custom assembly tool thread that drew responses from Dmitri Bourlatchkov and Yufei Gu. Robert Stupp opened a DISCUSS thread on enabling advisory Copilot PR review for documentation and test omissions that drew responses from Jean-Baptiste Onofré, Dmitri Bourlatchkov, Yong Zheng, and Yufei Gu — exactly the kind of AI-assistance governance conversation that mirrors the Iceberg and Parquet AI contribution policy work happening in parallel. Robert also opened a Guardrails for security-sensitive changes thread that drew a response from Jean-Baptiste Onofré — a conversation that lands with extra weight in the same release window as the four CVE disclosures.

Tornike Gurgenidze opened a DISCUSS thread on storage credential-vending SPI changes, pushing the SPI surface that lets vendors plug in alternative credential vending strategies. Srinivas Rishindra opened a DISCUSS thread on event persistence architecture and a global sanitization pipeline. EJ Wang continued the AI-readability conversation with a DISCUSS thread on linking the AI-generated Code Wiki from project docs. Dmitri Bourlatchkov opened a DISCUSS thread on adjusting the renameTable response code to 204 that drew responses from Nándor Kollár and Yufei Gu — a small REST API conformance question that matters for client interoperability. Anand Kumar Sankaran’s feat: Configurable STS session names thread drew a reply from Dmitri Bourlatchkov.

Apache Arrow

Arrow’s Rust subproject ran its tightest release week of the year. Andrew Lamb opened Apache Arrow Rust 58.3.0 RC1, Apache Arrow Rust 57.3.1 RC1, and Apache Arrow Rust 56.2.1 RC1 within a single planned cluster of patch and minor releases. All three drew rapid verification from Ed Seidl, Bryce Mecum, Raúl Cumplido, and L. C. Hsieh, and Andrew posted the three RESULT messages confirming all three votes had passed. The arrow-rs project running three concurrent release votes in a single week is a real engineering benchmark — it reflects the maintenance load of supporting multiple active release lines (58.x as the current minor, 57.x and 56.x as supported back-versions) and the verification community has reached the scale where this kind of parallel release cadence is actually sustainable. Andrew’s earlier heads-up that planned patch releases were coming this week set the cadence expectations clearly.

Beyond Rust releases, the design surface stayed lively. Antoine Pitrou opened a DISCUSS thread on field/schema/custom metadata restriction to UTF8 that drew engagement from Rusty Conover, Raphael Taylor-Davies, and Dewey Dunnington. The thread sits at the intersection of cross-language compatibility and forward extensibility — Arrow metadata that is strictly UTF8 is easier to interoperate across languages, but the constraint also limits what extension types can carry through. Richie Black opened a DISCUSS thread on column default value metadata changes to FlightSql.proto — a JDBC interoperability question that matters for cross-system data engineering as Flight SQL adoption grows.

The pyarrow-stubs donation vote that has been building since Rok Mihevc opened it in April drew further engagement as Rok closed out the vote and confirmed the donation could move toward formal acceptance. The donation effectively brings type stubs for pyarrow into the official Arrow project rather than relying on a community-maintained external repository — a small but meaningful signal that the Python community is investing in pyarrow’s type-checking surface as much as its runtime behavior.

The Erlang language binding moved forward. Benjamin Philip continued advancing the Arrow Erlang grant documents thread with Sutou Kouhei, working through the IP grant paperwork that the ASF requires for code donations. Erlang adoption is a niche audience for Arrow, but the grant work is the same procedural foundation that every language expansion follows, and the visibility of the process is what makes Arrow’s multi-language footprint sustainable. Antoine Pitrou’s announcement of the Apache Arrow / Parquet meetup in Paris drew enthusiasm from Sutou Kouhei and Marc Deveaux — the kind of cross-project meetup that reflects how interleaved the Arrow and Parquet communities have become.

Mandukhai Alimaa’s DISCUSS thread on a canonical BigDecimal extension type and Andrew Lamb’s arrow-rs security policy discussion continued threading toward production hardening. The security policy work in particular reflects a project that is being deployed in commercial-grade scenarios where formal vulnerability disclosure paths matter as much as the underlying code quality. The Nishant Avasthi-led DISCUSS on adding Apache Arrow support for IBM Db2 via ADBC drew a response from Ian Cook — another quietly significant adoption signal, since Db2 brings a category of enterprise mainframe and traditional database workloads into Arrow’s data interchange footprint.

Apache Parquet

Parquet finally shipped its first Java release of the year. Gang Wu opened the parquet-java 1.17.1 RC0 vote, drew +1s from Steve Loughran, Fokko Driesprong, Russell Spitzer, Daniel Weeks, and Xinli shang, and announced the 1.17.1 release. The release closes the gap that opened after parquet-java 1.17.0 shipped in January 2026, and resolves the long-running DISCUSS thread on a new parquet-java release that Manu Zhang opened in late March. The release cadence question has been one of the more honest community conversations of the year — parquet-java has historically shipped less frequently than its sister projects, and Manu’s thread surfaced the real costs (slower bug-fix delivery, encoding feature lag) that justified the patch release effort.

Russell Spitzer’s DISCUSS thread on GH-3547 automated release for Parquet drew engagement from Arnav Balyan, Gang Wu, and Fokko Driesprong on the broader question of how to bring parquet-java’s release infrastructure closer to the cadence of the format and Rust projects. Automated release tooling is what eventually makes monthly-class release cadences sustainable, and the Polaris release engineering work has been one of the visible reference implementations the Parquet community can point at when designing its own approach.

The community welcomed Ed Seidl as a new committer, with Micah Kornfield announcing the recognition and congratulations following from Andrew Lamb, Gang Wu, and Raúl Cumplido. Ed has driven multiple format-level threads through 2026, including the path_in_schema optionality proposal and engaged commentary on the FlatBuffer footer redesign work. The committer recognition reflects sustained spec-design work over the year.

The format-level conversation continued at high intensity. Will Edwards opened a DISCUSS thread on how readers handle Parquet files with future extensions — a forward-compatibility question that becomes more urgent as the format adds Variant, Geospatial, and now the proposed File logical type. Daniel Weeks opened a DISCUSS thread on supporting non-contiguous pages that drew engagement from Andrew Bell, Adrian Garcia Badaracco, Micah Kornfield, Will Edwards, and Andrew Lamb. Non-contiguous pages would let writers split a column’s pages across the file rather than requiring them adjacent — a non-trivial format change with implications for how readers do projection pushdown and how object storage range reads can be coalesced. Andrew Bell’s separate Wide Schemas thread drew responses from Andrew Lamb, Adrian Garcia Badaracco, and Steve Loughran — wide-table workloads are precisely the AI/ML use case that has been pulling format design forward all year.

Micah Kornfield’s DISCUSS thread on remaining open spec-level questions for ALP drew engagement from Andrew Lamb on closing out the ALP encoding work that passed an earlier vote. Arnav Balyan opened a DISCUSS thread on adding AGENTS.md to parquet-java that drew responses from Aaron Niskode-Dossett, Andrew Lamb, and Micah Kornfield, and a separate DISCUSS thread on an AI tooling policy for Parquet that drew a response from Fokko Driesprong. The AGENTS.md and AI tooling policy threads are the Parquet community’s version of the same conversation Iceberg and Polaris have been running in parallel — how AI-assisted contribution fits into Apache governance and what guardrails the project wants to set.

Martin Prammer’s Datasets Project — Raincloud thread drew responses from Arnav Balyan on the broader question of how the Parquet community thinks about reference datasets for testing and benchmarking. Dewey Dunnington’s Geography test files with statistics thread continued the geospatial spec stabilization work. Julien Le Dem ran the regular Parquet sync on May 6, with the meeting notes thread setting the agenda for the format-level work that played out across the rest of the week.

Cross-Project Themes

The clearest pattern this week is the maturation of release engineering across all four projects. Iceberg cut a major 1.11.0 RC. Polaris shipped a patch release coordinated with four CVE disclosures and immediately planned the next monthly minor. Arrow ran three Rust release votes in parallel. Parquet shipped its first patch release of the year and started seriously planning automated release tooling. Each project is operating at a release cadence that would have been hard to sustain a year ago, and the cumulative effect is a lakehouse stack where every component is shipping at predictable, professional intervals. That is the difference between a research stack and infrastructure, and the lakehouse stack has clearly crossed into the latter.

The second pattern is the consistent attention to AI-assisted contribution governance. Iceberg has the published AI contribution policy work that Holden Karau, Kevin Liu, Steve Loughran, and Sung Yun pushed through March. Polaris ran the AI-generated Code Wiki linking thread and the advisory Copilot PR review thread this week. Parquet ran the AGENTS.md and AI tooling policy threads in parallel. These conversations are not happening in isolation — they reflect a coordinated community position that AI tools are welcome in Apache contribution flows but require explicit governance, disclosure, and review patterns. The pattern is consistent enough across projects that it looks like an emerging Apache-wide norm rather than four parallel one-offs.

The third pattern is the continued translation of AI workload pressure into format-level proposals. Iceberg’s compact bitmap format, V4 aggregate column stats, and partition tuples work all push on metadata efficiency at scale. Parquet’s wide schemas thread, the File logical type proposal that’s still resolving, and the non-contiguous pages discussion all target the AI workloads where data shapes don’t match the assumptions the format was originally designed around. Arrow’s BigDecimal canonical extension type and the metadata UTF8 thread both push on cross-language interop for AI-pipeline data. These are not coincidences — they’re four projects responding to the same pressure from the same workloads, on the same timeline.

The fourth pattern is the visible security maturation across the stack. Polaris’s four CVE disclosures landed with formal coordinated advisories, separate patches, and clear remediation guidance. Iceberg’s KMS/Vault credential management work, Arrow’s security policy discussion, and Parquet’s careful handling of forward-compatibility questions all reflect a shared posture that the lakehouse stack is being deployed in environments where formal security disclosure paths are required, not optional. The CVEs being public is itself a healthy signal — projects with bad security posture don’t disclose, they hide.

Looking Ahead

Watch for the Iceberg 1.11.0 vote to close and the release to ship in the coming days. The Polaris 1.5.0 planning email is likely the next major release-side artifact, with the 1.4.1 security work as the floor and 1.5.0 feature scoping as the ceiling. The Iceberg V4 design work — partition tuples, compact bitmap format, aggregate column stats — is converging toward formal proposals that should land in the coming weeks alongside the long-anticipated formal single-file commits write-up. The Parquet release engineering automation thread should converge into a concrete proposal, and the AGENTS.md and AI tooling policy discussions should harden into something the community can adopt.

On the Arrow side, the field/schema metadata UTF8 thread and the FlightSql column default value proposal both look ready to mature into more formal proposals, and the pyarrow-stubs donation should land formally. The Iceberg Community Meetup Europe series across Barcelona, Erlangen, London, Amsterdam, and Basel — plus the Atlanta and Seattle North American meetups — will continue translating the dev-list conversations into in-person community building. Iceberg Summit 2026 session recordings will continue rolling out on YouTube, and the next round of Apache board reports across all four projects will set the formal narrative for what shipped in May.

Resources & Further Learning

Get Started with Dremio

• Try Dremio Free — Build your lakehouse on Iceberg with a free trial

• Build a Lakehouse with Iceberg, Parquet, Polaris & Arrow — Learn how Dremio brings the open lakehouse stack together

Free Downloads

• Apache Iceberg: The Definitive Guide — O’Reilly book, free download

• Apache Polaris: The Definitive Guide — O’Reilly book, free download

Books by Alex Merced

• Architecting an Apache Iceberg Lakehouse

• Enabling Agentic Analytics with Apache Iceberg and Dremio

• The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI

• The Book on Using Apache Iceberg with Python

This is Part 2 of a 15-part Apache Iceberg Masterclass. Part 1 covered why table formats exist. This article breaks down exactly how each format organizes its metadata.

The metadata structure of a table format determines everything: how fast queries start planning, how efficiently concurrent writes are handled, how schema changes propagate, and how much overhead accumulates over time. Two formats can both claim “ACID support” and “time travel” while having fundamentally different mechanisms under the hood.

Table of Contents

1. What Are Table Formats and Why Were They Needed?

2. The Metadata Structure of Current Table Formats

3. Performance and Apache Iceberg’s Metadata

4. Technical Deep Dive on Partition Evolution

5. Technical Deep Dive on Hidden Partitioning

6. Writing to an Apache Iceberg Table

7. What Are Lakehouse Catalogs?

8. Embedded Catalogs: S3 Tables and MinIO AI Stor

9. How Iceberg Table Storage Degrades Over Time

10. Maintaining Apache Iceberg Tables

11. Apache Iceberg Metadata Tables

12. Using Iceberg with Python and MPP Engines

13. Streaming Data into Apache Iceberg Tables

14. Hands-On with Iceberg Using Dremio Cloud

15. Migrating to Apache Iceberg

Apache Iceberg: The Metadata Tree

Iceberg organizes metadata into a tree with four levels. Each level adds specificity and enables pruning at query planning time.

Level 1: Catalog pointer. The catalog (a REST catalog, Dremio Open Catalog, AWS Glue, or Hive Metastore) stores a pointer to the current metadata.json file. This pointer is the single source of truth for the table’s current state.

Level 2: Metadata file (metadata.json). A JSON file containing the table’s schema (with column IDs), partition spec, sort order, table properties, and a list of snapshots. Each snapshot represents a complete, immutable version of the table. When the table is updated, a new metadata.json is created with the new snapshot appended to the list.

Level 3: Manifest list (Avro). Each snapshot points to exactly one manifest list. The manifest list is a table of contents: it lists all the manifest files that make up this snapshot and includes partition-level summary statistics for each manifest. These summaries let the query planner skip entire manifests that cannot contain data matching the query filter.

Level 4: Manifest files (Avro). Each manifest file tracks a set of data files and delete files. For each file, the manifest stores the file path, file size, row count, partition values, and column-level statistics (min value, max value, null count, NaN count, distinct count). These per-file statistics enable file-level pruning during query planning.

The key insight is that each level progressively narrows the search space. A query engine using Dremio or Spark reads the catalog pointer (1 request), loads the metadata file (1 read), checks the manifest list to skip irrelevant manifests (1 read, many skips), then reads only the relevant manifests to find the actual data files to scan. For a petabyte table, this can reduce planning from minutes of directory listing to milliseconds of metadata traversal.

Delta Lake: The Sequential Transaction Log

Delta Lake uses a simpler, linear structure. All metadata lives in the _delta_log/ directory alongside the data.

JSON commit files (000001.json, 000002.json, ...) record each transaction as a set of actions: add (a new data file), remove (a file marked for deletion), metaData (schema or property change), and protocol (version requirements). Each commit file is sequentially numbered.

Parquet checkpoint files are created every 10 commits (by default). A checkpoint is a Parquet file that summarizes the cumulative state of the table at that version, essentially a snapshot of all currently-active add actions. This prevents readers from having to replay hundreds of small JSON files.

_last_checkpoint is a small file pointing to the most recent checkpoint. The read process is: find the latest checkpoint, load it, then replay any JSON commits after it.

The tradeoff: Delta’s log is simple and easy to reason about, but it does not have the multi-level pruning that Iceberg’s manifest tree provides. File-level statistics exist in the add actions but are not organized hierarchically. For very large tables (millions of files), the planning phase can be slower because there is no intermediate pruning layer equivalent to Iceberg’s manifest list.

Apache Hudi: The Timeline

Hudi stores metadata in the .hoodie/ directory as a sequence of “instants” on a timeline. Each instant represents an operation (commit, compaction, rollback, clean) and transitions through three states: REQUESTED, INFLIGHT, and COMPLETED.

The timeline is split into two parts:

Active timeline contains recent instants that are needed for current read and write operations. The file naming pattern is <timestamp>.<action_type>.<state>. For example, 20250429010500.commit.completed indicates a completed write operation.

Archived timeline contains older instants that have been moved to .hoodie/archived/ to keep the active timeline lean. Hudi 1.0 introduced an LSM-based timeline that compacts archived instants into Parquet files for efficient long-term storage.

Hudi’s timeline tracks more granular operation types than other formats: commit (COW write), delta_commit (MOR write), compaction, clean (garbage collection), rollback, savepoint, and replace (clustering). This granularity reflects Hudi’s focus on complex write patterns like CDC pipelines.

Apache Paimon: Snapshots and LSM Trees

Paimon’s metadata is organized around snapshots and buckets. Each partition is divided into a fixed number of buckets, and each bucket contains an independent LSM (Log-Structured Merge) tree.

The snapshot metadata tracks which data files and changelog files belong to each bucket at each point in time. Inside each bucket, the LSM tree structure contains multiple “sorted runs” (levels) of Parquet files. When data is written, it lands in level 0 as a small sorted file. Background compaction merges small files into larger ones at higher levels.

This is fundamentally different from the other formats because Paimon’s metadata structure is designed for continuous streaming writes rather than batch commits. The LSM tree handles high-frequency inserts and updates efficiently by buffering writes in memory and flushing them as sorted runs.

DuckLake: SQL Database as Metadata

DuckLake takes the most radical departure. Instead of storing metadata as files in object storage, all metadata lives in a traditional SQL database (PostgreSQL, MySQL, SQLite, or DuckDB itself).

The metadata database contains tables for: schemas, snapshots, data files, column statistics, and table properties. When a query engine needs to plan a query, it issues a single SQL query against the metadata database instead of reading multiple metadata files from object storage.

The tradeoff is a dependency on a running database process for metadata management. The benefit is dramatically simpler metadata access patterns and the ability to use SQL for metadata operations like listing snapshots, finding files, and checking statistics.

Side-by-Side Comparison

For teams building on multiple engines, Iceberg’s metadata structure provides the best combination of planning efficiency and engine independence. Dremio uses Iceberg’s metadata tree to achieve fast query planning even on tables with millions of files, and its Columnar Cloud Cache caches frequently-accessed metadata locally to further reduce planning latency.

Part 3 covers how query engines use Iceberg’s metadata specifically for performance optimization.

Books to Go Deeper

• Architecting the Apache Iceberg Lakehouse by Alex Merced (Manning)

• Lakehouses with Apache Iceberg: Agentic Hands-on by Alex Merced

• Constructing Context: Semantics, Agents, and Embeddings by Alex Merced

• Apache Iceberg & Agentic AI: Connecting Structured Data by Alex Merced

• Open Source Lakehouse: Architecting Analytical Systems by Alex Merced

Free Resources

• FREE - Apache Iceberg: The Definitive Guide

• FREE - Apache Polaris: The Definitive Guide

• FREE - Agentic AI for Dummies

• FREE - Leverage Federation, The Semantic Layer and the Lakehouse for Agentic AI

• FREE with Survey - Understanding and Getting Hands-on with Apache Iceberg in 100 Pages

Most teams start building a semantic layer the wrong way: they open their BI tool, create a few calculated fields, and call it done. Six months later, three dashboards define “churn” differently, nobody trusts the numbers, and the data team is debugging metric discrepancies instead of building new features.

A well-built semantic layer prevents all of that. Here’s how to do it right.

Start With Metrics, Not Data Models

Before writing a single line of SQL, sit down with stakeholders from Sales, Finance, Marketing, and Product. Agree on the top 5-10 business metrics your organization uses to make decisions.

For each metric, document:

• The calculation: Revenue = SUM(order_total) WHERE status = ‘completed’ AND refunded = FALSE

• The owner: Who is accountable for this definition?

• The grain: Daily? Monthly? Per customer?

• The refresh cadence: Real-time? Daily batch? Weekly?

This exercise is harder than it sounds. You will discover that “Monthly Active Users” has three competing definitions. That’s the point. The semantic layer can’t resolve disagreements that haven’t been surfaced yet.

Output: A metric glossary. This becomes the source document for everything you build next.

Map Your Data Sources

Inventory every system that feeds into your analytics:

Source TypeExamplesAccess PatternTransactional databasesPostgreSQL, MySQL, SQL ServerFederated query (read-only)Cloud data lakesS3 (Parquet/Iceberg), Azure Data LakeDirect scan or catalogSaaS platformsSalesforce, HubSpot, StripeAPI extraction or replicationSpreadsheetsGoogle Sheets, ExcelOne-time import or scheduled sync

Not all sources need to be replicated into a central store. Federation lets you query data where it lives without the cost and complexity of ETL pipelines. Platforms like Dremio connect to dozens of sources and present them in a single namespace, so your semantic layer can span everything without data movement.

Design the Three-Layer View Structure

The most effective semantic layer architecture uses three layers of SQL views, commonly called the Medallion Architecture.

Bronze Layer (Preparation)

Create one view per raw source table. Apply no business logic. Just make the data human-readable:

• Rename cryptic columns: col_7 → OrderDate, cust_id → CustomerID

• Cast types to standard formats: strings to dates, integers to decimals

• Normalize timestamps to UTC

• Avoid using SQL reserved words as column names (Timestamp, Date, Role will force double-quoting in every downstream query. Use EventTimestamp, TransactionDate, UserRole instead.)

Bronze views should be boring. Their only job is to make raw data safe to work with.

Silver Layer (Business Logic)

This is where your metric glossary becomes code. Silver views join Bronze views, deduplicate records, filter invalid data, and apply business rules.

Example:

CREATE VIEW silver.orders_enriched AS
SELECT
    o.OrderID,
    o.OrderDate,
    o.Total AS OrderTotal,
    c.Region,
    c.Segment
FROM bronze.orders_raw o
JOIN bronze.customers_raw c ON o.CustomerID = c.CustomerID
WHERE o.Total > 0 AND o.Status = 'completed';

Each Silver view encodes exactly one business concept. “Revenue” is defined in one place. Every dashboard, notebook, and AI agent that needs revenue queries this view. No exceptions.

Gold Layer (Application)

Gold views are pre-aggregated for specific consumers. A BI dashboard gets monthly_revenue_by_region. An AI agent gets customer_360_summary. A finance report gets quarterly_financial_summary.

Gold views don’t add new business logic. They aggregate and reshape Silver views for performance and usability.

Document Everything — or Let AI Help

An undocumented semantic layer is a semantic layer nobody uses. Every table and every column should have a description that explains:

• What the data represents

• Where it comes from

• Any known limitations or caveats

This is tedious work. Modern platforms accelerate it with AI. Dremio’s generative AI, for example, can auto-generate Wiki descriptions by sampling table data, and suggest Labels (tags like “PII,” “Finance,” “Certified”) for governance and discoverability. The AI provides a 70% first draft. Your data team fills in the domain-specific context.

This documentation serves two audiences: human analysts browsing the catalog, and AI agents that need context to generate accurate SQL. Both benefit from rich, accurate descriptions.

Enforce Access Policies at the Layer

Security should be embedded in the semantic layer, not applied after the fact in each tool. Two patterns:

Row-Level Security: Filter what data a user can see based on their role. A regional manager sees only their region’s data. The SQL view applies the filter automatically.

Column Masking: Mask sensitive columns (SSN, email, salary) for roles that don’t need them. Analysts see ****@email.com. Data engineers see the full value.

The advantage of enforcing policies at the semantic layer: every downstream query inherits the rules, whether the query comes from a dashboard, a Python notebook, or an AI agent. No gaps.

Start Small, Then Expand

Don’t try to model your entire data landscape on day one. Start with:

• 3-5 core metrics from your glossary

• The 2-3 source systems those metrics depend on

• One Bronze → Silver → Gold pipeline per metric

Validate by running the same question across two different tools (a BI dashboard and a SQL notebook, for example). If both return the same number, the semantic layer is working. If they don’t, fix the Silver view definition before adding more.

Once the first metrics are stable, expand incrementally. Add new sources, new Silver views, new Gold views. Each addition is low-risk because the layered structure isolates changes.

What to Do Next

Pick the metric your organization argues about the most. Define it explicitly in a Silver view. Test it against the current dashboards. If the numbers match, you’ve validated the approach. If they don’t, you’ve just found the inconsistency that’s been silently costing your organization trust.

Try Dremio Cloud free for 30 days

This week pushed three concrete lines forward at once. Vercel open-sourced an AI security harness, TinyFish made paid web search and fetch APIs free for AI agents, and Jama Software shipped the first Model Context Protocol server for engineering management. Underneath that, Z.ai’s GLM-5.1 went live inside a trusted execution environment, and Anthropic previewed a proactive assistant for its work product. Here is what shipped and why each piece matters.

AI Coding Tools: Vercel Ships deepsec, TinyFish Drops Search Behind a Paywall

Vercel open-sourced deepsec, an AI-powered security harness that uses Claude and Codex to scan large codebases for vulnerabilities. The tool runs CLI-first, scales to over 1,000 concurrent sandboxes, and works with any pluggable coding agent through Vercel’s AI Gateway or your own subscription. The pitch is straight: most coding agents handle one repo at a time, but security audits need to fan out across dozens of services in parallel. deepsec treats the agent as a worker pool and scales horizontally.

TinyFish made its Web Search and Fetch APIs free for all developers and AI agents on May 5. The free tier supports Claude Code, Cursor, Codex, and other major agent frameworks, with no credit card required and what TinyFish calls generous rate limits. Web access has been a paid bottleneck for agent workflows since 2024, and a free tier from a vendor that already serves the agent ecosystem will pull pricing pressure across the rest of the search-API market.

Anthropic also previewed a proactive assistant called Orbit for Claude Cowork. Orbit will pull insights from Gmail, Slack, GitHub, Calendar, Drive, and Figma, then surface them on its own without the user asking. The product is reportedly a Max-tier feature, and Orbit Apps were also referenced in the leaks. The combination of always-on context and proactive surface area is the next step beyond chat-only agent products.

AI Processing: GLM-5.1 Runs FP8 Inside a Trusted Execution Environment

Z.ai’s GLM-5.1 went live on the 0G Private Computer running FP8 inside a Trusted Execution Environment on May 5. The model is a 754-billion-parameter Mixture-of-Experts release with 40 billion active parameters per token, shipped under the MIT license on April 7. Running it inside a TEE means the weights and prompts stay encrypted from the host operating system and cloud provider, which closes the residual trust gap that has slowed enterprise self-hosting of large open-weight models.

The 0G deployment matters for a specific reason. GLM-5.1 was trained entirely on Huawei Ascend 910B chips with no Nvidia or AMD GPUs, scores 58.4 percent on SWE-Bench Pro, and sustains autonomous task execution for over 8 hours. Putting that capability behind a TEE on a third-party serving platform is the first time a frontier-tier open-weight model has been delivered with hardware-backed confidentiality outside a hyperscaler.

Air Street’s State of AI report for May noted that the UK AI Security Institute now estimates frontier cyber-offence capability is doubling every four months, with both Anthropic’s Claude Mythos Preview and OpenAI’s GPT-5.5 clearing a 32-step end-to-end cyber-attack range in a single month. The compute side of the picture stayed dense as well. Anthropic raised an additional $40 billion from Google plus $5 billion from Amazon, packaged with $100 billion of AWS spend and chip deals with Google and Broadcom reportedly worth hundreds of billions.

Standards & Protocols: First MCP Server for Engineering Management

Jama Software launched the first MCP Server for engineering management software on May 4. Jama Connect 9.35 lets engineers work in Claude, Codex, Cursor, GitHub Copilot, Visual Studio, or any MCP-compatible tool while keeping the existing Traceability Information Model, permissions, lifecycle workflows, and audit trails intact. The pitch from CTO Jim Davidson is that AI engineering agents need Spec Driven Development to deliver compliant velocity gains, and MCP is now the standard pipe for that integration.

Unity AI also entered open beta this week with built-in MCP Server support, alongside the AI Gateway for third-party AI integrations. Game studios get a built-in agent tuned for Unity workflows plus the option to plug in any MCP-compatible client. The pattern across both releases is clear: vertical product vendors are no longer asking whether to support MCP. They are shipping it as a default integration surface alongside their native UIs.

The Model Context Protocol’s 2026 roadmap sets the priorities behind these adoptions. Lead Maintainer David Soria Parra named four focus areas: stateless transport for horizontal scaling, server discovery through .well-known URLs, task lifecycle for retry semantics and result expiry, and enterprise-readiness work covering audit trails, SSO, and gateway behavior. The June 2026 specification cycle is targeted for the stateless transport changes. Agentic AI Foundation governance, two specification releases through 2025, and a 500-plus public server count have moved MCP from an experiment into the production layer it was designed to be.

Resources to Go Further

The AI landscape changes fast. Here are tools and resources to help you keep pace.

Try Dremio Free: Experience agentic analytics and an Apache Iceberg-powered lakehouse. Start your free trial

Learn Agentic AI with Data: Dremio’s agentic analytics features let your AI agents query and act on live data. Explore Dremio Agentic AI

Join the Community: Connect with data engineers and AI practitioners building on open standards. Join the Dremio Developer Community

Book: The 2026 Guide to AI-Assisted Development: Covers prompt engineering, agent workflows, MCP, evaluation, security, and career paths. Get it on Amazon

Book: Using AI Agents for Data Engineering and Data Analysis: A practical guide to Claude Code, Google Antigravity, OpenAI Codex, and more. Get it on Amazon

The release wave that defined late April carried straight into early May, with Arrow shipping two more votes in seven days, Polaris settling into post-1.4.0 stabilization mode, and the Iceberg dev list staying focused on V4 design follow-ups from the summit. The clearest story of the week is Arrow’s release engineering: the arrow-rs 58.2.0 vote that opened on April 28 closed cleanly on May 2, and the Arrow .NET 23.0.0 vote opened the same day and passed by May 5. Two votes, two passing results, four days apart — a cadence that would have been unimaginable a year ago when the project was still navigating its full-stack release cycle. Iceberg’s design lists stayed in absorption mode as contributors continued to translate post-summit alignments into formal specification work, and Parquet’s dev list remained dense with format-level threads that have been simmering since the ALP encoding vote closed in April.

Apache Iceberg

Iceberg’s dev list ran quieter this week than the Arrow and Polaris lists, but the design conversations that have anchored 2026 continued to advance in the background. The V4 metadata.json optionality direction — the proposal to treat catalog-managed metadata as a first-class supported mode while preserving static-table portability through explicit opt-in semantics — is still the project’s defining specification conversation, with Anton Okolnychyi, Yufei Gu, Shawn Chang, Steven Wu, and Russell Spitzer continuing to push edge cases on portability guarantees and Spark driver behavior. The single-file commits proposal that Russell Spitzer and Amogh Jahagirdar have been advancing remains on track for a formal write-up that should land on the dev list in the coming weeks.

Péter Váry’s efficient column updates proposal for wide tables continues to attract collaboration. Anurag Mantripragada and Gábor Kaszab are working alongside Péter on POC benchmarks for both the Iceberg-native and Parquet-native approaches, with the latency and metadata footprint improvements making this one of the more practically grounded V4 proposals on the list. The design — write only the columns that change on each commit, then stitch the result at read time — is squarely aimed at petabyte-scale feature stores with thousands of embedding and model-score columns, and that workload pressure is precisely what’s pulling the V4 spec design forward.

The labels in LoadTableResponse proposal that Andrei Tserakhau drove through March continues to anchor the catalog-managed metadata conversation. The design lets each catalog (Polaris, Unity Catalog, Lakekeeper) surface internal metadata such as ownership, cost attribution, and semantic context through a standard optional field on table loads, without forcing requirements onto catalogs that don’t track that data. The cross-implementation POCs that Andrei published — Polaris (PR #4048), Unity Catalog (PR #1417), Lakekeeper (PR #1676), and the PyIceberg client (PR #3191) — remain useful reference points as the spec change progresses through review. Iceberg Summit 2026 session recordings continued rolling out on the project’s YouTube channel, and the published AI contribution policy that Holden Karau, Kevin Liu, Steve Loughran, and Sung Yun pushed through March remains the next concrete deliverable to track.

Apache Polaris

Polaris transitioned from release-week intensity into stabilization mode this week. The 1.4.0 release that Adnan Hemani announced on April 23, followed by the Python CLI 1.4.0 release on April 28, gave the project its first major release pair as a graduated top-level project. The post-launch issues that Alexandre Dutra surfaced — the Helm chart repo inconsistency, the release workflow failure in step 4, the Artifact Hub request, and the KMS-related upgrade bug — are exactly the kind of friction a project surfaces in its first independent release cycle. Yufei Gu has continued to triage most of the upgrade-path issues, and the Helm packaging questions are converging toward resolution.

Design discussions stayed active alongside the post-release stabilization. EJ Wang’s DISCUSS thread on AGENTS.md for Polaris — the proposal to add agent-readable repository metadata so coding agents can pick up the project conventions consistently — continued building toward a concrete implementation proposal, which the previous newsletter flagged as the next deliverable to watch. ITing Lee’s proposal to add OpenLineage to Polaris has accumulated the volume of review feedback from Adnan Hemani, Jean-Baptiste Onofré, Yufei Gu, and Michael Collado that it needs to move toward an implementation RFC. Yufei’s thread on narrowing the scope of SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION drew further engagement from Dmitri Bourlatchkov and Dennis Huo, and Alexandre Dutra’s URL path decoding and PolarisPrivilege grant validation threads continued to be active points of discussion.

Jean-Baptiste Onofré’s confirmation that Polaris is back on a monthly release cadence means a 1.4.1 patch release or 1.5.0 planning email is the natural next step. Given the volume of upgrade-path issues that surfaced after 1.4.0, a quick 1.4.1 to address the KMS bug and Helm packaging fixes seems the more likely path before the project moves on to 1.5.0 feature scoping.

Apache Arrow

Arrow’s release engine kept running. Andrew Lamb’s arrow-rs 58.2.0 RC1 vote that opened on April 28 closed on May 2, with the release approved by 6 +1 votes (4 binding) and immediately published to crates.io. Bryce Mecum, Ed Seidl, Jeffrey Vo, Raúl Cumplido, and L. C. Hsieh carried the verification work, with L. C. Hsieh casting the final binding +1 from an Intel Mac on April 29. The 58.2.0 release continues the monthly arrow-rs cadence that has held since 58.1.0 shipped in March, and 59.0.0 remains scheduled as a major release that may include breaking changes.

Curt Hagenlocher opened the Arrow .NET 23.0.0 RC0 vote on May 2 — the same day arrow-rs 58.2.0 was approved — and the vote passed on May 5 with 5 binding +1s from Bryce Mecum, Adam Reeve, Raúl Cumplido, Sutou Kouhei, and Curt himself. Sutou Kouhei verified on Debian sid with .NET SDK 8.0.413, and Curt ported verify_rc.sh to Powershell as part of the validation. Curt is now working through the post-vote release tasks, including a 401 issue with the GitHub release download script that he flagged for follow-up. The .NET 23.0.0 release continues the steady cadence the .NET implementation has settled into post the 22.0.0 cycle.

Beyond releases, the design conversations stayed lively. The pyarrow-stubs donation vote that Rok Mihevc opened on April 14 continued building toward a final tally. Emil Sadek’s ADBC Logo Proposal drew further engagement from Nic Crane, Julian Hyde, and Rusty Conover, and Benjamin Philip’s Arrow Erlang grant documents thread continued the project’s expansion into more language ecosystems. Andrew Lamb’s arrow-rs security policy discussion and Mandukhai Alimaa’s canonical BigDecimal extension type proposal both continued to draw input as the project tightens its production posture.

Apache Parquet

Parquet’s lists stayed dense. Manu Zhang’s DISCUSS thread on a new parquet-java release continued attracting input from Steve Loughran, Aaron Niskode-Dossett, Fokko Driesprong, Julien Le Dem, Gang Wu, and Rahil C, with the conversation now narrowing on a target version and ship date for what would be the next parquet-java release after 1.17.0. Ismaël Mejía’s thread soliciting code reviews for Java performance optimization work continued with Steve Loughran picking up the review load.

The format-level proposals continued evolving. Will Edwards’s DISCUSS thread on an alternative to the FlatBuffer footer with a lightweight byte-offset index kept drawing design feedback from Andrew Lamb, Ed Seidl, Jan Finis, Alkis Evlogimenos, Raphael Taylor-Davies, and Andrew Bell. Ed Seidl’s proposal to make path_in_schema optional continued attracting commentary from Gang Wu, Steve Loughran, and Micah Kornfield. Andrew Lamb’s thread on where VariantJsonParser should live — the cross-project boundary question between Parquet and Iceberg’s variant tooling — kept moving with input from Steve Loughran and Gang Wu.

The Geospatial work continued threading toward closure. Milan Stefanovic’s Geospatial CRS string format clarification drew further input from Dewey Dunnington and Micah Kornfield, and Jan Finis’s question on RLE bitpack page-edge validity continued the kind of spec-edge clarification work that matters for cross-implementation interoperability. The Parquet sync that Julien Le Dem ran on April 22 set the agenda for the design work that’s now playing out across the dev list.

Cross-Project Themes

This week’s clearest pattern is the rhythm of post-graduation Polaris finding its operational footing alongside Arrow’s well-established release cadence. Two Arrow votes in four days plus the Polaris 1.4.x stabilization wave plus Iceberg’s quiet absorption of summit alignments plus Parquet’s dense format-level work make the lakehouse stack feel less like four separate projects and more like one coordinated platform. The arrow-rs 58.2.0 release in particular landed inside a single five-day vote window — proposed April 28, approved May 2, published to crates.io the same day — which is a useful benchmark for how tight Apache release engineering can run when the verification community is engaged.

The second pattern is the continued translation of post-summit alignments into spec work. The V4 metadata.json optionality direction, the labels-in-LoadTableResponse proposal, the AGENTS.md thread for Polaris, the OpenLineage RFC, the Parquet footer redesign work, and the Geospatial spec clarifications are all converging on the same broader question: what does the lakehouse stack look like when the workloads it powers shift from analytical SQL to AI agents and ML feature engineering? Each design conversation makes more sense if you assume the next decade’s workload mix looks meaningfully different from the last decade’s.

The third pattern is enterprise-readiness work surfacing in real time. Polaris’s KMS upgrade bug, Helm packaging issues, OAuth2 Manager v2 design, and credential-subscoping scope discussion are all the work of a project being deployed at scale rather than a project being built. The visible triage on the dev list rather than behind closed doors is a healthy signal.

Looking Ahead

Watch for a Polaris 1.4.1 patch release vote to address the KMS bug and Helm packaging issues that surfaced after 1.4.0, with 1.5.0 planning to follow. The AGENTS.md discussion should firm into a concrete implementation proposal, and the Polaris OpenLineage RFC has the volume of feedback it needs to move toward an implementation. On the Iceberg side, the formal V4 single-file commits write-up, the V4 metadata.json optionality direction, and the published AI contribution policy remain the next concrete deliverables to track. The labels-in-LoadTableResponse spec PR (apache/iceberg#15750) should converge toward merge as the cross-catalog POCs validate the design.

On the Arrow side, the pyarrow-stubs donation vote should close in the coming days, and arrow-go and arrow-cpp release planning will shape what ships in May and June. For Parquet, Manu Zhang’s parquet-java release thread should converge on a target version, the path_in_schema optionality proposal looks ready for a formal vote, and the FlatBuffer-footer alternative is on track for a more formal design document. Iceberg Summit 2026 session recordings will continue rolling out on YouTube — the V4 design talks and production case studies from Apple, Bloomberg, and Pinterest are particularly worth catching as they land.

Resources & Further Learning

Get Started with Dremio

• Try Dremio Free — Build your lakehouse on Iceberg with a free trial

• Build a Lakehouse with Iceberg, Parquet, Polaris & Arrow — Learn how Dremio brings the open lakehouse stack together

Free Downloads

• Apache Iceberg: The Definitive Guide — O’Reilly book, free download

• Apache Polaris: The Definitive Guide — O’Reilly book, free download

Books by Alex Merced

• Architecting an Apache Iceberg Lakehouse

• Enabling Agentic Analytics with Apache Iceberg and Dremio

• The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI

• The Book on Using Apache Iceberg with Python

This is Part 1 of a 15-part Apache Iceberg Masterclass. This article covers the fundamental question: what problem do table formats solve, and why does the choice between them matter?

A data lake without a table format is a collection of files. It has no concept of a transaction, no mechanism to prevent two writers from producing corrupted state, and no efficient way to determine which files belong to the current version of a table. Table formats exist because the gap between “a pile of Parquet files” and “a reliable analytical table” is enormous, and bridging it requires a formal metadata specification.

Table of Contents

1. What Are Table Formats and Why Were They Needed?

2. The Metadata Structure of Current Table Formats

3. Performance and Apache Iceberg’s Metadata

4. Technical Deep Dive on Partition Evolution

5. Technical Deep Dive on Hidden Partitioning

6. Writing to an Apache Iceberg Table

7. What Are Lakehouse Catalogs?

8. Embedded Catalogs: S3 Tables and MinIO AI Stor

9. How Iceberg Table Storage Degrades Over Time

10. Maintaining Apache Iceberg Tables

11. Apache Iceberg Metadata Tables

12. Using Iceberg with Python and MPP Engines

13. Streaming Data into Apache Iceberg Tables

14. Hands-On with Iceberg Using Dremio Cloud

15. Migrating to Apache Iceberg

The World Before Table Formats

Before table formats, data lakes relied on a simple convention: data was organized into directories in object storage (S3, ADLS, GCS), and the Hive Metastore tracked which directories corresponded to which partitions.

This approach had five critical problems:

No atomic commits. If a Spark job wrote 500 new Parquet files and failed after writing 300, readers could see the 300 partial files. There was no mechanism to make all 500 files visible at once or none of them. Cleanup required manual intervention or custom garbage collection scripts.

Expensive query planning. To determine which files to scan, the engine issued LIST requests against object storage. S3 returns up to 5,000 objects per request. A table with 100,000 files required 20+ sequential HTTP calls before query execution could even start. At Netflix, query planning for large tables could take minutes just from directory listing.

Schema changes required rewrites. Adding a column to a Hive table meant either rewriting every file (expensive) or accepting that old files had a different schema than new files (confusing). Renaming a column was not supported without a full table rewrite because Hive mapped columns by position, not by identity.

No time travel. Once data was overwritten, the previous version was gone. There was no snapshot history, no ability to roll back a bad write, and no way to reproduce a query result from last Tuesday.

Exposed partitioning. Users had to know the physical partition layout. A table partitioned by year and month required queries to explicitly filter on those columns using the exact partition column names (WHERE year = 2024 AND month = 3). If partitioning changed, every downstream query broke.

What a Table Format Actually Is

A table format is a specification that defines how to organize metadata about data files so that query engines can treat them as reliable, transactional tables. It sits between the query engine and the physical files.

The core responsibilities of every table format:

• File tracking: Maintain an explicit list of which data files belong to the current version of the table, eliminating directory listing

• Atomic commits: Make all changes to a table visible to readers at once through a single metadata pointer swap

• Schema management: Track the table schema and its evolution history, allowing safe column adds, drops, renames, and reorders

• Partition management: Define how data is partitioned and enable query pruning without exposing the physical layout to users

• Snapshot history: Maintain a history of table states for time travel, rollback, and auditing

• Statistics: Store column-level min/max values and other statistics to enable file skipping during query planning

The data files themselves are still standard Parquet or ORC. The table format adds a metadata layer on top that gives those files the properties of a database table.

The Five Table Formats

Five table formats exist today, each born from a different problem and optimized for a different workload.

Apache Iceberg

Iceberg started at Netflix in 2017, created by Ryan Blue to solve Netflix’s petabyte-scale query planning problems. It uses a three-layer metadata tree: a metadata.json file points to a manifest list, which points to manifest files, which track individual data files with column-level statistics.

Iceberg’s defining feature is its formal specification. Any engine that follows the spec can read and write Iceberg tables correctly. This makes Iceberg the most engine-neutral format. Spark, Trino, Flink, Dremio, Snowflake, BigQuery, Athena, StarRocks, and DuckDB all support it.

Iceberg also introduced hidden partitioning and partition evolution, which are covered in depth in Parts 4 and 5 of this series.

Delta Lake

Delta Lake was created at Databricks in 2019. It stores metadata as a sequential transaction log (_delta_log/) of JSON and Parquet checkpoint files. Each commit appends a new log entry describing which files were added or removed.

Delta Lake’s design prioritizes simplicity within the Spark ecosystem. Its strongest features are Liquid Clustering (adaptive data organization that replaces static partitioning) and UniForm (automatic generation of Iceberg-compatible metadata so other engines can read Delta tables as Iceberg).

Apache Hudi

Hudi originated at Uber in 2016, designed specifically for Change Data Capture (CDC) pipelines that needed to upsert millions of records per hour. Hudi uses a timeline-based metadata architecture where each commit, compaction, and rollback is an “action instant.”

Hudi offers both Copy-on-Write (rewrite entire files on update) and Merge-on-Read (write deltas and merge at read time) table types, plus record-level indexing for fast point lookups. It excels when your primary workload involves frequent row-level updates and deletes.

Apache Paimon

Paimon evolved from Flink Table Store at Alibaba and entered Apache incubation in 2023. It uses LSM-tree based storage internally, making it the most streaming-native table format.

Tables in Paimon are divided into partitions and then further into buckets, each containing an independent LSM tree. This structure enables high-throughput streaming writes with millisecond-level latency. Paimon supports multiple merge engines (deduplication, partial update, aggregation) that determine how records with the same primary key are resolved.

DuckLake

DuckLake is the newest entry, released by DuckDB Labs and MotherDuck in 2025. It takes a fundamentally different approach: instead of storing metadata as files in object storage, DuckLake stores all metadata in a standard SQL database (PostgreSQL, MySQL, SQLite, or DuckDB itself).

This means a single SQL query resolves all metadata (schema, file list, statistics) instead of the multiple HTTP requests required by file-based metadata formats. The tradeoff is a dependency on a running database for the metadata layer and currently limited engine support (primarily DuckDB).

Where Each Format Excels

The key insight: each format reflects the priorities of the team that built it. Netflix needed multi-engine reads at petabyte scale (Iceberg). Uber needed high-frequency upserts (Hudi). Alibaba needed real-time streaming from Flink (Paimon). Databricks needed Spark-optimized simplicity (Delta). DuckDB Labs wanted SQL-native metadata management (DuckLake).

Why Iceberg Has Become the Default

Iceberg has achieved the broadest adoption for three reasons:

1. Specification-first design. Iceberg’s spec is independent of any engine or vendor. Any team can build a conforming implementation. This created a network effect: more engine support attracted more users, which attracted more engine support.

2. No engine dependency. Unlike Delta Lake’s historical Spark dependency or Paimon’s Flink focus, Iceberg was designed from day one to work across engines. A table written by Spark can be read by Dremio, Trino, Flink, or Snowflake without conversion.

3. Industry convergence. Snowflake, AWS (Athena, EMR), Google (BigQuery), and Databricks (via UniForm) have all adopted Iceberg as an interoperability standard. When the major cloud vendors align on a format, it becomes the safe choice for long-term investments.

That said, Iceberg is not universally superior. Hudi’s record-level indexing makes it faster for point lookups on upsert-heavy tables. Paimon’s LSM-tree architecture handles continuous streaming ingestion with lower latency than Iceberg’s batch-oriented commit model. DuckLake’s SQL-based metadata is simpler for single-engine, local-first analytics.

The rest of this series focuses on Iceberg because its design decisions and capabilities represent the state of the art for multi-engine analytical lakehouses. Part 2 examines the metadata structures of all five formats in detail.

Books to Go Deeper

To learn more about Apache Iceberg and the lakehouse architecture, check out these resources:

• Architecting the Apache Iceberg Lakehouse by Alex Merced (Manning)

• Lakehouses with Apache Iceberg: Agentic Hands-on by Alex Merced

• Constructing Context: Semantics, Agents, and Embeddings by Alex Merced

• Apache Iceberg & Agentic AI: Connecting Structured Data by Alex Merced

• Open Source Lakehouse: Architecting Analytical Systems by Alex Merced

Free Resources

• FREE - Apache Iceberg: The Definitive Guide

• FREE - Apache Polaris: The Definitive Guide

• FREE - Agentic AI for Dummies

• FREE - Leverage Federation, The Semantic Layer and the Lakehouse for Agentic AI

• FREE with Survey - Understanding and Getting Hands-on with Apache Iceberg in 100 Pages

If you manage a modern data stack, you likely spend the majority of your time and compute budget moving data around. You pull data from an operational database, stage it in object storage, transform it, load it into a data warehouse, and finally extract it into BI extracts. This DIY approach creates fragile pipelines, delayed insights, and vendor lock-in.

Dremio exists to eliminate this complexity. As a mature platform with 11 years of engineering development behind it, it is a unified analytics solution that allows you to query data where it lives, govern it securely, and interact with it using built-in Agentic AI.

To understand what Dremio does, you must view it as a three-part platform: a Federated Query Engine, an Iceberg Lakehouse Platform, and an Agentic AI Layer.

Pillar 1: The Federated Query Engine

At its core, Dremio is an execution engine built on the principle of “Query, Don’t Move.”

Instead of forcing you to centralize all your data into a single proprietary warehouse, Dremio acts as a logical abstraction layer. When a user or BI dashboard submits a SQL query, Dremio parses the request, identifies the underlying data sources, and generates optimized sub-queries. It pushes down filters and aggregations to the source systems, retrieves the minimal necessary data, and executes the final joins in memory.

This architecture eliminates the serialization tax and allows for Zero-Copy Data Movement. While many other platforms have historically struggled to scale query federation, Dremio is able to scale it effortlessly. This is because of Apache Arrow’s high-speed in-memory columnar execution, Dremio’s intelligent pushdowns, and Iceberg-based Reflections. These features give Dremio a massive performance advantage over other query federation tools that do not leverage them. You bypass complex, multi-stage ETL pipelines entirely while maintaining interactive analytics speed.

Pillar 2: The Iceberg Lakehouse Platform

While federation is a great starting place to operationalize your data analytics rapidly, you ideally want the majority of your analytics to operate directly from your data lake using Apache Iceberg tables. Shifting workloads to Iceberg provides three major advantages:

1. Reduction in costs: You rely on cheaper object storage (like Amazon S3, ADLS, or Google Cloud Storage) while eliminating the need for duplicative storage and expensive ETL pipelines.

2. Tool interoperability: Open standards ensure better collaboration between teams, allowing data engineers, analysts, and data scientists to interact with the exact same data using different compute engines.

3. Autonomous performance management: Dremio automatically optimizes your Iceberg tables and accelerates their performance with background Reflections. This makes a lakehouse feel as fast and easy to use as a traditional warehouse, but without the premium costs.

By natively supporting Apache Parquet and Apache Iceberg, Dremio brings relational database capabilities (like ACID transactions, schema evolution, and time travel) directly to your object storage.

To manage this open ecosystem securely, Dremio integrates tightly with Apache Polaris. Polaris serves as a neutral, open catalog that provides centralized governance, role-based access control (RBAC), and credential vending. It ensures that whether you query data using Dremio, Apache Spark, or Apache Flink, every engine respects the same security policies.

However, querying raw files on object storage can occasionally bottleneck at large scales. Dremio solves this with Autonomous Reflections. Instead of relying on data engineers to manually build and maintain materialized views or OLAP cubes, Dremio monitors query patterns and automatically materializes optimized data structures in the background. When a user runs a query, the engine transparently routes it to the Reflection, delivering sub-second BI performance directly on the lakehouse.

Pillar 3: The Agentic AI Layer

A fast query engine is useless if users cannot find or understand the data. Dremio bridges this gap by integrating artificial intelligence deeply into the platform.

The foundation of this layer is the AI-powered semantic layer. It maps raw tables and columns into clean, business-friendly concepts through SQL Views, tags, wikis, lineage and a knowledge graph with built-in semantic search capabilities to leverage it. This governed semantic layer ensures that both human analysts and AI agents interpret the data identically.

For human users, Dremio includes a built-in AI Agent. Users simply type a natural language request, such as “Show top customers by revenue,” and the agent instantly translates it into a highly optimized SQL query based on the context embedded in the semantic layer. But it goes beyond just translation (the agent immediately executes the query and can automatically generates interactive data visualizations or insightsbased on the results).

For system automation, Dremio provides a Model Context Protocol (MCP) Server. The Dremio MCP Server allows external AI assistants and local IDEs to securely connect to the lakehouse with already built in ability to leverage Dremio’s semantic layer. The server registers tools for semantic discovery and query execution, enabling AI agents to autonomously research and analyze data on your behalf.

Finally, Dremio brings Generative AI directly into your data pipelines through Native AI SQL Functions. Functions like AI_COMPLETE, AI_GENERATE, and AI_CLASSIFY allow you to process unstructured data directly within a SELECT statement. You can extract structured fields from raw PDF blobs or classify customer sentiment without ever moving the data to an external machine learning service.

Conclusion

Dremio is not a traditional data warehouse. It is a unified platform that eliminates data silos through a federated query engine, secures your object storage with an Iceberg-based lakehouse, and accelerates insights with an Agentic AI layer.

By building on open standards like Apache Iceberg, Apache Parquet, Apache Arrow, and Apache Polaris, you maintain full control of your data. You achieve interactive BI performance without vendor lock-in.

Ready to build your open data architecture? Take the next step:

• Try the free trial

• Learn more about Dremio at a workshop or webinar (Events and Workshops)

• Download free books:

  • FREE - Apache Iceberg: The Definitive Guide

  • FREE - Apache Polaris: The Definitive Guide

  • FREE - Agentic AI for Dummies

  • FREE - Leverage Federation, The Semantic Layer and the Lakehouse for Agentic AI

  • FREE with Survey - Understanding and Getting Hands-on with Apache Iceberg in 100 Pages

  • FREE - The Apache Iceberg Digest: Vol1

Ask three teams in your company how they calculate “revenue” and you’ll get three answers. Sales counts bookings. Finance counts recognized revenue. Marketing counts pipeline value. All three call it “revenue.” All three get different numbers. Nobody knows which one is right.

This is the problem a semantic layer solves.

What a Semantic Layer Actually Is

A semantic layer is a logical abstraction between your raw data and the people (or AI agents) querying it. It maps technical database objects — tables, columns, join paths — to business-friendly terms like “Revenue,” “Active Customer,” or “Churn Rate.”

It’s not a database. It doesn’t store data. It’s a layer of definitions, calculations, and context that ensures every query against your data produces consistent results, regardless of which tool or person runs it.

The concept isn’t new. Business Objects introduced “universes” in the 1990s — metadata models that let users drag and drop business concepts instead of writing SQL. What’s changed is scope. Modern semantic layers are universal (not tied to one BI tool), AI-aware (they provide context to language models), and governance-integrated (they enforce access policies alongside definitions).

What a Semantic Layer Contains

A complete semantic layer includes six components:

ComponentWhat It DoesVirtual datasets (Views)SQL-defined business logic applied once and reused everywhereMetric definitionsCanonical calculations for KPIs (e.g., MRR = SUM of active subscription revenue)DocumentationHuman- and machine-readable descriptions of tables, columns, and relationshipsLabels and tagsCategorization for governance (PII, Finance) and discoveryJoin relationshipsPre-defined join paths so users don’t need to know foreign keysAccess policiesRow-level security and column masking enforced at the layer

The key insight: these components serve both human analysts and AI agents. When an AI generates SQL from a natural language question, it consults this same layer to understand what “revenue” means, which tables to join, and which columns to filter.

How It Works in Practice

Here’s what happens when someone queries data through a semantic layer:

1. A user (or AI agent) asks: “What was revenue by region last quarter?”

2. The semantic layer translates:

   • “Revenue” → SUM(orders.total) WHERE orders.status = 'completed'

   • “Region” → customers.region

   • “Last quarter” → WHERE order_date BETWEEN '2025-10-01' AND '2025-12-31'

3. The query engine generates optimized SQL against the underlying data sources

4. Results are returned using business terms, not raw column names

The user never writes SQL. The AI never guesses at column names. The metric definition is applied identically whether the query runs in a dashboard, a Python notebook, or a chat interface.

Why It Matters Now More Than Ever

Three trends are making semantic layers essential, not optional.

AI agents need business context. Large language models generating SQL will hallucinate column names, use incorrect aggregation logic, and join tables wrong unless they have explicit definitions to work from. A semantic layer provides that grounding. This is why platforms like Dremio embed a semantic layer directly into the query engine — it’s the context that makes the AI accurate instead of confidently wrong.

Self-service analytics demands accessibility. Business users want to query data without filing a ticket. Exposing raw database schemas to non-technical users creates more problems than it solves. A semantic layer presents data in terms people already understand.

Governance requires centralized definitions. GDPR, CCPA, and industry regulations require organizations to know what data they have, who can access it, and how it’s used. A semantic layer centralizes these definitions and enforces access policies in one place instead of across dozens of tools.

Common Misconceptions

“It’s just a data catalog.” A data catalog is an inventory — it tells you what data exists. A semantic layer defines what data means and how to calculate it. You need both. They’re complementary, not interchangeable. (See: Semantic Layer vs. Data Catalog)

“It’s just a BI tool feature.” Some BI tools include semantic models (Looker’s LookML, Power BI’s datasets). But these are tool-specific. If your organization uses three BI tools, you maintain three separate semantic models. A universal semantic layer defines metrics once and serves them to every tool.

“It adds a performance penalty.” Modern semantic layers don’t just translate queries — they optimize them. Dremio, for example, uses Reflections (pre-computed, physically optimized data copies) to accelerate queries that pass through its semantic layer. The result is often faster than querying raw tables directly.

What to Do Next

Pick your organization’s five most important metrics. Ask two different teams how each one is calculated. If the answers don’t match, that’s your signal. You don’t have a semantic layer problem — you have a trust problem, and a semantic layer is how you fix it.

Try Dremio Cloud free for 30 days

The term “semantic layer” has been part of the data industry’s vocabulary for over 35 years. It first appeared in a 1991 patent filing by Business Objects, and it has since been reinvented, abandoned, and reinvented again across three distinct eras of data architecture. Today, it sits at the center of one of the most consequential design debates in the industry: should the semantic layer be a standalone product you bolt onto your stack, or a native capability of the platform that already manages your data?

This guide covers the full arc: what a semantic layer is, where it came from, how it split into two competing architectural approaches, and why the choice between them determines whether your AI agents produce accurate answers or hallucinated nonsense.

What a Semantic Layer Actually Is

A semantic layer is an abstraction that maps the physical structure of your data (table names, column names, join logic, filter conditions) to the business terms that people actually use (revenue, churn rate, active customer, cost per acquisition). It sits between the raw data and every consumer of that data: BI dashboards, AI agents, Python notebooks, Excel spreadsheets, and custom applications.

The semantic layer has three core responsibilities:

Metric consistency. When the finance team says “revenue,” they mean recognized revenue net of refunds. When the sales team says “revenue,” they mean bookings including pending deals. Without a semantic layer, both teams write their own SQL, get different numbers, and spend the next two weeks arguing about which dashboard is right. A semantic layer defines “revenue” once, in one place, and every downstream consumer uses that definition.

Access governance. The semantic layer controls who sees what. A marketing analyst querying customer data should not see Social Security numbers. A regional manager should only see data for their region. These rules (row-level security, column masking, role-based access) are defined at the semantic layer and enforced consistently regardless of which tool is doing the querying.

Query abstraction. Business users and AI agents should not need to know that “customer churn rate” requires joining three tables, filtering out test accounts, calculating a 90-day rolling window, and dividing by the active customer count from the prior period. The semantic layer encapsulates that logic in a reusable definition. Consumers ask for “churn rate” and get the right answer.

The Origin Story: Business Objects, 1991

The semantic layer was invented to solve a simple problem: business users could not write SQL.

In 1991, Business Objects filed a patent for a “relational database access system using semantically dynamic objects.” The product feature was called “Universes.” It worked like this: a data architect would build a metadata model that mapped physical database tables and join paths into business-friendly objects (”Customer,” “Product,” “Sales Amount”). Report builders could then drag and drop these objects to create queries without touching SQL.

This was a significant advance. Before Universes, generating a report from a relational database required either a developer who understood the schema or a business user willing to learn SQL. Business Objects eliminated that requirement entirely.

IBM’s Cognos followed with “Framework Manager,” which served the same purpose: map the physical database into a logical, business-friendly model. SAP built InfoProviders and BEx queries on top of SAP BW. Microsoft introduced SQL Server Analysis Services.

Every major enterprise BI vendor in the 1990s built some version of a semantic layer. But they all shared the same fundamental limitation: the semantic layer was proprietary and locked to a single vendor’s BI tool. If you built your metrics in Business Objects Universes, those definitions did not carry over to Cognos. If you modeled your data in SSAS, Tableau could not read it. The semantic layer existed, but it was a walled garden.

OLAP Cubes: The Implicit Semantic Layer

Running parallel to the relational semantic layer was the OLAP (Online Analytical Processing) cube. Products like SQL Server Analysis Services, Cognos TM1, and Oracle Essbase pre-computed data into multidimensional structures: dimensions (Customer, Product, Time), measures (Revenue, Quantity, Profit), and hierarchies (Year > Quarter > Month > Day).

The cube itself functioned as a semantic layer. Business users did not query tables; they navigated dimensions. They did not write SQL; they used MDX (Multidimensional Expressions) or simply clicked through pivot-table interfaces. The business logic was baked into the cube’s structure.

OLAP cubes worked well for their era. Pre-computing aggregations meant that analytical queries returned in seconds, even on the hardware of the early 2000s. But they had three fatal weaknesses:

1. Rigidity. Adding a new dimension or changing a hierarchy required rebuilding the cube, which could take hours for large datasets. Business requirements change faster than cubes can be rebuilt.

2. Cost. Cubes stored pre-aggregated copies of data. For large organizations, this meant maintaining terabytes of redundant, pre-computed data on expensive storage.

3. Specialization. Operating an OLAP cube required specialized skills (MDX, cube design, aggregation strategies) that most data teams did not have.

As cloud data warehouses like Snowflake, BigQuery, and Redshift made raw compute cheap and fast, the need for pre-aggregation declined. You could run the analytical query directly against the detail data and get results in seconds. The cube’s primary value proposition, speed through pre-computation, was no longer unique.

The Self-Service Era and the Loss of the Semantic Layer

The 2010s brought a dramatic shift. Self-service BI tools like Tableau and Power BI connected directly to databases, bypassing the semantic layer entirely. This was marketed as democratization: give every analyst direct access to the data, and they will find their own insights.

For small teams, this worked. For organizations with more than a handful of analysts, it created a problem that the industry calls “metric drift.” Without a centralized semantic layer, each analyst wrote their own SQL. Each SQL query embedded its own business logic. Revenue was calculated five different ways by five different people, and no one could agree on which number was correct.

The first response to metric drift came from Looker, founded in 2012, which introduced LookML as a code-based semantic layer. You defined your metrics, dimensions, and relationships in version-controlled modeling files. This was a meaningful evolution because it separated the semantic logic from the BI tool’s proprietary report format. Google acquired Looker for $2.6 billion in 2019, validating that the semantic layer was worth owning. But LookML was still tied to Looker’s ecosystem. If you used Tableau or Power BI as your primary BI tool, LookML did not help.

The broader industry realization was clear: skipping the semantic layer does not eliminate the need for one. It just distributes the problem across every team and every dashboard, where it becomes harder to find and harder to fix.

Dremio: The Semantic Layer Built Into the Query Engine From Day One

While Looker was coupling the semantic layer to a BI tool, a different approach was emerging. Dremio was founded in 2015 by Tomer Shiran and Jacques Nadeau, creators and contributors to the Apache Drill project. When Dremio publicly launched in July 2017, it introduced what it called a “governed, self-service semantic layer” as a core architectural component, not an add-on.

The key difference: Dremio’s semantic layer was integrated directly into a high-performance query engine. From its first release, the platform shipped with:

• Virtual Datasets (Views). SQL-defined business logic that users could create, share, and layer on top of any connected data source. No data movement required.

• Reflections. Patented, transparent materialized views that the query optimizer substitutes automatically. Users query the governed view; Dremio serves the fastest available Reflection behind the scenes.

• Federated access. The semantic layer worked across data sources (S3, HDFS, relational databases) from the start, not just against a single warehouse.

Dremio added Wikis and Labels (Tags) in subsequent releases, providing Markdown-formatted documentation and classification metadata directly attached to datasets in the catalog. This meant the semantic layer was not just a set of views; it included the context that made those views discoverable and understandable.

This was architecturally distinct from every other semantic layer on the market. AtScale (founded 2013) and Cube (open-sourced 2019) built the semantic layer as a separate product. Dremio built it into the same platform that executed the queries and managed the catalog. That design decision would become increasingly important as AI agents entered the picture.

The Modern Resurgence: Two Divergent Paths

By the early 2020s, the semantic layer was firmly back. dbt Labs acquired Transform (the creators of MetricFlow) in February 2023 to build a code-based metrics layer. Cube had open-sourced its API-first semantic layer in 2019 and launched Cube Cloud commercially in 2021. AtScale had been building its enterprise virtualization layer since 2013.

The market had split into two fundamentally different architectural forms, and the choice between them has significant consequences for how your data platform operates.

Path 1: The semantic layer as a standalone product. Companies like AtScale (2013), Cube (2019), and dbt (MetricFlow, 2023) built the semantic layer as a separate service that sits between your data warehouse and your BI tools. You deploy it as its own infrastructure, manage it as its own system, and integrate it with your existing stack.

Path 2: The semantic layer as a platform feature. Dremio (2017) integrated the semantic layer directly into its query engine and data catalog from the start. There is no separate service to deploy. The semantic layer is a native capability of the same platform that stores, governs, and queries your data.

Both approaches solve the metric consistency problem. They differ in how they solve it, what they require operationally, and how well they extend to AI-driven analytics.

Path 1: The Semantic Layer as a Standalone Product

Three standalone semantic layer products dominate the current market. Each targets a different architecture and team profile.

AtScale (Founded 2013)

AtScale, founded by veterans of the Yahoo data team, positions itself as a “universal semantic layer” for large enterprises. It creates a virtualization layer across multiple data warehouses (Snowflake, BigQuery, Databricks), presenting a unified semantic model to BI tools. Its strongest feature is native MDX and DAX compatibility, which makes it the only option for organizations with heavy Excel and SSAS dependencies.

AtScale excels when you have data spread across multiple warehouses and need a single semantic model that works across all of them. The tradeoff is infrastructure complexity and licensing cost. AtScale requires dedicated infrastructure, and its enterprise pricing model reflects its positioning.

Cube (Open-Sourced 2019)

Cube started as Statsbot in 2016 before pivoting to become an open-source, API-first semantic layer in 2019. It provides REST, GraphQL, SQL, MDX, and DAX APIs, making it the most flexible option for embedded analytics and customer-facing dashboards. Cube Cloud launched commercially in 2021. Cube’s pre-aggregation engine can deliver sub-second responses for complex queries by pre-computing results and caching them.

Cube excels when your primary consumer is a custom application, not a BI tool. The tradeoff is operational overhead: Cube runs as a separate server, requires its own infrastructure, and demands expertise in designing pre-aggregation strategies to achieve optimal performance.

dbt Semantic Layer (MetricFlow, 2023)

The dbt Semantic Layer is powered by MetricFlow, the technology dbt Labs acquired when it purchased Transform in February 2023. It lets teams define metrics as code in YAML files within their existing dbt projects. Metrics are version-controlled, reviewed via pull requests, and deployed alongside your dbt transformations. In late 2025, dbt Labs moved MetricFlow to an Apache 2.0 license, signaling a commitment to open, portable metric definitions.

The dbt Semantic Layer excels when your team is already a dbt shop and wants metrics managed in the same Git-based workflow as transformations. The tradeoff is that it requires dbt Cloud for the serving layer, lacks native caching (relying on the underlying warehouse for query execution), and is less suited for high-concurrency embedded applications.

The Structural Tradeoff of Standalone Products

All three standalone products share the same architectural limitation: they exist as a separate layer of infrastructure that must integrate with your data catalog, your governance system, and your query engine.

This means:

• Another system to operate. You deploy it, monitor it, upgrade it, and debug it.

• Governance is a separate concern. Access policies defined in your catalog or warehouse must be replicated or synced with the semantic layer. Any gap is a security risk.

• No native execution. Standalone semantic layers define metrics but do not execute queries. They translate user requests into SQL and send that SQL to an external engine. If the engine and the semantic layer disagree on the data model, you get wrong results.

• Sync lag. When you change a table schema, add a column, or update governance rules, the semantic layer must be updated separately. Until it is, your definitions are stale.

For teams with a single data warehouse, a strong DevOps practice, and a primary use case that matches one of these products, standalone semantic layers work well. For teams managing federated data across multiple sources, or teams building AI-driven analytics, the gap between “definition” and “execution” creates friction that compounds over time.

Path 2: The Semantic Layer as a Platform Feature

The alternative is to build the semantic layer into the same platform that manages your data catalog, governs access, and executes queries. This is the approach Dremio takes.

In Dremio, the semantic layer is not a separate product you bolt on. It is a native set of capabilities (views, wikis, labels, lineage, knowledge graph) that are integrated with the Open Catalog (built on Apache Polaris), the MPP query engine (built on Apache Arrow), and the governance system (Fine-Grained Access Control, row-level security, column masking).

This matters because the three activities that define a semantic layer, defining metrics, governing access, and executing queries, all happen in the same system. There is no handoff, no sync, no governance gap.

How Dremio’s Semantic Layer Works

Dremio’s AI Semantic Layer is built from five components that work together: views, wikis, labels, lineage, and the knowledge graph.

Views (Virtual Datasets)

Views are the foundation. A view is a SQL-defined virtual dataset that encapsulates business logic: joins, filters, calculations, and transformations. You write the SQL once, and every consumer (BI tool, AI agent, Python notebook) queries the view instead of the raw tables.

Dremio recommends a three-layer architecture for views:

• Preparation Layer. One view per source table. Handles type casting, column renaming, null handling. A direct 1:1 mapping of raw data into clean, standardized form.

• Business Layer. Shared business logic. This is where you define “active customer” (customers with at least one order in the last 90 days, excluding test accounts), “revenue” (order_amount minus refunds, in USD), and every other metric that needs a single definition.

• Application Layer. Tailored datasets for specific consumers. A marketing dashboard view joins customer demographics with campaign performance. An AI agent view exposes the most commonly asked metrics with rich column-level documentation.

Because views are virtual, they do not copy or move data. They execute against the underlying data at query time, using Dremio’s federated query engine to pull from S3, PostgreSQL, Snowflake, MongoDB, or any connected source. Change the underlying data, and the view reflects it immediately.

Wikis

Wikis are Markdown-formatted documentation attached directly to spaces, sources, folders, tables, views, and columns. They serve two audiences: human analysts browsing the catalog, and AI agents generating SQL.

A wiki for a view called analytics.customer_health might contain:

## Customer Health Score

Composite metric combining purchase frequency, support ticket volume,
and NPS survey responses over the trailing 90 days.

**Owner:** Customer Success team
**Refresh:** Updated daily by the ETL pipeline
**Filters:** Excludes test accounts (account_type != 'test')
**Churn definition:** Score below 30 for two consecutive months

Dremio can also auto-generate wiki content. The platform samples table data, analyzes column distributions, and produces context-rich descriptions using generative AI. This is particularly valuable for large data estates where manually documenting hundreds of tables is impractical.

Labels

Labels classify and organize data assets. You tag a table as PII, Finance, Marketing, Approved, or Draft. Labels serve two purposes: they improve discoverability (semantic search returns results filtered by label), and they integrate with governance rules (all PII-labeled columns automatically apply masking policies).

Like wikis, labels can be AI-suggested. Dremio analyzes column names, data patterns, and content to recommend labels like contains-email or likely-PII.

Lineage

Dremio automatically tracks the flow of data from source to view to consumer. You can see which raw tables feed into which business views, and which dashboards or AI queries consume those views.

Lineage is critical for impact analysis. Before changing the schema of a source table, you can trace all downstream dependencies and understand exactly what will break. Without automated lineage, this analysis requires manually reading SQL definitions and hoping you did not miss one.

Knowledge Graph

The knowledge graph is the newest addition to Dremio’s semantic layer. It operates at a higher level than individual wikis and labels, building a connected graph of entity relationships, metric definitions, and usage patterns.

The knowledge graph works in three ways:

1. Pattern detection. It analyzes query patterns across your organization to detect implicit definitions. If 80% of queries that reference “active customers” use the same WHERE clause (last_order_date > CURRENT_DATE - INTERVAL '90' DAY AND account_type != 'test'), the knowledge graph surfaces that pattern as a candidate definition.

2. User-defined context. You can provide business context as structured markdown files. These files define entities, relationships, and business rules that the knowledge graph ingests and makes available to AI agents.

3. Relationship mapping. The knowledge graph connects related entities (customers are related to orders, orders contain products, products belong to categories) and exposes those relationships to AI agents, enabling more accurate multi-table SQL generation.

Semantic Search

Semantic search lets users and AI agents discover data assets using natural language. Instead of browsing a schema tree looking for a table called dwh_fact_cust_ord_line_item, you search for “customer orders by product category” and find the relevant view, complete with its wiki documentation and labels.

Semantic search indexes wikis, labels, column names, table descriptions, and view definitions. It is the entry point for both human exploration and AI agent data discovery.

Why the Integrated Approach Changes Everything for AI

The reason the platform-versus-product distinction matters more now than it did five years ago is AI. Specifically, AI agents that generate SQL from natural language questions.

An AI agent that receives the question “What was our customer churn rate by region last quarter?” needs three things to produce an accurate answer:

1. Context. What does “churn rate” mean in this organization? What table contains the data? Which columns are relevant? What filters should be applied? The semantic layer’s wikis, labels, views, and knowledge graph provide this context.

2. Access. Can this user see the churn data? Are there row-level filters based on their role? Are any columns masked? The governance system enforces these rules.

3. Execution speed. The user expects an answer in seconds, not minutes. The query engine needs to be fast enough for interactive use.

In a standalone semantic layer architecture, these three capabilities live in three different systems: the semantic layer product provides context, the data catalog or warehouse provides governance, and a separate query engine provides execution. The AI agent must coordinate across all three, and any mismatch between them produces wrong answers, security violations, or slow responses.

In Dremio’s architecture, all three are co-located. The AI Agent reads the wikis, labels, and view definitions from the semantic layer, generates SQL that respects governance rules, and executes the query on the built-in MPP engine. The entire loop happens within a single governed platform.

Dremio’s MCP Server extends this to external AI tools. ChatGPT, Claude Desktop, or any custom agent that supports the Model Context Protocol can connect to Dremio and query through the same governed semantic layer. The external AI agent receives the same business context, respects the same governance rules, and gets the same fast query execution as the built-in AI Agent.

The semantic layer teaches the AI your business language so it generates the right SQL, not generic SQL.

Platform vs Product: A Side-by-Side Comparison

When a Standalone Product Fits

• You use a single data warehouse (Snowflake, BigQuery) and your semantic layer needs are limited to consistent BI metrics

• Your team is already deeply invested in dbt and wants metrics alongside transformations

• You are building customer-facing embedded analytics and need Cube’s pre-aggregation performance

• You have heavy Excel/MDX requirements that only AtScale supports

When the Platform Approach Fits

• Your data lives across multiple sources (S3, PostgreSQL, Snowflake, MongoDB) and you need federated access

• You want governance rules defined once and enforced everywhere, including for AI agents

• You are building or planning AI-driven analytics (AI Agent, MCP, natural language querying)

• You want to eliminate the operational overhead of managing a separate semantic layer product

• You need the semantic layer, the catalog, and the query engine to operate as a single governed system

Building Your Semantic Layer: A Practical Framework

If you are starting from scratch or migrating from an ad-hoc metric landscape, here is a practical sequence:

Step 1: Identify your top 10 metrics. Not all metrics need to be in the semantic layer on day one. Start with the metrics that cause the most confusion: revenue, churn, active users, cost per acquisition, NPS. These are the metrics where two teams have two different SQL queries and two different numbers.

Step 2: Build the layered view architecture. For each metric, create the three-layer view stack in Dremio. Preparation views clean the source data. Business views encode the agreed-upon logic. Application views tailor the output for specific consumers.

Step 3: Add wikis and labels. Document each view and its columns. Define what the metric means, who owns it, how it is calculated, and what filters are applied. Tag columns with labels like PII, Finance, or Approved.

Step 4: Configure governance. Apply Fine-Grained Access Control: row-level security for multi-tenant data, column masking for sensitive fields, role-based access for views. These rules are enforced at query time for every consumer, including AI agents.

Step 5: Connect AI interfaces. Enable the Dremio AI Agent for your team. Set up the MCP Server for external AI tools. The wikis and labels you added in Step 3 become the context that makes AI-generated SQL accurate.

Step 6: Expand. Add the next 10 metrics. Build knowledge graph definitions for complex entity relationships. Let autonomous Reflections learn from query patterns and accelerate the most common queries automatically.

The semantic layer is not a one-time project. It is a living system that grows with your organization’s data needs. Start small, prove value on the metrics that matter most, and expand from there.

Try Dremio Cloud free for 30 days to build your semantic layer on top of your existing data sources with zero data movement and native AI agent support.

Free Resources to Go Deeper

• FREE - Agentic AI for Dummies

• FREE - Leverage Federation, The Semantic Layer and the Lakehouse for Agentic AI

Three weeks past the Iceberg Summit, the lakehouse projects shifted from in-person alignment back into shipping mode. Polaris cut its 1.4.0 release and immediately followed up with a Python CLI 1.4.0, Arrow shipped its 24.0.0 major release and kicked off an arrow-rs 58.2.0 vote, and Parquet’s design lists stayed dense with proposals on footers, page encoding, and a new java release discussion. Iceberg’s dev list was quieter this week as contributors digested summit follow-ups and continued narrowing on V4 design questions in the background.

Apache Iceberg

The post-summit wave of formal proposals continued translating into design work this week. The V4 metadata.json optionality direction that has anchored multiple syncs — treating catalog-managed metadata as a first-class supported mode while keeping static-table portability through explicit opt-in semantics — is still the defining V4 design conversation, with Anton Okolnychyi, Yufei Gu, Shawn Chang, and Steven Wu continuing to push edge cases on portability and Spark driver behavior. The single-file commits proposal that Russell Spitzer and Amogh Jahagirdar have been advancing remains on track for a formal write-up, with the latency and metadata footprint reductions driving urgency.

Péter Váry’s efficient column updates proposal for wide tables continued attracting collaboration. The design — write only the columns that change on each commit, then stitch the result at read time — is squarely aimed at petabyte-scale feature stores with thousands of embedding and model-score columns, and the I/O savings make it one of the more practically grounded V4 proposals on the list. Anurag Mantripragada and Gábor Kaszab are working alongside Péter on POC benchmarks to support the formal proposal that should land on the dev list in the coming weeks.

On the Rust side, the Iceberg Rust 0.9.0 release shipped earlier this development cycle and continues to anchor downstream adoption discussions, with its DataFusion integration making it a serious option for teams that want Iceberg without a JVM dependency. Iceberg Summit 2026 session recordings are also rolling out on the project’s YouTube channel this week, giving the global community access to the V4 design talks, the vendor panel, and the production case studies from Apple, Bloomberg, Pinterest, and others. The AI contribution policy that Holden Karau, Kevin Liu, Steve Loughran, and Sung Yun pushed through March is still expected to land as published guidance covering disclosure requirements and code provenance standards.

Apache Polaris

Polaris had its biggest release week of the year. Adnan Hemani announced Apache Polaris 1.4.0 on April 23, the project’s first major release as a graduated top-level project. Dmitri Bourlatchkov, Yufei Gu, Xi Wen, and Alexandre Dutra all weighed in with congratulations and follow-up notes on packaging and distribution. Right behind it, Adnan kicked off and shepherded the Apache Polaris Python CLI 1.4.0 RC2 vote, which collected binding +1s from Yufei Gu, Honah J., and Jean-Baptiste Onofré, with Yong Zheng adding non-binding support. The Python CLI 1.4.0 release shipped on April 28, completing the back-to-back release pair. Jean-Baptiste also confirmed in a HEADS UP note that the project is now back on a monthly release cadence after the graduation transition.

The release had its share of post-launch fires. Alexandre Dutra opened threads on Helm chart repo inconsistency after the 1.4.0 release, a release workflow failure in step 4, and an Artifact Hub request for official status. A GitHub thread on KMS-related errors after bumping to 1.4.0 surfaced a real upgrade bug that drew immediate attention. Yufei Gu took the lead on triaging most of these, and the discussions are doing exactly what a healthy post-release cycle should — surfacing rough edges before they reach more users.

Design discussions stayed active alongside the release work. EJ Wang’s DISCUSS thread on AGENTS.md for Polaris opened a conversation about adding agent-readable repository metadata, picking up engagement from Yufei Gu. Yufei separately started a discussion on narrowing the scope of SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION, which Dmitri Bourlatchkov and Dennis Huo dug into. ITing Lee’s proposal to add OpenLineage to Polaris continued attracting feedback from Adnan Hemani, Jean-Baptiste Onofré, Yufei Gu, and Michael Collado. Alexandre Dutra’s URL path decoding thread and his PolarisPrivilege fields and grant validation discussion both kept multiple contributors engaged through the week, and Selvamohan Neethiraj raised a PolarisPrincipal user attributes server-side bug that Alexandre and Yufei traced through.

Apache Arrow

Arrow had its own back-to-back release week. Raúl Cumplido announced Apache Arrow 24.0.0 on April 22, closing out the 24.0.0 RC0 vote that spanned mid-April. Matt Topol followed with the Apache Arrow Go 18.6.0 RC0 vote on April 22 and announced the release result on April 28, with Pedro Matias, Ian Cook, David Li, and Bryce Mecum carrying the verification work. Andrew Lamb then opened the arrow-rs 58.2.0 RC1 vote on April 28, with Bryce Mecum, Ed Seidl, Jeffrey Vo, and Raúl Cumplido moving quickly through verification — finishing what last week’s newsletter flagged as the next ship to watch.

Beyond releases, the design conversations stayed lively. Emil Sadek opened a DISCUSS thread on an ADBC Logo Proposal with Nic Crane, Julian Hyde, and Rusty Conover weighing in on visual identity for the database connectivity standard. Benjamin Philip kicked off a new DISCUSS thread on Arrow Erlang’s grant documents, continuing the project’s expansion into more language ecosystems. The pyarrow-stubs donation vote that Rok Mihevc opened on April 14 stayed active, drawing additional support this week with Rok pushing for a final tally. Mandukhai Alimaa’s earlier proposal for a canonical BigDecimal extension type and Andrew Lamb’s arrow-rs security policy discussion both continued generating engagement as the project tightens its production posture.

Apache Parquet

Parquet’s lists were as dense as any project’s this week. Ismaël Mejía opened a thread soliciting code reviews for Java performance optimization work, with Steve Loughran picking it up immediately. Manu Zhang’s DISCUSS thread on a new parquet-java release drew sustained engagement from Steve Loughran, Aaron Niskode-Dossett, Fokko Driesprong, Julien Le Dem, Gang Wu, and Rahil C — covering both the timing question and what should ship in the next release. Julien Le Dem’s Parquet sync on April 22 drew Manu Zhang and Micah Kornfield into the agenda discussion.

The format-level proposals continued to evolve. Will Edwards’s DISCUSS thread on an alternative to the FlatBuffer footer with a lightweight byte-offset index kept pulling in design feedback from Andrew Lamb, Ed Seidl, Jan Finis, Alkis Evlogimenos, Raphael Taylor-Davies, Andrew Bell, and others. Ed Seidl’s proposal to make path_in_schema optional attracted commentary from Gang Wu, Steve Loughran, and Micah Kornfield. Andrew Lamb’s thread on where VariantJsonParser should live — touching the boundary between Parquet and Iceberg’s variant tooling — continued with Steve Loughran and Gang Wu. Jan Finis’s question on whether a too-long RLE bitpack at the end of a page is valid drew careful answers from Raphael Taylor-Davies and Micah Kornfield, the kind of spec-edge clarification that matters for cross-implementation interop. Milan Stefanovic’s Geospatial CRS string format clarification continued threading toward closure with Dewey Dunnington and Micah Kornfield.

Cross-Project Themes

This week’s clearest pattern is post-graduation Polaris finding its operational rhythm. The 1.4.0 release plus the Python CLI 1.4.0, the return to a monthly cadence, and the visible upgrade-path bugs and Helm packaging issues are all the work of a project growing into its TLP independence. The fact that contributors are surfacing problems publicly and triaging them on the dev list — rather than routing through a parent project — is itself the marker of a healthy graduation.

The release wave across projects also reflects how synchronized the lakehouse stack has become. Arrow 24.0.0 plus arrow-rs 58.2.0 plus arrow-go 18.6.0 plus Polaris 1.4.0 plus Polaris Python CLI 1.4.0 all landing within a single week is a coordination story. Engines and tools downstream of these libraries — Spark, Trino, Dremio, DataFusion, DuckDB, Snowflake — can pick up the new versions in a coherent batch rather than chasing staggered upgrades across half a dozen vendors. The format-level design work in Parquet (footers, optional path_in_schema, variant tooling location) and the V4 design work in Iceberg (metadata.json optionality, single-file commits, efficient column updates) are also starting to rhyme: both communities are picking apart assumptions baked into v1 and v2 spec design and asking what a leaner, AI-workload-aware format looks like.

Looking Ahead

Watch the arrow-rs 58.2.0 RC vote close out in the coming days. Polaris should publish 1.4.1 or move toward 1.5.0 planning given the monthly cadence commitment, and the AGENTS.md discussion is likely to firm into a concrete proposal. The Polaris OpenLineage RFC has the volume of feedback it needs to move toward implementation. On the Iceberg side, the formal V4 single-file commits write-up and the published AI contribution policy remain the next concrete deliverables to track. Iceberg Summit 2026 talk recordings will continue rolling out on YouTube, and the parquet-java release discussion should converge on a target version.

Resources & Further Learning

Get Started with Dremio

• Try Dremio Free — Build your lakehouse on Iceberg with a free trial

• Build a Lakehouse with Iceberg, Parquet, Polaris & Arrow — Learn how Dremio brings the open lakehouse stack together

Free Downloads

• Apache Iceberg: The Definitive Guide — O’Reilly book, free download

• Apache Polaris: The Definitive Guide — O’Reilly book, free download

Books by Alex Merced

• Architecting an Apache Iceberg Lakehouse

• Enabling Agentic Analytics with Apache Iceberg and Dremio

• The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI

• The Book on Using Apache Iceberg with Python

Week of April 23–29, 2026

This week, Google split its eighth-generation TPU into two specialized chips. SpaceX disclosed rights to acquire Cursor for $60 billion. Google Cloud Next 2026 framed enterprise software around autonomous agents, and the Model Context Protocol moved deeper into production-grade territory.

AI Coding Tools: SpaceX Eyes Cursor at $60B and Google Pushes Agent Platforms

SpaceX announced on April 22 that it has rights to buy AI coding tool Cursor for $60 billion later this year, with an alternative $10 billion partnership option. The move positions Elon Musk’s space and AI properties to compete with Anthropic and OpenAI ahead of a planned Wall Street debut. Cursor, made by San Francisco startup Anysphere, has wide distribution to expert software engineers, which is part of what makes it attractive to Musk’s company. Read the AP report.

Google Cloud Next 2026 ran April 22–24 in Las Vegas and made coding agents the centerpiece. Google rebranded its AI platform as the Gemini Enterprise Agent Platform, billed as a one-stop shop for autonomous agents with 200+ foundation models and enterprise governance. The platform supports a new Agents CLI that takes agents from creation to production through a single command-line tool. See the announcements.

Cursor 2.0 also gained attention this month for supporting up to eight parallel AI agents working on different sections of a codebase at the same time. Claude Code, meanwhile, now powers GitHub Copilot’s enterprise tier with multi-agent coordination that splits large tasks into parallel subtasks. The category leaders are converging on the same pattern: agents that read codebases, plan changes across multiple files, write the code, and run the tests.

AI Processing: Google Splits Its TPU Into Training and Inference Chips

Google Cloud announced on April 22 that its eighth-generation TPU is splitting into two specialized chips. The TPU 8t targets model training and the TPU 8i targets inference. Google reports up to 3x faster AI model training and 80% better performance per dollar over the previous generation, with the ability to link more than 1 million TPUs in a single cluster. Read the TechCrunch coverage.

Google also confirmed a partnership with Nvidia to extend Falcon, the software-based networking technology Google created and open-sourced in 2023 under the Open Compute Project. The work aims to make Nvidia-based systems perform better inside Google Cloud, a notable detente given Google’s TPU sales push.

The Nvidia chip rival market is also booming. AI chip startups raised $8.3 billion globally in 2026, according to Dealroom, with Cerebras Systems pulling in $1 billion in February and $500 million rounds going to MatX, Ayar Labs, and Etched. European companies like Axelera and Olix raised rounds north of $200 million. The argument: GPUs were not purpose-designed for AI inference, and novel system architectures bring big savings in energy and cost. See the CNBC report.

Standards & Protocols: MCP Hits Production Scale and Agentic Foundations Mature

The Model Context Protocol crossed a clear adoption threshold this month. MCP downloads now run at roughly 110 million per month across OpenAI, Google, LangChain, and other frameworks, according to a recent Anthropic keynote on the protocol’s direction. By Q2 2026, community-built MCP servers exist for GitHub, Slack, PostgreSQL, Stripe, Figma, Docker, Kubernetes, and over 200 other tools. See the Wikipedia summary.

The 2026 MCP roadmap published in March identified four priorities. First is streamable HTTP transport scalability. Second is the Tasks primitive lifecycle, including retry semantics and expiry policies. Third is governance maturation. Fourth is enterprise readiness covering audit trails, SSO-integrated auth, gateway behavior, and configuration portability. Stateful sessions fight with load balancers and horizontal scaling needs better support, so the working groups are evolving the existing transport rather than adding new ones. Read the roadmap.

Google Cloud Next 2026 also gave standards work a public showcase. A breakout session covered “Generative UI for any agent, anywhere: A2UI, AG-UI, MCP Apps, and more.” Interoperability between agent UI standards is now part of mainstream cloud roadmaps.

The Agentic AI Foundation launched in December 2025 under the Linux Foundation. Founding contributions came from Anthropic’s MCP, OpenAI’s AGENTS.md, and Block’s Goose. AAIF held its first MCP Dev Summit North America in New York earlier this month, drawing about 1,200 attendees, double the prior event. The next AAIF events are AGNTCon + MCPCon Europe on September 17–18 in Amsterdam and AGNTCon + MCPCon North America on October 22–23 in San Jose.

Resources to Go Further

The AI landscape changes fast. Here are tools and resources to help you keep pace.

Try Dremio Free — Experience agentic analytics and an Apache Iceberg-powered lakehouse. Start your free trial

Learn Agentic AI with Data — Dremio’s agentic analytics features let your AI agents query and act on live data. Explore Dremio Agentic AI

Join the Community — Connect with data engineers and AI practitioners building on open standards. Join the Dremio Developer Community

Book: The 2026 Guide to AI-Assisted Development — Covers prompt engineering, agent workflows, MCP, evaluation, security, and career paths. Get it on Amazon

Book: Using AI Agents for Data Engineering and Data Analysis — A practical guide to Claude Code, Google Antigravity, OpenAI Codex, and more. Get it on Amazon

A bad data model doesn’t announce itself. It hides behind slow dashboards, conflicting numbers, confused analysts, and AI agents that generate wrong SQL. By the time someone identifies the model as the root cause, the team has already built dozens of reports on top of it.

Here are seven modeling mistakes that create downstream pain — and how to avoid each one.

Mistake 1: No Defined Grain

The grain declares what one row in a fact table represents. “One row per order line item.” “One row per daily user session.” “One row per monthly account balance.”

Without a declared grain, aggregation produces wrong numbers. If some rows represent individual transactions and others represent daily summaries, a SUM query double-counts or under-counts depending on the mix.

Fix: Before designing any fact table, write down the grain in one sentence. Share it with your team. If you can’t state the grain clearly, the table isn’t ready for production.

Mistake 2: Cryptic Naming

Columns named c1, dt, amt, flg, and cat_cd save keystrokes during development but cost hours during analysis. Every analyst who encounters these names must either read the ETL code, ask the engineer, or guess.

AI agents have the same problem. An agent asked to calculate “total revenue” can’t identify the right column if it’s called amt3 instead of revenue_usd.

Fix: Use descriptive, business-friendly names. customer_name, order_date, revenue_usd, is_active, product_category. Include units where ambiguous (weight_kg, duration_minutes). Use snake_case consistently.

Mistake 3: Skipping the Conceptual Model

Going straight from a stakeholder request to CREATE TABLE skips the alignment step. Engineers build what they understand from the request. Stakeholders assumed something different. The gap surfaces weeks or months later when reports don’t match expectations.

Fix: For every new business domain, create a conceptual model first. List the entities, name the relationships, and get business stakeholder sign-off before writing any SQL.

Mistake 4: Over-Normalizing for Analytics

Third Normal Form (3NF) is correct for transactional systems where writes are frequent and consistency matters. Applied to an analytics workload, it creates queries with 10-15 joins that run slowly and break easily.

Fix: Separate your transactional model from your analytical model. Keep the OLTP system in 3NF. Build a denormalized star schema (or a set of wide views) for analytics. Different workloads deserve different models.

Mistake 5: Under-Documenting

A data model without documentation is a puzzle that only its creator can solve. And even they forget the details after a few months.

Without documentation, every new team member reverse-engineers the model from scratch. Every AI agent generates SQL based on guesses. Every analyst interprets column meanings differently, leading to metric discrepancies that take weeks to reconcile.

Fix: Document at three levels:

• Column level: What does each column mean? Where does it come from?

• Table level: What grain does this table use? Who maintains it?

• Model level: How do tables connect? What business process does this model represent?

Platforms like Dremio make this practical with built-in Wikis for every dataset and Labels for classification (PII, Certified, Raw, Deprecated). The documentation lives next to the data, not in a separate spreadsheet that goes stale.

Mistake 6: One Model for Every Workload

A model designed for a transactional application doesn’t serve analytics well. A model designed for analytics doesn’t serve a machine learning feature store well. Trying to make one model serve every use case leads to compromises that serve no use case well.

Fix: Build purpose-specific models layered on top of shared source data. The Medallion Architecture does this naturally:

• Bronze: Raw data from sources (shared foundation)

• Silver: Business logic layer (shared across analytics and ML)

• Gold: Purpose-built views (one for dashboards, one for ML features, one for AI agents)

Each Gold view is tailored to its consumer without duplicating the transformation logic in Silver.

Mistake 7: Ignoring Governance

Data models don’t exist in a vacuum. They contain PII, financial data, health records, and other sensitive information. Ignoring governance creates compliance risk and erodes trust.

Common governance gaps:

• No access controls (everyone sees everything)

• No classification (no one knows which columns contain PII)

• No ownership (no one knows who to ask about table X)

• No lineage (no one knows where the data came from)

Fix: Integrate governance from day one:

• Tag columns by sensitivity (PII, financial, public)

• Assign ownership per table or domain

• Apply row and column-level access policies

• Document data lineage from source to consumption

In Dremio, Fine-Grained Access Control enforces row and column-level policies, Labels classify datasets, and the Open Catalog tracks lineage. Governance is part of the platform, not an afterthought.

What to Do Next

Pick one of these seven mistakes. Check whether your current data model has it. Fix it. Then move to the next one. Data modeling is iterative — no team gets it perfect on the first pass. The goal is not perfection but continuous improvement: clearer names, better documentation, tighter governance, and models that match what your consumers actually need.

Try Dremio Cloud free for 30 days

The conventional wisdom for data platform modernization goes like this: pick a target system, build ETL pipelines for every source, migrate everything, validate the data, retrain your users, and then start getting value. That process takes six to eighteen months. During that time, analysts are waiting and leadership is asking why the investment has not produced results yet.

There is a better sequence. Instead of making everyone wait for a full migration, you start producing value on day one and migrate to Apache Iceberg at your own pace. The key is treating federation, the semantic layer, AI access, and Iceberg migration as four independent phases, each delivering value on its own, rather than a single all-or-nothing project.

Phase 1: Connect Your Data Where It Lives

Sign up for Dremio Cloud and you get a lakehouse project with a pre-configured Open Catalog right away. From there, start connecting your existing data sources through Dremio’s federated query engine: PostgreSQL, MySQL, MongoDB, S3, Snowflake, BigQuery, Redshift, AWS Glue, Unity Catalog, and more.

No data copying. No ETL pipelines. Dremio queries your data where it already lives, using predicate pushdowns to push filtering work down to each source system.

The result: by the end of day one, your team has unified SQL access across every connected source. An analyst can join a PostgreSQL customer table with an S3-based event stream in a single query, without waiting for a data engineer to build a pipeline first.

Phase 2: Build a Semantic Layer Over Everything

Raw source tables have cryptic column names, inconsistent types, and zero business context. Before anyone can get reliable answers, whether human or AI, you need a curated layer on top.

Dremio’s AI Semantic Layer uses SQL views organized in three tiers:

• Bronze/Raw views map to raw sources. They standardize column names, cast data types, and apply basic filters. One Bronze view per source table.

• Silver/Business views apply business logic. This is where you define what “active customer” means (purchased in the last 90 days, not on a trial), join data across sources, and compute metrics.

• Gold/Application views serve specific consumers: a dashboard, a report, or an AI agent. Each Gold view is optimized for its use case.

Dremio’s AI Agent can help you come up with the SQL to generate these views efficiently.

Govern Access and Document Everything

Grant users access to specific views using Role-Based Access Control (RBAC) at the folder, dataset, and column level. For sensitive data, add Fine-Grained Access Control (FGAC) via UDFs for row-level security and column-level masking.

Then enrich every dataset with Wikis (human-readable documentation explaining what each column means) and Tags (categorical labels for discoverability). Dremio can auto-generate Wiki descriptions and suggest Tags by sampling your table data and schema. You review and refine the output instead of writing everything from scratch.

This metadata is not just for humans. It is what the AI Agent reads when generating SQL. Better documentation means more accurate answers.

Phase 3: Turn On Agentic Analytics

With a governed semantic layer in place, you are ready for AI. This is the important part: you do not need to complete the Iceberg migration first. Agentic analytics works on federated data from the moment the semantic layer exists.

Dremio’s built-in AI Agent lets users type plain-English questions in the console. The agent writes SQL, executes it against your governed views, returns results, generates charts, and suggests follow-up questions. It respects every RBAC and FGAC policy in your catalog. Users can only get answers about data they are authorized to see.

For teams that want to use external tools, Dremio’s MCP (Model Context Protocol) server lets ChatGPT, Claude Desktop, or custom agents connect directly to your Dremio environment. External tools get the same semantic context and security controls as the built-in agent.

At this point your organization has unified data access, a governed semantic layer, and AI-powered analytics, and you have not migrated a single table to Iceberg yet.

Phase 4: Migrate to Iceberg, One Dataset at a Time

Federation gets you access, but a full Apache Iceberg lakehouse gets you more: Autonomous Reflections that optimize query performance based on actual usage patterns, end-to-end caching, automated table maintenance (compaction, clustering, vacuuming), and interoperability with every Iceberg-compatible engine (Spark, Flink, Trino). Your data stays in your storage, in an open format, with no vendor lock-in.

The migration pattern is deliberately incremental:

1. Pick one dataset to migrate (start with the highest-volume or most-queried table)

2. Build an Iceberg pipeline to land that data in your object storage (S3 or Azure)

3. Update the Bronze view to point to the new Iceberg table instead of the legacy federated source

4. Silver and Gold views stay unchanged. They reference the Bronze view, which now reads from Iceberg instead of the old source.

5. Every consumer is unaffected. Dashboards, reports, and AI agents continue to work exactly as before.

Repeat for the next dataset whenever you are ready. There is no deadline and no big-bang cutover.

Why the View Layer Makes Migration Invisible

This is the architectural insight that makes the whole journey work. The semantic layer acts as a contract between physical data storage and every consumer above it.

When you swap a Bronze view’s underlying source from PostgreSQL to an Iceberg table, every Silver view, Gold view, dashboard, report, and AI agent that depends on it continues to work without changes. The view contract (column names, data types, business logic) is preserved. Only the physical source pointer changes.

This means:

• No dashboard rewiring

• No report migration

• No API endpoint changes

• No AI Agent reconfiguration

• No user communication (beyond governance notifications if your policies require them)

The migration happens underneath the abstraction layer. Everyone above it is oblivious.

The Tradeoffs

This phased approach is not free of costs.

Federation introduces network latency. Queries that join a PostgreSQL table in one region with an S3 bucket in another will be slower than queries against co-located Iceberg tables. Reflections and caching mitigate this for repeated queries, but the first execution of a new query pattern will feel it.

Iceberg migration still requires building ingest pipelines. Dremio does not eliminate that work. What it does is decouple the pipeline work from the analytics timeline. Your analysts and AI agents are productive while engineers build migration pipelines in the background.

Autonomous Reflections need a 7-day observation window before they start optimizing. Day-one performance on brand-new Iceberg tables relies on baseline optimizations (C3 caching, predicate pushdowns, vectorized execution). The system gets faster as it learns your query patterns.

And Dremio is an analytical engine, not a transactional database. Your OLTP workloads stay in PostgreSQL, MongoDB, or whatever system runs your application. You query those systems through federation, not as a replacement.

Start Today, Migrate Over Time

The traditional approach forces you to choose: spend months migrating, or keep running fragmented analytics on scattered data. Dremio eliminates that choice. Connect your sources, build your semantic layer, enable AI access, and start migrating to Iceberg when you are ready. Each phase delivers value independently, and the view layer ensures that migration never disrupts the people who are already getting answers.

Try Dremio Cloud free for 30 days and start the journey from wherever your data lives today.

Free Resources to Go Deeper

• FREE - Apache Iceberg: The Definitive Guide

• FREE - Apache Polaris: The Definitive Guide

• FREE - Agentic AI for Dummies

• FREE - Leverage Federation, The Semantic Layer and the Lakehouse for Agentic AI

• FREE with Survey - Understanding and Getting Hands-on with Apache Iceberg in 100 Pages

Two weeks past the Iceberg Summit, the San Francisco in-person alignments are now translating into formal proposals and code on the dev lists. Iceberg’s V4 design work continued consolidating, Polaris kept moving toward its 1.4.0 milestone, Parquet’s Geospatial spec picked up a cleanup commit from a new contributor, and Arrow’s release engineering and Java modernization discussions stayed active.

Apache Iceberg

The post-summit V4 design work continued as the defining thread on the Iceberg dev list this week. The V4 metadata.json optionality discussion that Anton Okolnychyi, Yufei Gu, Shawn Chang, and Steven Wu drove through March kept narrowing on practical design questions. The concrete direction emerging from the summit is to treat catalog-managed metadata as a first-class supported mode while preserving static-table portability through explicit opt-in semantics, rather than the current implicit assumption that the root JSON file is always present.

Russell Spitzer and Amogh Jahagirdar’s one-file commits design moved toward a formal spec write-up this week. The approach replaces manifest lists with root manifests and introduces manifest delete vectors, enabling single-file commits that cut metadata write overhead dramatically for high-frequency writers. The in-person sessions at the summit cleared the last design disagreements about inline versus external manifest delete vectors, and the community is now aligning on the implementation plan.

Péter Váry’s efficient column updates proposal for AI and ML workloads drew steady engagement. The design lets Iceberg write only the columns that change on each write for wide feature tables, then stitch the result at read time. For teams managing petabyte-scale feature stores with embedding vectors and model scores, the I/O savings are meaningful. Anurag Mantripragada and Gábor Herman are working alongside Péter on POC benchmarks to support the formal proposal.

The AI contribution policy that Holden Karau, Kevin Liu, Steve Loughran, and Sung Yun pushed through March is moving toward published guidance. The summit provided the in-person alignment that async debate rarely produces, and a working policy covering disclosure requirements and code provenance standards for AI-generated contributions is expected on the dev list in the next couple of weeks. Polaris is navigating the same question in parallel, and the two communities are likely to converge on a shared approach given their overlapping contributor base.

Apache Polaris

The Polaris 1.4.0 release is in active scope finalization as the project’s first release since graduating to top-level status on February 18. Credential vending for Azure and Google Cloud Storage is the headline feature, alongside catalog federation that lets one Polaris instance front multiple catalog backends across clouds. The schedule-driven release model calls for a release intent email to the dev list about a week before the RC cut, so watch the list for that thread shortly.

The Apache Ranger authorization RFC from Selvamohan Neethiraj remained the most active governance discussion. The plugin lets organizations running Ranger with Hive, Spark, and Trino manage Polaris security within the same policy framework, eliminating the policy duplication that arises when teams bolt separate authorization onto each engine. It is opt-in and backward compatible with Polaris’s internal authorization layer, which lowers the enterprise adoption barrier considerably.

On the community side, Polaris’s blog continued its post-graduation cadence with a Sunday April 4 post on building a fully integrated, locally-running open data lakehouse in under 30 minutes using k3d, Apache Ozone, Polaris, and Trino. The Polaris PMC also shipped a March 29 post covering automated entity management for catalogs, principals, and roles. With incubator overhead behind it, release velocity has picked up noticeably from the 1.3.0 release on January 16.

Apache Arrow

Arrow’s release calendar shows arrow-rs 58.2.0 landing this month, following 58.1.0 in March which shipped with no breaking API changes. The cadence has held at roughly one minor version per month, with 59.0.0 already scheduled for May as a major release that may include breaking changes. The Rust implementation has become one of the most actively maintained segments of the Arrow ecosystem, with a DataFusion integration drawing engines that want Arrow without a JVM dependency.

Jean-Baptiste Onofré’s JDK 17 minimum proposal for Arrow Java 20.0.0 continued drawing input from Micah Kornfield and Antoine Pitrou. The practical rationale is coordination: setting JDK 17 as Arrow’s Java baseline aligns with Iceberg’s own upgrade timeline and effectively raises the minimum across the entire lakehouse stack in a single coordinated move. The decision is expected before the 20.0.0 release cycle formally opens.

Nic Crane’s thread on using LLMs for Arrow project maintenance continued generating discussion. The framing — AI as a resource for maintainers, not just contributors — is distinct from how Iceberg and Polaris are approaching their AI policies. Arrow’s angle is practical: a lean maintainer group managing a growing issue backlog needs help triaging, and LLMs can do that work without introducing the code-provenance concerns that matter for contributions. Google Summer of Code 2026 student proposals that landed in early April are being sorted this week, with interest concentrated in compute kernels and Go and Swift language bindings.

Apache Parquet

Parquet’s week centered on hardening the Geospatial spec that was adopted earlier this year. Milan Stefanovic merged PR #560 on April 20, clarifying the Geospatial spec wording for coordinate reference systems. The change documents existing CRS usage practice for the default OGC:CRS84 system and removes ambiguity caught during implementation reviews. Small spec-hardening commits like this are how a new type goes from “shipped” to “production-reliable” across engines.

The community blog effort continued alongside the spec work. The Native Geospatial Types blog that Jia Yu and Dewey Dunnington published on February 13 remains the community’s reference explainer, and Andrew Lamb has been coordinating with Aihua Xu on the companion Variant blog post. Spotlighting recent additions through the Parquet blog is part of a deliberate push to give the project the same kind of voice that DataFusion and Arrow have built.

The ALP encoding that cleared its acceptance vote in the prior week moved into implementation discussion. Engine teams across Spark, Trino, Dremio, and DataFusion are comparing notes on how to integrate ALP into their Parquet readers, with compression gains for float-heavy ML feature stores as the immediate benefit. The File logical type proposal for unstructured data (images, PDFs, audio) also kept advancing in community discussion, extending Parquet’s scope beyond pure analytics.

Cross-Project Themes

The summit’s downstream effect is now visible across every dev list. Iceberg’s V4 work, Polaris’s 1.4.0 scope, Arrow’s JDK 17 decision, and Parquet’s Geospatial cleanup are running in parallel, and the cross-project coordination on shared questions like AI contribution policy and Java baselines has intensified. The JDK 17 alignment is the clearest case: moving Arrow Java 20.0.0, Iceberg’s next major, and downstream engines to the same floor in a single window removes years of compatibility friction.

The second pattern is the steady expansion of format scope to meet AI workloads. Iceberg’s efficient column updates, Parquet’s File logical type, the Geospatial spec hardening, and Polaris’s multi-cloud federation all respond to the same pressure: the lakehouse stack is being asked to power AI pipelines, not just analytical queries. Each project is making changes that only make sense if you assume the next decade’s workloads look different from the last.

Looking Ahead

Watch for the V4 single-file commits formal spec write-up and the metadata optionality vote on the Iceberg dev list, along with a published AI contribution policy. The Polaris 1.4.0 release intent email should land in the coming days. Arrow’s JDK 17 baseline decision for Java 20.0.0 is close to a vote, and arrow-rs 58.2.0 should ship before the end of the month. Iceberg Summit 2026 session recordings are also rolling out on the project’s YouTube channel.

Resources & Further Learning

Get Started with Dremio

• Try Dremio Free — Build your lakehouse on Iceberg with a free trial

• Build a Lakehouse with Iceberg, Parquet, Polaris & Arrow — Learn how Dremio brings the open lakehouse stack together

Free Downloads

• Apache Iceberg: The Definitive Guide — O’Reilly book, free download

• Apache Polaris: The Definitive Guide — O’Reilly book, free download

Books by Alex Merced

• Architecting an Apache Iceberg Lakehouse

• Enabling Agentic Analytics with Apache Iceberg and Dremio

• The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI

• The Book on Using Apache Iceberg with Python

Three stories defined the past week: Anthropic shipped Claude Opus 4.7, Moonshot open-sourced Kimi K2.6 with 300-agent swarms, and Amazon committed another $25 billion to Anthropic alongside a $100 billion AWS spend. Here is what you need to know.

AI Coding Tools: Opus 4.7 Ships With a 1M Context Window

Anthropic released Claude Opus 4.7 on April 16, a new flagship model focused on agentic coding and long-horizon work. The model scores 87.6% on SWE-bench Verified and 64.3% on SWE-bench Pro, jumping from 80.8% on Opus 4.6. It runs with a full 1 million token context window and high-resolution image support for charts and dense documents.

The model landed across major platforms the same week. Claude Opus 4.7 arrived on Amazon Bedrock on launch day in four regions, with up to 10,000 requests per minute per account. GitHub Copilot began rolling out Opus 4.7 to Copilot Pro+ users with a 7.5x premium request multiplier until April 30. The model is replacing both Opus 4.5 and Opus 4.6 in the Copilot model picker.

Claude Code shipped Opus 4.7 the same day with new controls. The update added an “xhigh” effort level between high and max, a /ultrareview multi-agent code review command, and Auto mode for Max subscribers. Anthropic also launched Claude Design, a new Anthropic Labs product for building prototypes, slides, and one-pagers in collaboration with the model. Pricing stays at $5 per million input tokens and $25 per million output tokens.

AI Models: Kimi K2.6 Opens the Door to 12-Hour Agent Runs

Moonshot AI released Kimi K2.6 on April 20 as an open-source agentic model built for long-horizon coding. The model has 1 trillion total parameters in a Mixture-of-Experts architecture with 32 billion active per forward pass. It supports text, image, and video input, a 256K context window, and thinking and non-thinking modes behind an OpenAI-compatible API.

The headline claim is stamina. Kimi K2.6 targets 12-hour autonomous coding sessions and agent swarms that scale to 300 sub-agents across 4,000 coordinated steps. On benchmarks, Moonshot claims SWE-Bench Pro at 58.6, SWE-bench Multilingual at 76.7, and BrowseComp at 83.2. The model matches or beats GPT-5.4 and Claude Opus 4.6 on several open-source leaderboards.

K2.6 is available immediately on Kimi.com, the developer API, Kimi Code CLI, Ollama, and Hugging Face. Day-one integrations cover Kilo Code, VS Code and JetBrains extensions, OpenClaw, Tencent CodeBuddy, and Genspark. The MIT-derived license allows commercial use and redistribution, a direct challenge to closed-source frontier labs.

AI Infrastructure: AWS Interconnect Reaches GA and Amazon Adds $25B to Anthropic

AWS Interconnect reached general availability on April 20 with two new capabilities. AWS Interconnect Multicloud provides Layer 3 private connections between AWS VPCs and other clouds, starting with Google Cloud, with Azure and OCI coming later in 2026. Traffic flows over the AWS global backbone with built-in MACsec encryption, never crossing the public internet. AWS also published the Interconnect specification on GitHub under Apache 2.0, so any cloud provider can become a partner.

Amazon announced a $25 billion investment in Anthropic on April 20, on top of the $8 billion already committed. The deal includes $5 billion immediately, with up to $20 billion tied to commercial milestones. Anthropic committed to spending more than $100 billion on AWS over 10 years, securing up to 5 gigawatts of Trainium chip capacity. One gigawatt is scheduled to come online this year using Trainium2 and Trainium3.

The structure mirrors the $50 billion Amazon-OpenAI deal from February. Anthropic is now valued at $380 billion, with annualized revenue climbing from $9 billion at the end of 2025 to more than $30 billion. Enterprise customers spending at least $1 million annually have doubled since February, crossing 1,000 accounts.

Standards and Protocols: Interconnect Spec Goes Open

The AWS Interconnect specification going to GitHub under Apache 2.0 is the standards story of the week. The move gives any cloud provider a published path to join the private connectivity mesh without negotiating bilateral deals. For AI workloads moving data between model training clusters in one cloud and inference infrastructure in another, the alternative has been either the public internet or expensive dedicated circuits.

The broader pattern is that hyperscale cloud providers are open-sourcing infrastructure specs to lock in network effects. Trainium chip access is exclusive, but the connectivity layer is open. This is the same playbook the Linux Foundation’s Agentic AI Foundation uses for MCP and A2A: open standards for the plumbing, proprietary value on top.

MCP and A2A also saw continued adoption this week. Claude Opus 4.7 ships with both protocols built in, and Kimi K2.6 supports tool calls and OpenAI-compatible APIs that slot into MCP-aware agent stacks. The layered architecture is holding: MCP handles agent-to-tool connections, A2A handles agent-to-agent coordination, and the new open models and frontier releases are all landing with both built in by default.

Resources to Go Further

The AI landscape changes fast. Here are tools and resources to help you keep pace.

Try Dremio Free — Experience agentic analytics and an Apache Iceberg-powered lakehouse. Start your free trial

Learn Agentic AI with Data — Dremio’s agentic analytics features let your AI agents query and act on live data. Explore Dremio Agentic AI

Join the Community — Connect with data engineers and AI practitioners building on open standards. Join the Dremio Developer Community

Book: The 2026 Guide to AI-Assisted Development — Covers prompt engineering, agent workflows, MCP, evaluation, security, and career paths. Get it on Amazon

Book: Using AI Agents for Data Engineering and Data Analysis — A practical guide to Claude Code, Google Antigravity, OpenAI Codex, and more. Get it on Amazon

Every major data platform now lists Apache Iceberg somewhere on its feature page. Snowflake has Iceberg Tables. Databricks has UniForm. BigQuery has BigLake. This is a genuinely good thing for the ecosystem because it gives users more choice and more portability.

But “supports Iceberg” and “Iceberg native” are not the same thing. The distinction matters when you are deciding where your data gravity should live, meaning where your primary analytical data is stored, optimized, and governed. A platform that bolts Iceberg support onto a proprietary core makes different engineering tradeoffs than one built on Iceberg from the ground up.

Dremio claims to be Iceberg native. Here are the three things that back up that claim.

Iceberg by Default, Not by Request

There is no “Dremio Table” format. When you create a table in Dremio, it is an Apache Iceberg table. There is no proprietary alternative, no internal format that gets better performance, no checkbox to opt into openness. Iceberg is the only option because it is the foundation of the platform.

This is a bigger deal than it sounds. Engineering priority follows the default format. When a platform’s default is a proprietary format, that is where the optimization budget goes: faster writes, smarter compaction, deeper query optimizer integration. Iceberg support becomes a secondary project staffed by a smaller team, maintained to a “good enough” standard.

Dremio’s engineers spend their time on one thing: making the Dremio query engine faster on Apache Iceberg tables and operations. Every query optimizer improvement, every table management feature, every caching enhancement targets Iceberg directly. There is no internal format competing for attention.

The practical result: you don’t face a choice between “fast but proprietary” and “open but slower.” In Dremio, Iceberg IS the fast format.

Iceberg-Native Acceleration

Speed is where the “supports Iceberg” vs. “Iceberg native” distinction gets concrete.

Most platforms that added Iceberg support did not build their query engines for it. Their engines are tuned for their proprietary format. So when customers ask how to make Iceberg queries faster, the answer is often: create materialized views stored in the platform’s proprietary format, in the platform’s managed storage.

Think about what that means. The whole point of adopting Iceberg was to avoid writing data into proprietary formats. But the performance optimization path pushes you right back into one.

Dremio does the opposite.

Reflections: Iceberg as the Faster Format

Dremio’s Reflections are pre-computed, optimized materializations stored as Apache Iceberg tables. When you query a federated source like PostgreSQL or MongoDB through Dremio, the engine can create a Reflection in Iceberg format that serves future queries faster.

Read that again: Dremio uses Iceberg to speed up non-Iceberg sources. The industry pattern is “use a proprietary format to speed up Iceberg.” Dremio flips it: “use Iceberg to speed up everything.”

Autonomous Reflections take this further. Dremio analyzes your query patterns over a 7-day window and automatically creates, manages, and drops Reflections without human intervention. Reflections on Iceberg tables support incremental updates, so only changed data gets reprocessed. No full rebuilds.

Why the Engine Is Fast on Iceberg Natively

Dremio’s speed on Iceberg is not just about caching tricks. It is architectural:

• Apache Arrow as native in-memory format: Dremio co-created Apache Arrow. Data stays in columnar Arrow format in memory, which eliminates the serialization overhead that traditional engines pay when converting between formats.

• Vectorized Parquet reader: Maximizes parallelism when reading Iceberg’s underlying Parquet files.

• Columnar Cloud Cache (C3): Stores frequently accessed data on local NVMe drives at executor nodes. Cloud storage latency becomes local-disk speed.

• Results and Query Plan Cache: Identical queries return instantly.

The combination means Dremio doesn’t need a proprietary format to deliver fast analytics. Iceberg plus Arrow plus aggressive caching is the performance strategy.

Operationalizing Iceberg

Apache Iceberg is a table format specification. It is not, by itself, a managed analytics experience. The gap between “I have Iceberg tables” and “I have a production lakehouse” is filled with operational work that never ends: choosing a catalog, configuring compaction jobs, monitoring small file accumulation, managing schema evolution, vacuuming obsolete snapshots, and building alerting for all of it.

Dremio closes this gap so that working with Iceberg feels like working with a traditional data warehouse: you onboard datasets, write SQL, use AI agents, and run analytics. The open format stays open underneath.

Built-In Apache Polaris Catalog

Dremio’s Open Catalog is built on Apache Polaris, the open Iceberg REST catalog standard. It tracks and governs your lakehouse assets with Role-Based Access Control (RBAC) at the folder, dataset, and column level, plus Fine-Grained Access Control (FGAC) via UDFs for row-level security and column-level masking.

Dremio can also connect to your external Iceberg catalogs (AWS Glue, Unity Catalog, and others) alongside the built-in one, so you govern everything from a single namespace.

Iceberg V3 Spec Support

Dremio tracks the latest Apache Iceberg spec features, including Variant type for semi-structured data and Geospatial types. Being Iceberg native means adopting new spec capabilities as they land, not waiting for them to be translated through a compatibility layer.

Automated Table Maintenance

Background jobs handle the operational work that makes DIY lakehouses “perpetually a work-in-progress”:

Maintenance TaskWhat Dremio AutomatesCompactionConsolidates small files into optimally-sized filesClusteringSorts data by frequently filtered columns for faster queriesSnapshot managementVacuums obsolete snapshots and orphan files to control storage costsManifest rewritingOptimizes metadata structure for faster query planning

These jobs run based on actual table usage patterns. No cron jobs. No custom scripts. No on-call rotations for your data lake.

The Tradeoffs

Being Iceberg native is a deliberate architectural choice, and it comes with boundaries.

Dremio is an analytical engine. OLTP workloads (high-frequency transactional writes) belong in purpose-built databases like PostgreSQL or MongoDB. You can query those systems through Dremio’s federation, but you should not try to replace them.

Dremio’s opinionated commitment to Iceberg also means it is not the right fit if your organization’s strategy centers on a different format. If Delta Lake is your standard and you have no plans to move, a Delta-native platform is the better match while Dremio can access Uniform tables in Unity Catalog for data unification use cases.

And Iceberg itself is still evolving. V3 features like Variant and Geospatial are new, and the tooling ecosystem around them is maturing. Being on the leading edge of the spec means occasional rough edges.

Where Your Data Gravity Should Live

It is a great thing that so many platforms now support Apache Iceberg. More support means more flexibility for everyone. But if your intention is to make Iceberg your primary analytics format, then “supports Iceberg” and “built for Iceberg” lead to very different outcomes.

A platform where Iceberg is the default format, where acceleration stays in Iceberg, and where the operational complexity of running a lakehouse is abstracted away is a platform built for that Iceberg-native use case. That is what Dremio is designed to be.

Try Dremio Cloud free for 30 days and see how an Iceberg-native platform handles the workloads you are running today.

FREE Books

• FREE - Apache Iceberg: The Definitive Guide

• FREE - Apache Polaris: The Definitive Guide

• FREE - Agentic AI for Dummies

• FREE - Leverage Federation, The Semantic Layer and the Lakehouse for Agentic AI

• FREE with Survey - Understanding and Getting Hands-on with Apache Iceberg in 100 Pages

Read the complete Open Source and the Lakehouse series:

• Part 1: Apache Software Foundation

• Part 2: What is Apache Parquet?

• Part 3: What is Apache Iceberg?

• Part 4: What is Apache Polaris?

• Part 5: What is Apache Arrow?

• Part 6: Assembling the Apache Lakehouse

• Part 7: Agentic Analytics on the Apache Lakehouse

If you drop ten thousand Parquet files into an S3 bucket, you have a data swamp. You do not have a database. To run SQL queries against those files safely, your engine needs to know exactly which files belong to which table, what the columns are, and which files to ignore. Historically, Apache Hive solved this by tracking directories. Apache Iceberg solves this by tracking files.

That shift from directory-listing to file-level metadata fundamentally changes how organizations scale analytics. Iceberg brings the reliability of a transactional database to cloud object storage.

The Directory Listing Bottleneck

Legacy data architectures treated cloud storage like a local hard drive. If an engine like Hive wanted to read a table, it asked the cloud provider to list all the files inside a specific directory.

Listing millions of files in Amazon S3 or Google Cloud Storage takes an incredibly long time. Worse, cloud providers aggressively throttle high-frequency listing requests. When concurrent writers update a heavily partitioned Hive table, metadata synchronization operations cause readers to see inconsistent, partial data. Scaling meant hitting a hard wall.

Iceberg architects recognized that the file system is the wrong place to store database state. They moved the state into a dedicated metadata tree.

The Iceberg Metadata Tree Architecture

When an engine queries an Iceberg table, it never asks S3 to list directories. File discovery becomes an instant, O(1) metadata lookup. The architecture works through a strict hierarchy of pointers.

The query begins at the Catalog, which holds a single pointer to the current metadata.json file. This ensures atomic commits; whichever engine successfully updates the catalog pointer wins the transaction. The metadata.json tracks the table schema and points to a Manifest List.

The Manifest List acts as a table of contents for a specific point in time (a snapshot). It points to multiple Manifest Files. Finally, these Manifest Files contain the explicit paths to the individual Parquet data files, along with statistics like minimum and maximum values for every column.

This strict tree structure means the engine knows exactly which Parquet files it needs to read before touching the raw data.

Schema and Partition Evolution

Data shapes change. In traditional data lakes, renaming a column or changing a partition strategy required a total table rewrite. Iceberg executes these changes in milliseconds as metadata operations.

Iceberg achieves Schema Evolution by assigning a unique ID to every column. It tracks schema changes against the ID, not the string name. If you delete a column named user_id and create a new column named user_id, Iceberg knows they are entirely different fields. You can add, drop, rename, and reorder columns with zero side effects on existing files.

Similarly, Iceberg features “hidden partitioning”. Engineers do not have to create physically derived columns just to partition data (e.g., extracting the year from a timestamp). Iceberg tracks the partition logic entirely in metadata. If you decide to change a table from monthly partitioning to daily partitioning, old data remains partitioned by month, and new data partitions by day. The engine handles the difference transparently.

Time Travel and Atomic Snapshots

Because Iceberg uses a tree of files where data is never updated in place, every write operation creates a brand new, immutable snapshot of the table.

When you run an UPDATE statement, Iceberg writes a new Parquet file containing the updated records, creates a new Manifest pointing to the new data, and generates a new Manifest List. The previous snapshot remains completely intact.

This architecture unlocks Time Travel. Analysts can append FOR SYSTEM_TIME AS OF to their SQL queries to read previous table states. If a faulty pipeline writes bad data, you do not need to rebuild the table from backups. You simply roll back the catalog pointer to the previous, healthy snapshot. Time travel does not duplicate data; the metadata simply points back to the underlying files that were valid at that exact moment.

Scaling the Open Source Lakehouse

Apache Iceberg provides the structure necessary to treat raw Parquet files like high-performance relational tables. However, a table format alone is incomplete. You need a centralized catalog mechanism to manage the root pointers, enforce security access, and resolve interoperability between multiple query engines.

That requirement leads directly to Apache Polaris, the open catalog standard designed to unify the Iceberg ecosystem.

Dremio executes natively against Iceberg tables, managing the metadata optimization lifecycle automatically. To see Iceberg transactions and time travel in action without building infrastructure, try Dremio Cloud free for 30 days.

Dimensional modeling works well when your source systems are stable and your business questions are predictable. But what happens when sources change constantly, new systems get added every quarter, and regulatory requirements demand a full audit trail of every attribute change?

Data Vault modeling was designed for exactly this scenario. Created by Dan Linstedt, it separates data into three distinct table types — Hubs, Links, and Satellites — each handling a different concern: identity, relationships, and descriptive context.

What Problem Data Vault Solves

Traditional dimensional models embed everything about a business entity in one dimension table. A dim_customers table contains the customer ID, name, address, segment, acquisition channel, and lifetime value. When a new source system provides additional customer attributes, you add columns to dim_customers. When business rules change how “segment” is calculated, you update the ETL pipeline that populates that table.

Over time, these dimension tables become fragile. They depend on multiple source systems. A change in one source breaks the ETL. Schema changes require coordinated updates across pipelines, tables, and downstream reports.

Data Vault solves this by decomposing entities into independent components that can evolve separately.

The Three Building Blocks

Hubs: Business Identity

A Hub stores unique business keys — the identifiers that define a business entity regardless of which source system provides them.

CREATE TABLE hub_customer (
    customer_hash_key BINARY(32),  -- Hash of the business key
    customer_id VARCHAR(50),        -- Natural business key
    load_date TIMESTAMP,
    record_source VARCHAR(100)
);

Hubs are immutable. Once a business key is loaded, it never changes. A customer who has customer_id = 'C-1042' always has that key. Hubs answer the question: What business concepts exist?

Links: Relationships

A Link stores relationships between Hubs. Every relationship — customer-to-order, order-to-product, employee-to-department — gets its own Link table.

CREATE TABLE link_customer_order (
    link_hash_key BINARY(32),
    customer_hash_key BINARY(32),
    order_hash_key BINARY(32),
    load_date TIMESTAMP,
    record_source VARCHAR(100)
);

Links are also immutable. Once a relationship is recorded, it stays. Links support many-to-many relationships by default. They answer the question: How are business concepts related?

Satellites: Descriptive Context

Satellites store the descriptive attributes of a Hub or Link, along with their change history.

CREATE TABLE sat_customer_details (
    customer_hash_key BINARY(32),
    effective_date TIMESTAMP,
    customer_name VARCHAR(200),
    email VARCHAR(200),
    city VARCHAR(100),
    segment VARCHAR(50),
    load_date TIMESTAMP,
    record_source VARCHAR(100)
);

Every time an attribute changes, a new Satellite row is inserted. This is equivalent to SCD Type 2 — full history is preserved without modifying existing rows. Different source systems can feed different Satellites for the same Hub, allowing attributes to arrive independently.

How a Data Vault Query Works

To reconstruct a business entity (like a current customer profile), you join the Hub to its current Satellite rows:

SELECT
    h.customer_id,
    s.customer_name,
    s.email,
    s.city,
    s.segment
FROM hub_customer h
JOIN sat_customer_details s ON h.customer_hash_key = s.customer_hash_key
WHERE s.effective_date = (
    SELECT MAX(effective_date)
    FROM sat_customer_details s2
    WHERE s2.customer_hash_key = s.customer_hash_key
);

This is more complex than querying dim_customers directly. That complexity is the primary criticism of Data Vault. In practice, teams build a presentation layer — star schema views on top of the vault — for business users and BI tools.

Platforms like Dremio make this practical. The raw vault tables live in the Bronze layer. Silver-layer views reconstruct business entities by joining Hubs, Links, and Satellites. Gold-layer views present dimensional star schemas for dashboards and AI agents. Users never query the vault tables directly.

When Data Vault Fits

Multiple source systems that change frequently. Adding a new source means adding new Satellites — not redesigning existing tables. The Hub and Link structure remains stable.

Regulated industries requiring full audit trails. Financial services, healthcare, and government often need to prove what data looked like at any point in time. Satellites provide that out of the box.

Large enterprises with parallel development teams. Hubs, Links, and Satellites can be loaded independently, enabling parallel ETL development without pipeline conflicts.

Long-term data warehouses with decades of history. The separation of structure (Hubs, Links) from content (Satellites) makes the vault resilient to business changes over time.

When Data Vault Doesn’t Fit

Small teams or simple source environments. If you have five source tables and one BI tool, Data Vault adds complexity without proportional benefit. A star schema is faster to build and easier to maintain.

Direct BI tool access. BI tools don’t speak Data Vault natively. You always need a presentation layer on top, which means building two models instead of one.

Speed-to-value projects. When the goal is “get a dashboard live this sprint,” Data Vault’s up-front design work slows you down.

FactorData VaultDimensional ModelSource flexibilityHighModerateAudit trailBuilt-inOptional (SCDs)Query simplicityLow (needs presentation layer)HighLearning curveHighModerateAdding new sourcesEasy (new satellites)Harder (redesign dimensions)BI tool compatibilityLowHigh

What to Do Next

If you’re evaluating Data Vault, start by counting your source systems and estimating how often they change schema. If the answer is “more than five sources” and “at least once a quarter,” Data Vault’s separation of concerns will likely save you from painful redesign cycles. If your environment is simpler than that, a well-designed dimensional model will get you to production faster.

Try Dremio Cloud free for 30 days

The Iceberg Summit wrapped in San Francisco, leaving behind a set of in-person alignments that are now surfacing as concrete proposals on the dev lists. Parquet’s ALP encoding vote closed, Polaris 1.4.0 planning accelerated, and Arrow’s engineering community tackled two interlinked decisions about its future Java baseline and AI tooling policy. The post-summit week is when talk becomes code.

Apache Iceberg

The two days in San Francisco established alignment on the discussions that have dominated the dev list all spring. The V4 metadata.json optionality thread drew the largest in-person audience of any design session, with Anton Okolnychyi, Yufei Gu, Shawn Chang, and Steven Wu working through the portability and static-table implications of making the root JSON file optional when a catalog manages metadata state. The direction that emerged favors catalog-managed metadata as a first-class supported mode, with portability guarantees preserved through explicit opt-in semantics rather than the current default assumption.

The one-file commits design — the work Russell Spitzer and Amogh Jahagirdar have been advancing through multiple proposals — is heading toward a formal spec write-up following alignment reached at the summit. The approach replaces manifest lists with root manifests and uses manifest delete vectors to enable single-file commits, promising dramatic reductions in commit latency and metadata storage footprint. This is one of the most consequential V4 changes for high-frequency write workloads, and the in-person sessions cleared the remaining design disagreements about inline versus external manifest delete vectors.

Péter Váry’s efficient column updates proposal for AI and ML workloads drew real interest at the summit. The design targets wide tables where only a subset of columns change on each write — embedding vectors, model scores, feature values — allowing Iceberg to write only the updated columns to separate files and merge at read time. For teams managing petabyte-scale feature stores, the I/O savings are significant. Péter indicated that a formal proposal with POC benchmarks would land on the dev list in the days following the summit.

The AI contribution policy that pulled in Holden Karau, Kevin Liu, Steve Loughran, and Sung Yun over the preceding weeks moved toward practical resolution. The summit provided the in-person clarity that async debate rarely does, and a working policy covering disclosure requirements and code provenance standards for AI-generated contributions is expected to be published on the dev list this week.

Apache Polaris

Polaris is one month past its February 18 graduation as a top-level Apache project, and the governance machinery is running. Jean-Baptiste Onofré’s first board report as a TLP covers the March 26 ASF board meeting, documenting community health, development progress, and strategic direction under Polaris’s own PMC. JB also joined the Apache Software Foundation board itself as a Dremio-nominated director, a governance milestone that deepens the open-source commitment across the entire ecosystem.

The Apache Ranger authorization RFC from Selvamohan Neethiraj remained the most active technical discussion thread. The design allows organizations running Ranger alongside Hive, Spark, and Trino to manage Polaris security within a unified governance framework, eliminating the policy duplication that arises when teams bolt separate authorization systems onto each engine. The plugin is opt-in and backward compatible with Polaris’s existing internal authorization layer, a design choice that lowers the enterprise adoption barrier considerably.

The 1.4.0 release — Polaris’s first as a graduated project — is now in active scope finalization. Credential vending for Azure and Google Cloud Storage is the headline feature, alongside catalog federation design that lets Polaris front for multiple catalog backends in multi-cloud deployments. With incubator overhead behind it, release velocity is expected to accelerate. Watch the dev list this week for a 1.4.0 milestone thread and vote timeline.

Apache Arrow

Jean-Baptiste Onofré’s thread proposing JDK 17 as the minimum version for Arrow Java 20.0.0 is approaching decision. Contributors including Micah Kornfield and Antoine Pitrou have been weighing in, and the practical rationale is compelling: setting JDK 17 as the floor would align Arrow’s Java modernization with Iceberg’s own upgrade timeline, effectively raising the minimum across the entire lakehouse stack in a single coordinated move. The decision is expected to land before the 20.0.0 release cycle formally opens.

The arrow-rs 58.2.0 release was on track for April, following the 58.1.0 shipment in March, which arrived with no breaking API changes. The Rust implementation has become one of the most actively maintained segments of the Arrow ecosystem, with a release cadence that matches growing adoption in query engines that want Arrow’s columnar format without a JVM dependency.

Nic Crane’s thread on using LLMs for Arrow project maintenance continued to generate thoughtful discussion. The framing — AI as a resource for maintainers rather than just contributors — is distinct from how Iceberg and Polaris are approaching the same question. Arrow’s angle is practical: a lean maintainer group managing a growing issue backlog needs help triaging, and LLMs can do that work without introducing the code-provenance concerns that matter for contributions. Google Summer of Code 2026 student proposals arrived this week, with interest concentrated in compute kernels and language bindings for Go and Swift, adding bandwidth to a project that will need it as the 20.0.0 cycle opens.

Apache Parquet

The ALP (Adaptive Lossless floating-Point) encoding specification vote closed this week, marking one of the most meaningful additions to the Parquet specification in recent memory. ALP encodes floating-point exponents and mantissas separately, delivering significantly better compression ratios for float-heavy columns. The practical beneficiaries are ML feature stores and scientific computing workloads, where columns full of embedding coordinates and model outputs are common. Months of careful spec review paid off.

The Variant type that shipped in February has been generating follow-on integration discussion across engine teams. Spark, Trino, and Dremio contributors compared notes on their implementation experiences this week, working through edge cases in semi-structured data handling that the spec leaves partially open. Getting these implementations to converge matters: Parquet’s value as a cross-engine format depends on consistent behavior, and Variant is novel enough that divergence between engines would fragment the ecosystem.

The File logical type proposal — which would allow Parquet files to natively embed unstructured data including images, PDFs, and audio as columnar records — continued advancing through community discussion. Alongside Variant, this proposal signals a deliberate effort to evolve Parquet from a purely analytical format into a unified storage layer capable of managing the diverse data shapes that AI and ML pipelines produce. The direction is ambitious and the community engagement is substantive.

Cross-Project Themes

The post-summit week is when the conversations that happened in person translate back into the formal proposals and vote threads that actually change the projects. Across all four lists, expect the next two weeks to be among the most active of 2026 as in-person alignments hit the dev list in concrete form.

The second theme connecting all four projects is the deliberate expansion of format scope to meet AI workload demands. Parquet’s ALP acceptance, the File logical type proposal, Iceberg’s efficient column updates for wide ML tables, Polaris’s Ranger integration and federation work, and Arrow’s JDK 17 modernization are all responses to the same underlying pressure: the lakehouse stack is being asked to power AI pipelines, not just analytical queries. The pace of that evolution is accelerating, and the summit put the community’s roadmap on the same page.

Looking Ahead

Watch the Iceberg dev list for the V4 metadata optionality formal proposal, the single-file commits spec write-up, and a published AI contribution policy. The Polaris 1.4.0 milestone thread and vote timeline should also land this week. Arrow’s JDK 17 decision for Java 20.0.0 will likely follow close behind. The summit session recordings will appear on YouTube in the weeks ahead — an excellent resource for anyone who missed San Francisco.

Resources & Further Learning

Get Started with Dremio

• Try Dremio Free — Build your lakehouse on Iceberg with a free trial

• Build a Lakehouse with Iceberg, Parquet, Polaris & Arrow — Learn how Dremio brings the open lakehouse stack together

Free Downloads

• Apache Iceberg: The Definitive Guide — O’Reilly book, free download

• Apache Polaris: The Definitive Guide — O’Reilly book, free download

Books by Alex Merced

• Architecting an Apache Iceberg Lakehouse

• Enabling Agentic Analytics with Apache Iceberg and Dremio

• The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI

• The Book on Using Apache Iceberg with Python