Unlocking Data Portability: Preventing Catalog Lock-in with REGISTER and UNREGISTER APIs

來源原標題

Unlocking Data Portability: Preventing Catalog Lock-in with REGISTER and UNREGISTER APIs

機構

Databricks

原始標籤

Platform · Product

原始連結

https://www.databricks.com/blog/unlocking-data-portability-preventing-catalog-lock-register-and-unregister-apis

摘要原文

As organizations embrace the lakehouse architecture, data is shifting from proprietary data warehouses into open storage and table formats, where it can be accessed across multiple engines, like Spark and Trino, without duplication. However, open formats are only one part of the openness equation. True openness also requires interoperability and flexibility in how data is managed and governed. For a lakehouse to fully deliver on its promise, organizations need the freedom to choose and move between catalogs as their architecture evolves. To support this portability, the REGISTER and UNREGISTER endpoints enable users to hand off a table between catalogs without rewriting, exporting, or copying a single file. REGISTER attaches an existing table to any IRC catalog. When moving a table, you can't just DROP it from the old catalog because that cleans up its underlying data and metadata. We added the UNREGISTER endpoint to the Apache Iceberg™ REST catalog specification so that you can tell the old catalog to forget about the table and return the exact pointer the next catalog needs to take over. In this post, we will take a closer look at how the open table format ecosystem is evolving, the key challenges these additions address, and how REGISTER, and the new UNREGISTER command, actually work. When an engine queries a table, it first asks the catalog to load the table to ensure it has the latest state. The catalog returns the table's current metadata, including the location of the table’s data in object storage. From there, the engine uses that metadata to find the schema and reads the Parquet straight from object storage. This coordination is essential for open table formats because all the important metadata - schema, history, statistics - and the data itself lives in storage, completely decoupled from compute. By acting as the central authority for that latest state, the catalog coordinates commits and ensures that two writers can never silently overwrite each other. Attaching an existing table to a catalog is simply a matter of handing over the metadata location from loading the table. This is exactly what REGISTER does through the endpoint already defined in the Iceberg REST specification. Figure 2: The REGISTER Operation The catch is that REGISTER alone would create a “split-brain” scenario, where two catalogs think they own the table and will coordinate commits. Because Iceberg catalogs are independent and do not communicate, REGISTER alone adds an entry to the new catalog while leaving the old one completely active. If two catalogs both believe they are the sole owner of the table, neither will throw an error, but the first write operation will fork the table, leading to inconsistent query results and silent data loss. Figure 3: The Split-Brain Scenario Historically, the ecosystem lacked a standard way to cleanly terminate a catalog's management of a table. Running DROP TABLE doesn’t work because it deletes the table’s data and metadata! Without UNREGISTER, the hand-off equation for REGISTER was missing its second half. We contributed UNREGISTER to the Apache Iceberg™ REST catalog specification to remove the table's entry from the managing catalog without touching a single underlying data file. Crucially, it returns the table's latest metadata location - the exact pointer the next catalog needs to assume commit coordination. By ensuring the original catalog explicitly relinquishes control, this eliminates the risk of a split-brain scenario. The request is an empty POST to the table resource: POST <uc-iceberg-rest-base>/v1/{prefix}/namespaces/{namespace}/tables/{table}/unregister The response is the pointer the next catalog needs: By combining these three steps - unregister, receive the location, and register - the handoff ensures the table has exactly one managing catalog at all times. (Note: A production migration still involves operational steps, like safely stopping writers and repointing jobs, which we will cover in a follow-up post.) With REGISTER and UNREGISTER, you now have the freedom to move your tables to other catalogs. We’re adding this capability because open-source portability is critical for our customers. Unity Catalog remains the most open lakehouse for managing your data, providing the unified governance the agentic era requires. It provides the context layer for your ontology, delivers best-in-class access control and observability across data and agents, and delivers flexibility across clouds, regions, and compute. To try out REGISTER and UNREGISTER on Unity Catalog in private preview, reach out to your account team. Subscribe to our blog and get the latest posts delivered to your inbox.