
KDnuggets
· 2 min read
One Table, Many Engines: Running BigQuery, Spark, and Trino on a Single Iceberg Copy
In Partnership with Google Cloud
Here is a common data engineering setup: BigQuery for analytics, Apache Spark for pipelines, and Trino for ad hoc questions that land on a Friday afternoon. Each engine gets its own copy of the data, while sync jobs do their best to keep this data aligned.
Despite best efforts, data doesn't always stay consistent, and the lag between syncs inevitably becomes an issue. Consider a scenario where BigQuery says that an order status is shipped, while the Spark copy reports paid and awaiting shipment. No one can trust either until the next sync job finishes. Even if it eventually does finish, you're still paying for compute time to perform the sync and storage for multiple copies. And if it doesn't finish? You pay for engineering time to trace and fix the pipeline issues.
Google Cloud's borderless Lakehouse and the Apache Iceberg open table format work together to help solve this. Every engine can read the same Parquet files and the same metadata layout without having to create separate copies. A shared catalog and commit protocol ensure that the engines agree on where the current version of the table lives, and on how a change gets committed without one writer trampling another.
To test it out, I built a project of my own: one orders table, one Cloud Storage bucket registered once in the Lakehouse runtime catalog. Apache Spark creates and loads the table, Trino appends to it, and BigQuery updates and merges rows in place. Then I made Spark and Trino write at the same moment, on purpose, to see what Iceberg does when two commits collide.
I ran every command here in a fresh Google Cloud project, and I've reported the outputs as they came back. You can find the full set of scripts in this repository.
Architecture Overview: One Bucket, One Catalog, Three Engines
Spark and Trino reach the Lakehouse runtime catalog through the standard Apache Iceberg REST catalog endpoint (https://biglake.googleapis.com/iceberg/v1/restcatalog).
Output:
Original source
This story was published by KDnuggets and written by KDnuggets. SyncAI.news shows a preview; the complete article is on the publisher's site.
Read the full story on kdnuggets.com


