feat(docker-compose): use Lakekeeper as the Iceberg Catalog when launching via docker compose#4274
feat(docker-compose): use Lakekeeper as the Iceberg Catalog when launching via docker compose#4274mengw15 wants to merge 19 commits intoapache:mainfrom
docker compose#4274Conversation
bobbai00
left a comment
There was a problem hiding this comment.
Generally looks good! Left some comments
docker compose
docker composedocker compose
|
@bobbai00 I am not sure if it is just a problem on my end, but the script to build the images appears not to work. I'm testing this PR, but when building the images, it fails and appears to be missing the License service that should be included. If needed, I can raise a Pr if this is a known issue. |
|
I tested this PR. Lakekeeper is functioning: the REST catalog handles namespace and table creation correctly, and workflow execution results are successfully stored in Iceberg via Lakekeeper. The only issue I hit is SQL init: docker compose up fails because bin/single-node/sql/ is empty. I see PR #4391 is addressing this, so this PR looks good overall. Separately, while trying other file input operators, I noticed an unrelated issue: file upload fails with a 413 error, likely due to a missing client_max_body_size setting in the single-node deployment nginx configuration. |
What changes were proposed in this PR?
This PR updates the single-node Docker Compose deployment to integrate Lakekeeper as the Iceberg REST catalog.
Changes to
bin/single-node/docker-compose.yml:lakekeeper-migrateinit container: runs database migration before Lakekeeper startslakekeeperservice (vakamo/lakekeeper:v0.11.0): REST catalog server with health checkslakekeeper-initcontainer: bootstraps default project, creates MinIO bucket, and configures warehouseChanges to
bin/single-node/.env:STORAGE_ICEBERG_CATALOG_TYPE=restIMAGE_REGISTRY,IMAGE_TAG,TEXERA_PORT,MINIO_PORT,TEXERA_SERVICE_LOG_LEVELvariablesAny related issues, documentation, discussions?
issue: close #4370
How was this PR tested?
Manually tested
Was this PR authored or co-authored using generative AI tooling?
co-authored with Claude