Skip to content

docs: Add docker compose setup with 3 node etcd cluster #151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
subkanthi merged 2 commits into from
Apr 28, 2026
Merged

docs: Add docker compose setup with 3 node etcd cluster #151

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ad6be9e
Added docker compose setup with 3 node etcd cluster
subkanthi Apr 27, 2026
82c903f
Added documentation to the etcd-cluster-setup for 3 node docker compo…
subkanthi Apr 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 17 additions & 10 deletions docs/etcd-cluster-setup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,20 +10,24 @@ This guide walks through setting up ice-rest-catalog backed by a 3-node etcd clu

## 1. Start the etcd Cluster

Use the provided docker-compose file to bring up etcd, MinIO, ice-rest-catalog, and ClickHouse:
Use the provided 3-node docker-compose file to bring up etcd, MinIO, ice-rest-catalog, and ClickHouse:

```bash
cd examples/docker-compose
docker compose -f docker-compose-etcd.yaml up -d
docker compose -f docker-compose-etcd-3-node.yaml up -d
```

This starts a single-node etcd by default. For a 3-node cluster, replace the `etcd` service definition with three separate nodes (etcd1, etcd2, etcd3) each with their own ports mapped to the host:
By default, etcd ports are only exposed on the compose network. To run `etcdctl` from the host
(used in step 4), uncomment the `ports` block under each etcd service in
`docker-compose-etcd-3-node.yaml`. The host ports are:

| Node | Client Port |
|-------|-------------|
| etcd1 | 12379 |
| etcd2 | 12479 |
| etcd3 | 12579 |
| Node | Host Port | Container Port |
|--------|-----------|----------------|
| etcd-1 | 12379 | 2379 |
| etcd-2 | 12479 | 2379 |
| etcd-3 | 12579 | 2379 |

For a single-node setup, use `docker-compose-etcd.yaml` instead.

## 2. Configure ice-rest-catalog

Expand All @@ -48,10 +52,13 @@ Start (or restart) ice-rest-catalog so it picks up the new config.

## 3. Insert Data

Use the ice CLI to create a table and insert a Parquet file:
Use the ice CLI to create a table and insert a Parquet file. The example `iris.parquet` lives
under `examples/scratch/` — see [examples/scratch/iris.parquet.txt](../examples/scratch/iris.parquet.txt)
for the source link if you don't already have it locally:

```bash
ice insert flowers.iris file://iris.parquet
cd examples/scratch
ice insert flowers.iris -p file://iris.parquet
```

## 4. Verify Replication with etcdctl
Expand Down
184 changes: 184 additions & 0 deletions examples/docker-compose/docker-compose-etcd-3-node.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@
# Iceberg REST catalog with a 3-member etcd cluster (static bootstrap).
#
# First-time / fresh data:
# docker compose -f docker-compose-etcd-3-node.yaml up
#
# Coming from a single-node stack (see docker-compose-etcd-1-instance.yaml):
# - If you do NOT need to keep etcd data: stop the 1-node stack, remove the old
# etcd volume if any, then start this file (3 fresh members).
# - If you MUST keep the existing data: you cannot re-bootstrap a new 3-node
# static cluster from the same data dir. Add members to the *running* 1-node
# cluster with `etcdctl member add` and start the new nodes with
# --initial-cluster-state=existing (see etcd "runtime reconfiguration" docs).
#
# Usage: docker compose -f docker-compose-etcd-3-node.yaml up
services:
etcd-1:
image: quay.io/coreos/etcd:v3.5.12
restart: unless-stopped
command:
- etcd
- --name=etcd-1
- --data-dir=/etcd-data
- --listen-client-urls=http://0.0.0.0:2379
- --advertise-client-urls=http://etcd-1:2379
- --listen-peer-urls=http://0.0.0.0:2380
- --initial-advertise-peer-urls=http://etcd-1:2380
- --initial-cluster=etcd-1=http://etcd-1:2380,etcd-2=http://etcd-2:2380,etcd-3=http://etcd-3:2380
- --initial-cluster-token=ice-rest-catalog-etcd
- --initial-cluster-state=new
volumes:
- etcd-1-data:/etcd-data
healthcheck:
test: ["CMD", "etcdctl", "--endpoints=http://127.0.0.1:2379", "endpoint", "health"]
interval: 5s
timeout: 3s
retries: 15
# Uncomment to run etcdctl from the host against this node:
# ports:
# - "12379:2379"

Copy link
Copy Markdown
Collaborator

@xieandrew xieandrew Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should add commented ports map like the single node setup:

# Uncomment to run etcdctl from the host against this cluster:
# ports:
#   - "12379:2379"

Port numbers should match the etc-cluster-setup docs (12379, 12479, 12579)

Copy link
Copy Markdown
Collaborator Author

@subkanthi subkanthi Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

etcd-2:
image: quay.io/coreos/etcd:v3.5.12
restart: unless-stopped
command:
- etcd
- --name=etcd-2
- --data-dir=/etcd-data
- --listen-client-urls=http://0.0.0.0:2379
- --advertise-client-urls=http://etcd-2:2379
- --listen-peer-urls=http://0.0.0.0:2380
- --initial-advertise-peer-urls=http://etcd-2:2380
- --initial-cluster=etcd-1=http://etcd-1:2380,etcd-2=http://etcd-2:2380,etcd-3=http://etcd-3:2380
- --initial-cluster-token=ice-rest-catalog-etcd
- --initial-cluster-state=new
volumes:
- etcd-2-data:/etcd-data
healthcheck:
test: ["CMD", "etcdctl", "--endpoints=http://127.0.0.1:2379", "endpoint", "health"]
interval: 5s
timeout: 3s
retries: 15
# Uncomment to run etcdctl from the host against this node:
# ports:
# - "12479:2379"

etcd-3:
image: quay.io/coreos/etcd:v3.5.12
restart: unless-stopped
command:
- etcd
- --name=etcd-3
- --data-dir=/etcd-data
- --listen-client-urls=http://0.0.0.0:2379
- --advertise-client-urls=http://etcd-3:2379
- --listen-peer-urls=http://0.0.0.0:2380
- --initial-advertise-peer-urls=http://etcd-3:2380
- --initial-cluster=etcd-1=http://etcd-1:2380,etcd-2=http://etcd-2:2380,etcd-3=http://etcd-3:2380
- --initial-cluster-token=ice-rest-catalog-etcd
- --initial-cluster-state=new
volumes:
- etcd-3-data:/etcd-data
healthcheck:
test: ["CMD", "etcdctl", "--endpoints=http://127.0.0.1:2379", "endpoint", "health"]
interval: 5s
timeout: 3s
retries: 15
# Uncomment to run etcdctl from the host against this node:
# ports:
# - "12579:2379"

minio:
image: minio/minio:RELEASE.2025-03-12T18-04-18Z
restart: unless-stopped
command: [ 'server', '/data', '--address', ':8999', '--console-address', ':9001' ]
environment:
MINIO_ROOT_USER: miniouser
MINIO_ROOT_PASSWORD: miniopassword
ports:
- '8999:8999' # 9000 is taken by clickhouse
- '9001:9001' # web console
volumes:
- minio:/data
minio-init:
image: minio/mc:RELEASE.2025-03-12T17-29-24Z
restart: on-failure # run once and exit
entrypoint: >
/bin/sh -c "
sleep 1; until /usr/bin/mc alias set local http://minio:8999 $$MINIO_ROOT_USER $$MINIO_ROOT_PASSWORD; do echo waiting for minio to start...; sleep 1; done;
/usr/bin/mc mb --ignore-existing local/bucket1;
exit 0;
"
environment:
MINIO_ROOT_USER: miniouser
MINIO_ROOT_PASSWORD: miniopassword
depends_on:
- minio
ice-rest-catalog:
image: altinity/ice-rest-catalog:${ICE_REST_CATALOG_TAG:-debug-with-ice-latest-master@sha256:9f5308309b98d2e0b76346cd29c22557076bfcb0070eb8f32b69b87932b9e37c}
pull_policy: ${ICE_REST_CATALOG_PULL_POLICY:-always}
restart: unless-stopped
ports:
- '5001:5000' # iceberg/http
configs:
- source: ice-rest-catalog-yaml
target: /etc/ice/ice-rest-catalog.yaml
depends_on:
etcd-1:
condition: service_healthy
etcd-2:
condition: service_healthy
etcd-3:
condition: service_healthy
minio-init:
condition: service_completed_successfully
clickhouse:
image: altinity/clickhouse-server:25.8.9.20496.altinityantalya-alpine
restart: unless-stopped
environment:
CLICKHOUSE_SKIP_USER_SETUP: "1" # insecure
ports:
- "8123:8123" # clickhouse/http
- "9000:9000" # clickhouse/native
configs:
- source: clickhouse-init
target: /docker-entrypoint-initdb.d/init-db.sh
volumes:
# for access to clickhouse-logs
- ./data/docker-compose/clickhouse/var/log/clickhouse-server:/var/log/clickhouse-server
# - ./config.xml:/etc/clickhouse-server/conf.d/config.xml
depends_on:
- ice-rest-catalog
configs:
clickhouse-init:
content: |
#!/bin/bash
exec clickhouse client --query $"
SET allow_experimental_database_iceberg = 1;

DROP DATABASE IF EXISTS ice;

CREATE DATABASE ice
ENGINE = DataLakeCatalog('http://ice-rest-catalog:5000')
SETTINGS catalog_type = 'rest',
auth_header = 'Authorization: Bearer foo',
storage_endpoint = 'http://minio:8999',
warehouse = 's3://bucket1';
"
ice-rest-catalog-yaml:
content: |
uri: etcd:http://etcd-1:2379,http://etcd-2:2379,http://etcd-3:2379
warehouse: s3://bucket1
s3:
endpoint: http://minio:8999
pathStyleAccess: true
accessKeyID: miniouser
secretAccessKey: miniopassword
region: minio
bearerTokens:
- value: foo
volumes:
minio:
etcd-1-data:
etcd-2-data:
etcd-3-data:
Loading