Pre-configure S3 buckets accessible from notebooks #74

New Issue

kert · 2026-03-21T20:57:01Z

kert commented

2026-03-21 20:57:01 +00:00

Problem

S3 buckets exist in RustFS but notebooks can't easily access them — no credentials are passed to the notebooks container, and there's no simple API.

Solution

1. Pass S3 credentials to notebooks container

Add to compose.yml notebooks environment:

environment:
  - PYTHONPATH=/home/kert/src
  - RUSTFS_ENDPOINT=http://rustfs:9000
  - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY}
  - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY}
  - NESSIE_S3_ACCESS_KEY=${NESSIE_S3_ACCESS_KEY}
  - NESSIE_S3_SECRET_KEY=${NESSIE_S3_SECRET_KEY}

2. Add notebooks to storage network

The notebooks container is only on gateway network. It needs storage network to reach rustfs and data network to reach nessie/trino.

3. Add stack.toml S3 config

[s3]
endpoint = "http://rustfs:9000"
region = "us-east-1"
buckets = ["lakehouse", "exports", "staging"]

4. Ensure standard buckets exist

Bootstrap script or init container that creates: lakehouse, exports, staging.

Files

compose.yml (networks + env vars)
stack.toml (S3 config)
src/conf/connect.py (S3 factory)

## Problem S3 buckets exist in RustFS but notebooks can't easily access them — no credentials are passed to the notebooks container, and there's no simple API. ## Solution ### 1. Pass S3 credentials to notebooks container Add to compose.yml notebooks environment: ```yaml environment: - PYTHONPATH=/home/kert/src - RUSTFS_ENDPOINT=http://rustfs:9000 - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY} - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY} - NESSIE_S3_ACCESS_KEY=${NESSIE_S3_ACCESS_KEY} - NESSIE_S3_SECRET_KEY=${NESSIE_S3_SECRET_KEY} ``` ### 2. Add notebooks to storage network The notebooks container is only on `gateway` network. It needs `storage` network to reach `rustfs` and `data` network to reach `nessie`/`trino`. ### 3. Add stack.toml S3 config ```toml [s3] endpoint = "http://rustfs:9000" region = "us-east-1" buckets = ["lakehouse", "exports", "staging"] ``` ### 4. Ensure standard buckets exist Bootstrap script or init container that creates: `lakehouse`, `exports`, `staging`. ## Files - `compose.yml` (networks + env vars) - `stack.toml` (S3 config) - `src/conf/connect.py` (S3 factory)

kert added this to the P14: Notebook DX — connections, snippets, templates, S3 buckets milestone 2026-03-21 20:57:01 +00:00

kert added the lakehouse config labels 2026-03-21 20:57:01 +00:00

kert referenced this issue

2026-03-21 20:57:01 +00:00

Retrofit existing notebooks to use conf.connect factories #78

kert referenced this issue from a commit

2026-03-22 00:25:05 +00:00

P14: connection factories, client wrappers, snippets, template

kert referenced this issue

2026-03-22 00:25:42 +00:00

P14: Notebook DX — connections, snippets, template, S3 #79

kert closed this issue