Wrap Nessie, Polaris, and S3 clients with auto-auth from env/config #73

New Issue

kert · 2026-03-21T20:57:01Z

kert commented

2026-03-21 20:57:01 +00:00

Problem

Nessie, Polaris, and S3 connections require manual HTTP setup with auth tokens, OAuth2 flows, and credential loading. Each notebook re-implements this.

Solution

Create thin client wrappers in src/api/clients/ that handle auth automatically:

`src/api/clients/nessie.py`

Reads cfg.lake.nessie.catalog_uri
Provides: list_refs(), get_content(ref, key), commit(), create_branch(), merge()
Returns Polars DataFrames where appropriate

`src/api/clients/polaris.py`

Reads cfg.lake.polaris.catalog_uri
OAuth2 client credentials from env (POLARIS_ROOT_SECRET)
Provides: list_catalogs(), create_catalog(), list_namespaces(), grant()

`src/api/clients/s3.py` (or extend rustfs client)

Reads env: RUSTFS_ACCESS_KEY, RUSTFS_SECRET_KEY
S3 endpoint from config or default http://rustfs:9000
Provides: list_buckets(), list_objects(bucket), get_object(), put_object()
Pre-configured buckets: lakehouse, gitea, gitea-lfs

Files

src/api/clients/nessie.py (new or extend existing)
src/api/clients/polaris.py (new)
src/api/clients/s3.py (new or extend rustfs)
Tests for each

## Problem Nessie, Polaris, and S3 connections require manual HTTP setup with auth tokens, OAuth2 flows, and credential loading. Each notebook re-implements this. ## Solution Create thin client wrappers in `src/api/clients/` that handle auth automatically: ### `src/api/clients/nessie.py` - Reads `cfg.lake.nessie.catalog_uri` - Provides: `list_refs()`, `get_content(ref, key)`, `commit()`, `create_branch()`, `merge()` - Returns Polars DataFrames where appropriate ### `src/api/clients/polaris.py` - Reads `cfg.lake.polaris.catalog_uri` - OAuth2 client credentials from env (`POLARIS_ROOT_SECRET`) - Provides: `list_catalogs()`, `create_catalog()`, `list_namespaces()`, `grant()` ### `src/api/clients/s3.py` (or extend rustfs client) - Reads env: `RUSTFS_ACCESS_KEY`, `RUSTFS_SECRET_KEY` - S3 endpoint from config or default `http://rustfs:9000` - Provides: `list_buckets()`, `list_objects(bucket)`, `get_object()`, `put_object()` - Pre-configured buckets: `lakehouse`, `gitea`, `gitea-lfs` ## Files - `src/api/clients/nessie.py` (new or extend existing) - `src/api/clients/polaris.py` (new) - `src/api/clients/s3.py` (new or extend rustfs) - Tests for each

kert added this to the P14: Notebook DX — connections, snippets, templates, S3 buckets milestone 2026-03-21 20:57:01 +00:00

kert added the api config labels 2026-03-21 20:57:01 +00:00

kert referenced this issue

2026-03-21 20:57:01 +00:00

Retrofit existing notebooks to use conf.connect factories #78

kert referenced this issue from a commit

2026-03-22 00:25:05 +00:00

P14: connection factories, client wrappers, snippets, template

kert referenced this issue

2026-03-22 00:25:42 +00:00

P14: Notebook DX — connections, snippets, template, S3 #79

kert closed this issue