Add connection factories to src/conf/ — duckdb, trino, nessie, polaris, bib, zotero #72

New Issue

kert · 2026-03-21T20:57:00Z

kert commented

2026-03-21 20:57:00 +00:00

Problem

Every notebook hand-rolls 5-10 lines of connection boilerplate. The same patterns repeat across acodb_explorer, pfs_calcs, bib_explorer, nessie_tutorial, etc.

Solution

Add connection factories to src/conf/__init__.py (or a new src/conf/connect.py):

from conf import connect

con = connect.duckdb()           # read-only DuckDB from cfg.db.aco
con = connect.duckdb('bib')      # DuckDB with bib database
store = connect.bib()            # bib.Store instance
con = connect.trino()            # Trino with catalog from cfg.lake.trino
client = connect.nessie()        # Nessie REST client (auto-auth)
client = connect.polaris()       # Polaris REST client (auto-OAuth2)
con = connect.zotero()           # SQLite read-only to Zotero DB
s3 = connect.s3()                # S3/RustFS client with creds from env

Each factory:

Reads config from stack.toml via cfg
Reads credentials from environment variables
Returns a ready-to-use connection/client
Has a q() helper for DuckDB/Trino that returns Polars DataFrames

Implementation:

# src/conf/connect.py
def duckdb(name: str = 'aco', *, read_only: bool = True):
    import duckdb as _duckdb
    return _duckdb.connect(str(path(f'db.{name}')), read_only=read_only)

def trino(*, catalog: str = '', user: str = 'kert'):
    import trino as _trino
    c = cfg.lake.trino
    return _trino.dbapi.connect(
        host=c.host, port=int(c.port),
        user=user, catalog=catalog or c.catalog,
    )

def bib():
    from bib.store import Store
    return Store(str(path('db.bib')))

def nessie():
    # Returns a simple HTTP client wrapper
    ...

def polaris():
    # Returns OAuth2-authenticated client
    ...

def s3(*, bucket: str = 'lakehouse'):
    # Returns boto3/RustFS client with env creds
    ...

Files

src/conf/connect.py (new)
src/conf/__init__.py (re-export connect)
Tests: tests/conf/test_connect.py

## Problem Every notebook hand-rolls 5-10 lines of connection boilerplate. The same patterns repeat across acodb_explorer, pfs_calcs, bib_explorer, nessie_tutorial, etc. ## Solution Add connection factories to `src/conf/__init__.py` (or a new `src/conf/connect.py`): ```python from conf import connect con = connect.duckdb() # read-only DuckDB from cfg.db.aco con = connect.duckdb('bib') # DuckDB with bib database store = connect.bib() # bib.Store instance con = connect.trino() # Trino with catalog from cfg.lake.trino client = connect.nessie() # Nessie REST client (auto-auth) client = connect.polaris() # Polaris REST client (auto-OAuth2) con = connect.zotero() # SQLite read-only to Zotero DB s3 = connect.s3() # S3/RustFS client with creds from env ``` Each factory: - Reads config from `stack.toml` via `cfg` - Reads credentials from environment variables - Returns a ready-to-use connection/client - Has a `q()` helper for DuckDB/Trino that returns Polars DataFrames ### Implementation: ```python # src/conf/connect.py def duckdb(name: str = 'aco', *, read_only: bool = True): import duckdb as _duckdb return _duckdb.connect(str(path(f'db.{name}')), read_only=read_only) def trino(*, catalog: str = '', user: str = 'kert'): import trino as _trino c = cfg.lake.trino return _trino.dbapi.connect( host=c.host, port=int(c.port), user=user, catalog=catalog or c.catalog, ) def bib(): from bib.store import Store return Store(str(path('db.bib'))) def nessie(): # Returns a simple HTTP client wrapper ... def polaris(): # Returns OAuth2-authenticated client ... def s3(*, bucket: str = 'lakehouse'): # Returns boto3/RustFS client with env creds ... ``` ## Files - `src/conf/connect.py` (new) - `src/conf/__init__.py` (re-export connect) - Tests: `tests/conf/test_connect.py`

kert added this to the P14: Notebook DX — connections, snippets, templates, S3 buckets milestone 2026-03-21 20:57:00 +00:00

kert added the config label 2026-03-21 20:57:00 +00:00

kert referenced this issue

2026-03-21 20:57:01 +00:00

Retrofit existing notebooks to use conf.connect factories #78

kert referenced this issue from a commit

2026-03-22 00:25:05 +00:00

P14: connection factories, client wrappers, snippets, template

kert referenced a pull request that will close this issue

2026-03-22 00:25:42 +00:00

P14: Notebook DX — connections, snippets, template, S3 #79

kert closed this issue