Files
stack/GROK_PLAN.md

23 KiB

GROK_PLAN: deep-focus pane, then Postgres reference manager

You are Grok, implementing this plan in /home/kert/stack. You have no prior context, so read this whole file first, then the spec: docs/superpowers/specs/2026-11-14-reference-manager-deep-focus-design.md. The spec says why. This file says what to do, in order. Do tasks in order, starting with Phase 0. Stop at each STOP and report to the user.

The user wants the pane (Phase A) running first. Phases B, C and D come after A is live and the user confirms.

Goal

A reviewer opens a source in a side pane, highlights a span, picks a label with one key, and the highlight is stored as a typed annotation that (1) sets a tag, (2) can be exported to a Zotero note, and (3) becomes labelled training data. No swivel chair: lex and lead open the pane in place.

Global constraints (binding)

Git

  • Commit straight to main in /home/kert/stack. No branches, worktrees, PRs, merge commits or git stash. The remote is gitea, not origin.
  • git add explicit paths only. Run git status --short before every commit. If anything you didn't stage is staged, stop and ask.
  • The working tree holds uncommitted drugs and market work from other sessions. The user has authorized you to review, test, stage and commit it (Phase 0 below). That authorization covers only the files Phase 0 lists. Everything else you did not write stays untouched: if git status --short shows other modified files (for example src/llm/dossier.py, evidence.py, rag.py, web/chat.html), don't stage, edit, format or revert them, and report them.
  • Commit messages: conventional prefix (feat(focus): ...), no Co-Authored-By or other trailer.
  • Never run a write or ingest command from a worktree.

Running things

  • Live commands: bash -c 'cd /home/kert/stack && set -a && . ./.env && set +a && uv run --no-sync <cmd>', called ENV below.
  • Tests: uv run --no-sync pytest -q -p no:cacheprovider -n auto <paths>. Lint: uv run --no-sync ruff format <paths> && uv run --no-sync ruff check <paths> (line length 88, rules E/F/I).
  • CI enforces --cov-fail-under=99: every branch of new Python needs a test. Mark code that needs live infrastructure # pragma: no cover the way llm/index.py::_engine does.
  • Tests must need no network, live DuckDB, Ollama or Postgres unless marked live (skipped when absent). Use SQLAlchemy against SQLite in-memory or a fake engine for unit tests, as tests/llm/test_migrate.py does. Read it first.
  • JS tests, when you add them: docker run --rm -v /home/kert/stack:/w:ro -w /w node:22-slim sh -c 'node --test tests/focus/*.test.mjs'. Pass means # fail 0.
  • Don't run docker compose down. Run compose only from /home/kert/stack. Probing compose-internal UIs: see ~/.claude/projects/-home-kert-stack/memory/compose_net_headless_probe.md.

Content rules

  • Never alter commenter names; show them as posted.
  • No AMA/CPT descriptor text; cpt_book items never get a viewer link.
  • No cloud LLM APIs anywhere. The llm module is local-only.
  • Never spoof a browser user agent.
  • A model suggestion is never stored as a human label. provenance is human or model.
  • hl:* tags belong to humans. llm tag replaces only llm:*, so keep the namespaces disjoint.

Existing code to read before Phase A

  • src/llm/api.py (the FastAPI app: _page, /ui/pdf/{key}, /pdf/{key}, /pdf/{key}/file, _KEY_RE, _VENDOR)
  • src/llm/web/pdf.html (the viewer you extend)
  • src/llm/pdfs.py (how a key resolves to files)
  • src/llm/migrate.py and tests/llm/test_migrate.py (idempotent DDL pattern)
  • src/llm/index.py (_engine, pg_url) and src/llm/source.py (the canonical extracted text the indexer uses)
  • src/lead/review/codes.py and src/lead/review/store.py (PRISMA screening labels today)
  • src/lead/explore/review_page.py (current labeling page)
  • src/bib/store.py and src/bib/tag.py (tag write path)
  • infra/gitea/custom/public/assets/dossier/ (lex ES modules; evidence drawer is in shell.js/view-sources.js)
  • tests/llm/test_api.py (how API tests call the app)

Phase 0: land the pending drugs and market work (do this first)

The user wants the uncommitted drugs and market work staged and committed, as part of this plan. It was written by other sessions, so verify before you commit, don't just stage it.

Files as of 2026-11-14 (re-run git status --short first; the list may have changed, and anything not in these groups is out of scope):

  • drugs: src/cli/drugs.py, src/drugs/tables.py, src/drugs/xwalk.py (new), tests/cli/test_drugs_cli.py, tests/drugs/test_imports.py, tests/drugs/test_tables.py, tests/drugs/test_xwalk.py (new)
  • market: src/cli/market.py, src/market/tables.py, src/market/lda.py (new), tests/cli/test_market_cli.py, tests/market/test_imports.py, tests/market/test_tables.py, tests/market/test_lda.py (new), tests/market/fixtures/lda_aafp_2024_q1.json (new), docs/docs/cli/market.md, market-entities.md, market-fts.md, market-gaps.md, market-prices.md (all new, under docs/docs/cli/)
  • yours, not theirs: GROK_PLAN.md and docs/superpowers/specs/2026-11-14-reference-manager-deep-focus-design.md (commit these together as docs(focus): reference manager + deep-focus design and plan)

Steps:

  1. Read each diff. Context: docs/superpowers/plans/2026-11-12-market-tier.md and 2026-11-13-drug-product-tier.md say what this work is for. Check the code against them.
  2. Run uv run --no-sync pytest -q -p no:cacheprovider -n auto tests/drugs tests/market tests/cli/test_drugs_cli.py tests/cli/test_market_cli.py, then ruff format and check on the touched paths, and check coverage of src/drugs and src/market (CI floor is 99%). If anything fails, fix only the failure inside these files and say what you changed. If a failure needs a design call, stop and ask.
  3. Content rules still apply: no CPT descriptor text (AMA), nothing from WNOT/Wakely or ACO-specific material (this repo hoards public and government data only), no secrets or API keys in fixtures (lda_aafp_2024_q1.json is a lobbying-disclosure fixture; check it holds only public filing data).
  4. Commit in logical groups, with explicit git add <paths> each time and git status --short before each commit: one commit for drugs, one for market (code, tests, fixture), one for the market CLI docs, one for the design and plan docs. Conventional prefixes (feat(drugs): ..., feat(market): ..., docs(market): ...), no trailers.
  5. Do not push unless the user says to (remote is gitea; main rejects force-push).
  6. Report the commit hashes and test results, then continue to Phase A.

Phase A: the pane, running (current stores, annotations in Postgres)

A1. Package and protocols

Create src/focus/__init__.py, src/focus/protocols.py, src/focus/protocols/*.yaml, tests/focus/__init__.py, tests/focus/test_protocols.py.

  • Protocol YAML shape: name, version (int), title, scope (free text), labels: [{id, text, key, tag}]. key is a single character for the keyboard shortcut; tag defaults to hl:<name>:<id>.
  • protocols.load(name) -> Protocol (frozen dataclass) validates: unique label ids and keys, key is one character, ids match ^[a-z0-9-]+$. protocols.available() lists names. Bad files raise ValueError with the filename.
  • Ship four protocols:
    • comment-stance: support, support-with-modifications, do-not-support.
    • cms-disposition: finalized-as-proposed, finalized-with-modifications, considered-for-future-rulemaking, other.
    • prisma-screen: build the label list from lead.review.codes.reason_codes at load time (read that module; use the real codes, don't invent). Write the YAML with a from: lead.review.codes marker and have protocols.load expand it; test it against the real codes.
    • code-family: one label per element of a code family. Look at src/pfs code-family modules and docs/superpowers/specs/2026-11-11-code-family-dossier-design.md for the element names. If they aren't clear, create the protocol with labels code, descriptor-change, payment-change, rationale, other, and flag it in your report as a placeholder.
  • TDD: tests first for validation errors, expansion, available().

Commit: feat(focus): review protocols as YAML label sets.

A2. Annotation store (Postgres)

Create src/focus/store.py, src/focus/schema.py, tests/focus/test_store.py.

Tables (idempotent DDL strings in schema.py, applied by schema.ensure(engine); also call it from llm.migrate.migrate at the end so stack llm migrate creates them):

CREATE TABLE IF NOT EXISTS focus_annotation (
  id uuid PRIMARY KEY,
  item_key text NOT NULL,
  attachment text NOT NULL DEFAULT '',
  protocol text NOT NULL,
  protocol_version int NOT NULL,
  label text NOT NULL,
  quote text NOT NULL,
  prefix text NOT NULL DEFAULT '',
  suffix text NOT NULL DEFAULT '',
  start_char int, end_char int,
  page int, rects jsonb,
  note text NOT NULL DEFAULT '',
  actor text NOT NULL,
  provenance text NOT NULL CHECK (provenance IN ('human','model')),
  model text,
  created_at timestamptz NOT NULL DEFAULT now(),
  deleted_at timestamptz
);
CREATE INDEX IF NOT EXISTS focus_annotation_item ON focus_annotation (item_key) WHERE deleted_at IS NULL;
CREATE TABLE IF NOT EXISTS focus_link (
  from_id uuid NOT NULL REFERENCES focus_annotation(id),
  to_id uuid NOT NULL REFERENCES focus_annotation(id),
  kind text NOT NULL CHECK (kind IN ('responds_to','supports','contradicts')),
  actor text NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now(),
  deleted_at timestamptz,
  PRIMARY KEY (from_id, to_id, kind)
);
CREATE TABLE IF NOT EXISTS focus_event (
  id bigserial PRIMARY KEY,
  annotation_id uuid NOT NULL,
  op text NOT NULL,
  payload jsonb NOT NULL,
  actor text NOT NULL,
  at timestamptz NOT NULL DEFAULT now()
);

The tests run on SQLite, which has no uuid, jsonb or timestamptz. So store.py takes the engine and uses SQLAlchemy Core with sqlalchemy.JSON and string uuids. Write the DDL per dialect in schema.py: Postgres DDL as above, SQLite-compatible DDL for tests. Test both strings are applied by ensure for their dialect (assert the Postgres branch by dialect name with a fake engine).

store.Store(engine) methods, each writing one focus_event row in the same transaction:

  • add(item_key, attachment, protocol, label, quote, prefix, suffix, start_char, end_char, page, rects, note, actor, provenance='human', model=None) -> Annotation. It validates: protocol exists, label is in the protocol, quote non-empty, provenance=='model' requires model.
  • list(item_key, protocol=None, include_deleted=False).
  • update_note(id, note, actor), relabel(id, label, actor), delete(id, actor) (sets deleted_at; never hard-deletes).
  • link(from_id, to_id, kind, actor), unlink(...) (soft).
  • history(annotation_id) returns events oldest first.

TDD covering every branch: unknown protocol, label not in protocol, empty quote, model without model id, soft delete hides from list, relabel records the event, link/unlink.

Commit: feat(focus): annotation store with event log.

A3. Tag driving

Create src/focus/tags.py, tests/focus/test_tags.py.

  • tag_for(protocol, label) -> str returns the tag field (default hl:<name>:<id>).
  • sync_item_tags(bib_store, item_key, annotations): the item's hl:* tag set becomes exactly the tags of its live human annotations. It never touches other tags. Read src/bib/tag.py and src/bib/store.py for the real tag API (Store.set_tags/similar); don't write raw SQL to bib.sqlite.
  • A missing bib row (a Zotero-only key) is not an error: return {"skipped": "no bib row"}.
  • Tests with a temp bib store: add, relabel and delete round-trip; non-hl: tags (llm:*, sup:*, source:*) are untouched.

Commit: feat(focus): highlights drive hl: tags.

A4. API

Create src/focus/api.py (a FastAPI APIRouter), mount it in src/llm/api.py with app.include_router(...) at /api/focus. Tests: tests/focus/test_api.py.

Endpoints (all validate the key with _KEY_RE, reuse the identity header the app already trusts for the actor; read how llm/api.py gets the SSO user and fall back to "unknown"):

  • GET /api/focus/protocols → names, titles, labels.
  • GET /api/focus/{key}/annotations?protocol= → live annotations.
  • POST /api/focus/{key}/annotations body: attachment, protocol, label, quote, prefix, suffix, start_char, end_char, page, rects, note. Always provenance='human'. Returns 201 with the annotation; calls tags.sync_item_tags.
  • PATCH /api/focus/annotations/{id} (label and/or note), DELETE /api/focus/annotations/{id}.
  • POST /api/focus/links and DELETE /api/focus/links.
  • GET /api/focus/{key}/text?attachment= → the canonical extracted text for anchoring, from the same source as the indexer (llm.source). Read it to find the right function. If none exists for a single item, add a small helper there with tests. Don't touch the dirty files listed in Global constraints (src/llm/dossier.py, evidence.py, rag.py, web/chat.html, if they are still modified).
  • 404 for a bad key or unknown annotation, 422 for validation errors, 400 for a label outside the protocol.

Tests use the repo's existing client pattern and an in-memory SQLite engine injected through a FastAPI dependency (get_store). Cover every status code.

Wire the engine: get_store builds the real engine via llm.index._engine(cfg) in production, # pragma: no cover. Note that tests override the dependency.

Commit: feat(focus): /api/focus router.

A5. The pane UI

Edit src/llm/web/pdf.html (read it fully first; preserve all current behavior: file list, search, page, docx and text renderers). Keep it framework-free like the other pages. Put new JS in src/llm/web/focus.js (served by adding it to _VENDOR-style allow-list in llm/api.py at /ui/focus.js; add to the allow-list test).

Behavior:

  1. A protocol picker in the header (?protocol= in the URL, default last used via URL only, no storage).
  2. Selecting text in the PDF text layer, text view or docx view opens a small toolbar next to the selection showing the protocol's labels with their key hints. Pressing a label key or clicking applies it. Esc cancels.
  3. On apply: compute quote, prefix and suffix (up to 32 chars each side), start_char/end_char as offsets into the /api/focus/{key}/text string (for PDFs the offsets come from matching the quote with its prefix/suffix against that text; if no unique match, leave offsets null and still store the quote), page and rects (PDF only, relative to the page viewport at scale 1). POST; paint the highlight with the label's color class.
  4. A right-hand panel lists the item's annotations for the protocol. Clicking one jumps to its page and scrolls to the highlight. Each has a delete button and an editable note.
  5. Link mode: a "link to…" button on a cms-disposition annotation, then click comment-stance annotations (from other items too: type the other item's key) to create responds_to links.
  6. Orphaned annotations (offsets and quote not found on re-render) show in the panel with an "orphaned" badge rather than vanishing.
  7. State in the URL: ?file=&page=&protocol=&ann=<id>. Opening with ann scrolls to it.
  8. Accessibility: toolbar buttons are real buttons; label colors never carry meaning alone (the text label always shows).

Extract the pure functions to src/llm/web/focus_anchor.mjs (selection → selectors; re-anchor by position, then quote with prefix/suffix, then fuzzy; offset math) and test them with node:test in tests/focus/anchor.test.mjs, run through tests/focus/test_js.py the way tests/lex/test_js.py does (copy that pattern). Cover: unique quote, repeated quote disambiguated by prefix/suffix, text changed slightly (fuzzy), absent (orphaned), unicode.

Commit in small steps (feat(focus): selection toolbar, feat(focus): annotation panel, feat(focus): link mode).

A6. Open it in place from lex and lead

  • lex (infra/gitea/custom/public/assets/dossier/, vanilla ES modules): in the evidence drawer, where a source with a library key currently shows an external link, add an "Open in focus pane" action that loads /ui/pdf/<key>?protocol=... in an iframe inside the drawer (sandbox="allow-scripts allow-same-origin"). Respect R10: never construct a citation URL; the pane is not a citation. Respect the AMA rule: no pane link for cite.kind == "cpt_book". Add a JS test in tests/lex/ following the existing .test.mjs style. lex lives on lex.<domain> and the llm service is a different host, so check how lex-api routes /api/policy in infra/traefik/dynamic/services.yml. You'll need a similar router for /ui/pdf, /ui/focus.js, /pdf/ and /api/focus on the lex host so the iframe is same-origin, or use the llm host and confirm Content-Security-Policy/X-Frame-Options allow it. Touching services.yml and auth-handler.conf changes live routing: follow the lex-S1 plan's Task 11 caution (docs/superpowers/plans/2026-11-13-lex-s1.md), validate YAML, and report before applying.
  • lead (src/lead/explore/, Streamlit): in review_page.py, replace the "go read the PDF elsewhere" step with st.components.v1.iframe to the pane for the current item's key. Keep the existing decision/reason form; when the user highlights with prisma-screen, the existing Store label write still happens via the form (don't remove it). The pane write is additive.

A7. Zotero note export and gold-label bridge

Create src/focus/export.py, tests/focus/test_export.py; add a stack focus export-notes command in src/cli/focus.py (register it where other CLIs are registered in src/cli/; look at src/cli/zot.py for the pattern).

  • export_notes(store, zot_db, *, protocol=None, dry_run=True): for each item key that exists in Zotero (zot.db.Db), render ONE note per (item, protocol): an HTML list of quote, label text and the user's note, each line carrying data-focus-id="<annotation id>". Idempotent: if a note with the marker <!-- focus:<protocol> --> exists, replace its body, else Db.add_note. See src/zot/db.py::add_note (line ~693). Never touch other notes or tags on the item.
  • Dry run by default; --write required to write. Never run this against the live zotero.sqlite while Zotero is running (the nightly zotero-sync workflow stops Zotero first); don't schedule it in this phase. Report the command and let the user wire it into the nightly job.
  • Skip items with no Zotero key (comments); count them in the summary.
  • Bridge: a one-way prisma-screen annotation is a piece of evidence for the existing gold label, not a replacement. Don't write labels.sqlite from export.py.

A8. Deploy and verify (STOP after this)

  1. Run the full touched test set plus tests/llm tests/lex tests/focus, ruff, and confirm coverage of src/focus is 100%.
  2. ENV stack llm migrate creates the tables on the live llm database. Verify with a read-only query that focus_annotation exists. Don't write test rows to the live database; use a throwaway key prefix ZZTEST and delete (soft) after, or skip live writes and say so.
  3. Roll the llm service per ~/.claude/projects/-home-kert-stack/memory/deploy_rollout.md (build and up from /home/kert/stack; COMMIT_SHA in .env). Roll lead too if you changed src/lead (it mounts ./src read-only, so it needs only a restart).
  4. Headless probe of the pane through the compose network (see compose_net_headless_probe.md): open /ui/pdf/<a real key>?protocol=comment-stance, select text, press 1, reload, and confirm the highlight persists. Capture a screenshot into $S (session scratchpad), not the repo.
  5. STOP. Report: what is live, the URL, what you couldn't verify, any placeholder (e.g. code-family labels), and the commands needed for the nightly note export. Wait for the user before Phase B.

Phase B: Postgres as the system of record (after the user confirms A)

Do not start without the user's go. Write a detailed task plan for B as docs/superpowers/plans/<date>-reference-manager-b.md, using this outline, and get the user's approval first.

  1. Schema + Pydantic models (src/ref/models.py, src/ref/schema.py): Work, Creator, Identifier(kind, value), Attachment, Note, Tag(name, provenance, confidence), Chunk, WorkEvent change log. Identity: DOI > URL > legacy key. to_langchain() adapter only at the edge; no LangChain types in the model.
  2. Migration bib.sqlite → Postgres: stack ref migrate --dry-run prints counts and identifier collisions first; resumable; verifies counts (items, attachments, notes, tags) against the source. The focus_annotation.item_key values must still resolve (keep the legacy key as an identifier).
  3. Writers move: bib.Store writes dual (SQLite + Postgres) behind a flag, then Postgres-only after parity checks; ingest (rex/comments, bib/federalregister, mail) uses the same upsert with a data-preserving rule (never blank an enriched field; see memory comment_upsert_clobber_incident.md, p47_seal_and_skip.md).
  4. Zotero sync one-way (src/ref/zotero_sync.py): Postgres → Zotero, match by URL/DOI never by key, preserve sup: tags (memory: zotero_rule_sync_incident.md, zotero_dedupe_2026_08.md), only items marked cited.
  5. DuckDB replica export: a read-only data/replica/ref.ro.duckdb of the work/annotation tables, so the portable single file survives. Follow conf.connect._replica_path.
  6. The nightly jobs and lead stop reading zotero.sqlite directly (LEAD_ZOTERO_DB, llm/pdfs.py).

Phase C: strip the Zotero container (after B)

  • infra/images/zotero.Dockerfile and the zotero service in compose.yml: keep only Zotero desktop + its healthcheck; remove anything unused; drop the GPU reservation and runtime: nvidia if nothing needs it; library = cited works only (small, so it loads fast).
  • A second container: a remote browser pane (KasmVNC/neko-style, a pinned image) on the gateway network, reachable from the pane UI as a split screen for pages behind bot protection. It carries no automation, no spoofed UA; it's for a human.
  • Follow ~/.claude/projects/-home-kert-stack/memory/new_subdomain_pattern.md if a new subdomain is needed.

Phase D: datasets and the predictive model (after B)

  • src/focus/datasets.py: export human-provenance annotations plus responds_to links as rows (comment_stance, code_family, docket, rule, cms_disposition); split by hash of docket id (stable, never random); write a manifest (row count, content hash, protocol versions). provenance='model' rows excluded unless include_model=True.
  • src/focus/train.py imports torch lazily and is the only module that does. Baseline first: logistic regression on stored embeddings; report held-out metrics per docket before anything heavier. The local GPU fleet is in memory gpu_fleet.md.
  • Model suggestions (provenance='model') come from the local judge chain only (llm.pool), shown differently in the pane until a human accepts.

Report format after each phase

State what's committed (hashes), what's live, test and coverage numbers, what you couldn't verify, and decisions you made that the user should check. If you hit something this plan got wrong, say so and propose the fix. Don't silently deviate.