23 KiB
GROK_PLAN: deep-focus pane, then Postgres reference manager
You are Grok, implementing this plan in /home/kert/stack. You have no prior context, so read this whole file first, then the spec: docs/superpowers/specs/2026-11-14-reference-manager-deep-focus-design.md. The spec says why. This file says what to do, in order. Do tasks in order, starting with Phase 0. Stop at each STOP and report to the user.
The user wants the pane (Phase A) running first. Phases B, C and D come after A is live and the user confirms.
Goal
A reviewer opens a source in a side pane, highlights a span, picks a label with one key, and the highlight is stored as a typed annotation that (1) sets a tag, (2) can be exported to a Zotero note, and (3) becomes labelled training data. No swivel chair: lex and lead open the pane in place.
Global constraints (binding)
Git
- Commit straight to
mainin/home/kert/stack. No branches, worktrees, PRs, merge commits orgit stash. The remote isgitea, notorigin. git addexplicit paths only. Rungit status --shortbefore every commit. If anything you didn't stage is staged, stop and ask.- The working tree holds uncommitted drugs and market work from other sessions. The user has authorized you to review, test, stage and commit it (Phase 0 below). That authorization covers only the files Phase 0 lists. Everything else you did not write stays untouched: if
git status --shortshows other modified files (for examplesrc/llm/dossier.py,evidence.py,rag.py,web/chat.html), don't stage, edit, format or revert them, and report them. - Commit messages: conventional prefix (
feat(focus): ...), noCo-Authored-Byor other trailer. - Never run a write or ingest command from a worktree.
Running things
- Live commands:
bash -c 'cd /home/kert/stack && set -a && . ./.env && set +a && uv run --no-sync <cmd>', called ENV below. - Tests:
uv run --no-sync pytest -q -p no:cacheprovider -n auto <paths>. Lint:uv run --no-sync ruff format <paths> && uv run --no-sync ruff check <paths>(line length 88, rules E/F/I). - CI enforces
--cov-fail-under=99: every branch of new Python needs a test. Mark code that needs live infrastructure# pragma: no coverthe wayllm/index.py::_enginedoes. - Tests must need no network, live DuckDB, Ollama or Postgres unless marked
live(skipped when absent). Use SQLAlchemy against SQLite in-memory or a fake engine for unit tests, astests/llm/test_migrate.pydoes. Read it first. - JS tests, when you add them:
docker run --rm -v /home/kert/stack:/w:ro -w /w node:22-slim sh -c 'node --test tests/focus/*.test.mjs'. Pass means# fail 0. - Don't run
docker compose down. Run compose only from/home/kert/stack. Probing compose-internal UIs: see~/.claude/projects/-home-kert-stack/memory/compose_net_headless_probe.md.
Content rules
- Never alter commenter names; show them as posted.
- No AMA/CPT descriptor text;
cpt_bookitems never get a viewer link. - No cloud LLM APIs anywhere. The llm module is local-only.
- Never spoof a browser user agent.
- A model suggestion is never stored as a human label.
provenanceishumanormodel. hl:*tags belong to humans.llm tagreplaces onlyllm:*, so keep the namespaces disjoint.
Existing code to read before Phase A
src/llm/api.py(the FastAPI app:_page,/ui/pdf/{key},/pdf/{key},/pdf/{key}/file,_KEY_RE,_VENDOR)src/llm/web/pdf.html(the viewer you extend)src/llm/pdfs.py(how a key resolves to files)src/llm/migrate.pyandtests/llm/test_migrate.py(idempotent DDL pattern)src/llm/index.py(_engine,pg_url) andsrc/llm/source.py(the canonical extracted text the indexer uses)src/lead/review/codes.pyandsrc/lead/review/store.py(PRISMA screening labels today)src/lead/explore/review_page.py(current labeling page)src/bib/store.pyandsrc/bib/tag.py(tag write path)infra/gitea/custom/public/assets/dossier/(lex ES modules; evidence drawer is inshell.js/view-sources.js)tests/llm/test_api.py(how API tests call the app)
Phase 0: land the pending drugs and market work (do this first)
The user wants the uncommitted drugs and market work staged and committed, as part of this plan. It was written by other sessions, so verify before you commit, don't just stage it.
Files as of 2026-11-14 (re-run git status --short first; the list may have changed, and anything not in these groups is out of scope):
- drugs:
src/cli/drugs.py,src/drugs/tables.py,src/drugs/xwalk.py(new),tests/cli/test_drugs_cli.py,tests/drugs/test_imports.py,tests/drugs/test_tables.py,tests/drugs/test_xwalk.py(new) - market:
src/cli/market.py,src/market/tables.py,src/market/lda.py(new),tests/cli/test_market_cli.py,tests/market/test_imports.py,tests/market/test_tables.py,tests/market/test_lda.py(new),tests/market/fixtures/lda_aafp_2024_q1.json(new),docs/docs/cli/market.md,market-entities.md,market-fts.md,market-gaps.md,market-prices.md(all new, underdocs/docs/cli/) - yours, not theirs:
GROK_PLAN.mdanddocs/superpowers/specs/2026-11-14-reference-manager-deep-focus-design.md(commit these together asdocs(focus): reference manager + deep-focus design and plan)
Steps:
- Read each diff. Context:
docs/superpowers/plans/2026-11-12-market-tier.mdand2026-11-13-drug-product-tier.mdsay what this work is for. Check the code against them. - Run
uv run --no-sync pytest -q -p no:cacheprovider -n auto tests/drugs tests/market tests/cli/test_drugs_cli.py tests/cli/test_market_cli.py, then ruff format and check on the touched paths, and check coverage ofsrc/drugsandsrc/market(CI floor is 99%). If anything fails, fix only the failure inside these files and say what you changed. If a failure needs a design call, stop and ask. - Content rules still apply: no CPT descriptor text (AMA), nothing from WNOT/Wakely or ACO-specific material (this repo hoards public and government data only), no secrets or API keys in fixtures (
lda_aafp_2024_q1.jsonis a lobbying-disclosure fixture; check it holds only public filing data). - Commit in logical groups, with explicit
git add <paths>each time andgit status --shortbefore each commit: one commit for drugs, one for market (code, tests, fixture), one for the market CLI docs, one for the design and plan docs. Conventional prefixes (feat(drugs): ...,feat(market): ...,docs(market): ...), no trailers. - Do not push unless the user says to (remote is
gitea;mainrejects force-push). - Report the commit hashes and test results, then continue to Phase A.
Phase A: the pane, running (current stores, annotations in Postgres)
A1. Package and protocols
Create src/focus/__init__.py, src/focus/protocols.py, src/focus/protocols/*.yaml, tests/focus/__init__.py, tests/focus/test_protocols.py.
- Protocol YAML shape:
name,version(int),title,scope(free text),labels: [{id, text, key, tag}].keyis a single character for the keyboard shortcut;tagdefaults tohl:<name>:<id>. protocols.load(name) -> Protocol(frozen dataclass) validates: unique label ids and keys, key is one character, ids match^[a-z0-9-]+$.protocols.available()lists names. Bad files raiseValueErrorwith the filename.- Ship four protocols:
comment-stance:support,support-with-modifications,do-not-support.cms-disposition:finalized-as-proposed,finalized-with-modifications,considered-for-future-rulemaking,other.prisma-screen: build the label list fromlead.review.codes.reason_codesat load time (read that module; use the real codes, don't invent). Write the YAML with afrom: lead.review.codesmarker and haveprotocols.loadexpand it; test it against the real codes.code-family: one label per element of a code family. Look atsrc/pfscode-family modules anddocs/superpowers/specs/2026-11-11-code-family-dossier-design.mdfor the element names. If they aren't clear, create the protocol with labelscode,descriptor-change,payment-change,rationale,other, and flag it in your report as a placeholder.
- TDD: tests first for validation errors, expansion,
available().
Commit: feat(focus): review protocols as YAML label sets.
A2. Annotation store (Postgres)
Create src/focus/store.py, src/focus/schema.py, tests/focus/test_store.py.
Tables (idempotent DDL strings in schema.py, applied by schema.ensure(engine); also call it from llm.migrate.migrate at the end so stack llm migrate creates them):
CREATE TABLE IF NOT EXISTS focus_annotation (
id uuid PRIMARY KEY,
item_key text NOT NULL,
attachment text NOT NULL DEFAULT '',
protocol text NOT NULL,
protocol_version int NOT NULL,
label text NOT NULL,
quote text NOT NULL,
prefix text NOT NULL DEFAULT '',
suffix text NOT NULL DEFAULT '',
start_char int, end_char int,
page int, rects jsonb,
note text NOT NULL DEFAULT '',
actor text NOT NULL,
provenance text NOT NULL CHECK (provenance IN ('human','model')),
model text,
created_at timestamptz NOT NULL DEFAULT now(),
deleted_at timestamptz
);
CREATE INDEX IF NOT EXISTS focus_annotation_item ON focus_annotation (item_key) WHERE deleted_at IS NULL;
CREATE TABLE IF NOT EXISTS focus_link (
from_id uuid NOT NULL REFERENCES focus_annotation(id),
to_id uuid NOT NULL REFERENCES focus_annotation(id),
kind text NOT NULL CHECK (kind IN ('responds_to','supports','contradicts')),
actor text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
deleted_at timestamptz,
PRIMARY KEY (from_id, to_id, kind)
);
CREATE TABLE IF NOT EXISTS focus_event (
id bigserial PRIMARY KEY,
annotation_id uuid NOT NULL,
op text NOT NULL,
payload jsonb NOT NULL,
actor text NOT NULL,
at timestamptz NOT NULL DEFAULT now()
);
The tests run on SQLite, which has no uuid, jsonb or timestamptz. So store.py takes the engine and uses SQLAlchemy Core with sqlalchemy.JSON and string uuids. Write the DDL per dialect in schema.py: Postgres DDL as above, SQLite-compatible DDL for tests. Test both strings are applied by ensure for their dialect (assert the Postgres branch by dialect name with a fake engine).
store.Store(engine) methods, each writing one focus_event row in the same transaction:
add(item_key, attachment, protocol, label, quote, prefix, suffix, start_char, end_char, page, rects, note, actor, provenance='human', model=None) -> Annotation. It validates: protocol exists, label is in the protocol,quotenon-empty,provenance=='model'requiresmodel.list(item_key, protocol=None, include_deleted=False).update_note(id, note, actor),relabel(id, label, actor),delete(id, actor)(setsdeleted_at; never hard-deletes).link(from_id, to_id, kind, actor),unlink(...)(soft).history(annotation_id)returns events oldest first.
TDD covering every branch: unknown protocol, label not in protocol, empty quote, model without model id, soft delete hides from list, relabel records the event, link/unlink.
Commit: feat(focus): annotation store with event log.
A3. Tag driving
Create src/focus/tags.py, tests/focus/test_tags.py.
tag_for(protocol, label) -> strreturns thetagfield (defaulthl:<name>:<id>).sync_item_tags(bib_store, item_key, annotations): the item'shl:*tag set becomes exactly the tags of its live human annotations. It never touches other tags. Readsrc/bib/tag.pyandsrc/bib/store.pyfor the real tag API (Store.set_tags/similar); don't write raw SQL tobib.sqlite.- A missing bib row (a Zotero-only key) is not an error: return
{"skipped": "no bib row"}. - Tests with a temp bib store: add, relabel and delete round-trip; non-
hl:tags (llm:*,sup:*,source:*) are untouched.
Commit: feat(focus): highlights drive hl: tags.
A4. API
Create src/focus/api.py (a FastAPI APIRouter), mount it in src/llm/api.py with app.include_router(...) at /api/focus. Tests: tests/focus/test_api.py.
Endpoints (all validate the key with _KEY_RE, reuse the identity header the app already trusts for the actor; read how llm/api.py gets the SSO user and fall back to "unknown"):
GET /api/focus/protocols→ names, titles, labels.GET /api/focus/{key}/annotations?protocol=→ live annotations.POST /api/focus/{key}/annotationsbody:attachment, protocol, label, quote, prefix, suffix, start_char, end_char, page, rects, note. Alwaysprovenance='human'. Returns 201 with the annotation; callstags.sync_item_tags.PATCH /api/focus/annotations/{id}(label and/or note),DELETE /api/focus/annotations/{id}.POST /api/focus/linksandDELETE /api/focus/links.GET /api/focus/{key}/text?attachment=→ the canonical extracted text for anchoring, from the same source as the indexer (llm.source). Read it to find the right function. If none exists for a single item, add a small helper there with tests. Don't touch the dirty files listed in Global constraints (src/llm/dossier.py,evidence.py,rag.py,web/chat.html, if they are still modified).- 404 for a bad key or unknown annotation, 422 for validation errors, 400 for a label outside the protocol.
Tests use the repo's existing client pattern and an in-memory SQLite engine injected through a FastAPI dependency (get_store). Cover every status code.
Wire the engine: get_store builds the real engine via llm.index._engine(cfg) in production, # pragma: no cover. Note that tests override the dependency.
Commit: feat(focus): /api/focus router.
A5. The pane UI
Edit src/llm/web/pdf.html (read it fully first; preserve all current behavior: file list, search, page, docx and text renderers). Keep it framework-free like the other pages. Put new JS in src/llm/web/focus.js (served by adding it to _VENDOR-style allow-list in llm/api.py at /ui/focus.js; add to the allow-list test).
Behavior:
- A protocol picker in the header (
?protocol=in the URL, default last used via URL only, no storage). - Selecting text in the PDF text layer, text view or docx view opens a small toolbar next to the selection showing the protocol's labels with their key hints. Pressing a label key or clicking applies it.
Esccancels. - On apply: compute
quote,prefixandsuffix(up to 32 chars each side),start_char/end_charas offsets into the/api/focus/{key}/textstring (for PDFs the offsets come from matching the quote with its prefix/suffix against that text; if no unique match, leave offsets null and still store the quote),pageandrects(PDF only, relative to the page viewport at scale 1). POST; paint the highlight with the label's color class. - A right-hand panel lists the item's annotations for the protocol. Clicking one jumps to its page and scrolls to the highlight. Each has a delete button and an editable note.
- Link mode: a "link to…" button on a
cms-dispositionannotation, then click comment-stance annotations (from other items too: type the other item's key) to createresponds_tolinks. - Orphaned annotations (offsets and quote not found on re-render) show in the panel with an "orphaned" badge rather than vanishing.
- State in the URL:
?file=&page=&protocol=&ann=<id>. Opening withannscrolls to it. - Accessibility: toolbar buttons are real buttons; label colors never carry meaning alone (the text label always shows).
Extract the pure functions to src/llm/web/focus_anchor.mjs (selection → selectors; re-anchor by position, then quote with prefix/suffix, then fuzzy; offset math) and test them with node:test in tests/focus/anchor.test.mjs, run through tests/focus/test_js.py the way tests/lex/test_js.py does (copy that pattern). Cover: unique quote, repeated quote disambiguated by prefix/suffix, text changed slightly (fuzzy), absent (orphaned), unicode.
Commit in small steps (feat(focus): selection toolbar, feat(focus): annotation panel, feat(focus): link mode).
A6. Open it in place from lex and lead
- lex (
infra/gitea/custom/public/assets/dossier/, vanilla ES modules): in the evidence drawer, where a source with a library key currently shows an external link, add an "Open in focus pane" action that loads/ui/pdf/<key>?protocol=...in an iframe inside the drawer (sandbox="allow-scripts allow-same-origin"). Respect R10: never construct a citation URL; the pane is not a citation. Respect the AMA rule: no pane link forcite.kind == "cpt_book". Add a JS test intests/lex/following the existing.test.mjsstyle. lex lives onlex.<domain>and the llm service is a different host, so check howlex-apiroutes/api/policyininfra/traefik/dynamic/services.yml. You'll need a similar router for/ui/pdf,/ui/focus.js,/pdf/and/api/focuson the lex host so the iframe is same-origin, or use the llm host and confirmContent-Security-Policy/X-Frame-Optionsallow it. Touchingservices.ymlandauth-handler.confchanges live routing: follow the lex-S1 plan's Task 11 caution (docs/superpowers/plans/2026-11-13-lex-s1.md), validate YAML, and report before applying. - lead (
src/lead/explore/, Streamlit): inreview_page.py, replace the "go read the PDF elsewhere" step withst.components.v1.iframeto the pane for the current item's key. Keep the existing decision/reason form; when the user highlights withprisma-screen, the existingStorelabel write still happens via the form (don't remove it). The pane write is additive.
A7. Zotero note export and gold-label bridge
Create src/focus/export.py, tests/focus/test_export.py; add a stack focus export-notes command in src/cli/focus.py (register it where other CLIs are registered in src/cli/; look at src/cli/zot.py for the pattern).
export_notes(store, zot_db, *, protocol=None, dry_run=True): for each item key that exists in Zotero (zot.db.Db), render ONE note per (item, protocol): an HTML list ofquote, label text and the user's note, each line carryingdata-focus-id="<annotation id>". Idempotent: if a note with the marker<!-- focus:<protocol> -->exists, replace its body, elseDb.add_note. Seesrc/zot/db.py::add_note(line ~693). Never touch other notes or tags on the item.- Dry run by default;
--writerequired to write. Never run this against the livezotero.sqlitewhile Zotero is running (the nightly zotero-sync workflow stops Zotero first); don't schedule it in this phase. Report the command and let the user wire it into the nightly job. - Skip items with no Zotero key (comments); count them in the summary.
- Bridge: a one-way
prisma-screenannotation is a piece of evidence for the existing gold label, not a replacement. Don't writelabels.sqlitefromexport.py.
A8. Deploy and verify (STOP after this)
- Run the full touched test set plus
tests/llm tests/lex tests/focus, ruff, and confirm coverage ofsrc/focusis 100%. ENV stack llm migratecreates the tables on the livellmdatabase. Verify with a read-only query thatfocus_annotationexists. Don't write test rows to the live database; use a throwaway key prefixZZTESTand delete (soft) after, or skip live writes and say so.- Roll the llm service per
~/.claude/projects/-home-kert-stack/memory/deploy_rollout.md(build and up from/home/kert/stack;COMMIT_SHAin.env). Roll lead too if you changedsrc/lead(it mounts./srcread-only, so it needs only a restart). - Headless probe of the pane through the compose network (see
compose_net_headless_probe.md): open/ui/pdf/<a real key>?protocol=comment-stance, select text, press1, reload, and confirm the highlight persists. Capture a screenshot into$S(session scratchpad), not the repo. - STOP. Report: what is live, the URL, what you couldn't verify, any placeholder (e.g.
code-familylabels), and the commands needed for the nightly note export. Wait for the user before Phase B.
Phase B: Postgres as the system of record (after the user confirms A)
Do not start without the user's go. Write a detailed task plan for B as docs/superpowers/plans/<date>-reference-manager-b.md, using this outline, and get the user's approval first.
- Schema + Pydantic models (
src/ref/models.py,src/ref/schema.py):Work,Creator,Identifier(kind, value),Attachment,Note,Tag(name, provenance, confidence),Chunk,WorkEventchange log. Identity: DOI > URL > legacy key.to_langchain()adapter only at the edge; no LangChain types in the model. - Migration
bib.sqlite→ Postgres:stack ref migrate --dry-runprints counts and identifier collisions first; resumable; verifies counts (items, attachments, notes, tags) against the source. Thefocus_annotation.item_keyvalues must still resolve (keep the legacy key as an identifier). - Writers move:
bib.Storewrites dual (SQLite + Postgres) behind a flag, then Postgres-only after parity checks; ingest (rex/comments,bib/federalregister,mail) uses the same upsert with a data-preserving rule (never blank an enriched field; see memorycomment_upsert_clobber_incident.md,p47_seal_and_skip.md). - Zotero sync one-way (
src/ref/zotero_sync.py): Postgres → Zotero, match by URL/DOI never by key, preservesup:tags (memory:zotero_rule_sync_incident.md,zotero_dedupe_2026_08.md), only items markedcited. - DuckDB replica export: a read-only
data/replica/ref.ro.duckdbof the work/annotation tables, so the portable single file survives. Followconf.connect._replica_path. - The nightly jobs and
leadstop readingzotero.sqlitedirectly (LEAD_ZOTERO_DB,llm/pdfs.py).
Phase C: strip the Zotero container (after B)
infra/images/zotero.Dockerfileand thezoteroservice incompose.yml: keep only Zotero desktop + its healthcheck; remove anything unused; drop the GPU reservation andruntime: nvidiaif nothing needs it; library = cited works only (small, so it loads fast).- A second container: a remote browser pane (KasmVNC/neko-style, a pinned image) on the
gatewaynetwork, reachable from the pane UI as a split screen for pages behind bot protection. It carries no automation, no spoofed UA; it's for a human. - Follow
~/.claude/projects/-home-kert-stack/memory/new_subdomain_pattern.mdif a new subdomain is needed.
Phase D: datasets and the predictive model (after B)
src/focus/datasets.py: export human-provenance annotations plusresponds_tolinks as rows(comment_stance, code_family, docket, rule, cms_disposition); split by hash of docket id (stable, never random); write a manifest (row count, content hash, protocol versions).provenance='model'rows excluded unlessinclude_model=True.src/focus/train.pyimports torch lazily and is the only module that does. Baseline first: logistic regression on stored embeddings; report held-out metrics per docket before anything heavier. The local GPU fleet is in memorygpu_fleet.md.- Model suggestions (
provenance='model') come from the local judge chain only (llm.pool), shown differently in the pane until a human accepts.
Report format after each phase
State what's committed (hashes), what's live, test and coverage numbers, what you couldn't verify, and decisions you made that the user should check. If you hit something this plan got wrong, say so and propose the fix. Don't silently deviate.