homelab/stack
1
0
Fork 0
Code Issues 243 Pull Requests 2 Actions 4 Packages Projects 1 Releases Wiki Activity

feat(bib): stack bib import-zotero — Zotero collection items with files into bib (refs #687)

Browse Source
This commit is contained in:
kert
2026-09-09 16:26:21 -04:00
parent 5009e9de6f
commit 0dfa04b888
5 changed files with 597 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
docs/superpowers/plans/2026-09-09-cpt-organizing-principles.md
View File

@@ -8,11 +8,11 @@
**Tech Stack:** Python 3.13 stdlib `zipfile` + `html.parser` (no new dependency; `lxml`/`bs4` only if already in `uv.lock` — check), DuckDB via `duckdb_batch`, SQLite `bib.Store`, Zotero SQLite read-only via `conf.connect.zotero()`, typer, pytest.
**Spec:** `docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md` §Decisions 1–3 (elements, lineage, derived families). This slice supplies the organizing principle the spec's family derivation lacked (Ruling: the CPT's own hierarchy outranks stem-token similarity).
**Spec:** `docs/superpowers/specs/2026-09-09-cpt-canonical-schema-design.md` (the books' structure and the derived schema) and `docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md` §Decisions 1–3. This slice supplies the organizing principle the family derivation lacked (Ruling: the CPT's own hierarchy outranks stem-token similarity).
## Global Constraints
- **Copyright.** CPT content is AMA-licensed material the user owns. It is stored only in the local DuckDB and replica for the user's own analysis. It is never indexed into the chat's pgvector corpus in this slice (the CPT bib items carry a tag the corpus indexer skips), never printed into docs, the tracker, or commit messages, and the notebook shows at most one code entry at a time as an illustration. Tests use a synthetic fixture in the manual's markup style, not copied CPT text.
- **The CPT items are corpus documents like any other.** They are imported into bib with their files and indexed into the chat corpus; the `llm:skip` tag is a generic capability and is NOT applied to them (Ruling C4). Tests use a synthetic fixture in the manual's markup style.
- Files are read in place from Zotero storage (`data/zotero/data/storage/<attKey>/<file>`, owned by the Zotero container uid — read-only, never chown) or from the bib copy `attach_file` makes.
- `pfs/cpt_epub.py` and `pfs/cpt_model.py` are pure (no DuckDB, no bib at import). Writes only inside `duckdb_batch("aco")`, `publish_replica("aco")` after.
- Hand families remain a minimum (Ruling 17); CPT subsection edges are additive to the slice-1 edges.
@@ -24,8 +24,8 @@
| File | Responsibility |
|---|---|
| `src/bib/zotero_import.py` (new) + `src/cli/bib.py` (modify) | `stack bib import-zotero --collection "AMA Coding Publications" --with-files [--tag …]`: Zotero items (+ attachments) → bib `Source` items tagged `source:ama module:coding year:YYYY llm:skip` |
| `src/llm/source.py` (modify) | `iter_corpus_refs` skips items tagged `llm:skip` |
| `src/bib/zotero_import.py` (new) + `src/cli/bib.py` (modify) | `stack bib import-zotero --collection "AMA Coding Publications" --with-files [--tag …]`: Zotero items (+ attachments) → bib `Source` items tagged `source:ama module:coding year:YYYY` |
| `src/llm/source.py` (modify) | `iter_corpus_refs` skips items tagged `llm:skip` (generic capability; not used for AMA items) |
| `src/pfs/cpt_model.py` (new) | dataclasses `CptSection`, `CptCode`, `CptInstruction`, `CptReference`, `CptEdition` |
| `src/pfs/cpt_epub.py` (new) | `parse_epub(path) -> CptEdition` (2021+ template; 2019 template via the same table classes) |
| `src/pfs/codetables.py` (modify) | DDL + writers/readers for `pfs.cpt_section`, `pfs.cpt_code`, `pfs.cpt_instruction`, `pfs.cpt_reference` |
@@ -50,10 +50,11 @@
def list_collection(zcon, name: str, *, with_files: bool) -> list[ZoteroBook] # sqlite3 connection from conf.connect.zotero(); resolves storage:<file> → storage/<attKey>/<file>
def import_books(store, books, *, tags: Sequence[str], copy_files: bool = True, dry_run=False) -> dict # Source items upserted by url (fallback url = f"zotero://select/library/items/{key}"), tags + year:YYYY, attach_file per attachment; {"created","updated","attached"}
```
`iter_corpus_refs` (and any other corpus walker) skips items tagged `llm:skip`. CLI: `stack bib import-zotero --collection "AMA Coding Publications" --with-files --tag source:ama --tag module:coding --tag llm:skip [--only "CPT"] [--dry-run]` (`--only` = title substring filter).
`iter_corpus_refs` (and any other corpus walker) skips items tagged `llm:skip`. CLI: `stack bib import-zotero --collection "AMA Coding Publications" --with-files --tag source:ama --tag module:coding [--only "CPT"] [--dry-run]` (`--only` = title substring filter).
- [ ] Failing tests: `list_collection` against a tiny sqlite built with the Zotero schema subset (`items, itemData, itemDataValues, fields, collections, collectionItems, itemAttachments, itemTypes`) — two books, one with a `storage:` attachment; `import_books` into a tmp `Store` creates Source items with the tags and attaches the file (copy); rerun → `updated/unchanged`, no duplicate attachment; `iter_corpus_refs` skips an item tagged `llm:skip`.
- [ ] Implement; run `uv run pytest tests/bib tests/llm/test_source.py -q`; live: `uv run stack bib import-zotero --collection "AMA Coding Publications" --with-files --only CPT --tag source:ama --tag module:coding --tag llm:skip` → report the bib keys of the six CPT editions (2018, 2019, 2021, 2022, 2023 Changes, 2024).
- [ ] After import: `stack llm index --collection corpus` picks the CPT items up on its next run (no special handling).
- [ ] Commit: `feat(bib): import Zotero collection items with files into bib; llm:skip tag honoured by the corpus indexer (refs #688)`.
---
@@ -89,7 +90,7 @@ Parsing rules (from the 2024/2022/2021 template): headings are `<div class="h1|h
- Modify: `src/pfs/codetables.py` (four DDLs, `write_cpt_edition(con, edition, item_key)`, `read_cpt_codes(con, year)`, `read_cpt_sections(con, year)`), Create: `src/pfs/cpt_load.py`, Modify: `src/cli/pfs.py`
- Test: `tests/pfs/test_cpt_load.py`, `tests/cli/test_pfs_cli.py` (extend)
Tables (all with `edition_year INTEGER, item_key VARCHAR` first): `pfs.cpt_section(sec_id, level, title, path, code_lo, code_hi, guideline)`, `pfs.cpt_code(code, sec_id, descriptor, stem, elements, tail, addon, resequenced, new, revised, telemedicine, parent)` (`elements` space-joined? No — `elements` is a `VARCHAR[]` list; DuckDB supports it), `pfs.cpt_instruction(code, kind, text, targets)` (`targets VARCHAR[]`), `pfs.cpt_reference(code, kind, years INTEGER[], text)`. Delete-then-insert per `edition_year`.
Tables (all with `edition_year INTEGER, item_key VARCHAR` first): `pfs.cpt_section(sec_id, level, title, path, code_lo, code_hi, guideline)`, `pfs.cpt_code(code, sec_id, descriptor, stem, elements, tail, addon, resequenced, new, revised, telemedicine, parent)` (`elements` space-joined? No — `elements` is a `VARCHAR[]` list; DuckDB supports it), `pfs.cpt_instruction(code, kind, text, targets)` (`targets VARCHAR[]`), `pfs.cpt_reference(code, kind, years INTEGER[], text)`, plus the appendix-derived tables from the schema spec `docs/superpowers/specs/2026-09-09-cpt-canonical-schema-design.md` §3: `pfs.cpt_change(code, kind, old_text, new_text, section)` from Appendix B, `pfs.cpt_crosswalk(current_code, former_code, year_deleted, citations)` from Appendix M, `pfs.cpt_list(appendix, code)` from Appendices D/E/F/G/K/N/P/T, `pfs.cpt_modifier(modifier, title, text)` from Appendix A. Delete-then-insert per `edition_year`.
`cpt_load.ingest(store, con, *, years: Sequence[int] | None) -> dict`: finds bib items tagged `source:ama` whose title matches `CPT (Professional )?(\d{4})` with an EPUB attachment, parses each, writes, returns rows per table per year. CLI `stack pfs cpt-ingest [--edition 2024]… [--all] [--dry-run]`; `--dry-run` parses and prints counts only.

70
docs/superpowers/specs/2026-09-09-cpt-canonical-schema-design.md Normal file
View File

@@ -0,0 +1,70 @@
# The canonical coding books as schema — CPT, CPT Changes, HCPCS, ICD-10-CM
Source review 2026-09-09 of the AMA books in the Zotero collection "AMA Coding Publications" (files on disk: CPT 2018 PDF; CPT 2019 PDF+EPUB; CPT 2021, 2022, 2024 Professional EPUB (+2024 PDF); CPT Changes 2023 EPUB; E/M Office Visit Compendium 2021 EPUB; HCPCS 2019 EPUB+PDF; ICD-10-CM 2016–2025 and ICD-10-PCS 2019/2022 EPUB; Coding with Modifiers 4th ed.; Netter's Atlas for CPT Coding 2015). The 2021/2022/2024 CPT EPUBs share one publisher template; 2019 uses the same table classes without `code_` ids.
Purpose: read the books' own organizing structure and turn it into the schema that P49's element model, lineage and family derivation should be built on. The Federal Register tells us what Medicare *pays for*; these books tell us what a code *is* and how the code set is *organized* — and that organization is the first principle for "code families".
## 1. What the CPT codebook says about its own structure
**Sections and number ranges** (Introduction › "Section Numbers and Their Sequences"): Evaluation and Management 99202–99499; Anesthesiology 00100–01999, 99100–99140; Surgery 10004–69990; Radiology 70010–79999; Pathology and Laboratory 80047–89398, 0001U–0419U; Medicine 90281–99199, 99500–99607, 0001A–0174A; then Category II (performance measurement, `NNNNF`) and Category III (emerging technology, `NNNNT`). Each section opens with **Guidelines** (definitions and reporting rules that apply to every code in the section), and the book states that placement in a section "may reflect historical or other considerations" — i.e. the hierarchy is a curated taxonomy, not a strict classification.
**Hierarchy below the section** (the book's own words in the E/M guidelines): a section is "divided into broad **categories** … most of the categories are further divided into two or more **subcategories** … further classified into **levels** identified by specific codes." In the EPUB this is `div.h1` (subsection, e.g. *Care Management Services*), `div.h2` (category, e.g. *Chronic Care Management Services*), `div.h3`/`h4` (subcategory), each with a `sec_N` id, and the chapter TOC prints every heading with its code span, e.g. `Care Management Services* (99490-99427)` → `Chronic Care Management Services* (99490-99437)`, `Complex Chronic Care Management Services* (99487-99489)`, `Principal Care Management Services* (99424-99427)`. The asterisk marks headings that carry their own guideline text. Spans are print-order, not numeric (resequenced codes).
**Code entry format** ("Format of the Terminology"): a code is a stand-alone description *unless* it is indented, in which case it inherits the parent's text before the semicolon — the book's example `25100 Arthrotomy, wrist joint; with biopsy` / `25105 with synovectomy`. Care-management codes use the "with the following required elements:" form: a stem sentence, a bulleted list of elements, and a closing time clause. Add-on entries ("each additional 20 minutes …") inherit the primary's stem and elements. In the EPUB: `td.td-w1[id=code_NNNNN]` → `div.table-para1` (stem), `div.table-slist*` (elements), `div.table-para2` (tail), `div.table-para1-sub` (indented child).
**Symbols** (Legend): `●` new code, `▲` revised code, `✚` add-on, `⦸` modifier-51 exempt, `▶ ◀` new or revised text, `★` telemedicine (audio-video), audio-only glyph, `⚡` FDA approval pending, `#` resequenced, duplicate-PLA glyph, `⇅` Category I PLA, `➲` citations (CPT Changes / CPT Assistant). In the EPUB every glyph is a `span.ama-en`.
**Parenthetical instructions** (Introduction › "Instructions"): notes with selected codes that "indicate that a code should not be reported with another code or codes"; explicitly "not all inclusive". Observed kinds, each a `div.table-para2` beginning with `(`:
- `(Use X in conjunction with Y)` — add-on binding;
- `(Do not report X, Y in the same calendar month with …)` — exclusivity within a period;
- `(Do not report X for service time reported with …)` — time double-counting exclusion;
- `(Do not report X more than twice per calendar month)` / `(… of less than 20 minutes … are not reported separately)` — frequency/threshold;
- `(… may be reported using 99487, 99489)` / `(For …, see …)` / `(… use NNNNN)` — cross-references;
- `(NNNNN has been deleted. To report, see …)` — deletion pointers.
ICD-10-CM has the same construct with different names (Excludes1 = mutually exclusive, Excludes2 = not included here, Code first / Use additional code = sequencing) — a useful cross-check that "instruction" is a first-class object in every canonical book.
**Per-code references**: `➲ CPT Changes: An Insider's View 2015, 2021, 2022` — the years the code was added or revised — and `➲ CPT Assistant Oct 14:3, Feb 15:3 …` — newsletter guidance citations. Both are `div.table-RT*` lines.
**Appendices that are authoritative lists** (2024 edition): A modifiers; **B Summary of Additions, Deletions, and Revisions** (the year's changes, with revised descriptors shown as strikethrough/underline diffs — descriptor-level change tracking); C clinical examples; **D add-on codes**; E modifier-51 exempt; F modifier-63 exempt; G moderate sedation included; H alphabetical clinical topics; I genetic testing modifiers; J electrodiagnostic nerves; K FDA-pending; L vascular families (a literal "family" taxonomy); **M renumbered codes citations crosswalk** (current ↔ former code, year deleted, 2007–2009); **N resequenced codes**; O MAAA/PLA; **P audio-video telemedicine codes**; Q COVID vaccines; R digital medicine services taxonomy; S AI taxonomy; **T audio-only telemedicine codes**. Appendices P/T carry the CPT Editorial Panel's own telemedicine criteria — the counterpart of the Medicare telehealth Steps in the FR.
**Annual cycle**: one edition per calendar year; codes effective January 1. *CPT Changes: An Insider's View* is the companion for each year: per section a tabular **Summary of Additions, Deletions, and Revisions**, then the new/revised entries reprinted with a shaded **Rationale** box and **Clinical Examples** (typical patient + description of procedure). This is the AMA's own "why" for each change — the CPT-side twin of the FR's Comment/Response.
## 2. What the other books add
- **HCPCS Level II** (CMS, annual + quarterly): letter-prefixed sections (A–V) by supply/service type; each entry = code, descriptor, coverage/payment indicators, cross-references to Pub 100 (IOM) and NCCI policy (the 2019 book reprints NCCI chapter 1 as an appendix); modifiers; table of drugs. G-codes are CMS's parallel to CPT for Medicare programmatic needs — the FR repeatedly says "we prefer CPT unless Medicare has a programmatic need" (CY2015 final ¶1249), and lineage often runs G-code → CPT (G2058 → 99439, G2064/5 → 99424/6).
- **E/M Office Visit Compendium 2021**: the 2021 E/M redesign explained — MDM vs time selection, prolonged services, and a tabular review of guideline changes; the elements of E/M levels are a distinct element vocabulary (history, exam, MDM components, time).
- **ICD-10-CM**: chapter → block → category → subcategory → code (7th-character extensions), official guidelines, instruction notes as above. Not a PFS object but the same schema shape; relevant later for diagnosis-conditioned coverage.
- **Coding with Modifiers**, **Netter's Atlas**: modifier semantics (Appendix A's narrative) and anatomy → code mapping; reference only.
## 3. Schema derived from the books
All tables in DuckDB schema `pfs`, keyed by `edition_year` with `item_key` = the edition's bib item (provenance). Text columns hold the book's text verbatim for the user's own analysis; the CPT items are indexed into the chat corpus like every other item in the library.
| table | grain | columns (beyond `edition_year, item_key`) | source |
|---|---|---|---|
| `cpt_section` | one heading | `sec_id, level (1–4), title, path (ancestor titles), parent_sec_id, code_lo, code_hi, has_guidelines, guideline_text` | body `div.hN`, chapter TOC spans |
| `cpt_code` | one code entry | `code, sec_id, category (I/II/III), stem, elements VARCHAR[], tail, descriptor_full (stem + elements + tail, child text expanded per the semicolon rule), parent_code (for indented/add-on rows), addon, resequenced, new, revised, mod51_exempt, telemedicine_av, audio_only, fda_pending, pla` | code rows + Legend glyphs |
| `cpt_instruction` | one parenthetical | `owner (code or sec_id), kind (use-with / not-with-period / not-with-time / frequency / cross-ref / deleted-pointer / other), text, targets VARCHAR[] (ranges expanded)` | `div.table-para2` beginning `(` and guideline-level notes |
| `cpt_reference` | one citation line | `code, kind (cpt-changes / cpt-assistant), years INTEGER[], text` | `div.table-RT*` |
| `cpt_change` | one change in one edition | `code, kind (added / revised / deleted), old_text, new_text, section` | Appendix B (strike/underline diff), cross-checked with CPT Changes summary tables |
| `cpt_rationale` | one rationale or clinical example | `code (or sec_id), kind (rationale / clinical-example), text` | *CPT Changes* rationale boxes and clinical examples |
| `cpt_crosswalk` | one renumbering | `current_code, former_code, year_deleted, citations` | Appendix M |
| `cpt_list` | one code in one appendix list | `appendix (D/E/F/G/K/N/P/T…), code` | Appendices D–T |
| `cpt_modifier` | one modifier | `modifier, title, text` | Appendix A |
HCPCS Level II gets the same shape later (`hcpcs_section`, `hcpcs_code`, `hcpcs_note`) from the CMS quarterly files already in the pipeline plus the book's coverage notes.
## 4. How the schema feeds P49
- **Elements**: `cpt_code.elements` *are* the required-elements list; `stem`/`tail` carry actor, time and period phrases. `pfs.extract` reads them as a source (`source="cpt"`) before the FR paragraph runs, so element extraction covers every CPT code, not only the ones the FR happens to reprint. The element vocabulary grows from the book's phrasing (e.g. E/M levels, "medical decision making", "total time on the date of the encounter").
- **Families**: the organizing principle is the CPT hierarchy. Family = the lowest heading that groups ≥ 2 codes (category or subcategory), named by the heading. Hand families keep their keys. Additional intra-family edges: `✚` + `use-with` (add-on → primary), the semicolon-inheritance parent (`parent_code`), and Appendix L vascular families. Cross-family edges are *not* created from `not-with` instructions (they express exclusivity across families, e.g. CCM vs PCM).
- **Lineage**: `cpt_change` gives added/revised/deleted per edition year with the descriptor diff; `cpt_reference` gives the CPT Changes years per code (the AMA's own revision history); `cpt_crosswalk` gives renumberings; `cpt_rationale` gives the AMA's reason. These are `source="cpt"` events beside the FR (`fr`) and RVU-file (`rvu`) events, and they anchor RVU events the same way FR events do.
- **Telehealth**: Appendix P/T membership per edition + the CPT criteria text sit beside the Medicare telehealth list and the FR Steps 1–3; the chat can show both bodies' reasoning for one code.
- **Chat**: the CPT items are corpus documents (chunked, embedded, code-anchored) so a question about a code retrieves the manual's guidelines and instructions, the FR's policy, the RVU valuation, and the comments' reaction together.
## 5. Gaps and decisions
- Editions on disk cover 2019, 2021, 2022, 2024 (EPUB) and 2018 (PDF only). 2020, 2023, 2025, 2026 codebooks are in Zotero as records without files; `since` years will have gaps until they land. Appendix B of each present edition still gives the previous year's changes (2024's Appendix B = changes effective 2024).
- The 2019 EPUB lacks `code_` ids; the parser keys on the `<b>NNNNN</b>` in `div.table-para` for that edition.
- PDFs are not parsed in this slice.
- `llm:skip` stays as a generic capability (an item the corpus should not embed) but is **not** applied to the AMA items; the seven tags added by the first import are removed.

231
src/bib/zotero_import.py Normal file
View File

@@ -0,0 +1,231 @@
"""Pull items (with files) from a Zotero collection into bib.
``bib.sync`` only pushes bib → Zotero. This is the reverse: read-only
access to the Zotero SQLite database (``conf.connect.zotero()``), so
material the user already curated in Zotero — the AMA CPT manuals, for
instance — can be cited by ``item_key`` like everything else in bib,
files included.
Usage::
from conf.connect import zotero, bib
zcon = zotero()
books = list_collection(zcon, "AMA Coding Publications", with_files=True)
stats = import_books(bib(), books, tags=["source:ama", "llm:skip"])
"""
from __future__ import annotations
import sqlite3
from dataclasses import dataclass
from pathlib import Path
from typing import TYPE_CHECKING, Sequence
if TYPE_CHECKING:
from bib.store import Store
_FIELD_NAMES = ("title", "url", "date", "publisher")
@dataclass(frozen=True)
class ZoteroBook:
"""One Zotero book item: metadata + resolved attachment paths."""
key: str
title: str
url: str
year: str
publisher: str
attachments: tuple[Path, ...]
def list_collection(
zcon: sqlite3.Connection,
name: str,
*,
with_files: bool,
storage_dir: Path | None = None,
) -> list[ZoteroBook]:
"""Book items in the named Zotero collection, ordered by item id.
Excludes anything in ``deletedItems`` (items or attachments). A
``storage:<file>`` attachment path resolves against *storage_dir*
(default ``conf.path("storage.zotero")``) at
``<storage_dir>/<attachment item key>/<file>`` — the layout the
Zotero desktop app uses on disk.
``with_files=True`` returns only items with at least one resolved
attachment — for a citation-plus-file import like the CPT editions,
a catalog stub with no book behind it isn't worth pulling in.
"""
if storage_dir is None:
from conf import path
storage_dir = path("storage.zotero")
storage_dir = Path(storage_dir)
rows = zcon.execute(
"SELECT i.itemID, i.key FROM items i "
"JOIN collectionItems ci ON ci.itemID = i.itemID "
"JOIN collections c ON c.collectionID = ci.collectionID "
"JOIN itemTypes it ON it.itemTypeID = i.itemTypeID AND it.typeName = 'book' "
"WHERE c.collectionName = ? "
"AND i.itemID NOT IN (SELECT itemID FROM deletedItems) "
"ORDER BY i.itemID",
(name,),
).fetchall()
books: list[ZoteroBook] = []
for item_id, key in rows:
meta = dict(
zcon.execute(
"SELECT f.fieldName, idv.value FROM itemData id "
"JOIN fields f ON f.fieldID = id.fieldID "
"JOIN itemDataValues idv ON idv.valueID = id.valueID "
"WHERE id.itemID = ? AND f.fieldName IN (?, ?, ?, ?)",
(item_id, *_FIELD_NAMES),
).fetchall()
)
att_rows = zcon.execute(
"SELECT a.key, ia.path FROM itemAttachments ia "
"JOIN items a ON a.itemID = ia.itemID "
"WHERE ia.parentItemID = ? "
"AND a.itemID NOT IN (SELECT itemID FROM deletedItems)",
(item_id,),
).fetchall()
attachments = tuple(
storage_dir / att_key / att_path[len("storage:") :]
for att_key, att_path in att_rows
if att_path and att_path.startswith("storage:")
)
if with_files and not attachments:
continue
date = meta.get("date", "") or ""
year = date[:4] if date[:4].isdigit() else ""
books.append(
ZoteroBook(
key=key,
title=meta.get("title", "") or "",
url=meta.get("url", "") or "",
year=year,
publisher=meta.get("publisher", "") or "",
attachments=attachments,
)
)
return books
def import_books(
store: "Store",
books: Sequence[ZoteroBook],
*,
tags: Sequence[str] = (),
copy_files: bool = True,
dry_run: bool = False,
) -> dict:
"""Upsert each :class:`ZoteroBook` into *store* as a ``Source`` item.
Dedup is by URL — Zotero items without one fall back to a stable
``zotero://select/library/items/<key>`` URL, so re-running the
import always resolves to the same bib item. Every item gets
*tags* plus ``year:<YYYY>`` when the book has a year; every
attachment is copied in via :meth:`Store.attach_file`, which is
idempotent by filename — re-running never duplicates a file.
``dry_run=True`` makes no writes: it reports what a real run would
do (bib keys are unknown for not-yet-created items, so ``items``
entries carry an empty ``key`` in that case).
Returns a summary dict: ``created``/``updated``/``unchanged``
counts items by :meth:`Store.upsert_status` outcome, ``attached``
counts newly-copied files (a re-run's dedup'd attachments don't
count), and ``items`` lists one record per book for reporting.
"""
from bib.item import Source
stats = {"created": 0, "updated": 0, "unchanged": 0, "attached": 0, "items": []}
for book in books:
url = book.url or f"zotero://select/library/items/{book.key}"
if dry_run:
existing = (
store._con()
.execute( # noqa: SLF001
"SELECT key FROM items WHERE url = ?", (url,)
)
.fetchone()
)
status = "unchanged" if existing else "created"
stats[status] += 1
n_new = 0
if copy_files:
have = (
{
r[0]
for r in store._con()
.execute( # noqa: SLF001
"SELECT a.filename FROM attachments a "
"JOIN items i ON i.id = a.item_id WHERE i.key = ?",
(existing[0],),
)
.fetchall()
}
if existing
else set()
)
n_new = sum(1 for p in book.attachments if p.name not in have)
stats["attached"] += n_new
stats["items"].append(
{
"key": existing[0] if existing else "",
"zotero_key": book.key,
"title": book.title,
"status": status,
"attached": n_new,
}
)
continue
item = Source(
title=book.title,
url=url,
date_published=book.year,
institution=book.publisher,
)
item_tags = [*tags, *([f"year:{book.year}"] if book.year else [])]
key, status = store.upsert_status(item, tags=item_tags)
stats[status] += 1
n_new = 0
if copy_files:
for att_path in book.attachments:
if not att_path.exists():
continue
dup = (
store._con()
.execute( # noqa: SLF001
"SELECT 1 FROM attachments a "
"JOIN items i ON i.id = a.item_id "
"WHERE i.key = ? AND a.filename = ?",
(key, att_path.name),
)
.fetchone()
)
store.attach_file(key, att_path)
if dup is None:
n_new += 1
stats["attached"] += n_new
stats["items"].append(
{
"key": key,
"zotero_key": book.key,
"title": book.title,
"status": status,
"attached": n_new,
}
)
return stats

64
src/cli/bib.py
View File

@@ -521,6 +521,70 @@ def watch_iom() -> None:
typer.echo(f"futurepdf.pdf: {status} (sha256={digest[:12]})")
@app.command(name="import-zotero")
def import_zotero(
collection: str = typer.Option(
..., "--collection", help="Zotero collection name to pull from."
),
with_files: bool = typer.Option(
False,
"--with-files/--no-with-files",
help="Only import items that have at least one attachment file.",
),
only: str = typer.Option(
"", "--only", help="Title substring filter (case-insensitive)."
),
tag: list[str] = typer.Option(
None,
"--tag",
help="Tag every imported item with this. Repeat for multiple.",
),
dry_run: bool = typer.Option(
False, "--dry-run", help="Report what would happen without writing."
),
) -> None:
"""Pull a Zotero collection's book items (with files) into bib.
The reverse of ``sync-zotero``: reads the Zotero SQLite database
read-only and upserts each book as a bib ``Source`` item (dedup by
URL), copying its attachment file(s) into bib storage. Tag with
``llm:skip`` to keep material out of the corpus indexer (see
``llm.source.iter_corpus_refs``) — the escape hatch for
copyrighted text imported for citation/metadata only.
"""
from bib import connect
from bib.zotero_import import import_books, list_collection
from conf.connect import zotero
zcon = zotero()
try:
books = list_collection(zcon, collection, with_files=with_files)
finally:
zcon.close()
if only:
needle = only.lower()
books = [b for b in books if needle in b.title.lower()]
typer.echo(f"{len(books)} book(s) matched in '{collection}'")
store = connect()
stats = import_books(store, books, tags=tag or [], dry_run=dry_run)
prefix = "would-" if dry_run else ""
for rec in stats["items"]:
typer.echo(
f" {rec['zotero_key']} -> {rec['key'] or '(new)'}: "
f'{prefix}{rec["status"]} "{rec["title"][:60]}" '
f"attachments={rec['attached']}"
)
typer.echo(
f"{prefix}created={stats['created']} {prefix}updated={stats['updated']} "
f"{prefix}unchanged={stats['unchanged']} {prefix}attached={stats['attached']}"
)
@app.command(name="sync-zotero")
def sync_zotero(
tag: str = typer.Option("", "--tag", help="Only sync items matching a tag."),

225
tests/bib/test_zotero_import.py Normal file
View File

@@ -0,0 +1,225 @@
"""bib.zotero_import — pull a Zotero collection's book items (with
files) into bib."""
from __future__ import annotations
import sqlite3
import pytest
from bib.store import Store
from bib.zotero_import import ZoteroBook, import_books, list_collection
def _zotero_db(tmp_path):
"""A tiny sqlite db with the Zotero schema subset list_collection needs:
two book items in "AMA Coding Publications" (one with a storage:
attachment, one without), plus a distractor item in another
collection and a deleted item that must never surface."""
db = tmp_path / "zotero.sqlite"
con = sqlite3.connect(db)
con.executescript(
"""
CREATE TABLE items (itemID INTEGER PRIMARY KEY, itemTypeID INTEGER, key TEXT);
CREATE TABLE itemTypes (itemTypeID INTEGER PRIMARY KEY, typeName TEXT);
CREATE TABLE fields (fieldID INTEGER PRIMARY KEY, fieldName TEXT);
CREATE TABLE itemDataValues (valueID INTEGER PRIMARY KEY, value TEXT);
CREATE TABLE itemData (itemID INTEGER, fieldID INTEGER, valueID INTEGER);
CREATE TABLE collections (collectionID INTEGER PRIMARY KEY, collectionName TEXT);
CREATE TABLE collectionItems (collectionID INTEGER, itemID INTEGER);
CREATE TABLE itemAttachments (itemID INTEGER, parentItemID INTEGER, path TEXT, contentType TEXT);
CREATE TABLE deletedItems (itemID INTEGER);
INSERT INTO itemTypes VALUES (1, 'book'), (2, 'note');
-- item 1: CPT 2018, has a pdf attachment (item 10)
INSERT INTO items VALUES (1, 1, 'CPT2018K');
-- item 2: CPT Changes 2020, no attachment
INSERT INTO items VALUES (2, 1, 'CPTCHG20');
-- item 3: a note item in the same collection — never a "book"
INSERT INTO items VALUES (3, 2, 'NOTEKEY1');
-- item 4: deleted book — must never surface
INSERT INTO items VALUES (4, 1, 'DELETED1');
-- item 10: the attachment for item 1
INSERT INTO items VALUES (10, 3, 'ATTKEY01');
INSERT INTO fields VALUES (1, 'title'), (2, 'url'), (3, 'date'), (4, 'publisher');
INSERT INTO itemDataValues VALUES
(100, 'CPT 2018'), (101, 'https://ama.example/cpt2018'),
(102, '2017-11-01'), (103, 'American Medical Association'),
(200, 'CPT Changes 2020');
INSERT INTO itemData VALUES
(1, 1, 100), (1, 2, 101), (1, 3, 102), (1, 4, 103),
(2, 1, 200);
INSERT INTO collections VALUES (1, 'AMA Coding Publications'), (2, 'Other');
INSERT INTO collectionItems VALUES (1, 1), (1, 2), (1, 3), (1, 4), (2, 1);
INSERT INTO itemAttachments VALUES (10, 1, 'storage:CPT 2018.pdf', 'application/pdf');
INSERT INTO deletedItems VALUES (4);
"""
)
con.commit()
con.close()
return db
@pytest.fixture
def zcon(tmp_path):
con = sqlite3.connect(_zotero_db(tmp_path))
yield con
con.close()
@pytest.fixture
def storage_dir(tmp_path):
d = tmp_path / "storage" / "ATTKEY01"
d.mkdir(parents=True)
(d / "CPT 2018.pdf").write_bytes(b"%PDF-1.4 fake cpt 2018")
return tmp_path / "storage"
@pytest.fixture
def store(tmp_path):
s = Store(":memory:", storage_dir=tmp_path / "bib-storage")
yield s
s.close()
class TestListCollection:
def test_two_books_one_with_attachment(self, zcon, storage_dir):
books = list_collection(
zcon, "AMA Coding Publications", with_files=False, storage_dir=storage_dir
)
assert {b.key for b in books} == {"CPT2018K", "CPTCHG20"}
cpt = next(b for b in books if b.key == "CPT2018K")
assert cpt.title == "CPT 2018"
assert cpt.url == "https://ama.example/cpt2018"
assert cpt.year == "2017"
assert cpt.publisher == "American Medical Association"
assert cpt.attachments == (storage_dir / "ATTKEY01" / "CPT 2018.pdf",)
changes = next(b for b in books if b.key == "CPTCHG20")
assert changes.attachments == ()
assert changes.url == ""
def test_excludes_notes_and_deleted_items(self, zcon, storage_dir):
books = list_collection(
zcon, "AMA Coding Publications", with_files=False, storage_dir=storage_dir
)
keys = {b.key for b in books}
assert "NOTEKEY1" not in keys
assert "DELETED1" not in keys
def test_with_files_filters_to_items_with_attachments(self, zcon, storage_dir):
books = list_collection(
zcon, "AMA Coding Publications", with_files=True, storage_dir=storage_dir
)
assert [b.key for b in books] == ["CPT2018K"]
def test_unknown_collection_returns_empty(self, zcon, storage_dir):
assert (
list_collection(zcon, "Nope", with_files=False, storage_dir=storage_dir)
== []
)
class TestImportBooks:
def _cpt_book(self, storage_dir):
return ZoteroBook(
key="CPT2018K",
title="CPT 2018",
url="https://ama.example/cpt2018",
year="2017",
publisher="American Medical Association",
attachments=(storage_dir / "ATTKEY01" / "CPT 2018.pdf",),
)
def test_creates_source_item_with_tags_year_and_attachment(
self, store, storage_dir
):
book = self._cpt_book(storage_dir)
stats = import_books(
store, [book], tags=["source:ama", "module:coding", "llm:skip"]
)
assert stats["created"] == 1
assert stats["updated"] == 0
assert stats["attached"] == 1
(rec,) = stats["items"]
assert rec["zotero_key"] == "CPT2018K"
item = store.get(rec["key"])
assert item.item_type == "source"
assert item.title == "CPT 2018"
assert item.url == "https://ama.example/cpt2018"
assert set(item.tags) == {
"source:ama",
"module:coding",
"llm:skip",
"year:2017",
}
attachments = (
store._con()
.execute(
"SELECT filename FROM attachments a "
"JOIN items i ON i.id = a.item_id WHERE i.key = ?",
(rec["key"],),
)
.fetchall()
)
assert [a[0] for a in attachments] == ["CPT 2018.pdf"]
def test_fallback_url_used_when_book_has_none(self, store):
book = ZoteroBook(
key="NOURLKEY",
title="No URL Book",
url="",
year="2020",
publisher="",
attachments=(),
)
import_books(store, [book], tags=["source:ama"])
item = store.get(store.list_items(query="No URL Book")[0].key)
assert item.url == "zotero://select/library/items/NOURLKEY"
def test_rerun_is_unchanged_with_no_duplicate_attachment(self, store, storage_dir):
book = self._cpt_book(storage_dir)
tags = ["source:ama", "module:coding", "llm:skip"]
first = import_books(store, [book], tags=tags)
second = import_books(store, [book], tags=tags)
assert second["created"] == 0
assert second["unchanged"] == 1
assert second["attached"] == 0
bib_key = first["items"][0]["key"]
rows = (
store._con()
.execute(
"SELECT filename FROM attachments a "
"JOIN items i ON i.id = a.item_id WHERE i.key = ?",
(bib_key,),
)
.fetchall()
)
assert len(rows) == 1 # no duplicate attachment
def test_dry_run_makes_no_writes(self, store, storage_dir):
book = self._cpt_book(storage_dir)
stats = import_books(store, [book], tags=["source:ama"], dry_run=True)
assert stats["created"] == 1
assert stats["attached"] == 1
assert store.list_items() == [] # nothing written
def test_dry_run_after_real_import_reports_unchanged(self, store, storage_dir):
book = self._cpt_book(storage_dir)
tags = ["source:ama"]
import_books(store, [book], tags=tags)
stats = import_books(store, [book], tags=tags, dry_run=True)
assert stats["unchanged"] == 1
assert stats["attached"] == 0
assert stats["items"][0]["key"] # bib key resolved from the existing url