zot: PRISMA screening workflow with Zotero annotations #258

New Issue

kert · 2026-03-25T19:31:21Z

kert commented

2026-03-25 19:31:21 +00:00

Objective

Use Zotero highlighting and annotation tools to facilitate systematic review screening and PRISMA flow.

Workflow

Title/abstract screening — Zotero tags: screen:include, screen:exclude, screen:uncertain
Full-text screening — after PDF retrieval, annotate inclusion/exclusion reasons
Data extraction — highlight key findings (effect sizes, sample sizes, outcomes) in PDFs
Quality assessment — tag with risk-of-bias assessment per Cochrane guidelines
PRISMA flow — auto-generate counts from tag distribution

Zotero Tools

Tags for screening decisions (already have module:skin-subs, type:rct, etc.)
Notes for extraction forms (structured per article)
PDF annotations (highlights, sticky notes) for in-document coding
Saved searches for screening queues (unscreened, included, excluded)
Related items for forward/backward citation links

Automation

Script to create saved searches for screening queues
Script to generate PRISMA flow diagram from tag counts
Script to export data extraction table from structured notes
Integration with bib.sqlite for tracking screening progress

Acceptance Criteria

Researcher can open Zotero, see screening queue, annotate, tag, and have progress tracked automatically

## Objective Use Zotero highlighting and annotation tools to facilitate systematic review screening and PRISMA flow. ## Workflow 1. **Title/abstract screening** — Zotero tags: `screen:include`, `screen:exclude`, `screen:uncertain` 2. **Full-text screening** — after PDF retrieval, annotate inclusion/exclusion reasons 3. **Data extraction** — highlight key findings (effect sizes, sample sizes, outcomes) in PDFs 4. **Quality assessment** — tag with risk-of-bias assessment per Cochrane guidelines 5. **PRISMA flow** — auto-generate counts from tag distribution ## Zotero Tools - **Tags** for screening decisions (already have module:skin-subs, type:rct, etc.) - **Notes** for extraction forms (structured per article) - **PDF annotations** (highlights, sticky notes) for in-document coding - **Saved searches** for screening queues (unscreened, included, excluded) - **Related items** for forward/backward citation links ## Automation - Script to create saved searches for screening queues - Script to generate PRISMA flow diagram from tag counts - Script to export data extraction table from structured notes - Integration with bib.sqlite for tracking screening progress ## Acceptance Criteria - Researcher can open Zotero, see screening queue, annotate, tag, and have progress tracked automatically

kert added this to the P29: Skin Substitutes — predatory pricing, distribution, and utilization analysis milestone 2026-03-25 19:31:54 +00:00

kert commented

2026-04-15 05:09:21 +00:00

Closed. LLM-driven PRISMA pipeline shipped for skin-subs.

Architecture

Model-agnostic by construction: every LLM call goes through a Provider protocol in src/prisma/llm.py. AnthropicProvider today (Messages API, prompt caching on cache_control markers); OpenAICompatProvider stub ready for vLLM/Ollama/llama.cpp-server — swap via PRISMA_LLM_PROVIDER env, no code change.

Modules

src/prisma/llm.py — LLMCall/LLMResult/LLMTool value objects; Provider protocol; Anthropic + OpenAI-compat backends. Lazy SDK imports so the core stack CLI stays importable without the prisma extras.
src/prisma/project.py — project bootstrap. Three Zotero anchor items per project carry the three PRISMA documents as notes: prisma:criteria, prisma:extraction-template, prisma:reasons. Edits happen in the Zotero UI; code reads via tag lookup. _LEGACY_COHORT_TAGS backfills project:<name> onto legacy cohort tags (module:skin-subs → project:skin-subs) on init.
src/prisma/export.py — Zotero item → YAML-frontmatter markdown. Abstract + optional PDF full text (via pdfminer → pypdf fallback) + notes + PDF annotations. Fulltext skipped at stage 2, included at stages 3/3+.
src/prisma/tools.py — three JSON-schema tool definitions (SCREEN_TOOL, ELIGIBILITY_TOOL, EXTRACT_TOOL). Tools are forced responses via LLMCall.force_tool=<name>.
src/prisma/ingest.py — tool-call payload → Zotero tags + note. Tag-surgery rule enforces stage mono-assignment (strips old stage:* before writing new).
src/prisma/screen.py — stage 2 loop. Cached prefix = system + criteria + reasons; per-item tail = item markdown.
src/prisma/eligibility.py — stage 3 loop. Same prefix, full-text body, resolves to stage:4-included on include.
src/prisma/extract.py — stage 3+. Cached prefix = extraction template; fills EXTRACT_TOOL schema; writes structured note tagged prisma:extracted.
src/prisma/flow.py — PRISMA 2020 Mermaid diagram + text summary from tag counts. No LLM calls.

Tag vocabulary

project:<name>                    cohort gate
stage:1-identified                implicit default
stage:{2-title-abstract,          explicit per stage
  3-full-text, 4-included}
screen:{include,exclude,uncertain}
screen:reason:<code>              from the reasons codebook
theme:<slug>                      free-form LLM-discovered
prisma:criteria                   anchor role tags
prisma:extraction-template
prisma:reasons
prisma:extracted                  marks stage-3+ completion
module:prisma                     anchor marker; excluded from queues

CLI

stack prisma init skin-subs       # bootstrap + backfill cohort tag
stack prisma ping-llm             # verify provider config
stack prisma export <itemID>      # markdown preview
stack prisma screen <proj> [-n N] # stage 2
stack prisma eligible <proj>      # stage 3
stack prisma extract <proj>       # stage 3+
stack prisma flow <proj>          # Mermaid or --text summary
stack prisma run <proj>           # all stages in one zotero hold

Verified e2e

init skin-subs → 3 anchors + 10,262 items backfilled with project:skin-subs
screen --limit 5 → 5 screened (2 exclude: editorial, wrong-population; 3 uncertain); themes discovered (amniotic-membrane, ocular-surface, diabetic-foot-ulcer, dermagraft, apligraf, etc.)
eligible --limit 2 → 2 assessed, gracefully unavailable-fulltext when PDFs missing
flow → Mermaid flowchart from live tag counts

Known follow-ups (not blocking)

Wire prisma run skin-subs into a cron via issue #343's scheduler once that lands.
Add a prisma export-extracted CLI that aggregates prisma:extracted notes into an analysis-ready Parquet for #246's manuscript analysis.
Validate OpenAICompatProvider once a local vLLM/Ollama endpoint is stood up.

Files added

src/prisma/__init__.py
src/prisma/llm.py
src/prisma/project.py
src/prisma/export.py
src/prisma/tools.py
src/prisma/ingest.py
src/prisma/screen.py
src/prisma/eligibility.py
src/prisma/extract.py
src/prisma/flow.py
src/cli/prisma.py
pyproject.toml (prisma extras)
.env / .env.example (PRISMA_LLM_* block + ANTHROPIC_API_KEY typo fix)

**Closed.** LLM-driven PRISMA pipeline shipped for `skin-subs`. ## Architecture Model-agnostic by construction: every LLM call goes through a `Provider` protocol in `src/prisma/llm.py`. `AnthropicProvider` today (Messages API, prompt caching on `cache_control` markers); `OpenAICompatProvider` stub ready for vLLM/Ollama/llama.cpp-server — swap via `PRISMA_LLM_PROVIDER` env, no code change. ### Modules - **`src/prisma/llm.py`** — `LLMCall`/`LLMResult`/`LLMTool` value objects; `Provider` protocol; Anthropic + OpenAI-compat backends. Lazy SDK imports so the core `stack` CLI stays importable without the `prisma` extras. - **`src/prisma/project.py`** — project bootstrap. Three Zotero anchor items per project carry the three PRISMA documents as notes: `prisma:criteria`, `prisma:extraction-template`, `prisma:reasons`. Edits happen in the Zotero UI; code reads via tag lookup. `_LEGACY_COHORT_TAGS` backfills `project:<name>` onto legacy cohort tags (`module:skin-subs` → `project:skin-subs`) on init. - **`src/prisma/export.py`** — Zotero item → YAML-frontmatter markdown. Abstract + optional PDF full text (via pdfminer → pypdf fallback) + notes + PDF annotations. Fulltext skipped at stage 2, included at stages 3/3+. - **`src/prisma/tools.py`** — three JSON-schema tool definitions (`SCREEN_TOOL`, `ELIGIBILITY_TOOL`, `EXTRACT_TOOL`). Tools are forced responses via `LLMCall.force_tool=<name>`. - **`src/prisma/ingest.py`** — tool-call payload → Zotero tags + note. Tag-surgery rule enforces stage mono-assignment (strips old `stage:*` before writing new). - **`src/prisma/screen.py`** — stage 2 loop. Cached prefix = system + criteria + reasons; per-item tail = item markdown. - **`src/prisma/eligibility.py`** — stage 3 loop. Same prefix, full-text body, resolves to `stage:4-included` on include. - **`src/prisma/extract.py`** — stage 3+. Cached prefix = extraction template; fills `EXTRACT_TOOL` schema; writes structured note tagged `prisma:extracted`. - **`src/prisma/flow.py`** — PRISMA 2020 Mermaid diagram + text summary from tag counts. No LLM calls. ### Tag vocabulary ``` project:<name> cohort gate stage:1-identified implicit default stage:{2-title-abstract, explicit per stage 3-full-text, 4-included} screen:{include,exclude,uncertain} screen:reason:<code> from the reasons codebook theme:<slug> free-form LLM-discovered prisma:criteria anchor role tags prisma:extraction-template prisma:reasons prisma:extracted marks stage-3+ completion module:prisma anchor marker; excluded from queues ``` ### CLI ``` stack prisma init skin-subs # bootstrap + backfill cohort tag stack prisma ping-llm # verify provider config stack prisma export <itemID> # markdown preview stack prisma screen <proj> [-n N] # stage 2 stack prisma eligible <proj> # stage 3 stack prisma extract <proj> # stage 3+ stack prisma flow <proj> # Mermaid or --text summary stack prisma run <proj> # all stages in one zotero hold ``` ## Verified e2e - `init skin-subs` → 3 anchors + 10,262 items backfilled with `project:skin-subs` - `screen --limit 5` → 5 screened (2 exclude: `editorial`, `wrong-population`; 3 uncertain); themes discovered (`amniotic-membrane`, `ocular-surface`, `diabetic-foot-ulcer`, `dermagraft`, `apligraf`, etc.) - `eligible --limit 2` → 2 assessed, gracefully `unavailable-fulltext` when PDFs missing - `flow` → Mermaid flowchart from live tag counts ## Known follow-ups (not blocking) 1. Wire `prisma run skin-subs` into a cron via issue #343's scheduler once that lands. 2. Add a `prisma export-extracted` CLI that aggregates `prisma:extracted` notes into an analysis-ready Parquet for #246's manuscript analysis. 3. Validate `OpenAICompatProvider` once a local vLLM/Ollama endpoint is stood up. ## Files added ``` src/prisma/__init__.py src/prisma/llm.py src/prisma/project.py src/prisma/export.py src/prisma/tools.py src/prisma/ingest.py src/prisma/screen.py src/prisma/eligibility.py src/prisma/extract.py src/prisma/flow.py src/cli/prisma.py pyproject.toml (prisma extras) .env / .env.example (PRISMA_LLM_* block + ANTHROPIC_API_KEY typo fix) ```

kert closed this issue

2026-04-15 05:09:22 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#258