feat: pydoc pincite — structured Zotero citations in docstrings
Some checks failed
CI / skinny-install (bib) (push) Successful in 34s
CI / skinny-install (cli) (push) Successful in 32s
CI / skinny-install (cms) (push) Successful in 27s
CI / skinny-install (conf) (push) Successful in 29s
CI / skinny-install (opps) (push) Successful in 31s
CI / skinny-install (perf) (push) Successful in 30s
CI / skinny-install (pfs) (push) Successful in 31s
Infra CI / notebooks (push) Successful in 7s
CI / skinny-install (aco) (pull_request) Successful in 52s
CI / lint-test (pull_request) Failing after 1m18s
CI / skinny-install (api) (pull_request) Successful in 26s
CI / skinny-install (bcda) (pull_request) Successful in 38s
CI / skinny-install (bib) (pull_request) Successful in 36s
CI / skinny-install (bls) (pull_request) Successful in 26s
CI / skinny-install (cli) (pull_request) Successful in 34s
CI / skinny-install (cms) (pull_request) Successful in 29s
CI / skinny-install (conf) (pull_request) Successful in 28s
CI / skinny-install (opps) (pull_request) Successful in 30s
CI / skinny-install (perf) (pull_request) Successful in 33s
CI / skinny-install (pfs) (pull_request) Successful in 33s
Infra CI / notebooks (pull_request) Successful in 7s
Infra CI / zotero (pull_request) Successful in 6s
Infra CI / docs (pull_request) Failing after 7s
Infra CI / api (pull_request) Successful in 8s
CI / skinny-install (aco) (push) Successful in 51s
CI / lint-test (push) Failing after 1m16s
CI / skinny-install (api) (push) Successful in 26s
CI / skinny-install (bcda) (push) Successful in 35s
CI / skinny-install (bls) (push) Successful in 23s
CI / skinny-install (ccw) (push) Successful in 32s
Infra CI / zotero (push) Successful in 10s
CI / skinny-install (rex) (push) Successful in 25s
Infra CI / docs (push) Failing after 9s
Infra CI / api (push) Successful in 9s
Infra CI / mc (push) Successful in 6s
CI / skinny-install (ccw) (pull_request) Successful in 32s
CI / skinny-install (rex) (pull_request) Successful in 25s
Infra CI / mc (pull_request) Successful in 7s
Some checks failed
CI / skinny-install (bib) (push) Successful in 34s
CI / skinny-install (cli) (push) Successful in 32s
CI / skinny-install (cms) (push) Successful in 27s
CI / skinny-install (conf) (push) Successful in 29s
CI / skinny-install (opps) (push) Successful in 31s
CI / skinny-install (perf) (push) Successful in 30s
CI / skinny-install (pfs) (push) Successful in 31s
Infra CI / notebooks (push) Successful in 7s
CI / skinny-install (aco) (pull_request) Successful in 52s
CI / lint-test (pull_request) Failing after 1m18s
CI / skinny-install (api) (pull_request) Successful in 26s
CI / skinny-install (bcda) (pull_request) Successful in 38s
CI / skinny-install (bib) (pull_request) Successful in 36s
CI / skinny-install (bls) (pull_request) Successful in 26s
CI / skinny-install (cli) (pull_request) Successful in 34s
CI / skinny-install (cms) (pull_request) Successful in 29s
CI / skinny-install (conf) (pull_request) Successful in 28s
CI / skinny-install (opps) (pull_request) Successful in 30s
CI / skinny-install (perf) (pull_request) Successful in 33s
CI / skinny-install (pfs) (pull_request) Successful in 33s
Infra CI / notebooks (pull_request) Successful in 7s
Infra CI / zotero (pull_request) Successful in 6s
Infra CI / docs (pull_request) Failing after 7s
Infra CI / api (pull_request) Successful in 8s
CI / skinny-install (aco) (push) Successful in 51s
CI / lint-test (push) Failing after 1m16s
CI / skinny-install (api) (push) Successful in 26s
CI / skinny-install (bcda) (push) Successful in 35s
CI / skinny-install (bls) (push) Successful in 23s
CI / skinny-install (ccw) (push) Successful in 32s
Infra CI / zotero (push) Successful in 10s
CI / skinny-install (rex) (push) Successful in 25s
Infra CI / docs (push) Failing after 9s
Infra CI / api (push) Successful in 9s
Infra CI / mc (push) Successful in 6s
CI / skinny-install (ccw) (pull_request) Successful in 32s
CI / skinny-install (rex) (pull_request) Successful in 25s
Infra CI / mc (pull_request) Successful in 7s
Centerpiece feature: machine-parseable :pincite: directives in function docstrings that link to specific pages, sections, or paragraphs of bibliography items, enabling a knowledge graph from regulatory source to function to table to column. Format: :pincite:`JX46GQ9L p.14` — NDC validation rules New module src/bib/pincite.py with 4 layers: - Parse: regex extraction of :pincite: directives, AST-based discovery of @nw.narwhalify functions, locator classification (page/section/cfr/ fr/chapter/paragraph) - Store: pincites table in bib.sqlite with upsert, query, tag sync (pin: and fn: namespaces), foreign key validation - Graph: CitationGraph with nodes (functions, items), edges (pincite rows), functions_citing/items_cited_by queries, mermaid and networkx export - Inject: format_pincite_block for References sections, inject_pincites_into_source for AST-located non-destructive docstring modification Schema: new pincites table with unique index on (fn_path, item_key, locator), FK cascade to items. Tag: new Tag.pin(key, locator) factory -> pin:KEY/locator Store: upsert_pincite, list_pincites, delete_pincites on Store class. 39 tests covering parse, classify, store, graph, inject, and check.
This commit is contained in:
@@ -62,6 +62,11 @@ from .item import Manual as Manual
|
||||
from .item import Regulation as Regulation
|
||||
from .item import Rule as Rule
|
||||
from .item import Source as Source
|
||||
from .pincite import CitationGraph as CitationGraph
|
||||
from .pincite import Pincite as Pincite
|
||||
from .pincite import build_citation_graph as build_citation_graph
|
||||
from .pincite import extract_all_pincites as extract_all_pincites
|
||||
from .pincite import parse_pincites as parse_pincites
|
||||
from .spider import crawl as crawl
|
||||
from .spider import crawl_all as crawl_all
|
||||
from .store import Store as Store
|
||||
|
||||
574
src/bib/pincite.py
Normal file
574
src/bib/pincite.py
Normal file
@@ -0,0 +1,574 @@
|
||||
"""Pinpoint citations — structured Zotero references in docstrings.
|
||||
|
||||
Embeds machine-parseable ``:pincite:`` directives in function docstrings
|
||||
that link to specific pages, sections, or paragraphs of bibliography
|
||||
items. Enables a knowledge graph from regulatory source to function
|
||||
to table to column.
|
||||
|
||||
Directive format::
|
||||
|
||||
:pincite:`JX46GQ9L p.14` — NDC validation rules
|
||||
:pincite:`9ASETLJ4 §3.2` — exclusion criteria
|
||||
|
||||
Components:
|
||||
|
||||
- **Parse**: regex extraction of ``:pincite:`` directives from docstrings
|
||||
- **Store**: upsert/query pincite records in ``bib.sqlite``
|
||||
- **Graph**: build a citation knowledge graph (nodes + edges)
|
||||
- **Inject**: write pincite blocks back into source files
|
||||
- **Check**: validate all cited item keys exist
|
||||
|
||||
Usage::
|
||||
|
||||
from bib.pincite import parse_pincites, build_citation_graph
|
||||
|
||||
pincites = parse_pincites(fn.__doc__, "aco.express.pharmacy.pharmacy_claims")
|
||||
graph = build_citation_graph(store)
|
||||
print(graph.to_mermaid())
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from bib.tag import Tag
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from bib.store import Store
|
||||
|
||||
|
||||
# ── Regex ────────────────────────────────────────────────────────────
|
||||
|
||||
# Zotero key charset: 23456789ABCDEFGHIJKLMNPQRSTUVWXYZ (no 0, 1, O, L)
|
||||
_PINCITE_RE = re.compile(
|
||||
r":pincite:`"
|
||||
r"([23456789A-HJ-NP-Z]{8})" # group 1: item key
|
||||
r"(?:\s+(.+?))?" # group 2: locator (optional)
|
||||
r"`"
|
||||
r"(?:\s*(?:--|—)\s*(.+))?" # group 3: note (optional)
|
||||
)
|
||||
|
||||
|
||||
# ── Data model ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class Pincite(BaseModel):
|
||||
"""A pinpoint citation linking a function to a bibliography item."""
|
||||
|
||||
fn_path: str
|
||||
item_key: str
|
||||
locator: str = ""
|
||||
locator_type: str = ""
|
||||
note: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class CitationNode:
|
||||
"""A node in the citation knowledge graph."""
|
||||
|
||||
id: str
|
||||
kind: str # "function" or "item"
|
||||
label: str
|
||||
metadata: dict = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class CitationEdge:
|
||||
"""An edge linking a function to a cited item."""
|
||||
|
||||
source: str # fn_path
|
||||
target: str # item_key
|
||||
locator: str = ""
|
||||
locator_type: str = ""
|
||||
note: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class CitationGraph:
|
||||
"""Knowledge graph of function → item citations."""
|
||||
|
||||
nodes: list[CitationNode] = field(default_factory=list)
|
||||
edges: list[CitationEdge] = field(default_factory=list)
|
||||
|
||||
def functions_citing(self, item_key: str) -> list[str]:
|
||||
"""Return fn_paths that cite a given item."""
|
||||
return [e.source for e in self.edges if e.target == item_key]
|
||||
|
||||
def items_cited_by(self, fn_path: str) -> list[str]:
|
||||
"""Return item_keys cited by a given function."""
|
||||
return [e.target for e in self.edges if e.source == fn_path]
|
||||
|
||||
def to_networkx_dict(self) -> dict:
|
||||
"""Export as networkx-compatible node-link dict."""
|
||||
return {
|
||||
"directed": True,
|
||||
"multigraph": False,
|
||||
"nodes": [
|
||||
{"id": n.id, "kind": n.kind, "label": n.label, **n.metadata}
|
||||
for n in self.nodes
|
||||
],
|
||||
"links": [
|
||||
{
|
||||
"source": e.source,
|
||||
"target": e.target,
|
||||
"locator": e.locator,
|
||||
"note": e.note,
|
||||
}
|
||||
for e in self.edges
|
||||
],
|
||||
}
|
||||
|
||||
def to_mermaid(self) -> str:
|
||||
"""Render as Mermaid flowchart."""
|
||||
lines = ["graph LR"]
|
||||
for n in self.nodes:
|
||||
if n.kind == "function":
|
||||
lines.append(f' {_mermaid_id(n.id)}["{n.label}"]')
|
||||
else:
|
||||
lines.append(f' {_mermaid_id(n.id)}("{n.label}")')
|
||||
for e in self.edges:
|
||||
label = e.locator or e.note or "cites"
|
||||
lines.append(
|
||||
f" {_mermaid_id(e.source)} -->|{label}| {_mermaid_id(e.target)}"
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _mermaid_id(s: str) -> str:
|
||||
"""Sanitise a string for use as a Mermaid node ID."""
|
||||
return re.sub(r"[^a-zA-Z0-9_]", "_", s)
|
||||
|
||||
|
||||
# ── Parse layer ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _classify_locator(locator: str) -> str:
|
||||
"""Classify a locator string into a type category."""
|
||||
if not locator:
|
||||
return ""
|
||||
loc = locator.strip()
|
||||
if loc.lower().startswith(("p.", "pp.")):
|
||||
return "page"
|
||||
if "CFR" in loc or "cfr" in loc:
|
||||
return "cfr"
|
||||
if re.match(r"\d+\s+FR\s+\d+", loc, re.IGNORECASE):
|
||||
return "fr"
|
||||
if loc.lower().startswith(("ch.", "ch ")):
|
||||
return "chapter"
|
||||
if loc.startswith("¶"):
|
||||
return "paragraph"
|
||||
if loc.startswith("§") or re.match(r"\d+[\d.]+", loc):
|
||||
return "section"
|
||||
return "other"
|
||||
|
||||
|
||||
def parse_pincites(docstring: str, fn_path: str) -> list[Pincite]:
|
||||
"""Extract ``:pincite:`` directives from a docstring.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
docstring : str
|
||||
The raw docstring text.
|
||||
fn_path : str
|
||||
Fully qualified function path for attribution.
|
||||
|
||||
Returns
|
||||
-------
|
||||
list[Pincite]
|
||||
Parsed pincite records, deduplicated by (key, locator).
|
||||
"""
|
||||
if not docstring:
|
||||
return []
|
||||
|
||||
results: list[Pincite] = []
|
||||
seen: set[tuple[str, str]] = set()
|
||||
|
||||
for m in _PINCITE_RE.finditer(docstring):
|
||||
key = m.group(1)
|
||||
locator = (m.group(2) or "").strip()
|
||||
note = (m.group(3) or "").strip()
|
||||
pair = (key, locator)
|
||||
if pair in seen:
|
||||
continue
|
||||
seen.add(pair)
|
||||
results.append(
|
||||
Pincite(
|
||||
fn_path=fn_path,
|
||||
item_key=key,
|
||||
locator=locator,
|
||||
locator_type=_classify_locator(locator),
|
||||
note=note,
|
||||
)
|
||||
)
|
||||
return results
|
||||
|
||||
|
||||
# ── AST discovery ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _is_narwhalify(node: ast.FunctionDef | ast.AsyncFunctionDef) -> bool:
|
||||
"""Check if a function is decorated with @nw.narwhalify."""
|
||||
for dec in node.decorator_list:
|
||||
if isinstance(dec, ast.Name) and dec.id == "narwhalify":
|
||||
return True
|
||||
if isinstance(dec, ast.Attribute) and dec.attr == "narwhalify":
|
||||
return True
|
||||
if isinstance(dec, ast.Call):
|
||||
func = dec.func
|
||||
if isinstance(func, ast.Name) and func.id == "narwhalify":
|
||||
return True
|
||||
if isinstance(func, ast.Attribute) and func.attr == "narwhalify":
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _path_to_module(path: Path, src_root: Path) -> str:
|
||||
"""Convert file path to dotted module name."""
|
||||
rel = path.relative_to(src_root)
|
||||
parts = list(rel.with_suffix("").parts)
|
||||
if parts[-1] == "__init__":
|
||||
parts = parts[:-1]
|
||||
return ".".join(parts)
|
||||
|
||||
|
||||
def extract_all_pincites(src_root: Path | None = None) -> list[Pincite]:
|
||||
"""Walk every ``@nw.narwhalify`` function and extract pincites.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
src_root : Path, optional
|
||||
Root of the source tree. Defaults to ``src/`` relative to
|
||||
the project root.
|
||||
|
||||
Returns
|
||||
-------
|
||||
list[Pincite]
|
||||
All pincites found across all narwhalify docstrings.
|
||||
"""
|
||||
if src_root is None:
|
||||
src_root = Path(__file__).parent.parent # src/
|
||||
|
||||
results: list[Pincite] = []
|
||||
for py_file in sorted(src_root.rglob("*.py")):
|
||||
if "__pycache__" in py_file.parts or ".egg-info" in str(py_file):
|
||||
continue
|
||||
try:
|
||||
source = py_file.read_text(encoding="utf-8")
|
||||
tree = ast.parse(source, filename=str(py_file))
|
||||
except SyntaxError:
|
||||
continue
|
||||
|
||||
module = _path_to_module(py_file, src_root)
|
||||
for node in ast.walk(tree):
|
||||
if not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
|
||||
continue
|
||||
if not _is_narwhalify(node):
|
||||
continue
|
||||
doc = ast.get_docstring(node) or ""
|
||||
if not doc:
|
||||
continue
|
||||
fn_path = f"{module}.{node.name}"
|
||||
results.extend(parse_pincites(doc, fn_path))
|
||||
|
||||
return results
|
||||
|
||||
|
||||
# ── Store operations ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
def upsert_pincites(store: Store, pincites: list[Pincite]) -> int:
|
||||
"""Insert or update pincite records and sync tags.
|
||||
|
||||
Returns the number of rows affected.
|
||||
"""
|
||||
con = store._con()
|
||||
# Ensure pincites table exists
|
||||
con.executescript(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS pincites (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
fn_path TEXT NOT NULL,
|
||||
item_key TEXT NOT NULL,
|
||||
locator TEXT NOT NULL DEFAULT '',
|
||||
locator_type TEXT NOT NULL DEFAULT '',
|
||||
note TEXT NOT NULL DEFAULT '',
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ','now')),
|
||||
FOREIGN KEY (item_key) REFERENCES items(key) ON DELETE CASCADE
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_pincites_unique
|
||||
ON pincites(fn_path, item_key, locator);
|
||||
CREATE INDEX IF NOT EXISTS idx_pincites_fn ON pincites(fn_path);
|
||||
CREATE INDEX IF NOT EXISTS idx_pincites_item ON pincites(item_key);
|
||||
"""
|
||||
)
|
||||
|
||||
count = 0
|
||||
for p in pincites:
|
||||
# Skip if item doesn't exist
|
||||
row = con.execute(
|
||||
"SELECT key FROM items WHERE key = ?", (p.item_key,)
|
||||
).fetchone()
|
||||
if row is None:
|
||||
continue
|
||||
|
||||
con.execute(
|
||||
"""
|
||||
INSERT INTO pincites (fn_path, item_key, locator, locator_type, note)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT (fn_path, item_key, locator)
|
||||
DO UPDATE SET locator_type = excluded.locator_type,
|
||||
note = excluded.note
|
||||
""",
|
||||
(p.fn_path, p.item_key, p.locator, p.locator_type, p.note),
|
||||
)
|
||||
count += 1
|
||||
|
||||
# Sync tags on the item
|
||||
pin_tag = Tag.pin(p.item_key, p.locator).label
|
||||
fn_tag = Tag.fn(p.fn_path.split(".")[-1]).label
|
||||
for tag_label in (pin_tag, fn_tag):
|
||||
try:
|
||||
store.add_tag(p.item_key, tag_label)
|
||||
except (KeyError, Exception):
|
||||
pass
|
||||
|
||||
con.commit()
|
||||
return count
|
||||
|
||||
|
||||
def list_pincites(
|
||||
store: Store,
|
||||
*,
|
||||
fn_path: str = "",
|
||||
item_key: str = "",
|
||||
) -> list[Pincite]:
|
||||
"""Query pincites with optional filters."""
|
||||
con = store._con()
|
||||
|
||||
# Table might not exist yet
|
||||
try:
|
||||
con.execute("SELECT 1 FROM pincites LIMIT 0")
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
sql = (
|
||||
"SELECT fn_path, item_key, locator, locator_type, note FROM pincites WHERE 1=1"
|
||||
)
|
||||
params: list[str] = []
|
||||
if fn_path:
|
||||
sql += " AND fn_path = ?"
|
||||
params.append(fn_path)
|
||||
if item_key:
|
||||
sql += " AND item_key = ?"
|
||||
params.append(item_key)
|
||||
sql += " ORDER BY fn_path, item_key"
|
||||
|
||||
rows = con.execute(sql, params).fetchall()
|
||||
return [
|
||||
Pincite(
|
||||
fn_path=r["fn_path"],
|
||||
item_key=r["item_key"],
|
||||
locator=r["locator"],
|
||||
locator_type=r["locator_type"],
|
||||
note=r["note"],
|
||||
)
|
||||
for r in rows
|
||||
]
|
||||
|
||||
|
||||
def check_pincite_keys(
|
||||
pincites: list[Pincite], store: Store
|
||||
) -> list[tuple[Pincite, str]]:
|
||||
"""Verify all pincite item_keys exist in bib.sqlite."""
|
||||
con = store._con()
|
||||
errors: list[tuple[Pincite, str]] = []
|
||||
for p in pincites:
|
||||
row = con.execute(
|
||||
"SELECT key FROM items WHERE key = ?", (p.item_key,)
|
||||
).fetchone()
|
||||
if row is None:
|
||||
errors.append((p, f"item key {p.item_key!r} not found in bib.sqlite"))
|
||||
return errors
|
||||
|
||||
|
||||
# ── Knowledge graph ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
def build_citation_graph(store: Store) -> CitationGraph:
|
||||
"""Build the citation knowledge graph from the pincites table.
|
||||
|
||||
Nodes: every function with pincites, every cited item.
|
||||
Edges: one per pincite row.
|
||||
"""
|
||||
pincites = list_pincites(store)
|
||||
|
||||
fn_ids: set[str] = set()
|
||||
item_ids: set[str] = set()
|
||||
nodes: list[CitationNode] = []
|
||||
edges: list[CitationEdge] = []
|
||||
|
||||
for p in pincites:
|
||||
if p.fn_path not in fn_ids:
|
||||
fn_ids.add(p.fn_path)
|
||||
short = p.fn_path.rsplit(".", 1)[-1]
|
||||
nodes.append(CitationNode(id=p.fn_path, kind="function", label=short))
|
||||
if p.item_key not in item_ids:
|
||||
item_ids.add(p.item_key)
|
||||
# Try to get title from store
|
||||
try:
|
||||
item = store.get(p.item_key)
|
||||
label = item.title[:60] if item.title else p.item_key
|
||||
except Exception:
|
||||
label = p.item_key
|
||||
nodes.append(CitationNode(id=p.item_key, kind="item", label=label))
|
||||
|
||||
edges.append(
|
||||
CitationEdge(
|
||||
source=p.fn_path,
|
||||
target=p.item_key,
|
||||
locator=p.locator,
|
||||
locator_type=p.locator_type,
|
||||
note=p.note,
|
||||
)
|
||||
)
|
||||
|
||||
return CitationGraph(nodes=nodes, edges=edges)
|
||||
|
||||
|
||||
# ── Injection ────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def format_pincite_block(pincites: list[Pincite]) -> str:
|
||||
"""Format pincites as a docstring References section.
|
||||
|
||||
Returns
|
||||
-------
|
||||
str
|
||||
Formatted block, e.g.::
|
||||
|
||||
References
|
||||
~~~~~~~~~~
|
||||
:pincite:`JX46GQ9L p.14` -- NDC validation rules
|
||||
"""
|
||||
if not pincites:
|
||||
return ""
|
||||
lines = ["References", "~~~~~~~~~~"]
|
||||
for p in pincites:
|
||||
parts = [f":pincite:`{p.item_key}"]
|
||||
if p.locator:
|
||||
parts.append(f" {p.locator}")
|
||||
parts.append("`")
|
||||
if p.note:
|
||||
parts.append(f" -- {p.note}")
|
||||
lines.append("".join(parts))
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def inject_pincites_into_source(
|
||||
source_path: Path,
|
||||
fn_name: str,
|
||||
pincite_block: str,
|
||||
*,
|
||||
dry_run: bool = False,
|
||||
) -> str | None:
|
||||
"""Insert or update a References section in a function's docstring.
|
||||
|
||||
Uses AST to locate the function, then text manipulation to
|
||||
insert the pincite block. Non-destructive: preserves all existing
|
||||
docstring content outside the References block.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
source_path : Path
|
||||
Python source file.
|
||||
fn_name : str
|
||||
Function name to target (unqualified).
|
||||
pincite_block : str
|
||||
Formatted pincite block to insert.
|
||||
dry_run : bool
|
||||
If True, return modified source without writing.
|
||||
|
||||
Returns
|
||||
-------
|
||||
str | None
|
||||
Modified source text if changes were made, None otherwise.
|
||||
"""
|
||||
source = source_path.read_text(encoding="utf-8")
|
||||
try:
|
||||
tree = ast.parse(source, filename=str(source_path))
|
||||
except SyntaxError:
|
||||
return None
|
||||
|
||||
# Find the target function
|
||||
target = None
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
|
||||
if node.name == fn_name:
|
||||
target = node
|
||||
break
|
||||
|
||||
if target is None:
|
||||
return None
|
||||
|
||||
doc_node = target.body[0] if target.body else None
|
||||
if not (
|
||||
isinstance(doc_node, ast.Expr)
|
||||
and isinstance(doc_node.value, ast.Constant)
|
||||
and isinstance(doc_node.value.value, str)
|
||||
):
|
||||
return None # No docstring
|
||||
|
||||
lines = source.splitlines(keepends=True)
|
||||
|
||||
# Detect indentation from the docstring body
|
||||
doc_start = doc_node.lineno - 1 # 0-indexed
|
||||
indent = " " * ((doc_node.col_offset // 4) + 1)
|
||||
|
||||
# Find the closing triple-quote line
|
||||
doc_end = doc_node.end_lineno - 1 # 0-indexed
|
||||
closing_line = lines[doc_end]
|
||||
|
||||
# Check for existing References section
|
||||
ref_start = None
|
||||
for i in range(doc_start, doc_end + 1):
|
||||
stripped = lines[i].strip()
|
||||
if stripped in ("References", "References:", "Sources", "Sources:"):
|
||||
ref_start = i
|
||||
break
|
||||
if stripped.startswith("~~") and ref_start is not None:
|
||||
continue
|
||||
|
||||
# Build indented block
|
||||
indented_lines = []
|
||||
for line in pincite_block.splitlines():
|
||||
if line:
|
||||
indented_lines.append(f"{indent}{line}\n")
|
||||
else:
|
||||
indented_lines.append("\n")
|
||||
|
||||
if ref_start is not None:
|
||||
# Replace from References header to end of docstring (before closing """)
|
||||
new_lines = lines[:ref_start] + indented_lines + [closing_line]
|
||||
if doc_end + 1 < len(lines):
|
||||
new_lines += lines[doc_end + 1 :]
|
||||
else:
|
||||
# Append before closing triple-quote
|
||||
# Insert a blank line separator + the block
|
||||
insert_lines = ["\n"] + indented_lines
|
||||
new_lines = lines[:doc_end] + insert_lines + lines[doc_end:]
|
||||
|
||||
new_source = "".join(new_lines)
|
||||
|
||||
if new_source == source:
|
||||
return None
|
||||
|
||||
if not dry_run:
|
||||
source_path.write_text(new_source, encoding="utf-8")
|
||||
|
||||
return new_source
|
||||
@@ -77,3 +77,19 @@ CREATE TABLE IF NOT EXISTS notes (
|
||||
content TEXT NOT NULL DEFAULT '',
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ','now'))
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS pincites (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
fn_path TEXT NOT NULL,
|
||||
item_key TEXT NOT NULL,
|
||||
locator TEXT NOT NULL DEFAULT '',
|
||||
locator_type TEXT NOT NULL DEFAULT '',
|
||||
note TEXT NOT NULL DEFAULT '',
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ','now')),
|
||||
FOREIGN KEY (item_key) REFERENCES items(key) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_pincites_unique
|
||||
ON pincites(fn_path, item_key, locator);
|
||||
CREATE INDEX IF NOT EXISTS idx_pincites_fn ON pincites(fn_path);
|
||||
CREATE INDEX IF NOT EXISTS idx_pincites_item ON pincites(item_key);
|
||||
|
||||
@@ -527,3 +527,61 @@ class Store:
|
||||
|
||||
items = [self.get(k) for k in keys]
|
||||
return format_bibliography(items, style=style)
|
||||
|
||||
# ── Pincites ─────────────────────────────────────────────────
|
||||
|
||||
def upsert_pincite(
|
||||
self,
|
||||
fn_path: str,
|
||||
item_key: str,
|
||||
locator: str = "",
|
||||
locator_type: str = "",
|
||||
note: str = "",
|
||||
) -> int:
|
||||
"""Insert or update a pincite record. Returns the row id."""
|
||||
from bib.pincite import Pincite, upsert_pincites
|
||||
|
||||
return upsert_pincites(
|
||||
self,
|
||||
[
|
||||
Pincite(
|
||||
fn_path=fn_path,
|
||||
item_key=item_key,
|
||||
locator=locator,
|
||||
locator_type=locator_type,
|
||||
note=note,
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
def list_pincites(
|
||||
self,
|
||||
*,
|
||||
fn_path: str = "",
|
||||
item_key: str = "",
|
||||
) -> list[dict[str, str]]:
|
||||
"""Query pincites with optional filters."""
|
||||
from bib.pincite import list_pincites as _list
|
||||
|
||||
return [p.model_dump() for p in _list(self, fn_path=fn_path, item_key=item_key)]
|
||||
|
||||
def delete_pincites(self, *, fn_path: str = "", item_key: str = "") -> int:
|
||||
"""Delete pincites matching filters. Returns count deleted."""
|
||||
con = self._con()
|
||||
try:
|
||||
con.execute("SELECT 1 FROM pincites LIMIT 0")
|
||||
except Exception:
|
||||
return 0
|
||||
|
||||
sql = "DELETE FROM pincites WHERE 1=1"
|
||||
params: list[str] = []
|
||||
if fn_path:
|
||||
sql += " AND fn_path = ?"
|
||||
params.append(fn_path)
|
||||
if item_key:
|
||||
sql += " AND item_key = ?"
|
||||
params.append(item_key)
|
||||
|
||||
cur = con.execute(sql, params)
|
||||
con.commit()
|
||||
return cur.rowcount
|
||||
|
||||
@@ -218,6 +218,26 @@ class Tag(BaseModel):
|
||||
"""
|
||||
return cls(namespace="col", value=f"{column_ref}={description}")
|
||||
|
||||
@classmethod
|
||||
def pin(cls, item_key: str, locator: str = "") -> Tag:
|
||||
"""Tag linking a bibliography item to a pinpoint citation.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
item_key : str
|
||||
8-char Zotero-compatible item key.
|
||||
locator : str
|
||||
Locator string (page, section, etc.), or empty for
|
||||
item-level citation.
|
||||
|
||||
Examples::
|
||||
|
||||
Tag.pin("JX46GQ9L", "p.14") # pin:JX46GQ9L/p.14
|
||||
Tag.pin("ABC12345") # pin:ABC12345
|
||||
"""
|
||||
value = f"{item_key}/{locator}" if locator else item_key
|
||||
return cls(namespace="pin", value=value)
|
||||
|
||||
@classmethod
|
||||
def fn(cls, qualified_name: str) -> Tag:
|
||||
"""Tag linking a bibliography item to an express function.
|
||||
|
||||
339
tests/bib/test_pincite.py
Normal file
339
tests/bib/test_pincite.py
Normal file
@@ -0,0 +1,339 @@
|
||||
"""Tests for bib.pincite — pinpoint citation system."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import textwrap
|
||||
|
||||
import pytest
|
||||
|
||||
from bib.pincite import (
|
||||
Pincite,
|
||||
_classify_locator,
|
||||
build_citation_graph,
|
||||
check_pincite_keys,
|
||||
format_pincite_block,
|
||||
inject_pincites_into_source,
|
||||
list_pincites,
|
||||
parse_pincites,
|
||||
upsert_pincites,
|
||||
)
|
||||
from bib.store import Store
|
||||
|
||||
# ── Fixtures ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def store(tmp_path):
|
||||
"""In-memory-like store using a temp database."""
|
||||
db = tmp_path / "test_bib.sqlite"
|
||||
s = Store(database=str(db))
|
||||
return s
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def store_with_items(store):
|
||||
"""Store with two items for pincite testing."""
|
||||
from bib.item import Source
|
||||
|
||||
store.create(Source(key="JX46GQ9L", title="Tag vocabulary item"))
|
||||
store.create(Source(key="9ASETLJ4", title="Nature citation standards"))
|
||||
return store
|
||||
|
||||
|
||||
# ── Parse ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TestParsePincites:
|
||||
def test_basic(self):
|
||||
doc = ":pincite:`JX46GQ9L p.14` -- NDC validation rules"
|
||||
result = parse_pincites(doc, "mod.fn")
|
||||
assert len(result) == 1
|
||||
assert result[0].item_key == "JX46GQ9L"
|
||||
assert result[0].locator == "p.14"
|
||||
assert result[0].note == "NDC validation rules"
|
||||
assert result[0].fn_path == "mod.fn"
|
||||
|
||||
def test_no_locator(self):
|
||||
doc = ":pincite:`JX46GQ9L`"
|
||||
result = parse_pincites(doc, "mod.fn")
|
||||
assert len(result) == 1
|
||||
assert result[0].locator == ""
|
||||
|
||||
def test_no_note(self):
|
||||
doc = ":pincite:`JX46GQ9L p.14`"
|
||||
result = parse_pincites(doc, "mod.fn")
|
||||
assert len(result) == 1
|
||||
assert result[0].note == ""
|
||||
|
||||
def test_section_locator(self):
|
||||
doc = ":pincite:`9ASETLJ4 §3.2` — exclusion criteria"
|
||||
result = parse_pincites(doc, "mod.fn")
|
||||
assert result[0].locator == "§3.2"
|
||||
assert result[0].locator_type == "section"
|
||||
assert result[0].note == "exclusion criteria"
|
||||
|
||||
def test_cfr_locator(self):
|
||||
doc = ":pincite:`ABC23456 42 CFR §414.22` -- payment rates"
|
||||
result = parse_pincites(doc, "mod.fn")
|
||||
assert result[0].locator == "42 CFR §414.22"
|
||||
assert result[0].locator_type == "cfr"
|
||||
|
||||
def test_fr_locator(self):
|
||||
doc = ":pincite:`DEF78923 90 FR 86252` -- final rule"
|
||||
result = parse_pincites(doc, "mod.fn")
|
||||
assert result[0].locator == "90 FR 86252"
|
||||
assert result[0].locator_type == "fr"
|
||||
|
||||
def test_multiple(self):
|
||||
doc = textwrap.dedent("""
|
||||
:pincite:`JX46GQ9L p.14` -- first
|
||||
:pincite:`9ASETLJ4 §3.2` -- second
|
||||
:pincite:`ABC23456 Ch. 12` -- third
|
||||
""")
|
||||
result = parse_pincites(doc, "mod.fn")
|
||||
assert len(result) == 3
|
||||
|
||||
def test_deduplication(self):
|
||||
doc = ":pincite:`JX46GQ9L p.14` -- first\n:pincite:`JX46GQ9L p.14` -- dupe"
|
||||
result = parse_pincites(doc, "mod.fn")
|
||||
assert len(result) == 1
|
||||
|
||||
def test_empty_docstring(self):
|
||||
assert parse_pincites("", "mod.fn") == []
|
||||
assert parse_pincites(None, "mod.fn") == []
|
||||
|
||||
def test_no_pincites(self):
|
||||
doc = "Just a plain docstring with no citations."
|
||||
assert parse_pincites(doc, "mod.fn") == []
|
||||
|
||||
def test_invalid_key_charset(self):
|
||||
# 0, 1, O, L are not valid Zotero key chars
|
||||
doc = ":pincite:`00000000 p.1` -- bad key"
|
||||
assert parse_pincites(doc, "mod.fn") == []
|
||||
|
||||
def test_em_dash(self):
|
||||
doc = ":pincite:`JX46GQ9L p.14` — em dash note"
|
||||
result = parse_pincites(doc, "mod.fn")
|
||||
assert result[0].note == "em dash note"
|
||||
|
||||
|
||||
# ── Classify locator ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TestClassifyLocator:
|
||||
def test_page(self):
|
||||
assert _classify_locator("p.14") == "page"
|
||||
assert _classify_locator("pp.8-12") == "page"
|
||||
|
||||
def test_section(self):
|
||||
assert _classify_locator("§3.2") == "section"
|
||||
|
||||
def test_cfr(self):
|
||||
assert _classify_locator("42 CFR §414.22") == "cfr"
|
||||
|
||||
def test_fr(self):
|
||||
assert _classify_locator("90 FR 86252") == "fr"
|
||||
|
||||
def test_chapter(self):
|
||||
assert _classify_locator("Ch. 12") == "chapter"
|
||||
|
||||
def test_paragraph(self):
|
||||
assert _classify_locator("¶4") == "paragraph"
|
||||
|
||||
def test_empty(self):
|
||||
assert _classify_locator("") == ""
|
||||
|
||||
|
||||
# ── Store operations ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TestUpsertPincites:
|
||||
def test_insert(self, store_with_items):
|
||||
pincites = [
|
||||
Pincite(fn_path="mod.fn", item_key="JX46GQ9L", locator="p.14", note="test"),
|
||||
]
|
||||
count = upsert_pincites(store_with_items, pincites)
|
||||
assert count == 1
|
||||
|
||||
result = list_pincites(store_with_items, fn_path="mod.fn")
|
||||
assert len(result) == 1
|
||||
assert result[0].item_key == "JX46GQ9L"
|
||||
|
||||
def test_upsert_idempotent(self, store_with_items):
|
||||
p = Pincite(fn_path="mod.fn", item_key="JX46GQ9L", locator="p.14")
|
||||
upsert_pincites(store_with_items, [p])
|
||||
upsert_pincites(store_with_items, [p])
|
||||
result = list_pincites(store_with_items, fn_path="mod.fn")
|
||||
assert len(result) == 1
|
||||
|
||||
def test_skips_missing_item(self, store_with_items):
|
||||
p = Pincite(fn_path="mod.fn", item_key="ZZZZZZZZ", locator="p.1")
|
||||
count = upsert_pincites(store_with_items, [p])
|
||||
assert count == 0
|
||||
|
||||
def test_tags_added(self, store_with_items):
|
||||
p = Pincite(fn_path="mod.fn", item_key="JX46GQ9L", locator="p.14")
|
||||
upsert_pincites(store_with_items, [p])
|
||||
item = store_with_items.get("JX46GQ9L")
|
||||
assert any("pin:" in t for t in item.tags)
|
||||
|
||||
|
||||
class TestListPincites:
|
||||
def test_by_fn(self, store_with_items):
|
||||
upsert_pincites(
|
||||
store_with_items,
|
||||
[
|
||||
Pincite(fn_path="a.b", item_key="JX46GQ9L", locator="p.1"),
|
||||
Pincite(fn_path="c.d", item_key="9ASETLJ4", locator="§2"),
|
||||
],
|
||||
)
|
||||
result = list_pincites(store_with_items, fn_path="a.b")
|
||||
assert len(result) == 1
|
||||
assert result[0].item_key == "JX46GQ9L"
|
||||
|
||||
def test_by_item(self, store_with_items):
|
||||
upsert_pincites(
|
||||
store_with_items,
|
||||
[
|
||||
Pincite(fn_path="a.b", item_key="JX46GQ9L", locator="p.1"),
|
||||
Pincite(fn_path="c.d", item_key="JX46GQ9L", locator="p.2"),
|
||||
],
|
||||
)
|
||||
result = list_pincites(store_with_items, item_key="JX46GQ9L")
|
||||
assert len(result) == 2
|
||||
|
||||
def test_empty_store(self, store):
|
||||
assert list_pincites(store) == []
|
||||
|
||||
|
||||
# ── Check keys ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TestCheckKeys:
|
||||
def test_valid(self, store_with_items):
|
||||
p = Pincite(fn_path="mod.fn", item_key="JX46GQ9L")
|
||||
errors = check_pincite_keys([p], store_with_items)
|
||||
assert len(errors) == 0
|
||||
|
||||
def test_invalid(self, store_with_items):
|
||||
p = Pincite(fn_path="mod.fn", item_key="NOTFOUND")
|
||||
errors = check_pincite_keys([p], store_with_items)
|
||||
assert len(errors) == 1
|
||||
assert "not found" in errors[0][1]
|
||||
|
||||
|
||||
# ── Knowledge graph ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TestCitationGraph:
|
||||
def test_build(self, store_with_items):
|
||||
upsert_pincites(
|
||||
store_with_items,
|
||||
[
|
||||
Pincite(fn_path="mod.fn_a", item_key="JX46GQ9L", locator="p.1"),
|
||||
Pincite(fn_path="mod.fn_b", item_key="JX46GQ9L", locator="p.2"),
|
||||
Pincite(fn_path="mod.fn_a", item_key="9ASETLJ4", locator="§3"),
|
||||
],
|
||||
)
|
||||
graph = build_citation_graph(store_with_items)
|
||||
assert len(graph.nodes) == 4 # 2 functions + 2 items
|
||||
assert len(graph.edges) == 3
|
||||
|
||||
def test_functions_citing(self, store_with_items):
|
||||
upsert_pincites(
|
||||
store_with_items,
|
||||
[
|
||||
Pincite(fn_path="a.b", item_key="JX46GQ9L"),
|
||||
Pincite(fn_path="c.d", item_key="JX46GQ9L"),
|
||||
],
|
||||
)
|
||||
graph = build_citation_graph(store_with_items)
|
||||
fns = graph.functions_citing("JX46GQ9L")
|
||||
assert set(fns) == {"a.b", "c.d"}
|
||||
|
||||
def test_items_cited_by(self, store_with_items):
|
||||
upsert_pincites(
|
||||
store_with_items,
|
||||
[
|
||||
Pincite(fn_path="a.b", item_key="JX46GQ9L"),
|
||||
Pincite(fn_path="a.b", item_key="9ASETLJ4"),
|
||||
],
|
||||
)
|
||||
graph = build_citation_graph(store_with_items)
|
||||
items = graph.items_cited_by("a.b")
|
||||
assert set(items) == {"JX46GQ9L", "9ASETLJ4"}
|
||||
|
||||
def test_to_mermaid(self, store_with_items):
|
||||
upsert_pincites(
|
||||
store_with_items,
|
||||
[
|
||||
Pincite(fn_path="mod.fn", item_key="JX46GQ9L", locator="p.1"),
|
||||
],
|
||||
)
|
||||
graph = build_citation_graph(store_with_items)
|
||||
mermaid = graph.to_mermaid()
|
||||
assert "graph LR" in mermaid
|
||||
assert "mod_fn" in mermaid
|
||||
|
||||
def test_to_networkx_dict(self, store_with_items):
|
||||
upsert_pincites(
|
||||
store_with_items,
|
||||
[
|
||||
Pincite(fn_path="mod.fn", item_key="JX46GQ9L"),
|
||||
],
|
||||
)
|
||||
graph = build_citation_graph(store_with_items)
|
||||
d = graph.to_networkx_dict()
|
||||
assert d["directed"] is True
|
||||
assert len(d["nodes"]) == 2
|
||||
assert len(d["links"]) == 1
|
||||
|
||||
|
||||
# ── Format + inject ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TestFormatPinciteBlock:
|
||||
def test_single(self):
|
||||
p = Pincite(fn_path="m.f", item_key="JX46GQ9L", locator="p.14", note="test")
|
||||
block = format_pincite_block([p])
|
||||
assert ":pincite:`JX46GQ9L p.14` -- test" in block
|
||||
assert block.startswith("References")
|
||||
|
||||
def test_no_locator(self):
|
||||
p = Pincite(fn_path="m.f", item_key="JX46GQ9L")
|
||||
block = format_pincite_block([p])
|
||||
assert ":pincite:`JX46GQ9L`" in block
|
||||
|
||||
def test_empty(self):
|
||||
assert format_pincite_block([]) == ""
|
||||
|
||||
|
||||
class TestInjectPincites:
|
||||
def test_append_to_docstring(self, tmp_path):
|
||||
source = textwrap.dedent('''
|
||||
def foo():
|
||||
"""Existing docstring."""
|
||||
return 1
|
||||
''').lstrip()
|
||||
f = tmp_path / "test_mod.py"
|
||||
f.write_text(source)
|
||||
|
||||
block = "References\n~~~~~~~~~~\n:pincite:`JX46GQ9L p.14` -- test"
|
||||
result = inject_pincites_into_source(f, "foo", block, dry_run=True)
|
||||
assert result is not None
|
||||
assert ":pincite:`JX46GQ9L p.14`" in result
|
||||
assert "Existing docstring." in result
|
||||
|
||||
def test_skip_no_docstring(self, tmp_path):
|
||||
source = "def foo():\n return 1\n"
|
||||
f = tmp_path / "test_mod.py"
|
||||
f.write_text(source)
|
||||
result = inject_pincites_into_source(f, "foo", "block")
|
||||
assert result is None
|
||||
|
||||
def test_skip_missing_function(self, tmp_path):
|
||||
source = "def bar():\n pass\n"
|
||||
f = tmp_path / "test_mod.py"
|
||||
f.write_text(source)
|
||||
result = inject_pincites_into_source(f, "foo", "block")
|
||||
assert result is None
|
||||
Reference in New Issue
Block a user