c9b6a28f07
Some checks failed
CI / lint (push) Successful in 45s
CI / notebooks-smoke (push) Successful in 1m31s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / mc (push) Has been skipped
Infra CI / notebooks (push) Successful in 57s
Infra CI / zotero (push) Successful in 14s
Infra CI / docs (push) Successful in 1m43s
CI / test (push) Has been cancelled
Deploy / report (push) Has been cancelled
Infra CI / mc (push) Has been cancelled
Infra CI / api (push) Has been cancelled
rex.comments.coordination: hti5 methodology — Jaccard similarity on character 5-gram shingles (threshold 0.45) over normalized text, union-find grouping with stable group ids, form-letter flag at >=3 members. Short texts (<200 chars) are excluded: two short 'I oppose' notes are agreement, not coordination. rex.comments.table: shared load path for skin_subs.rulemaking_comments — outer-joins the classification (#254) and coordination (#255) JSONL caches onto the comment identity rows, so whichever pass runs first populates the table and the other enriches it without clobbering. classify_comments.py refactored onto it; new analyze_coordination.py driver. Run against CMS-2025-0304 (the CY2026 OPPS skin-sub docket): of the 384 skin-sub-relevant comments, 203 (53%) are form letters across 31 coordinated groups — largest campaigns 72 and 36 members. Table loaded: 384 rows, coordination columns filled, classification columns NULL until the #254 LLM run (blocked on API credits). 59 comments tests green.
82 lines
2.7 KiB
Python
82 lines
2.7 KiB
Python
"""Coordination detection over skin-sub-relevant comments (#255).
|
|
|
|
No LLM involved: near-duplicate detection via Jaccard similarity on
|
|
character 5-gram shingles (hti5 methodology, threshold 0.45),
|
|
union-find grouping, and form-letter flagging by group size. Results
|
|
land in a JSONL cache and the docket's slice of
|
|
skin_subs.rulemaking_comments (merging the classification cache from
|
|
classify_comments.py when present).
|
|
|
|
Usage:
|
|
uv run python dev/scripts/analyze_coordination.py --docket CMS-2025-0304
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import json
|
|
from collections import Counter
|
|
from pathlib import Path
|
|
|
|
ROOT = Path(__file__).resolve().parents[2]
|
|
BIB_PATH = ROOT / "data" / "bib.sqlite"
|
|
DEFAULT_DOCKET = "CMS-2025-0304"
|
|
|
|
|
|
def main() -> int:
|
|
parser = argparse.ArgumentParser(description=__doc__)
|
|
parser.add_argument("--docket", default=DEFAULT_DOCKET)
|
|
parser.add_argument(
|
|
"--no-load", action="store_true", help="analyze but skip the DuckDB load"
|
|
)
|
|
args = parser.parse_args()
|
|
|
|
from rex.comments.classify import is_relevant
|
|
from rex.comments.coordination import analyze
|
|
from rex.comments.table import load_comments, read_cache
|
|
|
|
comments = load_comments(args.docket, BIB_PATH)
|
|
relevant = [c for c in comments if is_relevant(c["text"])]
|
|
print(f"{args.docket}: {len(comments)} comments, {len(relevant)} skin-sub relevant")
|
|
|
|
results = analyze({c["key"]: c["text"] for c in relevant})
|
|
|
|
by_id = {c["key"]: c["comment_id"] for c in relevant}
|
|
groups = Counter(
|
|
r["coordination_group"] for r in results.values() if r["coordination_group"]
|
|
)
|
|
form = sum(1 for r in results.values() if r["is_form_letter"])
|
|
print(f"coordinated groups: {len(groups)}, form-letter comments: {form}")
|
|
for gid, n in groups.most_common(10):
|
|
print(f" group {by_id.get(gid, gid)}: {n} members")
|
|
|
|
base = ROOT / "data" / "cms"
|
|
base.mkdir(parents=True, exist_ok=True)
|
|
coordination_path = base / f"comments-coordination-{args.docket}.jsonl"
|
|
with coordination_path.open("w") as fh:
|
|
for key, rec in sorted(results.items()):
|
|
fh.write(json.dumps({"key": key, **rec}) + "\n")
|
|
print(f"cache → {coordination_path}")
|
|
|
|
if args.no_load:
|
|
return 0
|
|
|
|
from conf.connect import duckdb_batch
|
|
from rex.comments.table import load_table
|
|
|
|
classified_path = base / f"comments-classified-{args.docket}.jsonl"
|
|
with duckdb_batch("aco") as con:
|
|
n = load_table(
|
|
con,
|
|
args.docket,
|
|
relevant,
|
|
classified=read_cache(classified_path),
|
|
coordination=results,
|
|
)
|
|
print(f"skin_subs.rulemaking_comments: {n} rows for {args.docket}")
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|