feat(bib): nightly zotero-sync workflow — mail updates froze in bib for 2 months
All checks were successful
CI / lint (push) Successful in 31s
CI / notebooks-smoke (push) Successful in 1m22s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / mc (push) Has been skipped
Deploy / report (push) Successful in 19s
CI / test (push) Successful in 13m20s

mail-poller ingests CMS mail into bib continuously, but the bib→Zotero
routing (stack bib sync-zotero) had no scheduler — last manual run was
Apr 29, so the Cmsupdates collection silently stopped updating while
bib accumulated a 212-item backlog (drained today: 270 items now in
the collection).

New generated workflow zotero-sync.yml (04:15 nightly + dispatch):
stop zotero (it holds an exclusive SQLite write lock) → docker exec
api: sync-zotero --tag source:email --no-hold → restart zotero
(if:always). Pre-flighted the exact exec path: idempotent re-run
skips all 270.
This commit is contained in:
kert
2026-07-10 15:32:30 -04:00
parent e8f6385bc5
commit 9c8a5f1e73
2 changed files with 94 additions and 0 deletions

View File

@@ -0,0 +1,45 @@
# DO NOT EDIT — generated by gen_config.py from stack.toml
# Re-generate: uv run python dev/scripts/gen_config.py
name: Zotero Sync
on:
workflow_dispatch:
schedule:
- cron: "15 4 * * *"
jobs:
zotero-sync:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: https://github.com/actions/checkout@v4
- name: Set up uv
run: curl -LsSf https://astral.sh/uv/install.sh | sh
env:
UV_INSTALL_DIR: /usr/local/bin
- name: Stop zotero (release SQLite write lock)
run: docker stop zotero
- name: Sync mail-ingested bib items into Zotero
run: |
docker exec api uv run --no-sync \
stack bib sync-zotero --tag source:email --no-hold
- name: Restart zotero
if: always()
run: docker start zotero
- name: File failure issue
if: failure()
env:
GITEA_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
run: |
uv sync --no-dev --quiet 2>/dev/null || true
uv run python -m api.diag.ci \
--workflow "Zotero Sync" --job "zotero-sync" \
--run "${{ github.run_number }}" \
--sha "${{ github.sha }}" \
--ref "${{ github.ref }}" || true

View File

@@ -580,6 +580,54 @@ jobs:
return (".gitea/workflows/notebooks-integration.yml", content) return (".gitea/workflows/notebooks-integration.yml", content)
def _gen_zotero_sync(runner: str, uv_version: str, **_kw: object) -> tuple[str, str]:
"""Nightly bib → Zotero routing for mail-ingested items.
mail-poller ingests CMS mail into bib continuously, but Zotero only
sees those items when `stack bib sync-zotero` pushes them — and that
had no scheduler, so the Cmsupdates collection silently froze for two
months (Apr 29 → Jul 10, 212-item backlog). sync writes directly into
zotero.sqlite, so the running Zotero app must be stopped for the
duration; the stop/start lives in workflow steps (visible, and
restart is if:always()) rather than the CLI's --hold-zotero, because
the sync runs docker-exec'd inside the api container, which has no
docker socket. 04:15 sits after the 03:30 notebooks-integration run
so the two never contend for zotero.sqlite.
"""
content = f"""\
{_HEADER}
name: Zotero Sync
on:
workflow_dispatch:
schedule:
- cron: "15 4 * * *"
jobs:
zotero-sync:
runs-on: {runner}
steps:
{_checkout_step()}
{_setup_uv_step(uv_version)}
- name: Stop zotero (release SQLite write lock)
run: docker stop zotero
- name: Sync mail-ingested bib items into Zotero
run: |
docker exec api uv run --no-sync \\
stack bib sync-zotero --tag source:email --no-hold
- name: Restart zotero
if: always()
run: docker start zotero
{_failure_step("Zotero Sync", "zotero-sync")}
"""
return (".gitea/workflows/zotero-sync.yml", content)
def _gen_release(runner: str, uv_version: str, **_kw: object) -> tuple[str, str]: def _gen_release(runner: str, uv_version: str, **_kw: object) -> tuple[str, str]:
content = f"""\ content = f"""\
{_HEADER} {_HEADER}
@@ -646,6 +694,7 @@ def emit(
_gen_rebuild_all, _gen_rebuild_all,
_gen_infra_ci, _gen_infra_ci,
_gen_notebooks_integration, _gen_notebooks_integration,
_gen_zotero_sync,
_gen_release, _gen_release,
): ):
path, content = gen_fn(**common) # type: ignore[arg-type] path, content = gen_fn(**common) # type: ignore[arg-type]