feat(bib): nightly zotero-sync workflow — mail updates froze in bib for 2 months
All checks were successful
CI / lint (push) Successful in 31s
CI / notebooks-smoke (push) Successful in 1m22s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / mc (push) Has been skipped
Deploy / report (push) Successful in 19s
CI / test (push) Successful in 13m20s
All checks were successful
CI / lint (push) Successful in 31s
CI / notebooks-smoke (push) Successful in 1m22s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / mc (push) Has been skipped
Deploy / report (push) Successful in 19s
CI / test (push) Successful in 13m20s
mail-poller ingests CMS mail into bib continuously, but the bib→Zotero routing (stack bib sync-zotero) had no scheduler — last manual run was Apr 29, so the Cmsupdates collection silently stopped updating while bib accumulated a 212-item backlog (drained today: 270 items now in the collection). New generated workflow zotero-sync.yml (04:15 nightly + dispatch): stop zotero (it holds an exclusive SQLite write lock) → docker exec api: sync-zotero --tag source:email --no-hold → restart zotero (if:always). Pre-flighted the exact exec path: idempotent re-run skips all 270.
This commit is contained in:
45
.gitea/workflows/zotero-sync.yml
Normal file
45
.gitea/workflows/zotero-sync.yml
Normal file
@@ -0,0 +1,45 @@
|
||||
# DO NOT EDIT — generated by gen_config.py from stack.toml
|
||||
# Re-generate: uv run python dev/scripts/gen_config.py
|
||||
|
||||
name: Zotero Sync
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
- cron: "15 4 * * *"
|
||||
|
||||
jobs:
|
||||
zotero-sync:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: https://github.com/actions/checkout@v4
|
||||
|
||||
- name: Set up uv
|
||||
run: curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
env:
|
||||
UV_INSTALL_DIR: /usr/local/bin
|
||||
|
||||
- name: Stop zotero (release SQLite write lock)
|
||||
run: docker stop zotero
|
||||
|
||||
- name: Sync mail-ingested bib items into Zotero
|
||||
run: |
|
||||
docker exec api uv run --no-sync \
|
||||
stack bib sync-zotero --tag source:email --no-hold
|
||||
|
||||
- name: Restart zotero
|
||||
if: always()
|
||||
run: docker start zotero
|
||||
|
||||
- name: File failure issue
|
||||
if: failure()
|
||||
env:
|
||||
GITEA_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
|
||||
run: |
|
||||
uv sync --no-dev --quiet 2>/dev/null || true
|
||||
uv run python -m api.diag.ci \
|
||||
--workflow "Zotero Sync" --job "zotero-sync" \
|
||||
--run "${{ github.run_number }}" \
|
||||
--sha "${{ github.sha }}" \
|
||||
--ref "${{ github.ref }}" || true
|
||||
@@ -580,6 +580,54 @@ jobs:
|
||||
return (".gitea/workflows/notebooks-integration.yml", content)
|
||||
|
||||
|
||||
def _gen_zotero_sync(runner: str, uv_version: str, **_kw: object) -> tuple[str, str]:
|
||||
"""Nightly bib → Zotero routing for mail-ingested items.
|
||||
|
||||
mail-poller ingests CMS mail into bib continuously, but Zotero only
|
||||
sees those items when `stack bib sync-zotero` pushes them — and that
|
||||
had no scheduler, so the Cmsupdates collection silently froze for two
|
||||
months (Apr 29 → Jul 10, 212-item backlog). sync writes directly into
|
||||
zotero.sqlite, so the running Zotero app must be stopped for the
|
||||
duration; the stop/start lives in workflow steps (visible, and
|
||||
restart is if:always()) rather than the CLI's --hold-zotero, because
|
||||
the sync runs docker-exec'd inside the api container, which has no
|
||||
docker socket. 04:15 sits after the 03:30 notebooks-integration run
|
||||
so the two never contend for zotero.sqlite.
|
||||
"""
|
||||
content = f"""\
|
||||
{_HEADER}
|
||||
name: Zotero Sync
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
- cron: "15 4 * * *"
|
||||
|
||||
jobs:
|
||||
zotero-sync:
|
||||
runs-on: {runner}
|
||||
steps:
|
||||
{_checkout_step()}
|
||||
|
||||
{_setup_uv_step(uv_version)}
|
||||
|
||||
- name: Stop zotero (release SQLite write lock)
|
||||
run: docker stop zotero
|
||||
|
||||
- name: Sync mail-ingested bib items into Zotero
|
||||
run: |
|
||||
docker exec api uv run --no-sync \\
|
||||
stack bib sync-zotero --tag source:email --no-hold
|
||||
|
||||
- name: Restart zotero
|
||||
if: always()
|
||||
run: docker start zotero
|
||||
|
||||
{_failure_step("Zotero Sync", "zotero-sync")}
|
||||
"""
|
||||
return (".gitea/workflows/zotero-sync.yml", content)
|
||||
|
||||
|
||||
def _gen_release(runner: str, uv_version: str, **_kw: object) -> tuple[str, str]:
|
||||
content = f"""\
|
||||
{_HEADER}
|
||||
@@ -646,6 +694,7 @@ def emit(
|
||||
_gen_rebuild_all,
|
||||
_gen_infra_ci,
|
||||
_gen_notebooks_integration,
|
||||
_gen_zotero_sync,
|
||||
_gen_release,
|
||||
):
|
||||
path, content = gen_fn(**common) # type: ignore[arg-type]
|
||||
|
||||
Reference in New Issue
Block a user