225ce01d6b
zotero.fhirworx.io rendered a blank desktop: KDE+Zotero draw on Xvfb :20, but kasmvnc serves :21, and the kasmxproxy mirror that bridges them never started. The stock kasmvnc-entrypoint.sh blocks on `until [ -S /tmp/.X11-unix/X21 ]` (a filesystem socket path) while entrypoint.sh's concurrent `rm -rf /tmp/.X*` unlinks it — Xvnc keeps its fd + abstract socket open and serves a blank :21 forever. A startup race, hence the intermittent "doesn't open" with every process reporting healthy. - kasmxproxy-guard.sh: supervised watchdog that probes :20/:21 via xdpyinfo (not the racy socket file) and (re)launches kasmxproxy, self-healing if it dies. Emits structured `zotero_event=` logs to container stdout + a heartbeat file. - zotero-healthcheck.sh + compose healthcheck: detect the real failure mode (bridge down / :21 unreachable / KDE dead / stale heartbeat) that container/nginx/Xvnc liveness all miss. - promtail: extract zotero_event into a Loki label. - Grafana: Zotero Desktop dashboard + alert on sustained bridge failure. - otel-cli baked in; OTLP tracing dormant by default (enabling it requires attaching zotero to the observability network — documented opt-in, left off to preserve gateway-only isolation). Verified live: guard self-heals on kasmxproxy kill; healthcheck passes healthy / fails on stale heartbeat.
86 lines
2.8 KiB
YAML
86 lines
2.8 KiB
YAML
# Grafana unified-alerting provisioning for the Zotero desktop bridge.
|
|
#
|
|
# Fires when the kasmxproxy-guard watchdog reports the :20->:21 mirror as
|
|
# genuinely failing (kasmxproxy won't start, or a display is down) for a
|
|
# sustained window — i.e. the browser would see a blank desktop. The
|
|
# watchdog's self-healing bridge_down -> bridge_up blips are intentionally
|
|
# NOT alerted (they recover within seconds); we key off bridge_start_failed
|
|
# and *_display_down, gated by `for: 5m`.
|
|
#
|
|
# NOTE: alert-rule provisioning JSON shifts between Grafana versions (this
|
|
# stack tracks grafana:latest). Verify in the UI after deploy:
|
|
# Alerting -> Alert rules -> "Zotero desktop bridge failing".
|
|
# Routing to an actual channel additionally needs a contact point /
|
|
# notification policy; unrouted, the rule still shows as Firing in the UI.
|
|
apiVersion: 1
|
|
|
|
groups:
|
|
- orgId: 1
|
|
name: zotero
|
|
folder: Zotero
|
|
interval: 1m
|
|
rules:
|
|
- uid: zotero-bridge-failing
|
|
title: Zotero desktop bridge failing
|
|
condition: C
|
|
for: 5m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
labels:
|
|
severity: warning
|
|
service: zotero
|
|
annotations:
|
|
summary: >-
|
|
Zotero KDE desktop bridge (:20->:21) is failing — the browser sees a
|
|
blank screen at zotero.fhirworx.io.
|
|
description: >-
|
|
kasmxproxy-guard reported bridge_start_failed or a display down for
|
|
5m+. The KasmVNC display the browser connects to is not being
|
|
driven. Check `docker logs zotero` and the "Zotero Desktop"
|
|
dashboard.
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange:
|
|
from: 600
|
|
to: 0
|
|
datasourceUid: loki
|
|
model:
|
|
refId: A
|
|
datasource:
|
|
type: loki
|
|
uid: loki
|
|
queryType: instant
|
|
expr: >-
|
|
sum(count_over_time({container="zotero",
|
|
zotero_event=~"bridge_start_failed|dst_display_down|src_display_down"}
|
|
[5m]))
|
|
- refId: B
|
|
relativeTimeRange:
|
|
from: 600
|
|
to: 0
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: B
|
|
type: reduce
|
|
reducer: last
|
|
expression: A
|
|
datasource:
|
|
type: __expr__
|
|
uid: __expr__
|
|
- refId: C
|
|
relativeTimeRange:
|
|
from: 600
|
|
to: 0
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: B
|
|
datasource:
|
|
type: __expr__
|
|
uid: __expr__
|
|
conditions:
|
|
- evaluator:
|
|
type: gt
|
|
params: [0]
|