Offsite Watchdog

External monitoring for a Docker Swarm cluster. Runs on an offsite server over VPN to detect and alert when the entire cluster goes down.

Problem

If all monitoring (Prometheus, Alertmanager, ntfy) runs inside the cluster, a full cluster outage means zero alerts. This stack solves that.

Components

Container	Purpose	Port
Uptime Kuma	Monitors cluster endpoints, fires alerts when they're down	3001
ntfy (offsite)	Push notification server independent of the cluster	9281

A Watchdog alert in Prometheus (configured in the infra repo) is always firing. Uptime Kuma monitors it via the Alertmanager API. If the heartbeat disappears, Uptime Kuma sends a notification through the offsite ntfy instance.

Architecture

                         ┌─── Docker Swarm Cluster ───┐
                         │                             │
                         │  Prometheus ──► Alertmanager │
                         │      │              │        │
                         │  Watchdog alert     │        │
                         │  (always firing)    ▼        │
                         │                   ntfy       │
                         │               (in-cluster)   │
                         │                              │
                         └──────────────────────────────┘
                                      │
                                   VPN link
                                      │
                         ┌─── Offsite Server ───────────┐
                         │                               │
                         │  Uptime Kuma ─── probes ──►   │
                         │      │         cluster URLs   │
                         │      │                        │
                         │      │  monitors Alertmanager │
                         │      │  API for Watchdog      │
                         │      │  heartbeat             │
                         │      │                        │
                         │      ▼                        │
                         │  ntfy (offsite, port 9281)    │
                         │      │                        │
                         │      ▼                        │
                         │  Push notification to phone   │
                         │                               │
                         └───────────────────────────────┘

Deployment

1. Clone and start on the offsite server

git clone https://gitea.ignohr.com/Intech/offsite-watchdog.git
cd offsite-watchdog
docker compose up -d

2. Cluster-side setup (infra repo)

The Watchdog alert and watchdog-silent Alertmanager receiver must be deployed in the main infra repo. These are already configured in:

• resources/monitoring/alert-rules.yml -- the Watchdog always-firing alert
• resources/monitoring/alertmanager.yml -- the watchdog-silent receiver

Deploy via:

docker stack deploy --compose-file services/monitoring/docker-compose.yml --with-registry-auth monitoring

3. Subscribe to the offsite ntfy topic

Open the ntfy app on your phone and add a new subscription:

• Server URL: http://<offsite-server-ip>:9281
• Topic: cluster-alerts (or any topic name you choose)

4. Configure Uptime Kuma

Access Uptime Kuma at http://<offsite-server-ip>:3001 and complete the initial setup.

a. Notification channel (offsite ntfy)

1. Go to Settings > Notifications > Setup Notification
2. Type: ntfy
3. Server URL: http://offsite-ntfy:9281 (Docker network name) or http://localhost:9281
4. Topic: cluster-alerts
5. Priority: 5 (max/urgent)
6. Check "Apply on all existing monitors"
7. Save

b. HTTP endpoint monitors

Create one monitor per critical service:

Monitor Name	URL	Interval	Retries	Retry Interval
Gitea	https://gitea.ignohr.com	60s	3	60s
Grafana	https://grafana.ignohr.com	60s	3	60s
Odoo	https://odoo.ignohr.com	60s	3	60s
OpenCode	https://opencode.ignohr.com	60s	3	60s
Vaultwarden	https://pwd.ignohr.com	60s	3	60s
ntfy (cluster)	https://ntfy.ignohr.com	60s	3	60s
MeshCentral	https://mesh.ignohr.com	60s	3	60s
Nextcloud	https://nextcloud.ignohr.com	60s	3	60s

c. Watchdog heartbeat (dead man's switch)

This is the primary "cluster down" detector:

1. Create a new monitor: Type = HTTP(s) - JSON Query
2. Name: Cluster Watchdog Heartbeat
3. URL: http://docker.ignohr.com:9093/api/v2/alerts?filter=alertname%3D%22Watchdog%22
4. JSON Query: $[0].status.state
5. Expected Value: active
6. Heartbeat Interval: 60s
7. Retries: 3
8. Retry Interval: 60s

Simpler alternative -- just check Alertmanager and Prometheus health:

Monitor Name	Type	URL	Expected Status
Alertmanager Health	HTTP(s)	http://docker.ignohr.com:9093/-/healthy	200
Prometheus Health	HTTP(s)	http://docker.ignohr.com:9090/-/healthy	200

d. Ping monitors (optional)

Monitor Name	Type	Target
dockerhost01	Ping	10.0.1.176
dockerhost02	Ping	10.0.1.182

What Gets Detected

Failure Scenario	Detected By
Entire cluster down	Watchdog heartbeat gone + all HTTP monitors fail
Single node down	In-cluster alerts (existing) + some HTTP monitors may fail
Docker Swarm broken	HTTP endpoint monitors fail
Prometheus crashed	Watchdog heartbeat disappears from Alertmanager
Alertmanager crashed	Alertmanager health check fails
Network/VPN down	All monitors fail simultaneously
Individual service down	Specific HTTP endpoint monitor fails