Keep it running
Last updated
Operating a Weald relay means watching process liveness, dependency readiness, storage growth and whether the running image is behind a security release. There is no admin panel and nothing to administer inside a workspace. This page lists what to alert on and what each signal means.
Health endpoints
The public listener answers liveness and nothing else. Readiness and metrics bind to a private listener, on loopback by default, so a self-hosted relay keeps its storage totals and security posture off the internet.
| Endpoint | Listener | Answers |
|---|---|---|
/healthz | public | Whether the process is alive. No state beyond that |
/readyz | private | Postgres and object storage reachability, access set mode, release check result |
/metrics | private | Prometheus metrics, with nothing content-derived in them |
Metrics carry nothing derived from content, because the relay does not know channel names, ticket titles or who is in a group. Per-group envelope counts are off by default. Consider what that label reveals before you enable it.
What to alert on
/healthzfailing, which is the process being down./readyzfailing, which usually points at Postgres or the bucket.- Storage crossing a threshold you picked, well before
WEALD_RELAY_MAX_STORAGE_GBif you set one. A relay at its storage ceiling refuses writes with a quota error, which members see as work that will not save. - The release check reporting that the running digest is behind a release carrying a security advisory.
What the numbers mean
Peak concurrent connections measures capacity and does not count people. One person with a Mac, a phone and two agents is several connections, and a workspace member who is asleep is none.
Storage grows as workspaces are used. It falls when a workspace compacts, which is a control in the Mac app. The relay cannot compact by itself, because compaction is a decision about content.
Backpressure and errors
A relay under load slows down and does not drop envelopes. A dropped envelope leaves a hole in an author chain, which raises a security alarm on another member's screen. Per-connection queues are bounded, and a full queue stops reading the socket, pushing back through TCP.
Every error a client receives carries a class, so a client branches on a code instead of a string.
| Class | Meaning | What the client does |
|---|---|---|
retry | Transient infrastructure trouble | Backs off and resends the same bytes |
reject | Permanently wrong as sent | Stops, keeps the payload, surfaces a defect |
denied | Well formed, not permitted now | Re-reads the state named in the error |
quota | Over a storage, seat or rate limit | Surfaces the limit and the lever that clears it |
version | Protocol version unsupported | Aborts the connection |
Maintenance mode
WEALD_RELAY_WRITE_MODE=read_only refuses new durable writes while leaving subscription, reconciliation, backup and export working. Clients are told with a non-content reason code, so the app can say the relay is in maintenance instead of showing an unexplained failure.
It names no billing system and contacts nothing outside the relay.
Troubleshooting, without content access
A support engineer cannot look inside a workspace, and neither can you as the operator. A relay can report its version, its connection state and its errors. It has never seen a message or a ticket, so it cannot explain one.
Set that expectation before the first support request. Troubleshooting covers the failures that are diagnosable from outside, and Support covers the content-free diagnostic bundle the client produces.