Keep it running
Last updated
Operating a Weald relay is watching four things: process liveness, dependency readiness, storage growth, and whether the running image is behind a security release. There is no admin panel and nothing to administer inside a workspace, so the operational surface is small. What follows is what to alert on and what each signal means.
Health endpoints
The public listener answers liveness and nothing else. Readiness and metrics bind to a private listener, on loopback by default, so a self-hosted relay does not publish its storage totals and security posture to the internet.
| Endpoint | Listener | Answers |
|---|---|---|
/healthz | public | Whether the process is alive. No state beyond that |
/readyz | private | Postgres and object storage reachability, access set mode, release check result |
/metrics | private | Prometheus metrics, with nothing content-derived in them |
Metrics exclude anything derived from content, which comes for free: the relay does not know channel names, ticket titles or who is in a group. Per-group envelope counts are off by default and are the one label worth thinking about before enabling.
What to alert on
/healthzfailing, which is the process being down./readyzfailing, which is almost always Postgres or the bucket rather than the relay.- Storage crossing a threshold you picked, well before
WEALD_RELAY_MAX_STORAGE_GBif you set one. A relay at its storage ceiling refuses writes with a quota error, which members see as work that will not save. - The release check reporting that the running digest is behind a release carrying a security advisory.
What the numbers mean
Peak concurrent connections is a capacity signal and not a headcount. One person with a Mac, a phone and two agents is several connections, and a workspace member who is asleep is none.
Storage is the number that grows without anybody deciding to grow it. It falls only when a workspace compacts, which is a control in the Mac app rather than something the relay can do on its own, because compaction is a decision about content.
Backpressure and errors
A relay under load slows down rather than dropping envelopes, because a dropped envelope is a hole in an author chain and therefore a security alarm on somebody else's screen. Per-connection queues are bounded and a full queue stops reading the socket, pushing back through TCP.
Every error a client receives carries a class, so a client branches on a code rather than on a string.
| Class | Meaning | What the client does |
|---|---|---|
retry | Transient infrastructure trouble | Backs off and resends the same bytes |
reject | Permanently wrong as sent | Stops, keeps the payload, surfaces a defect |
denied | Well formed, not permitted now | Re-reads the state named in the error |
quota | Over a storage, seat or rate limit | Surfaces the limit and the lever that clears it |
version | Protocol version unsupported | Aborts the connection rather than continuing |
Maintenance mode
WEALD_RELAY_WRITE_MODE=read_only refuses new durable writes while leaving subscription, reconciliation, backup and export working. Clients are told with a non-content reason code, so the app can say the relay is in maintenance instead of showing an unexplained failure.
It names no billing system and contacts nothing. It is a local switch for a local reason.
Troubleshooting, without content access
A support engineer cannot look inside a workspace, and neither can you as the operator. A relay can report its version, its connection state and its errors. It cannot explain a message or a ticket, because it has never seen one.
Set that expectation before the first support request. Troubleshooting covers the failures that are diagnosable from outside, and Support covers the content-free diagnostic bundle the client produces.