Your monitoring fired. That is not the same as your monitoring working
A monitor has two jobs. It has to notice, and it has to tell you. Most of the monitoring failures we have had in five weeks of running three agents around the clock were not failures to notice. They were monitors that had stopped being connected to the thing they watched, and kept reporting “fine” — because “fine” is what you get when you count the bad events in a stream that has gone quiet.
Below are ten of them, dated, with what the monitor said, what was true, and what we changed. They fall into a handful of shapes, and the shapes are the useful part. Every one of these was caught by a person reading something they were not supposed to have to read.
1. The process lived. The alert path died.
2026-08-15. Our watchers get their chat-notification token from an environment variable set with launchctl setenv. Those variables do not survive a reboot. After one, the watcher kept running on schedule, found the condition it was looking for, wrote alert suppressed to stderr because the token was empty, and exited zero. Scheduler status: healthy. Process: alive. Notifications: gone, silently, until someone noticed nothing had arrived in a while.
Process alive is not the same as alert path alive. The fix was to stop depending on the session environment at all — each script reads its own .env — plus a login-time job that restores the variables for anything that still needs them.
2. Zero bad events, because zero events
2026-08-30. We wanted the product agent to stop calling the chat API directly and use a script instead. The first draft of the acceptance condition was “zero raw API calls in the window.” The window included 04:00–06:59, during which the agent’s session logs — 568 files — contained no events at all. The agent was not running. Zero raw calls, condition satisfied, full marks.
“Bad thing happened zero times” scores stopped and healthy identically. Every such condition now carries a liveness term ANDed in: the process completed at least once in the window, and the check prints both terms.
2026-09-06. Same shape, one level down. A session health check computed its warning flag as any(warn for check in checks). On 09-02 the session store moved from files to SQLite. The glob that fed checks found nothing, any() of an empty list is False, and the check reported all-clear for four days. It now warns when its source is empty and prints the population it examined, and the control for it is “delete the source directory and confirm it complains.”
2026-09-08. Same shape, in a file. A gateway watchdog counted idle-timeout lines in the gateway’s error log. A gateway restart rewrote the service definition so that stderr went to /dev/null; the log file froze at its last line. The watchdog kept counting a file that no longer changed: zero new idle-timeouts, healthy, for three and a half hours. It now checks the log’s modification time first and reports instrument stopped if the file is older than the window, which is a different word from healthy on purpose.
2026-09-07. Same shape, reversed: the file was changing, just not because of the job we cared about. Five scheduled jobs shared one log file, so the monthly model-training job always looked freshly run — the daily jobs kept the timestamp new. We had no evidence the monthly job had ever fired. The fire-check now wants a job-specific artifact, the model directory that job alone creates or the scheduler’s run counter for that job, and ignores shared logs.
Four incidents, one rule. A pass condition needs three terms, ANDed and all printed: zero failures, at least one completed run in the window, and an input that was modified inside the window by the thing being watched.
3. Exit zero, nothing written
2026-09-03. A nightly job that scores the day’s predictions ran two nights in a row: exit 0, run count 2, scheduler happy. It had written zero rows both nights, because a column it needed was missing from the input. Exit code does not distinguish “wrote 138 rows” from “wrote none.” Liveness for a job that produces a file is the file: row count and a hash, compared to yesterday’s.
4. Connected is not receiving
2026-09-05. A market-data recorder held WebSocket connections to three channels per symbol. Status: connected. For three days, two of the three channels — trades and liquidations — received nothing, while reconnecting roughly 1,378 times per symbol. The order book channel worked, so the process had data flowing and looked busy. The monitor watched the connection state. It now watches receive counts per channel, against what the channel should carry: trades arrive by the hundreds per minute, so zero in a minute is the alarm, not the absence of a disconnect.
5. The alarm that rang into a file
2026-09-06. A credential-expiry monitor had been writing any_warn: true to its log directory every day since 08-26. Twelve days. Nobody read the directory, both agents kept working, and the warning was in any case about a credentials file that had been abandoned in an April migration to the system keychain. The monitor was broken and had been telling a directory about it.
Two rules came out of this. A warning that only lands in a log is not monitoring; warnings go to the ops channel, with a cooldown. And a warning that repeats for N days is treated as evidence about the monitor, not the target: the message now says “same warning for 12 days — the watched path may be stale.”
6. The check that cannot fail
2026-09-03. A backtest judged whether a limit order would have filled by asking whether the day’s high–low range crossed the limit price. It does, nearly always; a daily bar spans most prices. The unfilled rate came out at 0 of 24, which looked like a result and was actually a granularity that could not produce a failure. Switching to one-minute bars did not fix it either: 20 of the 24 orders “filled” in minute zero, because the limit price was the bar’s open, so the test had become “did the price move one tick in the first minute.” Only after offsetting the limit 25 basis points from the open did an unfilled order appear at all. Fills are now judged on one-minute bars with that offset, and when the data cannot answer, the cell is blank rather than zero.
This one is not a monitor in the operational sense, but it is the same disease: a metric whose construction guarantees it passes is not measuring anything, and the number it produces will be quoted as if it were.
7. The terminal state we guessed
2026-09-22. A watcher over a long-running research job was written to stop watching on states matching STOPPED_*, PARTIAL_*, *COMPLETE*, or *FAIL*. The job’s actual terminal state was RESOURCE_PROFILE_PASS. The watcher did not recognize it, kept waiting, and was silent for two hours until a scheduled check at 17:00 noticed the job had finished. The set of terminal states is now copied from the runner’s source, not guessed from what sounds terminal, and any state name outside the known set triggers a notification of its own.
The shapes
alert path dead, process alive 08-15 env var lost on reboot
zero events read as zero bad events 08-30 agent not running
09-06 empty check list → all clear
09-08 frozen log file → no hits
freshness borrowed from a sibling 09-07 shared log
exit 0, no output 09-03 zero rows written
connected ≠ receiving 09-05 0 trades for 3 days
alarm written to a file 09-06 12 days unread
metric that cannot fail 09-03 daily bar always crosses
terminal state guessed 09-22 unknown state = silenceSeven shapes, ten instances, five weeks. None was found by the monitor. All were found by someone reading a raw log, a directory listing, or a scheduler status by hand, usually while looking for something else.
What we now require of a monitor
These are not clever. They are the checklist we run against every new watcher before it is allowed to say “healthy,” and each line is there because of a date above. They are also the same habit that catches bad fixes — run it against what already exists, and count — pointed at the monitor instead of the fix.
Liveness is ANDed, not implied. The three terms from section 2, in one expression, all printed. A monitor that cannot show its own denominator is reporting a mood.
The population, every time. “0 of 0 checked” and “0 of 412 checked” are different lines and must look different. Empty input is a warning, never a pass.
Evidence from the thing itself. A file’s row count and hash. A per-channel receive count. A job-specific artifact. Not the exit code, not a shared log’s timestamp, not the connection state.
Warnings go to a person, with a cooldown. And the same warning repeating for N days is reported as a warning about the monitor.
States copied from source, not guessed. The set of terminal states comes from the runner’s code, and a state outside the set is its own notification.
A control that breaks it. Before trusting a monitor, make the condition true — delete the input directory, empty the token, freeze the log — and confirm it fires. A monitor that has never been seen to fire has never been seen to work. Then run it over everything that already exists, and count.
The uncomfortable summary is that “healthy” is the cheapest word a system can produce. It is what you get from a monitor that is working, and it is also what you get from a monitor whose input went away, whose output went nowhere, whose metric cannot fail, or whose idea of “finished” was a guess. Those four look identical on a dashboard. They are only distinguishable in the raw material, which is why every one of these was found there.
For the incident that started this — a monitor that was working perfectly and still made things worse — see Our idle detector sent 144 alerts in one night. The watchers, fire-checks, and health scripts with these rules built in ship as part of the enrai Agent Pack ($33 on Gumroad).