Watchtower: canary probe & health alerts

Blengi watches itself, so the operator finds out about problems before visitors do — not the other way round. Three monitors run from the scheduler and raise alerts through the same channels as the Widget Monitor: bell, browser push and (when configured) WhatsApp, throttled so an error storm pages once, not hundreds of times.

The canary probe

pitchbar:canary runs a real conversation against every published agent every five minutes — over public HTTP, through the proxy, Octane, retrieval, the LLM and the SSE stream. If a visitor would hit an error, the canary hits it first.

• The probe's conversation is playground-flagged (no billing, no analytics, no content gaps, no operator alerts) and deleted the moment the probe finishes — the only residue is one visitor row per agent.
• A single failed probe is counted but not alerted — blips self-heal. Two consecutive failures (ten minutes apart at most) raise a canary_failed alert naming the agent, the failing stage (init / stream / latency) and the detail.
• A workspace at its message quota is reported as a skip, never an outage.
• Opt-in per install: CANARY_ENABLED=true — each probe spends one real LLM turn per agent. Budgets: CANARY_TTFT_BUDGET_MS (default 25000), CANARY_TOTAL_BUDGET_MS (default 90000), CANARY_BASE_URL (defaults to the app URL).

The queue heartbeat

Every five minutes pitchbar:queue-heartbeat drops a tiny job onto each monitored queue (OPS_MONITORED_QUEUES, default analytics,default,integrations,index,crawl). The job's only work is stamping "this queue consumed something at time T". A stamp proves consumption, not dispatch — a dead worker leaves the job queued and the stamp ages. It is the idle-queue half of the signal; the other half is below.

pitchbar:health-scan checks the stamps: one older than 15 minutes raises a queue_stalled alert naming the queue. This permanently closes the silent-dead-worker class of failure, which twice went unnoticed for over a month.

What proves the worker is alive

Two signals, because one is not honest on its own (card #576). Every job a worker finishes stamps its own queue — throttled to one write per queue per fifteen seconds, so a worker clearing a hundred jobs a minute writes four times. That is the busy-queue signal. The five-minute QueueHeartbeatJob remains the idle-queue signal: on a queue nobody is using, it is the only traffic there is. Together they are honest in all four cases:

	Worker alive	Worker dead
Queue idle	heartbeat arrives and stamps — quiet	nothing stamps — alerts
Queue busy	every job stamps — quiet	nothing stamps — alerts

Before #576 only the heartbeat existed, and it rode the same FIFO it was measuring: with 887 jobs ahead of it, the probe dispatched at 12:00 had still not had its turn at 12:57, so the stamp aged and the operator was paged "worker likely dead" three times while five workers cleared 108 jobs a minute. The probe was reporting queue depth wearing the costume of worker health.

The same card gave this scanner the discipline every other Watchtower check has had since #537: one incident, one page. A stall that is already open is not re-alerted on the next five-minute pass, and the incident is resolved when the queue recovers, so the next genuine stall pages again.

A deep queue is still a real problem

#576 stops a backlog from being reported as a dead worker, but the backlog itself still matters: on 2 September 2026 one customer's 4,077-document source put 3,843 IndexDocumentJobs in front of everything, each an embedding round-trip of four to five seconds, and a single worker takes queues in strict priority — so every other customer's crawl waited hours. Queue depth is visible on /admin/queue-health; the two fixes that matter are below.

Before restarting anything, check whether the worker is actually working: pm2 logs pitchbar-queue --lines 20 --nostream. Lines scrolling past mean the fix is throughput, not a restart.

Where an indexing job's time actually goes

Before adding workers, it is worth knowing what one job spends its time on. Measured against the live Cloudflare account (card #575):

Step	Was	Now
ensureCollection — 1 GET + 3 metadata-index creates	3,423 ms, every job	once per worker generation
deleteByFilter with nothing to delete	700 ms	skipped on a first index
embed, 1 chunk	181 ms	unchanged
embed, 20 chunks	523 ms	unchanged

Three quarters of every indexing job was one question — "does this index exist at this dimension?" — whose answer cannot change while a run is in flight. It is now memoised per (index, dimension), in the process and in the shared cache, and forgotten when vector:rebuild-index drops the index. Because the key carries the dimension, switching CLOUDFLARE_EMBED_MODEL still triggers a real check, so the dimension-mismatch guard is intact.

The delete pass is skipped only when the document has no chunk rows. Chunk rows are always written before their vectors are upserted, so "no rows" proves "no vectors" — every document that has been indexed before still has its old vectors removed first.

Scaling the workers

ecosystem.config.cjs reads two environment variables (card #574):

• PITCHBAR_QUEUE_WORKERS — general workers on the full priority list analytics, default, integrations, index, crawl. Default 2.
• PITCHBAR_INDEX_WORKERS — workers on index alone. Default 3. Embedding is network-bound, so this scales almost linearly; each PHP worker costs roughly 75MB.

crawl is deliberately left to the general workers alone: Cloudflare Browser Rendering enforces its own concurrency limit, and CrawlSourceJob already staggers page dispatch by two seconds for that reason. Widening it there buys errors, not speed.

The health scan

Besides heartbeats, the five-minute scan checks two more signals:

• Error traces — visitor turns that died with a server-side exception raise a turn_errors alert with a sample of the question that failed.
• Log fatals — hard PHP failures (e.g. "Maximum execution time exceeded") never produce a trace because the process dies; the scan tails laravel.log and raises a log_fatals alert for fresh ERROR entries. A cursor guarantees each entry alerts once.

Watchdog alerts are recorded inline, never via the queue — an alert about a dead queue must not depend on that queue.

When a log fatal pages, and when it only records

Every fresh ERROR entry becomes a Widget-Monitor row, but not every entry rings the platform admin's phone. The first sighting after a quiet spell is recorded at warning severity — visible on the monitor, no push — because a single line is very often a transient a queued job has already retried past (a provider timeout that healed fifteen seconds later, for instance). The scan escalates to a paging error when one scan finds three or more new entries — the hot-path-fatal storm the detector exists for — or when new entries keep arriving within thirty minutes of the previous sighting.

Entries count regardless of their environment marker — a local.ERROR line written by an artisan command run in an oddly-configured shell is still recorded. The debounce is what keeps such one-offs off the phone; filtering by marker was tried and dropped, because a box whose scheduler resolves a different environment than its workers would go blind to the real fatals.

Scripted QA runs

Beyond "is it up?", the Watchtower can answer "is it still giving the right answers?". Each agent carries QA scenarios (agent → QA scenarios): a scripted conversation — one or more visitor messages — plus assertions on the final answer: text it must contain, text it must not contain, whether it must ask a follow-up question, and whether product cards are forbidden.

pitchbar:qa-run executes every enabled scenario nightly against the live agent through the full public stack, on a playground-flagged conversation that is deleted afterwards. Failures raise a qa_failed alert carrying the scenario name, the broken rule and an excerpt of the offending answer, and the verdict is shown next to each scenario on its page. This turns a client's manual test script into a permanent regression suite: a knowledge edit or model change that silently breaks an answer is caught the next night, not the next time a human happens to re-test.

The widget reports its own deaths

Every way a chat turn can die client-side now phones home: stream timeouts, HTTP failures on the stream endpoint, network failures and — the subtlest one — a stream that closes without a completion event (the signature of a server-side crash mid-answer). These arrive as client_stalled entries on the Widget Monitor with the precise reason.

Crucially, an expired widget session can still deliver telemetry: the events endpoint accepts a genuinely-signed token up to seven days past expiry (signature validation unchanged — forgeries are rejected). A dying widget is exactly the one whose distress call must not be dropped at the door.

Running the scheduler

All of this rides the Laravel scheduler. Self-hosted installs must run it — either a system cron calling php artisan schedule:run every minute, or a supervised php artisan schedule:work process (the shipped ecosystem.config.cjs includes a pitchbar-scheduler PM2 app for exactly this; activate it with pm2 start ecosystem.config.cjs --only pitchbar-scheduler). Without a scheduler, none of the recurring tasks — monitors, re-crawl sweeps, digests — ever run.