How to Plan Monitoring and Alerting With AI
Decide what deserves to wake somebody up. Plan monitoring from the journey that has to work rather than the components you happen to have — signals sorted into page, ticket and dashboard, and an alert for the failure that makes no noise at all.
Get the Monitoring Plan PromptThe graphs were unusually calm
The webhook dispatcher stopped on Friday evening. Nothing went red. There were no errors, because a process that isn't running doesn't throw any. There was no latency spike, because nothing was being served. The traffic panel didn't drop, because it counts requests coming in and this thing sends them out. If anything the dashboards looked better than usual: flat, tidy, green.
On Monday a customer asked why they'd received nothing since Thursday. That question was the monitoring. And the awkward part isn't that an alert failed; it's that the alerting was working exactly as designed. Every rule on that service asked "is this number too high?" and the number was zero, and zero is an excellent number. Nothing anywhere in that system was built to ask whether a number arriving at all was the point.
An alert is a machine with permission to interrupt a person
That framing does more work than any taxonomy. A dashboard waits to be visited. A metric sits there. An alert is different in kind: it runs unattended, evaluates every thirty seconds, forever, and when it decides, it takes something from somebody — an evening, a night's sleep, the thread they were holding in their head. And unlike a person reading a chart, it has no option to conclude "I can't tell yet." It fires or it doesn't.
Which makes the failure mode of a bad alert unusual. A wrong dashboard is ignorable. A wrong alert *trains* — it teaches the person who answered it that a page from this service is probably nothing, and that lesson is permanent, transfers to every other alert on the same rota, and gets applied at exactly the moment you needed it not to be. You cannot fix that later by writing a better alert. The budget you're spending isn't compute, it's credibility, and it's the only thing that makes the real page work.
So the plan is mostly a series of refusals. Most signals are not alerts. Most alerts are not pages. The interesting output is the short list of things a machine may wake somebody for, and the longer list of things it may not.
A model helps with the shape of that. Give it your system boundary, the journeys that matter, the ways they fail, what you already collect and who is on call, and it'll draft the signals, ask what happens when each one is wrong, and reliably notice the failure mode you left out. What it can't do is know your traffic. NewPrompt's part is narrow: it builds the prompt, in your browser. It has no connection to your monitoring platform; it reads no metric, log or trace; it creates no dashboard and no alert rule; it detects nothing; it sends no notification; it validates no threshold; and it neither knows nor touches your on-call rotation or escalation. Any number a draft produces — five percent, five hundred milliseconds, eighty percent CPU — is a plausible-looking guess, and the owners confirm every one against real telemetry and real incident history. A plan is not coverage, and it doesn't promise that anything gets detected.
Step 1: Start from the journey, not the components
Write the sentence that has to stay true. "An event happens and the customer's endpoint receives it, signed, within a minute." That sentence is the service; everything you monitor exists to notice when it stops being true. Start instead from an inventory — here are our four boxes, let's watch each one — and you'll end up with alerts about CPU and disk on machines nobody is being hurt by, and no alert at all for the one path that matters.
Then draw the boundary, and be honest about what's outside it. A webhook platform can't fix a customer's endpoint being down. That failure is real, it's worth knowing about, and it is not an incident on your side. Services that don't draw that line end up paging their own engineers for other companies' outages, which is a specific and demoralising way to lose a rota.
Now list the ways the journey breaks, derived from the journey rather than recalled from a list. The thing dies. The queue fills faster than it drains. One customer is broken. Everyone is broken. A thing that looks like one failure is actually another. Each of those wants a different signal, and at least one of them — reliably — is invisible in everything you currently collect.
Step 2: Ask what the failure looks like from outside
For each failure mode, the question is what a user would notice, not what the machine would be doing. That distinction sorts nearly everything that follows.
A symptom is what's happening to the person: requests failing, the thing not arriving, the queue not draining. A cause is an internal condition that might produce a symptom: CPU at ninety percent, memory climbing, a pod restarting. Pages go to symptoms, for two reasons. Symptoms are nearly complete — they catch the failure whose cause you never thought of, which is most of them. And they're definitionally worth waking someone for, because someone is already being hurt. A cause is a hypothesis about harm; a symptom is harm.
Causes are not useless — they're what you look at once a symptom fires, which is what dashboards are for. The exception: an imminent, unrecoverable exhaustion with no symptom until it's too late — a certificate expiring, a disk filling — earns a page on the cause, because by the time there's a symptom you've missed it.
Sort what's left into leading and lagging. Lagging says the damage is done: failures, breaches, complaints. Leading says the damage is coming while there's still room: a queue climbing, utilisation drifting up, a certificate two weeks out. You want both, and they get different treatment — leading indicators are imprecise by nature (a queue can grow and drain harmlessly), so they belong in tickets and dashboards, not on the pager.
Step 3: Sort every signal into page, ticket, or dashboard
One test, and it isn't "is this real" or "is this important" — both of those are usually yes, which is why they sort nothing. The test is: **when this fires at an inconvenient hour, does the person it wakes have something they can do about it right now, and does it get worse if they don't?**
Three questions make that operational. Is somebody being hurt, or will they be before the next working day? Does the recipient have a response that changes the outcome? Would ignoring it be *wrong*? Fail any one and it isn't a page. It might be a ticket, it might be a panel on a dashboard, and it might be nothing.
Two heuristics catch most of the rest. An alert whose runbook says "investigate" doesn't have a runbook — it has a hope that the person woken will figure it out, which is a bad thing to find out at speed. And an alert that has fired forty times and been acted on zero times is not an asset that's a bit noisy; it's a training programme, and what it's teaching is precisely the thing you can't afford.
Which is why deleting is part of this work rather than a tidy-up afterwards. The plan that only adds alerts gets worse every quarter, because the credibility is shared and the additions are individually reasonable.
Step 4: Say when the machine is allowed to interrupt
For every page, the condition — and the condition is not just a number. It's what's measured, over what window, with what minimum before the measurement means anything.
Don't let a draft hand you the number. Five percent, five hundred milliseconds, eighty percent CPU are the placeholders every model reaches for, and they are not thresholds, they're the shape of a threshold. A real one is derived: from what the service promises, from what a normal bad day already looks like in the data you have, and from what you'd be embarrassed to have not noticed. Nobody outside your team has those. So the number is a blank with a name beside it and a sentence saying which query answers it. That is a real state, and it beats a confident 5% that everyone quietly stops believing by March.
Then two things that break conditions specifically because nobody is watching them. Freshness is the one almost nobody writes down: your pipeline lags, so the newest bucket is always partial, and "no errors in the last minute" reads identically to "the data has not landed." It is silent in both directions at once — suppressing the error alert while firing the traffic-drop one — and neither looks like a data problem from the outside. Exclude the trailing bucket, and alert on the age of your newest sample, because a monitoring system that has stopped receiving anything looks exactly like a peaceful night. Volume is the older lesson and the shorter one: a ratio wants a floor under it and a window long enough that the sample means something. What differs here is the cost of getting it wrong. A person reading a chart can look at four data points and decide not to decide. A rule has no such move — it evaluates anyway, at 3am, and every time thin arithmetic tips it, it takes a night from somebody and spends a little more of the reason they pick up.
Step 5: Alert on the silence too
Nearly every rule you will write is presence-based: something happened, and there was too much of it. Errors above a line. Latency past a point. Failures per minute. All of them require the thing to be running enough to produce a number.
So the failure they cannot see is the one where it stops entirely. A worker that dies emits no errors, no latency, no throughput — nothing at all — and every presence-based rule reports green, correctly, forever. This isn't an edge case. For anything that runs on a schedule, consumes a queue, or sends rather than serves, it's the most likely serious failure and the least likely to be noticed, because the absence of a signal is exactly what a healthy quiet night looks like.
Invert it. The job emits a heartbeat every cycle, and the alert fires when the last one is older than the cycle plus a tolerance. Add a work-outcome signal, because a worker that runs and processes nothing for six hours is also broken and the heartbeat is happy. Add backlog age, because a worker that runs but can't keep up delivers everything, late, and never fails a success rate — a ratio literally cannot see it. And one level up, the same trick protects the whole thing: a rule that always fires, routed somewhere outside your stack, so that when it stops arriving somebody hears about it. Without that, an outage of your monitoring is the quietest night you'll ever have.
Absence alerts have one property: they must be tested, because they're the only kind that can be broken and green simultaneously. A heartbeat rule nobody has ever seen fire is a belief, not a control.
Step 6: Decide what happens when it fires, and when it should stop existing
Every page needs a name and a first move. The owner is a person, not a team — teams don't wake up. The runbook doesn't have to be long, but it has to start with something specific enough to do half-asleep: is the process alive, did a deploy just land, is this one customer or all of them.
Then the machinery that keeps one failure from arriving as forty notifications, and the four words for it do different jobs. **Dedup** is identity: the same alert, still firing, notifies once instead of every thirty seconds. **Grouping** bundles distinct alerts that share something — one customer with two hundred endpoints down is one notification with two hundred rows. **Silencing** is a human deciding, for a bounded time, not to hear something. **Inhibition** is causal and automatic: the cluster-down alert suppresses the fifty service alerts underneath it, because they're all the same sentence.
Silencing deserves its own caution. It's a deliberate blind spot, and the bound is the part that gets forgotten — a silence with no expiry is an alert you deleted without noticing. And be careful what you silence during deploys: pausing the noisy backlog alert is sensible, but a deploy that briefly stops a worker and a deploy that kills it permanently look identical for the first ten minutes, so the absence alert is exactly the one that must stay awake.
Finally, put a date on the review. One question per alert: how many times did this fire, and how many times did somebody act? An alert that has never fired is untested rather than reliable. An alert that fires and is never actioned should be deleted, and the deletion belongs in the document — otherwise somebody re-adds it next quarter, reasonably, because its absence looks like an oversight.
Common mistakes
Most of these produce a monitoring setup that looks thorough and is quiet at exactly the wrong moment.
- Watching the components instead of the journey. An inventory-driven plan monitors every box and misses the one path a customer actually cares about — and the boxes will all be green while the path is broken.
- Paging on causes. CPU at ninety percent might be exactly what you're paying for; a service can serve errors at ten percent. A cause-based page is a guess about harm dressed as a certainty, and it's the biggest single source of pages nobody acts on.
- The heartbeat nobody tested. Absence alerts are the only kind that can be broken and green at the same time, so an untested one is a belief with a dashboard. It costs ten minutes in staging and it is the alert most likely to be the one that mattered.
- Alerting on somebody else's outage. Your customer's endpoint is down. It's real, it's worth a digest, and nobody on your side has an action at 1am. Page for it and you've volunteered your rota for other companies' incidents.
- The silence with no end date. Somebody mutes something for a maintenance window and the mute outlives the window, so the alert is now deleted and nobody knows it. Unbounded silences are how a control disappears without anyone deciding to remove it.
- Reading the plan as coverage. It is a draft that owners still have to hold against real telemetry and real incident history — NewPrompt reads no metrics, creates no rules, notices nothing and pages nobody, and no plan on a page has ever detected an outage.
Monitoring a webhook delivery service
A platform that posts events to customer-registered endpoints. It earns the space because it fails in the way that defeats ordinary alerting: it sends rather than serves, so when it stops, every number it produces is zero, and zero looks wonderful. Everything below is fictional.
One journey, five failure modes and six signals: three pages, one digest, one warning and one panel — plus the heartbeat the service never had, the ratio that needed a floor before it could mean anything at 3am, a silence that must never cover the one alert that matters, and an alert deleted for firing 41 times and being acted on noneMONITORING PLAN — webhook delivery plan v1, service-scoped
THE JOURNEY THAT HAS TO WORK
An event happens in our product. The customer's endpoint receives it, signed,
within a minute. That sentence is the whole service. Everything below exists
to notice when it stops being true.
BOUNDARY
Ours the dispatcher, the delivery queue, the retry worker, our egress.
Not ours the customer's endpoint. When theirs is down we retry and report.
That is a fact about their week, not an incident in ours — and
conflating the two is how this service pages people for other
companies' outages.
FAILURE MODES (from the journey, not from a list of components)
F1 dispatcher stops nothing is sent, and nothing complains
F2 queue drains slower than it fills delivered, late, still 'success'
F3 one customer's endpoint down theirs
F4 all endpoints failing ours — egress, DNS, or the signer
F5 signing key rotated badly delivered, rejected, looks like F3
TELEMETRY WE HAVE request logs, delivery attempts table, queue depth
TELEMETRY WE DO NOT HAVE any heartbeat from the dispatcher; the age of the
oldest undelivered event. F1 and F2 are unnoticeable
until we add these, and F1 is the one that took the
weekend.
──────────────────────────────────────────────────────────────
S1 dispatcher heartbeat PAGE owner: Ana (platform)
Fires when no heartbeat for longer than one cycle plus tolerance.
Cycle is 30s. Tolerance TBD — Ana. Long enough that a slow GC
does not page; short enough that F1 is minutes, not a weekend.
Why page a dead dispatcher is total, silent, and gets worse by waiting.
Runbook [link] — first check: is the process alive, and did a deploy
just finish?
Test kill the dispatcher in staging; assert the page fires inside the
window. An absence alert nobody has ever seen fire is a belief.
^ this is the alert the service did not have, and the reason F1 lasted
from Friday to Monday. Every other signal here is presence-based: an
error rate, a latency, a count. A process that is not running emits
none of them — no errors, because it throws nothing; no latency,
because it serves nothing. Silence and health are the same shape, and
only an alert that fires on absence can tell them apart.
S2 oldest undelivered event, age PAGE owner: Ana
Fires when age > threshold, sustained. Threshold TBD — Ana.
Derive it not from a round number. The promise is 'within a minute', so
the threshold is the point past which we are breaking it badly
enough to tell a customer. Ana has 90 days of delivery-attempt
timestamps; the number comes out of what a normal bad Tuesday
already looks like, plus what we would be embarrassed to admit.
Why page F2 is invisible in a success rate — every event eventually
delivers, so the ratio stays green while the queue is an hour
deep. Age catches it. Rate cannot.
Runbook [link]
S3 delivery success ratio, global PAGE owner: Ana
Fires when ratio below TBD over TBD window, AND at least N attempts in it.
The AND matters more than the ratio. At 3am we do maybe 40 attempts an
hour: one retryable blip is 2.5% and a genuinely dead egress is
100%, and a bare ratio cannot always tell them apart at that
volume. Without a minimum count this pages on arithmetic.
Freshness our metrics pipeline lags ~90s. The newest bucket is always
partial, so 'zero failures in the last minute' and 'the data
has not arrived' are the same reading. Exclude the trailing
bucket; alert separately when the newest sample gets older than
TBD — Ana. Floor is the ~90s lag plus margin; the number is the
point past which a stale read would fool S3 into calling an
outage a quiet night.
Runbook [link]
S4 per-customer failure ratio NO ALERT — digest owner: Priya (support)
Why not it is real, it matters, and it is F3: their endpoint, their
certificate, their firewall. Nobody on our side has an action at
1am. Priya gets a daily digest and emails them.
^ the honest test is not 'is this true' or 'is this important'. Both yes.
It is: when this fires at an inconvenient hour, does the person woken
have something to do about it? No. So it is not a page, and calling it
one anyway is how the pages that matter stop being read.
S5 signature rejects WARNING — ticket owner: Ana
Fires when any endpoint rejects on signature after a key rotation.
Why warn F5 masquerades as F3 — it looks like the customer is broken and
it is us. Rare, not urgent, and worth a ticket the same day.
S6 dispatcher CPU DASHBOARD ONLY
Not an alert. It is a cause, and a weak one: the dispatcher can be at 95% and
perfectly healthy, or at 10% and dead. It explains S1-S3 once they fire. It
cannot stand in for them.
REMOVED 'dispatcher CPU above 80% for 5m'
It existed for nine months. It fired 41 times. It was acted on zero times.
It is gone, and this line is here so nobody adds it back next quarter
thinking it was an oversight.
^ 41 pages that taught the on-call rota that a page from this service is
probably nothing. That is the cost, and it does not sit on the alert
that produced it — it sits on S1, which now has to be believed.
──────────────────────────────────────────────────────────────
GROUPING by customer. One customer with 200 registered endpoints going
down is one notification with 200 rows, not 200 notifications.
DEDUP S1 firing continuously notifies once, not every 30 seconds.
INHIBITION S1 inhibits S2 and S3. A dead dispatcher makes the backlog age
climb and the ratio collapse, so one failure would otherwise
arrive as three pages saying the same sentence. S1 is the one
that means something; the other two are it, arriving later.
SILENCING during a deploy, S2 is silenced for the deploy window — 10 min
today, Ana owns it if that changes. The queue backs up on
purpose and drains after.
S1 is NOT silenced. Ever.
^ a deploy that kills the dispatcher and a deploy that briefly pauses it
look identical for the first ten minutes. Silence the heartbeat during
deploys and you have blinded yourself in exactly the window where F1 is
most likely. Every silence is a bounded blind spot, and the bound is
the part that gets forgotten — this one expires, and the expiry is not
optional.
REVIEW Ana, quarterly, one question per alert: how many times did this
fire, and how many times did somebody do something? An alert
that has never fired is untested. An alert that fires and is
never actioned is training.
WHERE THIS STOPS
S1 pages. What Ana does next — triage, mitigation, who else gets called,
whether this becomes an incident — is not in this document and is not the
same skill. This plan makes the failure visible and hands it to a person.
Everything after that handoff belongs somewhere else.
Nothing here has been wired up. Six signals, three of which do not exist yet,
and every threshold on the page is TBD with a name against it. Fictional
service, fictional numbers, fictional people.Where this fits in NewPrompt
The Monitoring & Alerting Plan Prompt is the closest starting point, and it carries the two rules this guide is built around: every page must be urgent *and* actionable or it's a dashboard item, and cover what breaks without throwing an error rather than just the exceptions. Its five sections — signals, thresholds, page-versus-dashboard, silent failures, the first dashboard to open — are a good spine. One thing about its shape: it's written for a release, asking what to watch for a specific change about to ship. That's a narrower question than this guide's. A service runs when nobody is watching and nothing is shipping, and the failure in the example above had no release anywhere near it.
Markdown Output Builder is where a prompt like that gets assembled in your browser, and it ships a genuine preset for this — five sections matching the resource's own, tables required, heading order pinned. Two gaps before you rely on it. There is nowhere to describe your service: the contract it emits opens by announcing that no source text exists to summarize, which leaves the goal field, or a paste afterwards. And it carries half of those two rules. The silent-failure one survives, because the preset section description does ask for failure modes that throw no error and the synthetic check that catches them. Urgent-and-actionable does not: it shrinks to the words no-noisy-pages inside one section description — and that is the rule doing all the work, demoted to an aside. Put it back.
Two edges, and the guides on the other side of them drew both. A staged rollout borrows whatever signals it finds and sets one release's gate against them; which signals exist at all, and which of them are allowed to wake anyone, is settled here instead — and the rollout side says as much itself. If a gate over there needs a signal nobody collects, that is a finding for this plan rather than a threshold to invent over there. And the DevOps Engineer Role Prompt is the stance rather than the artifact — it puts the model in a seat that thinks in symptoms users feel rather than in machine internals, which is Step 2's whole argument, and it's worth pasting above the plan prompt if the drafts keep coming back full of CPU.
One level up, the AI Deployment & Release Workflow ends on monitoring: its last step is setting up what you'll watch before you ship. That's the release-shaped version of this. NewPrompt supplies the prompts and the order; you run them. Every threshold, and every question about who gets woken, belongs to the people carrying the pager.
The measure of whether this worked isn't how many alerts you end up with. It's what happens the next time one fires at an inconvenient hour — whether the person who wakes finds something to do, or finds the thing they've now seen forty times and learned to swipe away. Every alert you add spends a little of that, every alert you delete pays some back, and no amount of care in the next rule recovers what the last forty spent. Which is why the most valuable line in most of these plans turns out to be the one that removes something.