Four Alerts That Could Never Fire, and How We Found Them
A week-long audit uncovered four alerting rules that never fired because they were built on faulty assumptions. From a Prometheus rule that only triggers when healthy to a curl check that ignores content, each case shows how misconfigured metrics or tests can mask outages. The article explains the…
When we set out to audit our own monitoring, the first question we asked wasn’t whether anything was alerting, but whether an alert would look different if the thing it watched had actually failed. In four cases the answer was a flat no: the alerts never turned red, no matter how hard the underlying system broke. Below we unpack each false negative, explain why it happened, and give you a quick checklist to spot the same problems in your own observability stack.
1. A Prometheus Rule That Only Fires While Healthy
The rule was designed to warn when a scheduled maintenance task stopped arriving. The expression used was increase(celery_tasks_total{task="datanika.run_maintenance"}[2h]) == 0. In Prometheus, a comparison against a scalar value like 0 is evaluated only when the series has a value; if the series is missing or zero the rule never evaluates to true. As a result the alert would never fire if the task stopped completely, because the series would simply disappear. The rule only worked when the task was partially stopped and still produced a non‑zero counter.
How we found it: the engineer who wrote the rule used a metric that was never exposed by the reverse‑proxy. The proxy only routed a specific set of paths to the backend; any other path returned the single‑page app’s shell with a 200 status. A curl -ff check on that path would succeed because 200 is not an error, so the rule never saw a failure.
2. An Exporter Scraped for Nothing
Our exporter exposed a counter that was never incremented because the process that produced the metric was not the one serving it. In a multi‑process or pre‑forked server, the child processes that handle requests may not share the same counter state as the parent that increments it. The alert, which watched the counter, would stay green even when the service was down, because the counter never changed.
Fix: ensure the metric is emitted by the same process that handles the traffic, or aggregate counters across all worker processes before exposing them to Prometheus.
3. A curl Check That Ignores Content
The CI pipeline used curl -sf https://example.com/api to verify the API was up. The -s flag silences progress, -f fails on HTTP status codes 400 and above, and -q would exit with a non‑zero status if the response body didn’t contain a specific string. However, the check was only looking at the status code; if the endpoint returned a 200 with an unexpected HTML page, the command succeeded and the pipeline considered the check passed. This meant that a misconfigured route that served the SPA shell could slip through unnoticed.
Fix: add a content check, such as grep -q "application/json" on the Content‑Type header, or assert on a known string in the body. Also, ensure that every backend route has an explicit proxy entry so that unmatched paths don’t fall back to the SPA.
4. A Pager That Alerted on a Non‑existent Failure
Our end‑to‑end test suite had an auto‑filer that opened a ticket and paged when a job failed. The job failed because an artifact upload timed out, but the test suite reported all specs as passing. The auto‑filer misinterpreted the timeout as a test failure and created an issue stating “only the gating tier can open this report.” The message was false at the moment it was filed, eroding trust in the alerting system.
Fix: tie alerts to the specific step that failed. In GitHub Actions, use if: steps.specs.outcome == 'failure' instead of a blanket failure() at the job level. This ensures the alert fires only when the actual test step fails, not when any part of the job stalls.
How to Spot These Issues in Your Own Stack
- Check alert expressions for bare scalar comparisons and verify the values that reach the threshold.
- Confirm that the process emitting a counter is the same process that serves the metric endpoint.
- Make
curlchecks assert on content as well as status codes. - Scope alerts to the exact step that can fail, not to the entire job or workflow.
Run these tests regularly: stop a worker and watch the metric disappear, point a curl at an unrouted path and confirm the check fails, or deliberately break a test step and see if the alert fires. If nothing turns red, you’ve uncovered another silent failure.
In short, an alert that never goes red is a silent failure. By validating the alert’s logic against real failure scenarios, you can ensure your observability stack truly reflects the health of your services.
Why it matters
False negatives in monitoring can let outages go unnoticed until they hit users, eroding confidence in the system. By identifying and correcting these hidden pitfalls, teams can trust that alerts truly signal problems, not just healthy states.
Key points
- Prometheus rules can silently fail if they compare against scalars and the series disappears.
- Exporters must expose metrics from the same process that handles traffic.
- Content checks are essential; status code checks alone can miss misrouted responses.
- Alerts should target the specific step that can fail, not the entire job.
- Regularly test alerts by inducing real failures to ensure they fire as expected.
Frequently asked questions
What does the "increase" function do in Prometheus?
It calculates the increase in a counter over a specified time window, useful for measuring event rates.
Why does a 200 status code sometimes hide problems?
A 200 status only indicates the HTTP request succeeded; the body may still contain an error page or incorrect content.
How can I make a curl command fail on content mismatch?
Pipe the output to grep and use the -q flag to exit with a non‑zero status if the expected string isn’t found.
