Is a CloudWatch alarm a real incident or a false alarm?

An alarm fires at 03:54 UTC. WriteIOPS on the Aurora writer crossed a 1,000 threshold. It is a real number, correctly measured and correctly compared against a real threshold, and no customer experienced anything at all. That is not a monitoring failure in the sense of a broken metric. It is a threshold describing a workload it was never fitted to.

What makes an alarm fire without an incident?#

Four causes, and they are worth naming because the remedies differ.

A static threshold against a bursty workload. The threshold was picked from a quiet period, and the workload has a spiky shape. Batch jobs, nightly aggregations, cache warms and index rebuilds all produce legitimate spikes that no user feels.

Platform activity, not application activity. Snapshots, backups, maintenance windows, and storage-layer operations generate real I/O against your resource without any request causing it.

The wrong statistic for the metric. A Sum over a 60 second period and an Average over 300 seconds describe the same spike very differently. A Maximum on a counter will catch every transient and alarm on all of them.

Missing data treated as breaching. A metric that reports sparsely, then goes quiet, will trip an alarm configured with TreatMissingData: breaching even though nothing happened at all.

How do you check an alarm against the request path?#

Read the alarm's own configuration first. Most of the answer is in there and almost nobody looks.

aws cloudwatch describe-alarms --alarm-names content-writer-WriteIOPS-High \  --query 'MetricAlarms[].{      Metric:MetricName,      Stat:Statistic,      Period:Period,      Threshold:Threshold,      EvalPeriods:EvaluationPeriods,      DatapointsToAlarm:DatapointsToAlarm,      Missing:TreatMissingData    }' --output table 

A single evaluation period with DatapointsToAlarm: 1 means one datapoint can page someone. That is correct for a hard failure and far too sensitive for a throughput counter.

Then the history, which tells you whether this is chronic:

# How often has this alarm fired? A pattern of brief transitions that# self-resolve is the definition of a threshold that needs tuning.aws cloudwatch describe-alarm-history \  --alarm-name content-writer-WriteIOPS-High \  --history-item-type StateUpdate --max-records 20 \  --query 'AlarmHistoryItems[].[Timestamp,HistorySummary]' --output text 

Now the check that actually decides it. Pull latency and errors from the request path over the identical window:

WINDOW="--start-time 2026-03-13T03:40:00Z --end-time 2026-03-13T04:10:00Z --period 60" # Database side: did anything actually get slower?for M in ReadLatency WriteLatency DatabaseConnections; do  echo "== $M"  aws cloudwatch get-metric-statistics \    --namespace AWS/RDS --metric-name $M \    --dimensions Name=DBClusterIdentifier,Value=content-writer \    --statistics Average $WINDOW \    --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Average]' --output textdone # Application side: did any user see it? This is the deciding evidence.aws cloudwatch get-metric-statistics \  --namespace AWS/ApplicationELB --metric-name TargetResponseTime \  --dimensions Name=LoadBalancer,Value=app/content-writer-alb/50dc6c495c0c9188 \  --statistics Average p95 $WINDOW \  --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Average,ExtendedStatistics.p95]' --output text 

If p95 response time and 5xx count are flat across the spike, the alarm described something real and harmless.

How do you find what caused the spike?#

Attribute it before you dismiss it. A harmless cause still has to be identified, or you are dismissing an alarm you do not understand.

-- Aurora: what ran in that window, ordered by how much work it did.-- A batch job shows as few calls with enormous row counts.SELECT calls,       rows,       round(total_exec_time::numeric / 1000, 1) AS total_s,       round(mean_exec_time::numeric, 1)         AS mean_ms,       left(query, 100)                          AS queryFROM pg_stat_statementsORDER BY rows DESCLIMIT 10; 

 calls |    rows    | total_s | mean_ms | query-------+------------+---------+---------+---------------------------------     3 | 18,442,910 |    41.2 | 13733.3 | INSERT INTO content_archive ... 92418 |     92,418 |    31.7 |     0.3 | SELECT id, slug FROM content ... 

Three calls moving 18 million rows is a batch job. Ninety thousand calls at 0.3 ms is the request path, and it never changed. That is the shape of a false alarm: one process did legitimate bulk work, and the counter reported it accurately.

Also rule out platform activity, which produces I/O no query explains:

aws rds describe-events --source-identifier content-writer \  --source-type db-cluster --duration 120 \  --query 'Events[].[Date,Message]' --output text 

How do you fix a false alarm?#

Change the alarm, and be specific about which of its properties was wrong.

If the workload is bursty but bounded, require persistence rather than a single datapoint. An alarm that needs three of five periods will ignore a two-minute batch job and still catch a sustained problem:

aws cloudwatch put-metric-alarm \  --alarm-name content-writer-WriteIOPS-High \  --namespace AWS/RDS --metric-name WriteIOPS \  --dimensions Name=DBClusterIdentifier,Value=content-writer \  --statistic Average --period 300 \  --evaluation-periods 5 --datapoints-to-alarm 3 \  --threshold 1000 --comparison-operator GreaterThanThreshold \  --treat-missing-data notBreaching 

If the workload has a daily or weekly shape, a static threshold will never fit it. Anomaly detection models the shape and alarms on deviation from it:

aws cloudwatch put-metric-alarm \  --alarm-name content-writer-WriteIOPS-anomalous \  --comparison-operator GreaterThanUpperThreshold \  --evaluation-periods 3 --datapoints-to-alarm 2 \  --threshold-metric-id ad1 \  --metrics '[    {"Id":"m1","MetricStat":{"Metric":{"Namespace":"AWS/RDS","MetricName":"WriteIOPS","Dimensions":[{"Name":"DBClusterIdentifier","Value":"content-writer"}]},"Period":300,"Stat":"Average"},"ReturnData":true},    {"Id":"ad1","Expression":"ANOMALY_DETECTION_BAND(m1, 3)","ReturnData":true}  ]' 

Best of all, alarm on the symptom rather than the cause. Nobody is paged because WriteIOPS is high. They are paged because requests got slow, and an alarm on p95 latency and 5xx rate is immune to every batch job you will ever run. Resource counters belong on dashboards, where they help you explain a latency alarm after it fires.

If the spike is genuinely undesirable even without user impact, fix the workload instead: schedule the batch off-peak, chunk it, or rate-limit it.

What does a real false alarm look like?#

Sherlocks AI investigated one on the content-writer Aurora PostgreSQL cluster. A CloudWatch alarm fired at 09:24 IST, 03:54 UTC, when WriteIOPS crossed a static 1,000 IOPS threshold.

The investigation found a short-lived write spike from an upstream application, a transient burst from a batch job. No customer impact: latency and error rates from the request path never moved across the window. The conclusion was that the alarm was correct about the number and wrong about its significance, and the remediation belonged to the alarm rather than to the database. The full writeup, including the paths that were ruled out, is in the RDS WriteIOPS alarm investigation.

The reason this matters beyond one alarm is that every false page erodes the response to the true ones. An alarm nobody trusts is worse than no alarm. When the spike is real, the same window-comparison method points at the actual constraint, as it does in the Aurora load surge guide and the EBS throughput throttling guide. More incidents worked end to end are on the AWS examples page, and the alarm semantics are documented in Amazon CloudWatch alarms.

CloudWatch false alarm FAQ#

How do I know an alarm was a false alarm? Compare it against the request path over the identical window. If p95 latency and 5xx rate never moved, no user experienced it.

What is the most common cause? A static threshold on a bursty counter. Batch jobs, snapshots and maintenance produce legitimate spikes nobody feels.

Does TreatMissingData matter? Yes. Set to breaching, a sparsely reporting metric that goes quiet will trip the alarm even though nothing happened.

Should I just raise the threshold? Only if the shape is flat. For a workload with a daily rhythm, requiring several datapoints or using anomaly detection fits better than a higher number.

What should I alarm on instead? Symptoms. Latency and error rate from the request path. Resource counters belong on dashboards that explain an alarm, not on pagers.

Is it safe to just delete a noisy alarm? Only after you have attributed the spike. Dismissing an alarm you cannot explain is how a real signal gets suppressed.