Alert info
- Severity
- Warning
- Detected by
- CloudWatch
- Alert
- WriteIOPS exceeded threshold (>1,000 IOPS)
- Time
- 2026-03-13 09:24 IST
- Service
- content-writer (Aurora PostgreSQL)
WriteIOPS on the Aurora writer breached the static 1,000 IOPS alarm at 09:24 IST (03:54 UTC). Investigate the cause.
Short-lived write spike from upstream application caused a WriteIOPS alarm. Investigation confirmed no customer impact: transient burst from a batch job.
- Peak IOPS
- 1,816 ops/sec
- Write Latency
- 0.2ms
- Duration
- ~2 min
Entities identified
Infrastructure
CloudWatch
alarm source
CloudWatch alarm tracking WriteIOPS.
content-sync
CronJob 54 3 * * *
Daily batch job for data synchronization.
Service
content-writer
application
Application writing to the Aurora cluster.
Data
aurora-writer
Aurora PostgreSQL
Aurora PostgreSQL writer instance in
ap-south-1.content table
write target
Database table for content.
aurora-replica
read replica
Aurora PostgreSQL read replica.
How they relate
The same relationships as text
- CloudWatch monitors aurora-writer. Monitors Aurora writer metrics.
- content-writer writes aurora-writer. Application connects to the Aurora writer.
- content-sync bulk writes aurora-writer. Batch job executes queries against the writer.
- content-sync INSERT/UPDATE content table
- aurora-writer persists content table
- aurora-writer replicates aurora-replica. Replicates data to the read replica.
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
Application write surge from user traffic or a connection stormRuled out
Analyze active DB sessions in pg_stat_activity to detect connection surges.
Sessions held flat at 21 through the alarm. No connection-pool exhaustion, no surge in application write sessions.
Connection count across the 03:52 to 03:56 UTC window.
time (UTC) sessions status
03:52 21 ok
03:54 21 ok (alarm)
03:56 21 ok
No connection storm. The write spike is not driven by a surge in application sessions.
Long-running transaction or runaway query amplifying write pressureRuled out
Query pg_stat_activity for long-running transactions and evaluate write latency.
Zero long-running transactions; write latency stayed flat at 0.2ms. A runaway query would show latency degradation and held locks.
Non-idle sessions older than 30s during the spike window.
SELECT pid, state, query_start, query
FROM pg_stat_activity
WHERE state != 'idle'
AND query_start < now() - interval '30 seconds'
ORDER BY query_start;
-- 0 rows returned
No long-running or idle-in-transaction sessions. No lock contention amplifying writes.
Latency through the IOPS peak.
WriteLatency 0.2 ms (flat through 03:54-03:56)
Latency flat at 0.2ms. A runaway query would degrade latency; this did not. Ruled out.
Aurora storage / I/O layer degradation (burst credits or replica strain)Ruled out
Check Aurora BurstBalance and AuroraReplicaLag metrics for degradation.
BurstBalance stayed above 80% and replica lag under 5ms. Ample I/O headroom, no throttling, no downstream strain.
I/O credit headroom and read-replica lag across the burst.
metric value status
BurstBalance >80% ok
AuroraReplicaLag <5 ms ok
Cluster absorbed the burst with headroom to spare. No storage degradation, no replica strain.
Scheduled batch job produced a normal, expected transient IOPS burstConfirmed
Compare CronJob schedules against the spike timing and verify job completion status.
The spike is precisely coincident with the content-sync cron at 03:54 UTC and lasts about 2 minutes. The burst is a known daily batch pattern, not an anomalous load event. Confirmed; identify the exact write source next.
Per-minute WriteIOPS around the alarm; peak then immediate recovery.
time (UTC) WriteIOPS status
03:52 198 normal
03:53 245 normal
03:54 1,816 ALARM
03:55 1,204 ALARM
03:56 312 ok
Sharp 1,816 peak for ~2 min then back to ~200 baseline. A transient burst, not sustained pressure.
The batch schedule lined up against the 03:54 UTC peak.
$ kubectl get cronjob content-sync -n prod -o jsonpath='{.spec.schedule}'
54 3 * * *
Schedule fires at 03:54 UTC, exactly the spike minute. The batch job is the burst source.
content-sync bulk INSERT/UPDATE drove the IOPS peak (benign, no impact)Root cause
Examine the content-sync job logs and metric correlation to determine impact.
content-sync-28472234 started 03:54:02 UTC and completed in 47s exit-code 0. Its bulk INSERT/UPDATE on the content table is the entire burst. Latency, sessions, burst balance, and replica lag all stayed healthy. Benign root cause: an expected daily batch burst, no customer impact. The defect is the static 1,000 IOPS threshold sitting below the normal batch peak.
content-sync run id, start time, duration, and exit code.
content-sync-28472234 1/1 47s 03:54:02 UTC exit 0
Job ran 47s and finished clean. Its write window covers the ~2 min burst exactly.
Every health signal across the spike window.
signal value verdict
WriteLatency 0.2 ms healthy
DB sessions 21 steady
BurstBalance >80% headroom
AuroraReplicaLag <5 ms unaffected
All signals green. The burst is a known, expected workload pattern with no customer impact. The alarm is a false positive from a too-low static threshold.
A different scheduled task drove the I/ORuled out
Check other CronJobs and scheduled tasks.
No other scheduled tasks coincided with the 03:54 UTC window.
Final RCA
content-sync bulk INSERT/UPDATE drove the IOPS peak (benign, no impact)
content-sync-28472234 started 03:54:02 UTC and completed in 47s exit-code 0. Its bulk INSERT/UPDATE on the content table is the entire burst. Latency, sessions, burst balance, and replica lag all stayed healthy. Benign root cause: an expected daily batch burst, no customer impact. The defect is the static 1,000 IOPS threshold sitting below the normal batch peak.
Remediation
- Raise the alarm threshold to 2,000 IOPS or add a suppression window for the 03:54–03:56 UTC daily batch window.
- Add anomaly detection instead of a static threshold to avoid false alarms from known batch patterns.
- Consider throttling the batch job writes if the daily IOPS spike grows over time (e.g., add batch sizing or rate limiting).
- Monitor BurstBalance trend: if it drops below 50% during spikes, consider upgrading the instance class.
The database itself was never at risk: every health signal (latency, sessions, burst balance, replica lag) stayed inside normal range through the spike. The actual defect is a static 1,000 IOPS alarm threshold set below the batch job's own daily peak, so the fix belongs to the alarm, not the workload. Raising the threshold or replacing it with anomaly detection that recognizes the recurring 03:54 UTC pattern removes the false positive without masking a real future write surge, and a suppression window scoped to the known batch job keeps the alarm useful for anything outside that pattern.
Frequently asked questions
Was the RDS WriteIOPS alarm a real incident?
No. CloudWatch alarmed when WriteIOPS on the Aurora writer hit 1,816, above the 1,000 IOPS threshold, but the investigation found no customer impact. Write latency stayed at 0.2ms, database sessions held steady at 21, BurstBalance stayed above 80%, and replica lag stayed under 5ms. The alarm was a false positive triggered by an expected daily batch job, not a degraded database.
What actually caused the WriteIOPS spike?
A scheduled CronJob, content-sync, runs daily at 03:54 UTC and performs bulk INSERT and UPDATE operations against the content table. The job started at 03:54:02 UTC, completed in 47 seconds with exit code 0, and its writes account for the entire 1,816 IOPS peak. The spike lasted about 2 minutes before WriteIOPS returned to its roughly 200 baseline.
How was a connection storm or runaway query ruled out?
Active database sessions in pg_stat_activity held flat at 21 through the alarm window, ruling out a connection surge. A query for sessions running longer than 30 seconds returned zero rows, ruling out a long-running transaction or runaway query. Write latency stayed at 0.2ms throughout, which a lock-holding or runaway query would have degraded.
What should change so this alarm does not fire again?
The static 1,000 IOPS threshold sits below the batch job's normal daily peak of around 1,816, so it alarms on expected behavior every time content-sync runs. The fix is to raise the threshold, add a suppression window for the 03:54 to 03:56 UTC batch, or replace the static threshold with anomaly detection that recognizes the recurring pattern.