Alert info
- Severity
- Critical
- Detected by
- APM Collector
- Alert
- api-gateway metric value dropped to 0.00 (threshold: 1)
- Time
- 2026-03-10 11:05 IST
- Service
- api-gateway
The "api-gateway down" alert fired on a value of 0.00 against a threshold of 1, with 2 alerts firing. But the service shows no 5xx, no restarts, and a healthy 200 on healthz. Verify whether this is a real outage or a monitoring gap before paging the on-call.
False-positive alert: service was healthy but APM Collector could not receive metrics due to connection refused. Resolved in 1 minute.
- Alert to RCA
- 1 min
- Firing Alerts
- 2
- Impact
- None (false positive)
Entities identified
Infrastructure
gateway endpoints
k8s Service
Service api-gateway: 3 endpoints registered (10.0.12.5/.6/.7:8080), AGE=2d. Traffic routed to all pods.
Alert rule
ApiGatewayDown
expr up{service="api-gateway"} == 0. 2 alerts firing.
Service
api-gateway pods
3/3 Running
api-gateway-5d8f9b7c4-{abc12,def34,ghi56}. All 3/3 Ready, RESTARTS=0, AGE=2d. No CrashLoop, no pending pods.
Data
/metrics
:8080 scrape target
api-gateway.platform:8080/metrics. Reachable from within the cluster; collector scrape times out post-restart (connection refused).
up metric
scrape pipeline
up{service="api-gateway"} pipeline reading.
External
APM Collector
monitoring ns
apm-collector-8f7d6c5b3-xyz99. APM Collector pod responsible for scraping metrics.
Gateway
api-gateway
platform ns
The alerted service. 3/3 replicas Running & Ready, RESTARTS=0, AGE=2d. healthz returns 200. Actually healthy throughout.
How they relate
The same relationships as text
- api-gateway backed by api-gateway pods
- api-gateway pods registered in gateway endpoints
- api-gateway pods exposes /metrics
- APM Collector scrapes /metrics. Scrape path from collector to the gateway metrics endpoint.
- APM Collector reports up metric. The collector reports the up metric into the pipeline.
- up metric evaluated by Alert rule. 0.00 crosses the threshold of 1, triggering the alert.
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
Genuine api-gateway outage (pods crashed or unschedulable)Ruled out
List api-gateway pods and check ready status, restarts, and age against the alert claim that the service is down.
All 3 replicas are 3/3 Running & Ready with RESTARTS=0 and AGE=2d. No CrashLoop, no OOMKill, no pending pods. The service is not down.
Every api-gateway replica is Running and fully Ready with zero restarts.
NAME READY STATUS RESTARTS AGE
api-gateway-5d8f9b7c4-abc12 3/3 Running 0 2d
api-gateway-5d8f9b7c4-def34 3/3 Running 0 2d
api-gateway-5d8f9b7c4-ghi56 3/3 Running 0 2d
3/3 Ready, no restarts in 2d. No outage and no crash loop. Ruled out.
Application regression from a recent deploy or config errorRuled out
Look for a deployment event near the alert window and re-probe the service health endpoint.
No deploy in the window; all 3 pods have run 2d with no restarts and healthz returns 200. No regression.
The health endpoint answers 200 while the alert claims the service is down.
$ curl -s -o /dev/null -w '%{http_code}' \
http://api-gateway.platform.svc.cluster.local:8080/healthz
200
Service is serving 200, AGE=2d, no deploy event. No code or config regression. Ruled out.
Traffic routing failure (endpoints missing or LB error)Ruled out
Query the Kubernetes service endpoints to confirm pods are registered and receiving traffic.
The service has all 3 endpoints registered (10.0.12.5/.6/.7:8080), AGE=2d. Routing is intact.
All three pod IPs are registered as endpoints on port 8080.
NAME ENDPOINTS AGE
api-gateway 10.0.12.5:8080,10.0.12.6:8080,10.0.12.7:8080 2d
3 endpoints registered, traffic routed. No routing or load-balancer failure. Ruled out.
Resource exhaustion (OOMKill / node pressure) on api-gateway podsRuled out
Check restart counts and OOM events on the api-gateway pods.
RESTARTS=0 on every api-gateway pod and no OOM events. The pods are not under resource pressure. Ruled out. (Note: the restart that matters is on the collector, not the gateway.)
Zero restarts across all gateway pods; no OOMKilled events in the window.
pod restarts (api-gateway): 0, 0, 0
OOMKilled events: none
No OOM, no restarts on the gateway. Resource exhaustion is on the monitoring side, not here. Ruled out.
Monitoring pipeline gap: APM Collector could not receive metricsConfirmed
Pull the APM Collector pod state and scrape logs for api-gateway to verify if the collector is successfully scraping metrics.
The apm-collector pod restarted 12m ago and could not re-establish its scrape to api-gateway.platform:8080/metrics (connection refused / timeout), so it reported up=0.00. The 0.00 is a missing read, not a real value. This is a false positive in the monitoring pipeline.
The APM Collector shows a restart 12 minutes ago, just before the alert window.
NAME READY STATUS RESTARTS AGE
apm-collector-8f7d6c5b3-xyz99 1/1 Running 1 (12m ago) 5d
The only thing that restarted is the collector itself, right before the alert. Suspect the scrape path.
After the restart the collector logs repeated scrape timeouts and emits 0.00 for the up metric.
[05:23:42Z WARN] Scrape target api-gateway.platform:8080/metrics - connection timeout
[05:23:42Z WARN] Reporting value=0.00 for up{service="api-gateway"}
[05:24:42Z WARN] Scrape target api-gateway.platform:8080/metrics - connection timeout
The collector cannot connect, so it reports 0.00. The service is up; the read is missing. False positive.
A single missed scrape with no fallback fires the alert immediatelyRoot cause
Inspect the alert rule and the scrape topology: is there a 'for' duration, and is there a second scrape source?
The ApiGatewayDown rule has no 'for' clause, so one 0.00 sample crosses the threshold of 1 and fires Critical. There is no fallback scrape source, so a single collector's restart is enough to read 0.00. The root cause is the monitoring pipeline, not the application: harden the rule and add redundancy.
The rule fires on a single sample equal to 0 with no sustained-duration guard.
- alert: ApiGatewayDown
expr: up{service="api-gateway"} == 0
# no 'for:' clause
labels: { severity: critical }
One missed scrape (05:23:42Z) is enough to fire. A 'for: 5m' would have absorbed it.
Only one collector scrapes api-gateway; no second collector or direct Prometheus scrape exists.
scrapers of api-gateway:1
redundant source:none
With redundancy, up=1 would hold through a single collector restart. The gap is the root cause.
Network policy or firewall is dropping collector trafficRuled out
Check network policies between monitoring and platform namespaces to see if traffic is explicitly dropped.
No network policies block the traffic. The connection refused is from a collector restart issue, not a network policy drop.
No new or restrictive network policies in place.
NAME POD-SELECTOR AGE
allow-ingress <none> 15d
Traffic is not blocked at the network layer.
Final RCA
A single missed scrape with no fallback fires the alert immediately
The ApiGatewayDown rule has no 'for' clause, so one 0.00 sample crosses the threshold of 1 and fires Critical. There is no fallback scrape source, so a single collector's restart is enough to read 0.00. The root cause is the monitoring pipeline, not the application: harden the rule and add redundancy.
Remediation
kubectl rollout restart deploy/apm-collector -n monitoring
# prometheus-rules.yaml
- alert: ApiGatewayDown
expr: up{service="api-gateway"} == 0
for: 5m # <-- Add this
labels:
severity: critical
kubectl describe pod apm-collector-8f7d6c5b3-xyz99 -n monitoring | grep -A5 "Last State"- Restart the APM Collector to force a fresh scrape connection.
- Add a for: 5m duration to the alerting rule to prevent transient scrape failures from firing alerts.
- Add a fallback scrape source (e.g., a second collector or direct Prometheus scrape) so a single collector failure doesn't cause false positives.
- Investigate why the APM Collector restarted: check for OOMKill or resource limits.
- Add a meta-alert for APM Collector health so scrape failures are flagged before they cause downstream false positives.
The application was never at risk here, the failure lived entirely in the monitoring pipeline: a single collector restart plus an alert rule with no for duration and no fallback scrape source was enough to page on a healthy service. A for: 5m clause absorbs exactly this kind of transient scrape gap without hiding a genuine outage, and a second scrape path, whether a redundant collector or a direct Prometheus scrape, means one collector's restart no longer reads as the whole service being down. Root-causing why the collector restarted in the first place closes the remaining gap, since the same false alarm recurs on every future restart until that is understood.
Frequently asked questions
Was the api-gateway down alert a real outage?
No. All 3 api-gateway pods were Running and Ready with zero restarts, the Service had all 3 endpoints registered, and an in-cluster curl to healthz returned 200. The alert fired because the APM Collector reported a value of 0.00, not because the service stopped serving traffic.
How do you distinguish a false positive down alert from a genuine outage in Kubernetes?
Check pod readiness, restart counts, and Service endpoints directly rather than trusting the metric alone. Here every api-gateway pod showed READY 3/3 with RESTARTS=0, all 3 endpoints were registered, and healthz answered 200, which together rule out a real outage and point at the monitoring pipeline instead.
Why did a single missed scrape trigger a critical api-gateway down alert?
The ApiGatewayDown rule had no for duration, so one 0.00 sample from a failed scrape was enough to cross the threshold of 1 and fire Critical immediately. A for: 5m clause would have absorbed the single missed scrape from the collector's restart 12 minutes earlier without paging anyone.
What caused the APM Collector to report api-gateway as down when it was healthy?
The apm-collector pod had restarted 12 minutes before the alert and failed to re-establish its scrape connection to api-gateway.platform:8080/metrics, logging repeated connection timeouts. With no fallback scrape source, that single collector's failure was reported as up=0.00 for the whole service.