Alert info
- Severity
- Critical
- Detected by
- Uptime Monitor
- Alert
- 503 Unexpected Status Code on https://content-api.example.com/
- Time
- 2026-03-15 14:32 IST
- Service
- content-api (prod)
A synthetic probe returned 503 from content-api.example.com in production. The edge and ingress are healthy, yet every request fails.
Deployment scaled replicas to 0, leaving the service with no healthy endpoints. Root cause identified in under 5 minutes.
- Alert to RCA
- 5 min
- Status
- Resolved
Entities identified
Infrastructure
content-api
Deployment
Deployment in prod namespace managing the content-api pods.
content-api-7d4f8b6c9a
ReplicaSet
Active ReplicaSet for the content-api deployment.
content-api pods
0/2 running
Compute pods running the content-api application.
Service
content-api
Service
ClusterIP Service for content-api in prod. Endpoints: <none>. No backing pods, so every request 503s.
Data
monitoring
metrics / RPS
Service monitoring dashboards, metrics, and traffic throughput.
External
Uptime Monitor
external probe
Synthetic probe on https://content-api.example.com/. Fired 503 Unexpected Status Code at 14:32 IST.
Gateway
ingress / LB
edge router
Routes external traffic to the content-api service. Healthy, but returns 503 because the backend service has no endpoints.
How they relate
The same relationships as text
- Uptime Monitor probes ingress / LB. Synthetic probe hits the edge and receives 503 Service Unavailable.
- ingress / LB routes content-api. Ingress routes to the Service, but the Service has no endpoints, so a 503 is returned.
- content-api endpoints content-api pods. Service selects pods by label. With 0 pods, endpoints are empty.
- content-api manages content-api-7d4f8b6c9a
- content-api-7d4f8b6c9a manages content-api pods. ReplicaSet manages pod replicas.
- content-api throughput monitoring. Request throughput fell to 0 RPS at T0.
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
A bad image / startup crash prevented pods from becoming ReadyRuled out
Query the Deployment replica state to see if spec.replicas > 0 while available is 0, which would indicate pods failing to start.
spec.replicas reads 0, not 2. The deployment is not requesting any pods, so there are none to crash. A bad image would leave desired > 0 with available 0.
Desired, current, and available replicas are all 0.
$ kubectl get deploy content-api -n prod \
-o jsonpath='{.spec.replicas} {.status.replicas} {.status.availableReplicas}'
0 0 0
Desired = 0 means no pods are requested. A crash would show desired > 0, available 0. Ruled out.
Node-level failure evicted or deleted all podsRuled out
Query the cluster for any existing pods (Pending, Terminating, or Running) for the content-api application.
The desired replica count is 0, written into the Deployment spec. An eviction leaves desired unchanged and triggers rescheduling. This is a controller write, not a node event.
No pods exist at all, not even Pending or Terminating ones waiting to reschedule.
$ kubectl get pods -n prod -l app=content-api
No resources found in prod namespace.
Eviction would leave Pending pods chasing the desired count of 2. There are none. Ruled out.
Ingress / load balancer or networking failure upstream of the serviceRuled out
Inspect the Service endpoints to see if backends are populated but perhaps unreachable due to network issues.
The Service endpoints are empty. The 503 originates from having no backends, not from a misrouted or unreachable network path. A networking failure would leave endpoints populated.
The Service has no endpoints at all.
$ kubectl get endpoints content-api -n prod -o wide
NAME ENDPOINTS AGE
content-api <none> altd
Empty endpoints means the problem is at the pod/replica layer, not ingress or networking. Ruled out.
The Service has no backing pods at allConfirmed
Check the Service endpoints list and request throughput metrics to confirm if traffic stopped because there are zero backing pods.
Endpoints are empty, no pods are running, and request throughput dropped to 0 RPS concurrent with the alert. The Service genuinely has nothing to route to. Confirmed; trace why the pods disappeared.
No endpoints registered behind the Service.
$ kubectl get endpoints content-api -n prod -o wide
NAME ENDPOINTS AGE
content-api <none> altd
With zero endpoints, ingress has no backend and returns 503 to every caller.
Throughput fell to 0 RPS at T0, in lockstep with the scale-down.
time rps
T0-6m 412
T0-4m 198 <- scale-down begins
T0 0 <- 503s start
Traffic stopped being served exactly when the pods went away. The cause is upstream in the workload.
Deployment was scaled to 0 replicasRoot cause
Read the Kubernetes events and the Deployment spec for content-api in prod around T0-4min.
The content-api Deployment was scaled from 2 to 0 replicas about 4 minutes before the alert. The ReplicaSet terminated all pods, the Service lost every endpoint, and ingress returned 503 to all traffic. Root cause. The trigger (manual action, CI/CD, or HPA/KEDA) could not be confirmed because the metrics API was unavailable.
A scale-down event was recorded ~4 minutes before the alert fired.
$ kubectl get events -n prod --sort-by=.metadata.creationTimestamp \
| grep -i content-api
Scaled down replica set content-api-7d4f8b6c9a to 0 from 2
The ReplicaSet was explicitly scaled to 0, terminating all pods. This is the proximate cause.
The Deployment spec confirms replicas were set to 0.
$ kubectl get deploy content-api -n prod \
-o jsonpath='{.spec.replicas} {.status.replicas} {.status.availableReplicas}'
0 0 0
Desired 0, current 0, available 0. An intentional controller write to zero, not a crash or eviction.
The endpoint returns HTTP 503 with no backend available.
$ curl -s -o /dev/null -w '%{http_code}' https://content-api.example.com/
503
End-to-end confirmation: zero replicas to zero endpoints to 503. Scaling back to 2 restores service.
Pods were manually deleted but ReplicaSet has not recreated them yetRuled out
Check the Deployment and ReplicaSet specs to see if they still desire pods.
The Deployment explicitly requested 0 replicas. This wasn't a case of deleted pods waiting to be recreated by the controller.
Desired replicas is 0, not > 0.
$ kubectl get deploy content-api -n prod \
-o jsonpath='{.spec.replicas} {.status.replicas} {.status.availableReplicas}'
0 0 0
The controller intentionally wants 0 pods, ruling out a simple pod deletion.
Final RCA
Deployment was scaled to 0 replicas
The content-api Deployment was scaled from 2 to 0 replicas about 4 minutes before the alert. The ReplicaSet terminated all pods, the Service lost every endpoint, and ingress returned 503 to all traffic. Root cause. The trigger (manual action, CI/CD, or HPA/KEDA) could not be confirmed because the metrics API was unavailable.
Remediation
kubectl scale deploy/content-api -n prod --replicas=2
kubectl rollout status deploy/content-api -n prod
kubectl get endpoints content-api -n prod
curl -s -o /dev/null -w '%{http_code}' https://content-api.example.com/- Scale the deployment back to 2 replicas immediately to restore service.
- Investigate who or what triggered the scale-down (check CI/CD pipeline logs,
kubectlaudit logs). - Add a Kubernetes admission webhook or OPA policy to reject Deployment updates that set replicas to 0 for production workloads.
- Fix the metrics API to enable HPA/KEDA monitoring and prevent blind spots in autoscaler investigations.
A production Deployment reaching zero replicas should never be reachable from a single unaudited write, so an admission webhook or OPA policy that rejects a replicas: 0 update on production workloads turns this class of outage into a rejected request instead of an incident. That control does not explain who or what made the change, so pairing it with kubectl audit logging on Deployment updates keeps the trigger (manual, CI/CD, or an autoscaler) attributable the next time. The metrics API outage that blocked the HPA and KEDA check in this investigation is a separate gap worth closing on its own, since without it a genuine autoscaler-driven scale-down and a manual one are indistinguishable after the fact.
Frequently asked questions
What causes a 503 Service Unavailable error in Kubernetes?
A 503 from a Kubernetes-backed service almost always means the Service has no endpoints to route to. In this incident content-api's Deployment was scaled from 2 replicas to 0, so the ReplicaSet terminated every pod, the Service's endpoint list went empty, and the ingress had no backend to forward requests to, returning 503 to every caller.
How do you tell a scaled-to-zero deployment apart from a crashing pod when a service returns 503?
Check the Deployment's desired replica count and the pod list. A crash loop shows desired replicas greater than zero with pods cycling through restarts. Here, spec.replicas read 0 and kubectl get pods returned no resources at all, not even pending ones, which rules out a crash and points to an explicit scale-down instead.
Does scaling a Kubernetes deployment back up immediately fix a 503 caused by zero replicas?
Yes. Once the Deployment is scaled back to its intended replica count, new pods start, register as Service endpoints, and the 503s stop as soon as they pass readiness checks. In this incident scaling back to 2 replicas and confirming endpoints and a 200 response was the full immediate fix.
Was an autoscaler like HPA or KEDA responsible for the 503 outage?
It could not be confirmed either way. The investigation tried to check HPA and KEDA ScaledObject status for content-api, but the metrics API was unavailable at the time, so whether the scale-down came from a manual action, a CI/CD pipeline, or an autoscaler remains unresolved.