Alert info
- Severity
- Warning
- Detected by
- PagerDuty
- Alert
- [FIRING:1] url-swapping studioAI-Images - Urgency: High
- Time
- 2026-04-17 10:19:44 UTC
- Service
- CADENCE-CLIENT-URL-SWAPPING-SERVICE (prod · us-east-1)
url-swapping-service alerted with a churned/pending task: desired=2, running=2, pending=1. Investigate the cause.
ECS AZ rebalancing failed to place a replacement task with TaskFailedToStart: MemberOf placement constraint unsatisfied. CPU burst hit 115.22% at 10:03Z, but app logs showed normal operation. Service entered churned state (desired=2, running=2, pending=1) with no confirmed full outage.
- CPU Peak
- 115.22%
- Stop Code
- TaskFailedToStart
- Memory
- ~4.2% avg
Entities identified
Infrastructure
PROD-ECS-CLUSTER
ECS cluster
EC2-backed ECS cluster in
us-east-1. Hosts the url-swapping service across multiple AZs; AZ rebalancing acts here.AZ Rebalancing
scheduler action
ECS AZ rebalance action in the cluster.
MemberOf constraint
placement rule
Placement constraints associated with ECS services and tasks.
stopped task
TaskFailedToStart
A task that stopped during the incident window.
task definition
1 vCPU / 4 GiB
ECS task definition specifying CPU and memory reservations.
Service
url-swapping-service
ECS service · prod
CADENCE-CLIENT-URL-SWAPPING-SERVICE. Churned during incident: desired=2, running=2, pending=1. Deployment updatedAt 10:13:21Z.
Data
CloudWatch
metrics
CloudWatch metrics for the ECS service.
application logs
CloudWatch Logs
Application logs for the url-swapping service.
External
PagerDuty
alert
Fired [FIRING:1] url-swapping studioAI-Images, Urgency High at 10:19:44Z. Original alert rule/threshold not retrievable.
How they relate
The same relationships as text
- PROD-ECS-CLUSTER hosts url-swapping-service
- AZ Rebalancing acts on PROD-ECS-CLUSTER
- AZ Rebalancing started task stopped task. AZ rebalance may start or stop tasks.
- MemberOf constraint blocked placement stopped task. Constraints determine task placement.
- stopped task churned to pending url-swapping-service. Service maintains desired count through tasks.
- CloudWatch CPU burst url-swapping-service. Monitors service metrics.
- task definition sizes url-swapping-service
- application logs emits url-swapping-service. Captures service output.
- url-swapping-service triggered alert PagerDuty
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
Memory pressure / OOM killed the taskRuled out
Query CloudWatch MemoryUtilization for the window and compare against the task definition limit.
Memory held 3.09% to 9.34% (avg ~4.2%), peaking at ~382 MiB of 4096 MiB. No OOM signal.
MemoryUtilization stayed low across the entire incident window.
MemoryUtilization
Range: 3.09% - 9.34%
Average: ~4.2%
Memory [#-------------------] 9% healthy
peak ~382 MiB of 4096 MiB limit
Memory was never close to the limit. Not an OOM event.
Application bug / exception crashed the url-swap pathRuled out
Scan CloudWatch Logs for the service and run explicit ERROR / Exception searches across the incident window.
Logs show normal wrapped_url_swap jobs, status 200, and url-updated lines throughout. Explicit searches returned 0 ERROR and 0 Exception.
Normal job execution with no error patterns in the window.
/ECS/PROD-ECS-CLUSTER-CADENCE-CLIENT-URL-SWAPPING
Running job "wrapped_url_swap" repeated INFO
logging-response-status status: 200 repeated INFO
url-updated repeated INFO
Explicit search for ERROR: 0 matches
Explicit search for Exception: 0 matches
The service kept doing normal work. The alert is not an application-level failure.
Downstream dependency (DB / API / external) failedRuled out
Look for trace/error evidence or a downstream dependency graph tied to the incident.
No downstream dependency graph was provided and no trace/error evidence tied the incident to an external API or database; logs show 200s. Available evidence points to an ECS scheduling problem.
No trace-level errors and no downstream failure signals found.
Nothing links the incident downstream. The problem is in ECS scheduling, not a dependency.
ECS scheduling/placement failure during AZ rebalanceConfirmed
Query ECS stopped tasks and service event history to identify reasons for task churn.
A replacement task failed to start during AZ rebalancing with a hard TaskFailedToStart, leaving the service in a churned/pending state that triggered the alert. Confirmed; narrow to the exact stop reason next.
A stopped task gives the strongest hard-failure signal.
createdAt = 2026-04-17T09:43:39Z
stoppedAt = 2026-04-17T10:13:05Z
stopCode = TaskFailedToStart
stoppedReason = TaskFailedToStart: MemberOf placement constraint unsatisfied
ECS tried to rebalance capacity but could not schedule the task onto a valid host.
Service was non-steady with a pending task that could not be placed.
desiredCount = 2
runningCount = 2
pendingCount = 1
deployment updatedAt = 2026-04-17T10:13:21Z
The failed replacement left desired=2, running=2, pending=1. Classic placement-failure shape.
CPU burst (115.22%) overloaded the task and drove the failureRuled out
Pull CloudWatch CPUUtilization and the task definition sizing to check for overload.
The burst is real (1-vCPU burstable behavior) and precedes the failure, but a CPU spike does not emit a MemberOf placement reason. Contributing to alert sensitivity at most, not the cause. Ruled out as primary; subordinate to the placement failure.
Sharp CPU burst ~12 min before the placement failure, then recovery.
CPUUtilization ........#@%..
10:02Z 103.69%
10:03Z 115.22% <- peak
10:06Z 71.18%
CPU [####################-] 115% burst above 1-vCPU baseline
Burstable 1-vCPU task briefly exceeded allocation. Raises alert sensitivity, does not cause a MemberOf failure.
Resource limits vs actual usage during the burst.
container cpu = 1024 # 1 vCPU
container memory = 4096 MiB
container memoryReservation = 4096 MiB
Sizing is normal; the burst is transient and unrelated to scheduling eligibility.
A MemberOf placement constraint cannot be satisfied by any hostRoot cause
Read the stopped task's stoppedReason and correlate with the AZ rebalance events and placement strategies.
AZ rebalancing started a replacement task in us-east-1b, but no host matched the MemberOf placement constraint, so the task stopped at 10:13:05Z with TaskFailedToStart. The service config shows placementConstraints=[], so the constraint originates elsewhere (task definition, capacity provider, or cluster).
Explicit stop reason names MemberOf, yet the service config carries no constraint.
stoppedReason = MemberOf placement constraint unsatisfied
placementConstraints = []
placementStrategy = spread(ecs.availability-zone, instanceId)
A hard MemberOf failure with an empty service-level constraint: the constraint is sourced outside the service config.
Rebalance events bracket the failure exactly.
09:43:39Z AZ Rebalancing; started 1 task in us-east-1b
10:13:05Z task stopped: TaskFailedToStart (MemberOf)
10:13:11Z started 1 task in us-east-1b to AZ Rebalance
10:13:21Z deployment updated
The placement failure happened during rebalance churn, not a clean steady-state deploy.
Exact source of the MemberOf constraint + AZ instance attributesNeeds access
Attempt to query the task definition placementConstraints, container-instance attributes, and capacity provider state.
The retrieved service config did not expose the constraint (placementConstraints=[]). Pinpointing whether it lives on the task definition, depends on a container-instance attribute absent in us-east-1b, or reflects missing capacity needs access beyond the available telemetry.
Service config shows placementConstraints=[]; the true source was not in the data pulled.
Stop at the evidence boundary and label the constraint source as needing access rather than guessing.
Service-level placement constraints blocked placementRuled out
Check the service configuration for placement constraints.
Service config shows placementConstraints=[]. The constraint must be elsewhere.
Final RCA
A MemberOf placement constraint cannot be satisfied by any host
AZ rebalancing started a replacement task in us-east-1b, but no host matched the MemberOf placement constraint, so the task stopped at 10:13:05Z with TaskFailedToStart. The service config shows placementConstraints=[], so the constraint originates elsewhere (task definition, capacity provider, or cluster).
Remediation
# Identify the MemberOf constraint source: check task definition and cluster config
aws ecs describe-task-definition --task-definition CADENCE-CLIENT-URL-SWAPPING-SERVICE \
--query 'taskDefinition.placementConstraints'
# Check container instance attributes in the cluster
aws ecs list-container-instances --cluster PROD-ECS-CLUSTER \
--query 'containerInstanceArns' | xargs -I{} \
aws ecs describe-container-instances --cluster PROD-ECS-CLUSTER --container-instances {}
# Review capacity provider strategy for AZ balance handling
aws ecs describe-clusters --clusters PROD-ECS-CLUSTER \
--include SETTINGS --query 'clusters[0].settings'- Audit the MemberOf placement constraint source: the service config shows placementConstraints=[], so the constraint may be on the task definition, capacity provider, or cluster-level setting.
- Verify container instance attributes across all AZs match the constraint requirements, especially after instance replacements or scaling events.
- Consider adding a CloudWatch alarm on RunningTaskCount or PendingTaskCount to detect service churn earlier than application-level alerts.
- Review AZ rebalancing behavior and whether managed termination protection or capacity provider settings can prevent placement failures during rebalances.
- Add instance draining safeguards to ensure replacement capacity is available before instances are removed during AZ rebalancing.
The service config carries no placementConstraints, so the MemberOf rule that blocked scheduling lives somewhere the service definition does not show: the task definition, a capacity provider, or a cluster-level setting, and none of those were retrievable in this investigation. The durable fix is closing that visibility gap before the next AZ rebalance runs, by auditing every layer that can attach a placement constraint and confirming container-instance attributes are present in every AZ the service can be scheduled into. Pairing that audit with a CloudWatch alarm on PendingTaskCount gives an earlier signal than waiting for a downstream, application-level page the next time a rebalance tries to place a task against a constraint the service itself cannot see.
Frequently asked questions
What does TaskFailedToStart: MemberOf placement constraint unsatisfied mean in ECS?
It means the ECS scheduler could not find any container instance that satisfied a MemberOf placement constraint when trying to place a task. In this incident the stopped task showed exactly this stopCode and stoppedReason at 10:13:05Z, roughly 30 minutes after it was created during an AZ rebalance, while the service-level config reported placementConstraints=[], meaning the constraint was defined somewhere other than the service itself.
Why would a service show placementConstraints=[] but still fail a MemberOf constraint?
Because placement constraints can be attached at the task definition, capacity provider, or cluster level, not only on the service. In this investigation the service API call returned an empty constraints list, yet the stopped task's stoppedReason explicitly named a MemberOf failure, which means the constraint originated outside what the service-level describe call exposes. Confirming the exact source needed task-definition and container-instance level access beyond what was retrieved.
Can a CPU burst cause an ECS MemberOf placement failure?
No. In this incident CPUUtilization peaked at 115.22% about ten minutes before the placement failure, but a CPU burst does not generate a MemberOf stoppedReason, which is a scheduling-eligibility error, not a resource-exhaustion error. Memory stayed at roughly 4.2% average the whole time. The burst was ruled out as the cause and treated only as a possible contributor to alert sensitivity.
Does an ECS placement failure during AZ rebalance always mean an outage?
Not necessarily. In this incident the service entered a churned state (desired=2, running=2, pending=1) after one replacement task failed to place, but running tasks kept serving traffic and application logs showed normal job execution with zero ERROR or Exception matches throughout. The evidence supported a scheduling failure during rebalance, not a confirmed full outage.