How do you fix an ECS task placement failure during an AZ rebalance?

The service says desired 2, running 2, pending 1. Nothing is down, the application logs look normal, and CPU shows a burst that turns out to be irrelevant. Something is trying to place a task and failing, and the failure is not where you would look for it.

Why is a placement failure so hard to see?#

Because ECS surfaces the outcome, not the cause.

A task that cannot be placed shows up as a stopped task with a generic reason, and the StoppedReason on the task itself is frequently empty or unhelpful. The scheduler's actual explanation goes to the service's event stream, which is a separate API call almost nobody makes first. So the visible artifact is a task that stopped, and the sentence explaining why is one level away.

This also makes it easy to blame the loudest metric on the dashboard instead. A CPU burst during the same window looks causal and usually is not: the burst is the surviving tasks absorbing traffic the missing task should have taken.

How do you find the real reason?#

Go to the service event stream first. This is the single highest-value command in the whole investigation.

CLUSTER=prodSERVICE=url-swapping-service # The scheduler's own words. Newest first, and the reason is in here# in plain English, unlike anything on the task.aws ecs describe-services --cluster $CLUSTER --services $SERVICE \  --query 'services[].events[:15].[createdAt,message]' --output text 

TaskFailedToStart: MemberOf placement constraint unsatisfieddesired=2  running=2  pending=1     # churned, not downCPUUtilization  115.22% at 10:03Z   # a burst, not the cause 

MemberOf placement constraint unsatisfied is the answer. Now read what that constraint is:

aws ecs describe-services --cluster $CLUSTER --services $SERVICE \  --query 'services[].{Constraints:placementConstraints,Strategy:placementStrategy,Desired:desiredCount,Running:runningCount,Pending:pendingCount}' # The task definition can carry its own constraints, separate from the# service's. Both have to be satisfiable, and people routinely forget the second.aws ecs describe-task-definition --task-definition $SERVICE \  --query 'taskDefinition.{Constraints:placementConstraints,Cpu:cpu,Memory:memory,Requires:requiresCompatibilities}' 

What does each constraint type actually require?#

Three shapes cover nearly everything, and each fails differently.

distinctInstance requires every task in the service to land on a different container instance. It fails the moment the number of eligible instances drops below the desired count. A service with desired 3 needs three instances, and losing one during a rebalance makes the third task unplaceable immediately.

memberOf with an expression restricts placement to instances matching a cluster query, such as attribute:ecs.availability-zone in [us-east-1a, us-east-1b] or a custom attribute. It fails when no instance carrying that attribute has room, which can happen because the instances went away or because the ones remaining are full.

Resource requirements are not constraints but behave like one. A task requesting 2 vCPU cannot be placed on instances with 1.5 vCPU free, no matter how many of them there are. Fragmentation across many instances looks like plenty of spare capacity in aggregate and satisfies nothing.

Check which instances could actually have taken the task:

# Registered instances, what they have left, and the attributes a# memberOf expression would be matching against.aws ecs list-container-instances --cluster $CLUSTER --query 'containerInstanceArns' --output text \  | xargs aws ecs describe-container-instances --cluster $CLUSTER --container-instances \  --query 'containerInstances[].{      Id:ec2InstanceId,      Status:status,      Running:runningTasksCount,      CPUleft:remainingResources[?name==`CPU`].integerValue|[0],      MEMleft:remainingResources[?name==`MEMORY`].integerValue|[0],      AZ:attributes[?name==`ecs.availability-zone`].value|[0]    }' --output table 

Aggregate free capacity is not the number that matters. A single instance able to hold the whole task is.

Why does an AZ rebalance break placement?#

Because it changes the instance set under a constraint that was written against the old one.

ECS and the underlying Auto Scaling group both work to spread capacity evenly across availability zones. A rebalance terminates an instance in an over-represented AZ and launches one elsewhere. During the gap, the eligible instance set is smaller, and any constraint that was exactly satisfied becomes unsatisfiable. Nothing in your configuration changed. The set it was evaluated against did.

This is why the failure is intermittent and why it resolves on its own so often that it gets closed without a cause. The window is exactly as long as the replacement takes to launch, join the cluster and pass its health check.

# Was there a rebalance? This is the confirming evidence.aws autoscaling describe-scaling-activities \  --auto-scaling-group-name ecs-prod-asg --max-records 15 \  --query 'Activities[].{Start:StartTime,Cause:Cause,Status:StatusCode}' --output table 

Look for an instance was taken out of service in response to a difference between desired and actual capacity or an AZ rebalance cause line stamped within a minute or two of the TaskFailedToStart event.

How do you fix an ECS placement failure?#

Match the fix to the constraint that failed rather than adding capacity reflexively.

If distinctInstance is the constraint, you need at least as many instances as tasks, with one spare to survive a rebalance. Either raise the ASG minimum, or drop to a spread strategy if strict distinctness was never a real requirement:

{  "placementConstraints": [],  "placementStrategy": [    { "type": "spread", "field": "attribute:ecs.availability-zone" },    { "type": "spread", "field": "instanceId" }  ]} 

spread expresses the same intent as distinctInstance for most services, high availability across instances and zones, but it is a preference rather than a hard requirement, so a rebalance degrades placement instead of blocking it.

If a memberOf expression is the constraint, widen it or guarantee the attribute survives a rebalance. An expression naming two AZs in a three-AZ ASG will break every time the third AZ is the one with capacity.

If resources are the constraint, the fix is capacity or a smaller task. Check the fragmentation first, because the answer is often that no single instance has room even though the cluster looks half empty.

Then make the failure visible next time, since a churned service is not a down service and most alerting misses it entirely:

# Pending tasks that never become running is the signal. Running count# alone stays green through this entire class of incident.aws cloudwatch put-metric-alarm \  --alarm-name url-swapping-service-pending-tasks \  --namespace ECS/ContainerInsights --metric-name PendingTaskCount \  --dimensions Name=ClusterName,Value=prod Name=ServiceName,Value=url-swapping-service \  --statistic Maximum --period 60 --evaluation-periods 5 \  --threshold 0 --comparison-operator GreaterThanThreshold 

What does a real ECS placement failure look like?#

Sherlocks AI investigated one on a url-swapping service. PagerDuty fired at 10:19:44 UTC with the service in a churned state: desired 2, running 2, pending 1.

ECS AZ rebalancing had tried to place a replacement task and failed with TaskFailedToStart: MemberOf placement constraint unsatisfied. The distractor was a CPU burst hitting 115.22 percent at 10:03 UTC, which looked like the obvious cause and was not: application logs showed entirely normal operation throughout, and the burst was the running tasks absorbing the missing one's share.

The service was never down, which is exactly why it was confusing. Two tasks were running the whole time and every request-path metric was healthy. What was broken was the scheduler's ability to satisfy a constraint against an instance set that a rebalance had just changed. The full writeup is in the ECS task placement failure investigation.

Reading these backwards, from event to constraint to instance set, is the method. It generalises to the rest of AWS, where the loudest metric and the actual cause are frequently different dimensions entirely, as in EBS throughput throttling. More worked incidents are on the AWS examples page, and the scheduler's rules are set out in the Amazon ECS task placement documentation.

ECS task placement failure FAQ#

Where is the real reason for a placement failure? In the service event stream, from describe-services. The stopped task itself usually carries no useful reason.

What does MemberOf placement constraint unsatisfied mean? No container instance matching the constraint expression had room for the task at that moment.

Why did it fail now when the same config worked an hour ago? An AZ rebalance or a scaling activity changed the eligible instance set. The constraint did not change, the set it evaluates against did.

Is a churned service the same as an outage? No, and that is the trap. Running count can stay at desired while a replacement repeatedly fails to place, so request-path metrics look healthy.

Should I use distinctInstance or spread? spread for most services. It expresses the same availability intent as a preference, so a rebalance degrades placement instead of blocking it.

Why is CPU spiking during the incident? Usually because the remaining tasks are absorbing the missing task's traffic. It is a consequence, not the cause.