ECS circuit breaker and CloudFormation: who owns the rollback

This post is about the question that started the whole series: when an ECS deployment fails while CloudFormation is driving, who puts things back - ECS, CloudFormation, or both, one after the other? We had once watched a service roll back twice in a row on another stack, every task replaced once by ECS and then again by CloudFormation, and that fear is why we built the test stack. Nothing in the AWS documentation says what happens to a service resource when ECS rolls a deployment back in the middle of a stack update. So we measured it: deployments where the new tasks never come good, first with no failure detector at all, then with the circuit breaker in both of its modes, on both deployment strategies. Every number below comes from a measured run (July 2026, eu-west-1).

The test stack

All runs below use the test stack from the first post in this series: one CloudFormation stack, three Fargate services running a small test container whose behaviour is switched by a MODE environment variable. This post uses two of them: web (two tasks, Strategy: BLUE_GREEN, behind an ALB with two target groups) and worker (two tasks, Strategy: ROLLING, no load balancer). The web runs here deploy a revision whose health endpoint answers 500 while everything else works, and the worker runs deploy one that exits five seconds after starting. The full setup, including the container listing, is in that first post.

The baseline: no failure detector at all

A plain service definition carries no failure detector at all, so the starting point is what happens when the new tasks never come good and nothing in the deployment machinery can notice. The two strategies fail very differently.

Blue/green: an invisible hang

For this run, the web service carried the full blue/green wiring and its deployment alarms, but no circuit breaker:

 1WebService:
 2  Type: AWS::ECS::Service
 3  Properties:
 4    # ... the full blue/green wiring from part one of the series
 5    DeploymentConfiguration:
 6      Strategy: BLUE_GREEN
 7      BakeTimeInMinutes: 1
 8      Alarms:
 9        Enable: true
10        Rollback: true
11        # attached, but they never fire for this failure - see below
12        AlarmNames: [!Ref FiveXxAlarm, !Ref TgAFiveXxAlarm, !Ref TgBFiveXxAlarm]
13      # no DeploymentCircuitBreaker

We deployed the unhealthy revision. ECS launched green tasks into the alternate target group, the ALB health checks failed, and the scheduler stopped and replaced them at a steady rate of about one task per minute: 18 green tasks over the 16 minutes we watched, with no sign of backoff, every one stopped with the same reason:

Task failed ELB health checks in (target-group .../web-a)

The deployment record stayed IN_PROGRESS at the SCALE_UP stage for the entire hang. It never reached a traffic shift, so production stayed on blue: our witness loops made 1,989 requests across both listeners during the run and every one was answered with a 200 by the old revision.

The unsettling part is that nobody notices. All three 5xx deployment alarms on the service stayed OK for the whole run, for two reasons: ALB metrics exclude health-check requests (that’s documented), and no client request ever reaches the green tasks because both listener rules still forward to blue. A failure that only fails health checks is invisible to users and to the 5xx metrics at the same time. The only signals left are the service events, the UnHealthyHostCount metric on the green target group, and a stack update that never finishes. The only timer that would eventually end the hang is CloudFormation’s own 36-hour limit on blue/green updates.

One more trap for anyone watching dashboards: runningCount inflated to 5 during the hang (two blue tasks plus three green ones mid-replacement). The running count is not a health signal.

Rolling: a crash loop with no stop events

The worker’s configuration is plain rolling with nothing attached at all:

1WorkerService:
2  Type: AWS::ECS::Service
3  Properties:
4    # ...
5    DeploymentConfiguration:
6      Strategy: ROLLING
7      MaximumPercent: 200
8      MinimumHealthyPercent: 100
9      # no circuit breaker, no alarms, and no health source of any kind

The same experiment on the worker service, with the crash revision, has a nastier signature. The old two tasks kept running throughout (Min 100 holds capacity), and ECS relaunched the crashing new tasks at about 2.4 launches per minute: 39 tasks over a 16-minute watch, again with no backoff. This is a genuine crash-loop churn scenario, and the event stream is the strange part. It is pure launch spam:

(service worker) has started 1 tasks: (task 0087d1509c95...)

repeated every 25 seconds or so, with no stop events at all. A task that exits by itself emits none (the health-check stops in the blue/green run did). The evidence that anything is wrong lives only in the stopped-task records, which age out of list-tasks after about an hour. The deployment record pinned at SCALE_UP here too, and runningCount oscillated between 2 and 4.

At that cadence, a crash loop left overnight is roughly 3,400 Fargate task launches a day. That number is noise rather than money - Fargate bills vCPU and memory seconds, not launches - but it isn’t free either: each short-lived task is billed for a minimum of one minute, and the loop holds the deployment’s surge capacity indefinitely, roughly doubling this service’s spend for as long as it runs. And unlike blue/green, we can’t point at a documented timer here: we found no stated handler timeout for a plain rolling update, and nothing fired within our watch window.

Manual recovery: cancelling the update

Both hangs end the way an on-call person would end them: aws cloudformation cancel-update-stack. We measured it on both options, and the difference between them previews the rest of this post.

Blue/green: the cancel hands the work to ECS

The CloudFormation handler cancelled the in-flight deployment through ECS’s own stop-service-deployment API, and ECS’s native blue/green rollback did the cleanup: green tasks stopped, the alternate target group emptied, the old task set made primary again. The two healthy blue tasks were never touched. The stack was redeployable 2 minutes 46 seconds after the cancel, parameters restored automatically, no continue-update-rollback needed.

Rolling: the rollback is a real deployment (again)

On a rolling service there is no old task set to re-promote, so the recovery is a full re-deployment of the old configuration: two fresh old-revision tasks started first, then the two healthy old tasks were stopped behind them, so capacity never dropped. It works, but it replaces two perfectly healthy tasks that the blue/green cancel left alone. Redeployable 3 minutes 30 seconds after the cancel.

One record-keeping gotcha: the cancelled rolling deployment’s record says only “Replaced by a new service deployment.” - no trace of the cancel or of who ordered it. The attribution lives in the CloudFormation events (“UPDATE_ROLLBACK_IN_PROGRESS … User Initiated”) and CloudTrail. The blue/green cancel, by contrast, records “rolling back due to user action” in the service events.

The circuit breaker, on paper

The deployment circuit breaker is the detector for exactly the failures above. It’s one block on the service resource:

 1WorkerService:
 2  Type: AWS::ECS::Service
 3  Properties:
 4    # ...
 5    DeploymentConfiguration:
 6      Strategy: ROLLING
 7      DeploymentCircuitBreaker:
 8        Enable: true
 9        Rollback: true
10        ThresholdConfiguration:
11          Type: COUNT
12          Value: 3

Attaching it to a running service is a quiet configuration update. No deployment, no task replaced - the same behaviour we measured for every DeploymentConfiguration-only change.

The documented mechanics, briefly. The breaker counts failures in two stages:

1. tasks that fail to reach RUNNING,
2. tasks replaced because they failed health checks, where the validated health sources are Elastic Load Balancing, Cloud Map and container health checks.

By default only consecutive failures count (resetOnHealthyTask is true). The threshold is either a COUNT, used exactly as given, or the default BOUNDED_PERCENT (50% of the desired count, rounded up, clamped between 3 and 200), so for a small service the default threshold is simply 3. When the breaker declares a failure and Rollback is true, ECS rolls back to the most recent deployment in the COMPLETED state. If no completed deployment exists, there’s nothing to roll back to: the failed deployment just stalls and launches nothing.

Two notes on the documentation itself. First, the page says the breaker is “only supported for Amazon ECS services that use the rolling update (ECS) deployment controller”, which reads as if blue/green were excluded. It isn’t: the native blue/green strategy runs on that same ECS controller, and the breaker demonstrably trips there (see next section). Second, the page contradicts itself about stage one. One passage says a deployment can complete “because there is more than one task that transitioned to the RUNNING state”, while another says stage two “is entered when there are one or more tasks in the RUNNING state”. On a service with one desired task, the difference between “more than one” and “one or more” is the whole question. We come back to that configuration, where the ambiguity matters most, in a later post in this series: “The deployment that lied: ECS singletons and false success”.

The circuit breaker, measured

We repeated both no-detector failures with the breaker attached (COUNT 3, Rollback: true).

Blue/green: tripped before the traffic shift

The trip came at exactly three failed tasks, about four minutes in, while the deployment was still in SCALE_UP. No traffic shift had happened, so production was never touched: all 980 witnessed requests were answered by blue with a 200. The whole thing was hands-off, 8 minutes 4 seconds from update to a redeployable stack, against roughly “however long it takes for someone to notice” plus a decision to cancel for the no-detector version of the same failure described earlier.

Rolling: tripped in just over two minutes

The breaker declared the failure 2 minutes 2 seconds after the ECS deployment record started. The old tasks were untouched, and structurally so: a crash loop never gets far enough to scale the old tasks down, so a breaker rollback on this configuration cannot replace them. Expect an extra task or two to be started and stopped while the breaker is counting, though - the blue/green run launched one more than the threshold, the rolling run two more, because detection lags the third failure by a few seconds.

Who owns the rollback (ECS-triggered)

Now the question the series exists to answer. ECS has rolled the service back on its own. CloudFormation is still mid-update, waiting on a service resource whose deployment just failed underneath it. What does it do?

The same thing, in every ECS-owned rollback we ran - alarm-triggered, breaker-triggered and hook-triggered, on both strategies:

1. The waiter holds. CloudFormation does not fail the service resource while ECS is still rolling back. It waits until the service is restored and steady.
2. Then the resource fails. The UPDATE_FAILED event (with a reason such as “ECS Deployment Circuit Breaker was triggered”) arrives only after the service is already back on the old revision. The lag ranged from 6 seconds to about 3 minutes in our runs.
3. The stack rollback is a no-op re-sync. CloudFormation re-updates the service during the stack rollback, but because the service already matches the old configuration, the update creates no new deployment and replaces no tasks. It took about 38 seconds each time.
4. The bookkeeping is automatic. CloudFormation deletes the task definition revision it created on the way in, and restores the stack parameters to their previous values.

The feared double rollback - ECS restoring the service and CloudFormation then re-deploying it a second time - did not appear in any of these runs, and we ran more than a dozen of them across the series. We want to be precise about what that is: an observation, made in July 2026 in eu-west-1, not a documented contract. Nothing in the docs promises this behaviour, which is exactly why we tested it. Within our runs, though, it was completely consistent.

Two smaller observations in the same category. The ~38-second re-sync matches what CloudFormation spends on any UpdateService call, deployment or not - the same floor shows up on quiet configuration changes. And four of our breaker-tripped stack updates, across different services and failure modes, reached UPDATE_ROLLBACK_COMPLETE in exactly 8 minutes 5 seconds (the blue/green run above came in at 8 minutes 4 seconds). Treat that consistency as a curiosity, not a number to build on.

One practical warning for pipeline authors: while a stack rollback is in progress, the stack’s outputs disappear from describe-stacks. Tooling that reads outputs mid-recovery (to find a load balancer address, for example) will get an empty list. Resolve such values from the resources themselves if you need them during an incident.

Detect-only: making CloudFormation the rollback owner

The breaker’s Rollback: false mode detects the failure but reverts nothing, which makes CloudFormation the rollback owner: the resource fails, and the stack rollback has real work to do. We measured it on both strategies. It works, and it is strictly worse.

On blue/green, the breaker still ended the replacement loop (six tasks launched in total, against the no-detector run’s 18 and counting), and CloudFormation failed the resource just 4 seconds after the deployment went STOPPED. However, its stack rollback then ran a full blue/green deployment of the old configuration: two fresh tasks, health checks, test and production traffic shifts, and a complete bake. It stopped the two healthy blue tasks and left the target groups swapped. The recovery segment took 15 minutes 11 seconds, against 42 seconds for the ECS-owned equivalent: 19 minutes 44 seconds end to end against 8 minutes 4 seconds. Even so, no requests were broken, including through the mid-recovery traffic shift. Takes longer, rotates healthy tasks, but it’s safe.

On rolling, detection was identical (2 minutes 2 seconds after the record started), and the recovery was the same pattern as the manual cancel: a real re-deployment of the old configuration, two fresh tasks up, two healthy old tasks stopped. 7 minutes 20 seconds end to end against 4 minutes 15 seconds ECS-owned, with four extra task replacements against zero.

There’s also a diagnostic trap in this mode. The deployment record is the only ECS-side place the two modes read differently: “Service deployment rolled back because the circuit breaker threshold was exceeded” against “Service deployment stopped because the circuit breaker threshold was exceeded”. The service event is the same in both (“deployment failed: tasks failed to start”), and CloudFormation’s error is the same in both (“ECS Deployment Circuit Breaker was triggered”). From the CloudFormation console alone you cannot tell whether ECS rolled anything back.

What the failure reasons actually say

Having watched every detector fire, a ranking of how useful their failure wording is to whoever is on call:

• Pause hooks: precise. The record names the stage and the hook: “Service deployment rolled back because POST_TEST_TRAFFIC_SHIFT lifecycle hook(s) failed. Hook ecs-pause-… returned FAILED status.” (Measured in the pause hooks post.)
• Deployment alarms: generic. CloudWatch alarms that ECS watches during a deployment (they get their own post, “ECS deployment alarms: the fine print”) report “rolled back because of one or more active alarms” - the firing alarm is never named, so diagnosis means checking alarm histories.
• Circuit breaker: misleading. The service event says “tasks failed to start”, even when the tasks started fine and failed health checks, as in every one of our web runs. The deployment record’s statusReason is the accurate one - read that, not the event.

Verdict: one owner, and it should be ECS

The answer to the title is that there should be exactly one rollback owner, and it should be ECS: circuit breaker on, Rollback: true, on every service. CloudFormation tolerates the ECS-owned rollback perfectly - it waits, fails the resource after the fact, re-syncs quietly and restores the parameters, with no second wave of task replacement in any run we measured. Giving CloudFormation the job instead is slower on both strategies and replaces healthy tasks for no benefit. And running without any detector isn’t a neutral default: on blue/green the failure is invisible, on rolling it churns tasks pointlessly for as long as nobody looks, and either way nothing recovers until a human notices and cancels.

Testing how your deployment machinery behaves before an incident does it for you is part of our DevOps and infrastructure as code work - if you don’t know who owns your rollbacks, we can help you find out.