One stack update, three ECS services: who rolls back what

Every failure in this series so far happened to one service at a time. Real stacks don’t deploy like that. A typical application template carries a web service, a worker fleet and a singleton, and one CloudFormation stack update moves all of them together. That opens questions the single-service tests can’t answer. If the web deployment fails, what happens to workers that already finished? If a worker fails while the web service is mid blue/green, who backs out first? So we ran the combinations, and two findings stand out. When one service fails, CloudFormation rolls the healthy ones back too, each with a full backward deployment. And the update where all three services failed at once was the easiest one to recover from.

The test stack, and a note about timings

Same lab as the rest of the series: one CloudFormation stack, three small Fargate services. A web service behind an ALB running native blue/green, with deployment alarms, a circuit breaker and a pause hook providing a smoke test phase. A worker pair running rolling with a circuit breaker. A singleton running rolling with MinimumHealthyPercent: 0 and a breaker at COUNT 2 (its history is in the false-success post). All three carry container health checks, per the health checks post. The first part of the series has the full setup and the container listing. Deployments are triggered by flipping image tag parameters, and the worker and the singleton share a tag, so one update genuinely moves all three services at once. All runs are from July 2026 in eu-west-1.

One note before the results: this post quotes fewer durations than the earlier parts, on purpose. How long a deployment or a rollback takes depends mostly on your container’s size, start-up time and health check settings, so absolute numbers from our tiny test container wouldn’t transfer to your stack. One timeline appears below as a reference for the ordering, and the other timings we quote (how quickly CloudFormation reacts, what its own polling adds) belong to the machinery itself, so those should look much the same whatever you deploy.

A clean combined deployment costs nothing

First, the baseline: both image tags flipped in one update, everything healthy. CloudFormation doesn’t queue the services. It starts all three service updates in parallel, within seconds of the update being issued, and each service then finishes on its own schedule. The worker pair and the singleton completed while the web service was still mid blue/green, waiting at its smoke hook. The web service is the critical path: the combined update finished within one second of a web-only update we’d measured earlier. Deploying everything together added nothing.

The rolling deployments post already showed this pattern on two services, along with the fact that an unchanged service is left completely alone. Both held here, with three services and a blue/green in the mix. Two more details are worth keeping:

• The completion contract survives. CloudFormation marks each service resource complete as that service stabilises, and the stack reaches UPDATE_COMPLETE only when the last one has. A pipeline waiting on the stack is waiting on every service in it, which is exactly what you want the waiter to mean.
• CloudFormation is not the slow part. If you’ve ever watched a multi-service update and thought CloudFormation was late to notice a finished service, it wasn’t. Across all our runs it confirmed a service resource within a minute of ECS declaring that service’s deployment complete. The wait you can see is on the ECS side: the completion check that declares a deployment done runs on its own cadence of roughly two to three minutes. A multi-service update doesn’t make CloudFormation poll any slower.

One service fails: the healthy ones are rolled back too

Now the first failure case: both tags flipped in one update, and the web service’s new revision fails its health checks (the alive-but-sick container from the health checks post). The workers and the singleton are quick and healthy, so they complete first. Then the web service’s breaker trips, and from there the sequence runs like this:

Elapsed time counts from the moment the update was issued, as everywhere in this series. The absolute durations are our lab’s (a tiny container, fast health checks, a one-minute bake) and yours will differ with your application - the ordering and the gaps between the layers are the transferable part.

The count we were watching sits in the two restore rows near the end: each sibling came back once. When we first tested the circuit breaker, the fear (from an older project) was a double restore: ECS rolls a service back, then CloudFormation re-deploys it to the same revision a second time. It didn’t happen on a single service, and it doesn’t happen here either. The failed service gets a re-sync, not a second deployment, and each completed sibling is restored exactly once.

But restored is restored, and it’s a full backward deployment each. The workers rolled back with the usual two-up-then-two-down replacement. The singleton stopped and restarted, its second gap of the update (the first was the forward roll it had just completed). For a failure that was never its own, the singleton went down twice.

That’s the real cost of bundling. A risky change that fails takes every other changed service in the update back with it, healthy or not. Keep genuinely unrelated changes out of the same stack update as a risky one, and if a singleton rides along, remember that the restore is another gap.

Readers of the last two posts may remember the stack finishing its rollback in exactly 8 minutes and 5 seconds, four separate times. That figure was constant only because those rollbacks had no real work in them: one quiet re-sync of the single failed service. Here the rollback contains genuine restore deployments, so it takes as long as they do. The constant was a property of empty rollbacks, not of CloudFormation.

We re-ran the same combined failure with the web revision failing after the traffic shift instead, during the bake (real 500s to real clients while the health checks stay green). The sibling picture didn’t change at all: completed first, restored once each, web re-sync quiet. The news was on the alarm side. The green-side alarm of the per-target-group pair fired, this time on the opposite target group from our earlier test, so the pair recommendation from the alarms post is now proven from both sides: whichever group holds the green tasks, its alarm fires, and the blue one stays silent. The FILLed alarms also cleared themselves five minutes after the bad traffic stopped, before the stack had even finished rolling back, so nothing blocked the next attempt.

The siblings fail: CloudFormation cancels the healthy deployment

Then the reverse experiment: the web service gets a healthy new revision, and the worker and singleton revisions crash on start. To make it harder, we left the web deployment paused at its smoke test hook while the siblings were failing.

The workers tripped their breaker first, despite needing three failures to the singleton’s two, and the head start is structural. The worker pair deploys start-then-stop: MinimumHealthyPercent: 100 keeps the old tasks running while MaximumPercent: 200 launches two replacement tasks straight away, so the workers were collecting crash failures from the first moments of the update, on two slots in parallel. The singleton deploys stop-then-start: with MinimumHealthyPercent: 0, ECS has to stop the old task before it launches the replacement, so the singleton’s first crash landed later, and every failure after that came from the same single slot. Starting sooner and failing in parallel beat the lower threshold. Each service’s ECS rollback then ran on its own, and CloudFormation held back until both had settled.

Then it acted, all at once: within a quarter of a second it failed both sibling resources with the breaker wording and cancelled the in-flight web deployment. Not through some CloudFormation-internal mechanism, either. It called ECS’s own stop-service-deployment, the same API you’d use yourself. The web service’s events say so plainly:

(service web) (deployment ecs-svc/42460709...) deployment failed:
rolling back due to user action (stop-service-deployment).

The “user” here is CloudFormation. If you ever need to know whether a person or the stack cancelled a deployment, the service events won’t tell you. The caller is in CloudTrail.

The paused hook deserves its own entry. It wasn’t answered and it didn’t time out. It was preempted: mid-rollback, the deployment record’s reason read “Service deployment rolled back by user.”, the stage moved backwards, and the hook entry collapsed to a bare {"status": "IN_PROGRESS"} with the hook id, expiry and timeout action all gone. Two consequences. If your pipeline polls hook details while it waits for a smoke result, it has to tolerate that entry losing every field except the status, because a sibling failure can cancel the deployment out from under it. And nothing ever waits for the hook timeout, so a sibling failure doesn’t cost you the pause window’s budget.

One race worried us here. What happens when CloudFormation’s cancel lands on a service that has just tripped its own breaker and is already mid-rollback? The documentation answers it: a stop request against a deployment that is already rolling back “continues the rollback operation”. The cancel merges into the rollback that’s already running. No conflict, no second rollback, and every path ends at the previous revision. We watched it happen live in the next test.

Everything fails at once: the cleanest recovery of the lot

The last planned case: crash-looping workers, a crash-looping singleton and an alive-but-sick web revision, all in one update. We expected the messiest recovery of the series. We got the cleanest.

The whole incident resolved automatically, faster than any of the failure cases above, with no stuck stack and no manual step. The reason is the pattern this series has been circling since the circuit breaker post: every rollback was ECS-owned. All three services rolled themselves back, so CloudFormation’s stack rollback had nothing left to restore. Three quiet re-syncs, done. The expensive failures are the ones where healthy, completed services need real restore deployments. When everything fails, nothing completed, so there’s nothing to restore.

The sequencing is worth knowing for the on-call runbook. CloudFormation rolled the stack back as soon as the first failure settled (the workers again) and cancelled the other two resources within a quarter of a second of that. The singleton’s cancel landed while its own breaker rollback was mid-flight and merged, exactly as the documentation promises. The web service’s cancel landed while it was two counted failures into its own trip.

Recovery was also complete: the moment the stack finished rolling back, a fresh combined deployment went straight through.

A failure after the blue/green flip has finished: the full reversal

Every case above tripped while the web deployment was still in flight. A real template carries far more than three services, though, and something in it can fail long after the web service has completed its flip. CloudFormation then has to reverse a finished blue/green rather than cancel a running one. To force that, we gave the stack a resource that’s guaranteed to fail late, which is a genuinely useful trick for testing rollback behaviour:

 1LateFailureHandle:
 2  Type: AWS::CloudFormation::WaitConditionHandle
 3
 4LateFailure:
 5  Type: AWS::CloudFormation::WaitCondition
 6  Properties:
 7    Handle: !Ref LateFailureHandle
 8    # Nothing ever signals the handle, so the update fails ten minutes in,
 9    # long after the web service has completed its traffic shift
10    Timeout: "600"
11    Count: 1

The web deployment completed normally, production moved to the new revision and the old tasks were gone. Then the WaitCondition failed (“WaitCondition timed out. Received 0 conditions when expecting 1”) and the reversal began.

The answer to the big question: the reverse is a complete blue/green deployment, run backwards. New tasks running the previous revision came up on the alternate target group. The test listener shifted to them. Then the pause hook fired, on the rollback deployment, and sat waiting for an answer, a full smoke test phase with its usual timeout. After we answered CONTINUE, production shifted back, the bake ran, and the now-old tasks were stopped. Zero broken requests throughout, but the choreography is the whole choreography, hook included.

Three consequences.

Your pipeline must answer pause hooks during stack rollbacks, not just during deployments. Unattended, this rollback sits at the hook until the timeout (15 minutes in our template, and the default is 24 hours), and the timeout’s action is itself ROLLBACK, fired at a rollback deployment. We didn’t run that path, and nothing about a rollback of a rollback sounds like behaviour you want to discover in production (most likely scenario is “rollback failed” state). Auto-continuing the hook whenever the active deployment is a CloudFormation-initiated restore is the sensible policy.

The reversal runs at the restored parameter values. CloudFormation puts the old parameters back before it re-updates the services, so if the failed update had also, for example, raised the bake time, the reversal bakes for the old duration, not the new one.

On the ECS side, the reversal is indistinguishable from an ordinary deployment. Its record ends as plain SUCCESSFUL, with no rollback marker at all (the cancelled case at least said “user action”). A service history that reads “deployment succeeded” can mean CloudFormation just put the old version back. Attribution lives only in the stack events and CloudTrail.

One honest gap: our forced failure produced no bad traffic, so we can’t say whether deployment alarms are armed during a reverse deployment. Treat that as unknown, but prepare for the fact that they’re most likely armed and could cause “rollback failed” state.

Who rolls back what

The one-owner verdict from the circuit breaker post survives every combined case: rollbacks belong to ECS, and CloudFormation is the sequencer and the completion waiter above them. In no case we ran did CloudFormation re-deploy a service that ECS had already rolled back.

And the rules we’re taking into our own pipelines:

• Bundle deliberately. Every changed, completed service in a failed update is restored with a full backward deployment, and a singleton pays a gap each time. Keep unrelated changes away from risky ones.
• Diagnose from ECS deployment records, not stack events. The stack names the real failure reason once, at best. The rest read as cancelled bystanders.
• Answer hooks on rollback deployments too. A reversal of a completed blue/green runs the full choreography, smoke test phase included, and it will wait for you.

Deployment machinery that behaves under failure, and not just in the ideal scenario, is part of our DevOps and infrastructure as code work. If your stack updates several services at once and you’ve never watched a combined failure, we can help you run that test before production runs it for you.