CloudFormation

Every failure in this series so far happened to one service at a time. Real stacks don’t deploy like that. A typical application template carries a web service, a worker fleet and a singleton, and one CloudFormation stack update moves all of them together. That opens questions the single-service tests can’t answer. If the web deployment fails, what happens to workers that already finished? If a worker fails while the web service is mid blue/green, who backs out first? So we ran the combinations, and two findings stand out. When one service fails, CloudFormation rolls the healthy ones back too, each with a full backward deployment. And the update where all three services failed at once was the easiest one to recover from.

This post is about the question that started the whole series: when an ECS deployment fails while CloudFormation is driving, who puts things back - ECS, CloudFormation, or both, one after the other? We had once watched a service roll back twice in a row on another stack, every task replaced once by ECS and then again by CloudFormation, and that fear is why we built the test stack. Nothing in the AWS documentation says what happens to a service resource when ECS rolls a deployment back in the middle of a stack update. So we measured it: deployments where the new tasks never come good, first with no failure detector at all, then with the circuit breaker in both of its modes, on both deployment strategies. Every number below comes from a measured run (July 2026, eu-west-1).

Rolling is the ECS deployment strategy most people start with, and the one everyone assumes they already understand. Old tasks out, new tasks in, two percentages to control the pace. The percentages are documented properly. What the documentation never actually defines is when a rolling deployment is done. Three separate layers each declare completion at their own moment (the deployment record, the service’s own events, and the CloudFormation resource), and on an update they disagree by well over a minute. We measured where each one lands, on two very different rolling services, and this post is what came out of it.

In July 2025, ECS introduced a native blue/green deployment type comprising two target groups, a test listener, a bake period and managed rollback, all of which are run by ECS itself with no involvement from CodeDeploy. The documentation covers the ideal scenario. However, it says much less about how the feature behaves with CloudFormation on top, such as where UPDATE_COMPLETE lands relative to the bake, how long each stage really takes and whether stack updates end up conflicting with the listener rules that the feature rewrites. We built a small test stack and ran the machinery repeatedly, documenting everything. This post opens a series on how ECS deployment strategies behave when driven by CloudFormation - every number in it comes from a measured run.

Validating an IaC template before you hand it to CloudFormation is one of the cheapest ways to avoid the dreaded UPDATE_ROLLBACK_IN_PROGRESS. You can catch a lot without ever creating a stack - but only if you understand what the tool you use actually checks. Here are the two we usually use, and where one stops and the other takes over.