ECS

We deployed a container that exits with code 1 five seconds after it starts. Not a subtle failure - the process dies almost immediately, every time. ECS marked the deployment SUCCESSFUL. The service events said “reached a steady state”, twice. CloudFormation took the stack to UPDATE_COMPLETE in 4 minutes 36 seconds, a perfectly ordinary time for this service. Every layer a pipeline could check said the release went out cleanly. Meanwhile the service was down before the first new task launched, and it stayed down, relaunching a dead container roughly every 50 seconds, indefinitely. Nothing was going to notice. This post is the story of that deployment: why it’s in-contract behaviour rather than a bug, and the configuration that closes the gap.

Deployment alarms are the easiest of the ECS failure detectors to describe. You name some CloudWatch alarms on the service, and if one goes into ALARM during a deployment, ECS fails that deployment and rolls it back. We ran that path repeatedly against both a blue/green service and a rolling one, and the mechanism was never the interesting part. What matters is how late the alarm arrives, what a quiet metric does to it afterwards, and the documented behaviour that switches monitoring off at the worst (or best?) possible moment. Every number below comes from a measured run (July 2026, eu-west-1).

This post is about the question that started the whole series: when an ECS deployment fails while CloudFormation is driving, who puts things back - ECS, CloudFormation, or both, one after the other? We had once watched a service roll back twice in a row on another stack, every task replaced once by ECS and then again by CloudFormation, and that fear is why we built the test stack. Nothing in the AWS documentation says what happens to a service resource when ECS rolls a deployment back in the middle of a stack update. So we measured it: deployments where the new tasks never come good, first with no failure detector at all, then with the circuit breaker in both of its modes, on both deployment strategies. Every number below comes from a measured run (July 2026, eu-west-1).

Rolling is the ECS deployment strategy most people start with, and the one everyone assumes they already understand. Old tasks out, new tasks in, two percentages to control the pace. The percentages are documented properly. What the documentation never actually defines is when a rolling deployment is done. Three separate layers each declare completion at their own moment (the deployment record, the service’s own events, and the CloudFormation resource), and on an update they disagree by well over a minute. We measured where each one lands, on two very different rolling services, and this post is what came out of it.

When we measured a full ECS blue/green deployment in the first post of this series, one number stood out: 28 seconds. That’s how long the new revision sits on the test listener before production traffic shifts to it, and it’s the only period in which a smoke test could stop a bad release before users see it. No realistic test suite fits in that time. Pause hooks are the fix. Announced in May 2026, they make the deployment stop at a stage you choose and wait for you, turning those 28 seconds into a smoke test phase of whatever length you need. Third-party coverage of the feature is still close to zero, so we measured the whole workflow on our test stack: the continue path, the rollback path, an alarm firing mid-pause, and one case where the hook is accepted and then does nothing at all. Every number below comes from a measured run (July 2026, eu-west-1).