Notes from the work

Eight posts ago we built a small test stack to answer one question: how do ECS deployments and CloudFormation actually behave together, underneath what the documentation says? The answer turned out to be mostly reassuring and occasionally alarming, and the alarming parts share a theme. An ECS deployment with default settings doesn’t fail loudly. It fails silently, succeeds falsely, or hangs invisibly. This closing part collects everything in one place: the catalogue of silent failures the series found, the CloudFormation configuration we ship to close them, the pipeline rules the template can’t express, and one last measurement that makes new services cheaper to add than we expected.

Every failure in this series so far happened to one service at a time. Real stacks don’t deploy like that. A typical application template carries a web service, a worker fleet and a singleton, and one CloudFormation stack update moves all of them together. That opens questions the single-service tests can’t answer. If the web deployment fails, what happens to workers that already finished? If a worker fails while the web service is mid blue/green, who backs out first? So we ran the combinations, and two findings stand out. When one service fails, CloudFormation rolls the healthy ones back too, each with a full backward deployment. And the update where all three services failed at once was the easiest one to recover from.

A container health check is four lines of task definition, and on most services it looks like paperwork. The load balancer already checks the web tasks. A container that crashes already fails its own deployment. So many people never bother with them, and we often skipped them too, until a release that started, stayed up and did nothing useful was marked successful by every layer. The obvious follow-up question was which of our other services would fail the same way. The answer turned out to be all of them. This post covers what a container health check costs, what it catches on each type of service, and the one place where it does noticeably less than you would expect.

We deployed a container that exits with code 1 five seconds after it starts. Not a subtle failure - the process dies almost immediately, every time. ECS marked the deployment SUCCESSFUL. The service events said “reached a steady state”, twice. CloudFormation took the stack to UPDATE_COMPLETE in 4 minutes 36 seconds, a perfectly ordinary time for this service. Every layer a pipeline could check said the release went out cleanly. Meanwhile the service was down before the first new task launched, and it stayed down, relaunching a dead container roughly every 50 seconds, indefinitely. Nothing was going to notice. This post is the story of that deployment: why it’s in-contract behaviour rather than a bug, and the configuration that closes the gap.

Deployment alarms are the easiest of the ECS failure detectors to describe. You name some CloudWatch alarms on the service, and if one goes into ALARM during a deployment, ECS fails that deployment and rolls it back. We ran that path repeatedly against both a blue/green service and a rolling one, and the mechanism was never the interesting part. What matters is how late the alarm arrives, what a quiet metric does to it afterwards, and the documented behaviour that switches monitoring off at the worst (or best?) possible moment. Every number below comes from a measured run (July 2026, eu-west-1).