AWS

A container health check is four lines of task definition, and on most services it looks like paperwork. The load balancer already checks the web tasks. A container that crashes already fails its own deployment. So many people never bother with them, and we often skipped them too, until a release that started, stayed up and did nothing useful was marked successful by every layer. The obvious follow-up question was which of our other services would fail the same way. The answer turned out to be all of them. This post covers what a container health check costs, what it catches on each type of service, and the one place where it does noticeably less than you would expect.

We deployed a container that exits with code 1 five seconds after it starts. Not a subtle failure - the process dies almost immediately, every time. ECS marked the deployment SUCCESSFUL. The service events said “reached a steady state”, twice. CloudFormation took the stack to UPDATE_COMPLETE in 4 minutes 36 seconds, a perfectly ordinary time for this service. Every layer a pipeline could check said the release went out cleanly. Meanwhile the service was down before the first new task launched, and it stayed down, relaunching a dead container roughly every 50 seconds, indefinitely. Nothing was going to notice. This post is the story of that deployment: why it’s in-contract behaviour rather than a bug, and the configuration that closes the gap.

Deployment alarms are the easiest of the ECS failure detectors to describe. You name some CloudWatch alarms on the service, and if one goes into ALARM during a deployment, ECS fails that deployment and rolls it back. We ran that path repeatedly against both a blue/green service and a rolling one, and the mechanism was never the interesting part. What matters is how late the alarm arrives, what a quiet metric does to it afterwards, and the documented behaviour that switches monitoring off at the worst (or best?) possible moment. Every number below comes from a measured run (July 2026, eu-west-1).

This post is about the question that started the whole series: when an ECS deployment fails while CloudFormation is driving, who puts things back - ECS, CloudFormation, or both, one after the other? We had once watched a service roll back twice in a row on another stack, every task replaced once by ECS and then again by CloudFormation, and that fear is why we built the test stack. Nothing in the AWS documentation says what happens to a service resource when ECS rolls a deployment back in the middle of a stack update. So we measured it: deployments where the new tasks never come good, first with no failure detector at all, then with the circuit breaker in both of its modes, on both deployment strategies. Every number below comes from a measured run (July 2026, eu-west-1).

Rolling is the ECS deployment strategy most people start with, and the one everyone assumes they already understand. Old tasks out, new tasks in, two percentages to control the pace. The percentages are documented properly. What the documentation never actually defines is when a rolling deployment is done. Three separate layers each declare completion at their own moment (the deployment record, the service’s own events, and the CloudFormation resource), and on an update they disagree by well over a minute. We measured where each one lands, on two very different rolling services, and this post is what came out of it.