ECS deployments with CloudFormation

We deployed a container that exits with code 1 five seconds after it starts. Not a subtle failure - the process dies almost immediately, every time. ECS marked the deployment SUCCESSFUL. The service events said “reached a steady state”, twice. CloudFormation took the stack to UPDATE_COMPLETE in 4 minutes 36 seconds, a perfectly ordinary time for this service. Every layer a pipeline could check said the release went out cleanly. Meanwhile the service was down before the first new task launched, and it stayed down, relaunching a dead container roughly every 50 seconds, indefinitely. Nothing was going to notice. This post is the story of that deployment: why it’s in-contract behaviour rather than a bug, and the configuration that closes the gap.

A container health check is four lines of task definition, and on most services it looks like paperwork. The load balancer already checks the web tasks. A container that crashes already fails its own deployment. So many people never bother with them, and we often skipped them too, until a release that started, stayed up and did nothing useful was marked successful by every layer. The obvious follow-up question was which of our other services would fail the same way. The answer turned out to be all of them. This post covers what a container health check costs, what it catches on each type of service, and the one place where it does noticeably less than you would expect.