ECS container health checks: catching alive-but-sick services

A container health check is four lines of task definition, and on most services it looks like paperwork. The load balancer already checks the web tasks. A container that crashes already fails its own deployment. So many people never bother with them, and we often skipped them too, until a release that started, stayed up and did nothing useful was marked successful by every layer. The obvious follow-up question was which of our other services would fail the same way. The answer turned out to be all of them. This post covers what a container health check costs, what it catches on each type of service, and the one place where it does noticeably less than you would expect.

The test stack

Everything here was measured on the same small CloudFormation stack as the rest of the series: three Fargate services on one tiny test container whose behaviour is switched by an environment variable (a web service behind an ALB running blue/green, a worker pair running rolling with no load balancer, and a single-task singleton). The first part of the series has the full setup and the container listing. For this post we added a container health check to all three task definitions and re-ran each service’s failure mode with the check present, so that every number below comes from the configuration we would actually ship.

Why a service with no health source can’t fail a deployment

The rule that makes alive-but-sick releases possible isn’t hidden. It’s just in an odd place. The clearest statement AWS publishes about when a task counts during a deployment is on the CloudFormation reference page for DeploymentConfiguration, written in terms of the minimum healthy percent. For services without a load balancer:

If a task has no essential containers with a health check defined, the service scheduler will wait for 40 seconds after a task reaches a RUNNING state before the task is counted towards the minimum healthy percent total.

If a task has one or more essential containers with a health check defined, the service scheduler will wait for the task to reach a healthy status before counting it towards the minimum healthy percent total.

That’s the whole mechanism. With no health source, “the process started and did not immediately exit” is the only thing ECS can check, and 40 seconds later the task counts. A release that boots, opens its socket and then does nothing at all is indistinguishable from a working one.

A load balancer is a health source, and so is Cloud Map. Everything else in a typical stack (queue consumers, schedulers, cron runners, anything internal) has none unless you give it one, and a container health check is the only health source you can add without putting a load balancer in front of the service.

What the check costs: nothing we could measure

Attaching a check is a task definition change, so it rolls the service, which makes the attach its own measurement.

On the worker pair, the roll with the check attached took 3m23s for the service resource. The same roll without a check, measured earlier in the series, also took 3m23s. Identical to the second, with the two-up-then-two-down pattern unchanged and both new tasks reporting healthy.

On the web service, a full blue/green with the check attached, a pause hook answered CONTINUE and a one-minute bake took 6m07s end to end, with 396 and 391 requests across the two listeners and not a single non-200 response. The same setup without the container check, measured in the pause hook tests, also took 6m07s. There was no cross-talk between the container check, the ALB target health, the deployment alarms and the hook.

The zero cost surprised us less once we watched the StartPeriod behave. The grace period only forgives failures, it doesn’t delay success: the documentation says that “if a health check succeeds within the startPeriod, then the container is considered healthy”, and with a 5-second interval our tasks went healthy seconds after the container started. Completion waited for the health status, not for the 15-second grace window to run out. On the web service the task had to satisfy both layers, its own check and the target group, which is also what the CloudFormation page says happens.

What they catch, by type of service

The failure mode that matters here is sick-but-alive: our test container’s health endpoint returns 500 while everything else about the process looks normal. Each type of service behaves differently, so each gets its own run. Elapsed time in the timelines is counted from the moment the stack update was issued. ECS’s own detection clock starts when the deployment record opens, 13 to 16 seconds later, and the trip figures we quote for comparison between runs are measured from there.

A worker fleet with no load balancer

Two tasks, rolling, MaximumPercent: 200 and MinimumHealthyPercent: 100 (the rolling deployments post covers what those numbers do), circuit breaker at COUNT 3 with rollback, container check attached. Then we deployed the sick revision (July 2026, eu-west-1):

The breaker tripped 2m49s after the record opened. Each sick task produced its own service event naming the task and the cause:

(service worker) (task 428502f2...) failed container health checks.
(service worker) has stopped 1 running tasks: (task 428502f2...).

The stops were graceful (exit 143), and each stopped task’s record carries the reason Task failed container health checks. It’s the only failure signature in this series that names both the individual task and the reason it was stopped.

The important part is what it cost. Nothing. The two old tasks were never stopped, because MinimumHealthyPercent: 100 held throughout, so a queue consumer would have kept consuming through the entire failed deployment. The recovery replaced no tasks at all, because the old pair already satisfied the desired count. The incident cost 6m44s of a stack being busy and zero seconds of service.

One caveat about the counterfactual: we didn’t re-run this without the check, so we can’t quote a measured false success on this exact configuration. The counting rule above says a task with no health source counts 40 seconds after it starts, and we did measure that same sick release completing a deployment successfully on a single-task service. A worker fleet with no health source will deploy this release “successfully” today.

A web service behind a load balancer

Same sick release, this time on the blue/green web service, where the ALB target group check and the container check can both see the problem. This is where our first reading of the results was wrong, so here is the corrected version.

The container check does act first. Forty seconds after a green task started, ECS launched a replacement for it: Amazon ECS replaced 1 tasks due to an unhealthy status. That’s documented scheduler behaviour, and the CloudFormation page describes it plainly, saying the scheduler “replace[s] unhealthy tasks by starting replacement tasks first and then stopping the unhealthy tasks”. But the stops are what the breaker counts, and every stop it counted carried the load balancer’s attribution instead:

(service web) (task 6f052ef7...) (port 8080) is unhealthy in (target-group web-a)
due to (reason Health checks failed with these codes: [500]).

Those stops landed 80 to 160 seconds after each task started, and the breaker tripped 3m56s after the record opened. The same failure with no container check, measured a day earlier, tripped at 3m55s. One second apart. So on an ALB-attached service, the container check does not speed up the breaker. It adds replacement launches on the green side and nothing else that the deployment machinery reacts to.

No user saw any of it. All 466 requests on the production listener and all 462 on the test listener returned 200 from the old revision, because the whole failure happened inside the scale-up stage. The test traffic shift never ran, the pause hook and the deployment alarms never engaged, and the blue tasks were never touched. CloudFormation then did its usual quiet re-sync before finishing the stack rollback (the circuit breaker post covers who owns that rollback).

So why put a check on a load-balanced service at all? Two reasons, and speed isn’t one of them. The first is consistency: one rule for every service reviews better than “checks everywhere except where an ALB happens to exist”. The second is sidecars. An ALB checks one port on one container, so a task with an asset server, a metrics exporter or a log shipper alongside the application has containers nothing is watching, and a per-container check is the only detector for a sick sidecar in an otherwise healthy task.

The documented truth tables hold a trap for that second reason. Task health comes from the essential containers that have a check defined, and if one essential container is UNKNOWN then the whole task is UNKNOWN. So a task where the application container has a check and an essential sidecar doesn’t will never report healthy at all. If you add checks to a multi-container task, add them to every essential container in it. We didn’t test this, but the health check documentation is explicit about it.

A container that crashes on start

The last case isn’t a type of service but a type of failure, and it’s the one where the check does the least visible work. We deployed a revision that exits five seconds after starting, on the worker pair, with the check attached. The breaker tripped 1m57s after the record opened, against 2m02s for the same test without a check. The tasks died before their first health check completed, so their health status stayed UNKNOWN throughout and the breaker counted plain container exits, exactly as it had before.

The check is still doing something here, quietly. A task that never reports healthy can never be counted towards the deployment, so a crash-looping release can’t complete one. What happens to a permanently UNKNOWN task is not something the documentation addresses, and this is the empirical answer (July 2026, eu-west-1): it never counts, and the deployment either fails through another detector or waits forever.

Scoping the command is the part that can hurt you

Everything above is an argument for putting a check on every service. The argument against comes entirely from what you write in the command.

Check liveness only. Never a dependency round-trip. A check that asks your broker or your database whether it’s reachable turns a transient dependency blip into simultaneous unhealthy status across every task in the fleet. ECS then does what it’s documented to do and replaces them all, and the replacement wave arrives in the middle of the dependency outage. That failure mode is worse than the one you were trying to catch.

What works instead is a check that asks one question: is this container’s own work loop still turning?

• A scheduler or cron-style process: have the tick touch a heartbeat file, and check the file’s age.
• A queue consumer: same pattern, driven by the worker’s own event loop or heartbeat signal.
• A web application: a local HTTP call to your own health endpoint, in addition to the ALB’s check, so the sidecars are covered too.

Our test stack uses the local-HTTP version, and the interval is short because short runs made the tests quicker:

 1WorkerTaskDef:
 2  Type: AWS::ECS::TaskDefinition
 3  Properties:
 4    ContainerDefinitions:
 5      - Name: worker
 6        Essential: true
 7        # Liveness only. No broker, no database, nothing over the network:
 8        # a dependency blip must not make every task in the fleet unhealthy at once.
 9        HealthCheck:
10          Command:
11            - CMD-SHELL
12            - python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:8080/health', timeout=2)" || exit 1
13          Interval: 5
14          Timeout: 2
15          Retries: 2
16          StartPeriod: 15

For a real service, be more generous: an interval in the tens of seconds, Retries of at least 2 so a single slow answer doesn’t stop a task, and a StartPeriod that comfortably covers your slowest cold start. Then add one line to the runbook: a lot of tasks going unhealthy at once means look at the dependencies, not at the tasks.

The verdict

Trip times are measured from the moment the deployment record opened. The last row is the only one we didn’t re-run, because we had already measured it on a single-task service and the counting rule says the same thing about any task with no health source.

So, put a check on everything, scope it to liveness, and don’t expect it to speed up the circuit breaker on a load-balanced service. They cost nothing measurable, they turn a vague failure into an event that names the task and the reason, and for the alive-but-sick release they’re the only thing standing between you and a deployment that reports success while the service does no work at all.

Getting the deployment safety machinery right, and knowing which parts of it are load bearing, is part of our DevOps and infrastructure as code work. If you aren’t sure what your pipeline would do with a release like this, we can help you find out.