The deployment that lied: ECS singletons and false success

We deployed a container that exits with code 1 five seconds after it starts. Not a subtle failure - the process dies almost immediately, every time. ECS marked the deployment SUCCESSFUL. The service events said “reached a steady state”, twice. CloudFormation took the stack to UPDATE_COMPLETE in 4 minutes 36 seconds, a perfectly ordinary time for this service. Every layer a pipeline could check said the release went out cleanly. Meanwhile the service was down before the first new task launched, and it stayed down, relaunching a dead container roughly every 50 seconds, indefinitely. Nothing was going to notice. This post is the story of that deployment: why it’s in-contract behaviour rather than a bug, and the configuration that closes the gap.

The configuration that makes it possible

Everything here was measured on the test stack from the first post in this series: one CloudFormation stack, three small Fargate services running the same test container, whose behaviour is switched by a MODE environment variable (ok, crash, or alive-but-sick). Deployments are stack updates flipping an image tag or mode parameter, watched by curl loops, event streams and deployment record probes.

The service in this story is the singleton: a stand-in for any workload where two copies must never run at once, celery beat being the classic example. A hard singleton forces one exact ECS configuration: DesiredCount: 1 with MaximumPercent: 100 and MinimumHealthyPercent: 0. Max 100 forbids a second task during a deployment. And because the minimum healthy count rounds up, any minimum percentage above 0 would round to one whole task that must keep running while Max 100 forbids its replacement - the scheduler would have nothing it is permitted to move. Min 0 is the only combination that can deploy at all. There is no alternative configuration for this workload.

We covered this configuration’s clean behaviour in the rolling deployments post: a deployment is a stop-then-start with zero overlap, a worst-case gap of 55 seconds, and an outage that begins about ten seconds in, when ECS stops the old task. Acceptable, when it works. This test was the one the configuration had not yet faced: a release that crashes on start, with no failure detector of any kind attached.

The prediction, and what actually happened

We thought we knew the answer. The same crash container on the two-task worker service with no detector hangs the deployment indefinitely - visible, annoying, but honest (the circuit breaker post tells that story). We predicted the singleton would do the same: a stack update that never ends until someone cancels it.

Here is what actually happened (July 2026, eu-west-1). Times are elapsed from the moment the stack update was issued:

No hang. No error. A deployment record with status SUCCESSFUL and a null reason, a stack green in normal time, and a service event stream that read like a healthy roll. At a snapshot just under eight minutes in, eight crash tasks had started and died, with a ninth launching - about 1.2 task launches per minute, around 1,700 Fargate launches a day if nobody notices. Oddly, barely a cost problem: Fargate bills vCPU and memory seconds with a one-minute minimum per task, not launches, so this loop costs about the same as one healthy singleton. The waste is operational - endless image pulls, log noise, and a service doing no work at all.

The “steady state” events fired twice during the deployment window and kept firing throughout the crash loop, right up until we intervened. Neither the running count nor steady-state events are health signals.

Why ECS believed it

It’s important to understand how it works because it’s not an issue with ECS.

With Max 100, the old task is stopped first. From that moment, the deployment’s completion target becomes one new task in the RUNNING state. There is no old capacity and no second task to wait for.

A crashing task spends a surprising amount of time looking alive to the control plane. Each ~50 second cycle: task created, RUNNING about 15 seconds later, process dead 5 seconds after that, then around 25 seconds as a zombie before ECS records the exit. The task is visibly RUNNING for roughly 30 seconds of every 50, dead for most of it.

ECS checks deployment completion periodically, not continuously. Sooner or later a check lands inside one of those windows: running count equals desired count, every task belongs to the new revision, deployment complete. In our run that happened on the third crash cycle, 2 minutes 57 seconds after the deployment record started. The detail that matters: the task the verdict rested on had started 16 seconds earlier and had already been dead for about 11 seconds. ECS evaluated a zombie.

The circuit breaker documentation even states the completion rule in passing: “The deployment has a chance of transitioning to the COMPLETED state because there is more than one task that transitioned to the RUNNING state.” Reaching RUNNING is the bar. Staying alive afterwards is not part of it.

Compare this to the other worker container in our test stack. With two tasks at Min 100 / Max 200, completion needs both new tasks up together, with the old ones scaled down behind them. The crash loop never satisfied that in our runs: ECS never got as far as stopping the old tasks, and the deployment record stayed in its scale-up stage indefinitely. So the deployment hangs where everyone can see it. Same container, same crash, opposite failure mode - and the difference is purely the Min and Max settings.

So the false success is as-documented behaviour for the settings we chose, and the settings are forced for a hard singleton (only one task running ever). That’s exactly what makes it dangerous: you can’t configure your way out of the problem. You can only add detectors to it.

A false success spreads

Before the fix, one more consequence. The circuit breaker’s documented rollback target is “the most recent deployment that is in a COMPLETED state”. After our crash deployment was marked successful, the singleton’s most recent completed deployment was the crash deployment - a breaker attached later would have rolled the next bad release back into the crash loop. A false success doesn’t just hide today’s failure, it becomes tomorrow’s recovery point, until a genuinely clean deployment resets the target.

Recovery itself was easy: this was a test environment where the controlled the behaviour, so we redeployed the working mode straight over the live crash loop. The working task was RUNNING 1 minute 11 seconds after the update was issued, the stack green in 2 minutes 57 seconds, and the loop didn’t interfere at all. Total incident: 22 minutes 32 seconds of downtime, about 18 and a half minutes of it after the false success - bounded only because we were watching. In production, with no detector, it’s broken until someone notices.

Does the circuit breaker save you?

On paper, it shouldn’t. The breaker docs describe two counting stages: stage one counts tasks that fail to reach RUNNING, and stage two counts tasks “replaced because of health check failures”, where the validated health sources are Elastic Load Balancing, Cloud Map, and container health checks. Our crash tasks do reach RUNNING, and the singleton had no health source of any kind. Under a strict reading, the breaker has nothing to count here: structurally blind. But our worker tests had already contradicted that reading - the breaker counted RUNNING-then-exit tasks on an equally check-free service (the circuit breaker post, which also covers the internally inconsistent stage wording). The only way to know was to run it.

We attached a breaker to the singleton with COUNT 2 and rollback enabled. CloudFormation accepted the threshold of 2 - the documented minimum of 3 applies only to the default BOUNDED_PERCENT type. The attach was a quiet configuration-only update with no task replaced, as measured in the rolling post. Then we deployed the crash release again:

The breaker does count these failures. Two dead tasks were enough, the third launch was stopped mid-flight, and ECS rolled the service back to the working revision on its own - total downtime 3 minutes 38 seconds, with no operator action. (The event wording “tasks failed to start” is misleading - the tasks started fine and then died - but the rollback is real.) CloudFormation held the one-owner pattern from the circuit breaker post: it failed the service resource only after ECS had already restored it, re-synced quietly, and finished in the same 8 minutes 5 seconds. No double task replacement.

But look at the margins. Counting from each deployment record’s start (the only clock the runs share - records begin 14 to 16 seconds after the stack update), the unprotected run was marked successful at 2m57s. The breaker tripped at 2m40s. Seventeen seconds. And the completion check fired on crash cycle three of the unprotected run, so its cadence is unknown and possibly variable. Was the trip a race narrowly won? At COUNT 3 the breaker would need one more ~50 second cycle, around 3m30s - past the moment the unprotected deployment had already completed. If completion and the breaker genuinely race, COUNT 3 should lose. So we ran exactly that as a deliberate stress test:

The decisive moment is that third window. At 2m57s after its record started - the precise relative moment the unprotected deployment was marked successful - crash task 3 was midway through a 30-second RUNNING window, running count equal to desired, all tasks new. In the unprotected run, ECS completed 16 seconds into the equivalent window. Here, the whole window passed, nothing completed, and the breaker tripped 29 seconds later. The only variable that changed between the runs was the armed breaker.

Our working model, then: while an armed circuit breaker is accumulating consecutive failures, ECS waits for the completion decision. The race isn’t one.

Closing the door properly: a container health check

The breaker made the crash detectable. It still leaves the deployment’s idea of “healthy” resting on RUNNING, and that leaves one failure mode wide open: the release that stays up but does nothing. A process that is alive and sick satisfies “one new task RUNNING” permanently - no zombie window needed, no failures for the breaker to count. Without a health source, that release is marked successful every single time, and it never recovers.

A container health check gives the task a health signal that ECS itself evaluates: a command run inside the container, with HEALTHY, UNHEALTHY and UNKNOWN states. We added a trivial liveness check to the singleton’s task definition and measured three things.

What it costs: nothing we could measure. On a clean roll with the check in place, every number sat within noise of the baseline, including the same 55-second gap. One detail matched the docs exactly: the first passing check makes the task HEALTHY immediately, and StartPeriod only forgives failures during startup rather than delaying anything.

On the fast crash: passive insurance. Our five-second crasher died before its first health check ever completed, so task health stayed UNKNOWN and the breaker counted container exits exactly as before (trip at 3m23s versus 3m26s without the check - statistically identical). The check didn’t participate in detection. Its value here is quieter: a task that never becomes HEALTHY can’t complete a deployment, so even if the completion-delay behaviour above changed tomorrow, the false success stays closed.

On alive-but-sick: it is the only detector. We deployed the mode that keeps the process up but failing its health endpoint. Without a check, this release false-succeeds every time and stays sick forever. With it, ECS emitted a per-task event each cycle - “(service singleton) (task 718fc935…) failed container health checks” - stopped each unhealthy task cleanly, counted three of them, and tripped the breaker 4 minutes 58 seconds after the record started. Total outage about 5 minutes 27 seconds, fully automatic, and the stack finished at exactly 8 minutes 5 seconds for the third time. The cycles are slower than crash cycles (roughly 90 seconds against 50 - detecting UNHEALTHY takes longer than noticing a dead container), the COUNT 2 argument again.

One wording trap for an on-call person: the trip event still says “tasks failed to start”, now doubly misleading - the tasks started and passed nothing. The per-task “failed container health checks” events are the accurate trail.

Two detectors did not apply here, for completeness. A pause lifecycle hook can’t help: on a rolling service the hook is accepted and then ignored, as of July 2026 (the pause hooks post). And deployment alarms need a metric that breaches, which a headless singleton with no load balancer rarely has within a deployment window.

Where things landed:

What deployment protection cannot do

All of this machinery - breaker, alarms, hooks, health checks - ends the moment the deployment is marked complete. A release that dies after completion is invisible to every deployment-time mechanism and to CloudFormation. Our crasher slipped through a zombie window, but a release that survives ten minutes and then falls over doesn’t need one: it completes the deployment legitimately, and on this configuration the “after completion” period starts almost immediately.

That gap belongs to your pipeline and your monitoring, by design. The check we settled on: after the CloudFormation waiter returns, verify that the new revision’s task has been continuously RUNNING for two minutes with zero container exits since completion. That turns the worst case from unbounded into roughly five to six minutes (the check window plus the 1m11s redeployment we measured). Standing monitoring covers everything later than that. Health checks across the other service types, and how to scope the check command so it doesn’t hurt you, are the subject of the next post: “ECS container health checks: catching alive-but-sick services”.

The configuration a singleton should ship with

This is the configuration we now consider the baseline for any hard singleton:

 1SingletonService:
 2  Type: AWS::ECS::Service
 3  Properties:
 4    ServiceName: singleton
 5    Cluster: !Ref Cluster
 6    LaunchType: FARGATE
 7    DesiredCount: 1
 8    TaskDefinition: !Ref SingletonTaskDef
 9    DeploymentController:
10      Type: ECS
11    DeploymentConfiguration:
12      Strategy: ROLLING
13      MaximumPercent: 100        # a hard singleton, never two at once
14      MinimumHealthyPercent: 0   # forced - any higher value rounds up to one task and blocks the deployment
15      DeploymentCircuitBreaker:
16        Enable: true
17        Rollback: true
18        ThresholdConfiguration:
19          Type: COUNT
20          Value: 2               # each extra count is 50 to 90 seconds of downtime

Plus a liveness check on the container (ours calls the app’s local health endpoint - scope yours to “this process is functioning”, never to its dependencies):

1HealthCheck:
2  Command:
3    - CMD-SHELL
4    - python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:8080/health', timeout=2)" || exit 1
5  Interval: 5
6  Timeout: 2
7  Retries: 2
8  StartPeriod: 15

Plus the two things no template can express: a pipeline step that verifies the new task is still running two minutes after completion, and standing monitoring on the work the singleton actually does (a heartbeat, not just a process).

And a review rule worth stealing: DesiredCount: 1, or any MinimumHealthyPercent: 0, without a circuit breaker and a health check should not pass template review. This configuration false-succeeds, it doesn’t fail - and a deployment that lies is worse than one that breaks loudly, because nobody comes to fix it.

Making deployment pipelines tell the truth is part of our DevOps and infrastructure as code work - if you’re running singletons on ECS and this post made you want to go and check something, we can help.