ECS deployments that fail safely: the configuration we ship

Eight posts ago we built a small test stack to answer one question: how do ECS deployments and CloudFormation actually behave together, underneath what the documentation says? The answer turned out to be mostly reassuring and occasionally alarming, and the alarming parts share a theme. An ECS deployment with default settings doesn’t fail loudly. It fails silently, succeeds falsely, or hangs invisibly. This closing part collects everything in one place: the catalogue of silent failures the series found, the CloudFormation configuration we ship to close them, the pipeline rules the template can’t express, and one last measurement that makes new services cheaper to add than we expected.

The catalogue of silent failures

Every entry below was measured on the series’ test stack (July 2026, eu-west-1), and each links to the part with the full story. They fall into four groups.

Releases that report success

• A service with no health source can’t fail its deployment. With no load balancer, no Cloud Map and no container health check, “started and didn’t exit straight away” is the only thing ECS can check, so a release that boots and does nothing useful deploys “successfully” on any worker fleet or scheduler. The counting rule behind it is in the health checks post.
• A crash-looping singleton is marked successful at every layer. On the one-task MinimumHealthyPercent: 0 configuration, a container that exits five seconds after starting produced SUCCESSFUL on the deployment record, “reached a steady state” in the events, and UPDATE_COMPLETE on the stack, while the service was down indefinitely. The false success also becomes the circuit breaker’s rollback target for the next deployment. The deployment that lied is that whole story.
• A CloudFormation-driven reversal reads as an ordinary success. When a stack rollback reverses a completed blue/green flip, the resulting ECS deployment record ends plain SUCCESSFUL, with no rollback marker. A service history full of successes can include CloudFormation quietly putting old versions back (the multi-service post).

Deployments that hang with nothing to see

• Blue/green with an unhealthy green side and no circuit breaker hangs indefinitely. Users see nothing (production never shifts), the 5xx alarms see nothing (ALB metrics exclude health checks, and no client traffic reaches green), and the only timer on the state is CloudFormation’s 36-hour blue/green limit (the circuit breaker post).
• A rolling crash loop emits no stop events at all. Tasks that exit by themselves produce pure launch spam (“has started 1 tasks”, every half minute, forever), and the evidence that anything is wrong ages out with the stopped-task records (same post).

Protection that switches itself off

• An alarm already in ALARM when the deployment starts disables alarm monitoring for that entire deployment. Documented, deliberate, and completely unreported by any event or record. The fix-forward retry after a failed release is exactly the deployment that runs unprotected (the alarms post).
• Throttling can eat the alarm signal. ECS polls alarms through DescribeAlarms under the shared account quota, and AWS’s own documentation warns the rollback “might not occur” under throttling (same post).
• A pause hook on a rolling service is accepted and silently ignored. No validation error, no pause, no trace: the deployment just rolls straight through (the pause hooks post).
• A too-old CLI or SDK silently drops the hook fields. A deployment can sit paused and waiting while your tooling, built on a client that predates the hook API, reports no hook at all (same post).
• A preempted hook loses its identity. If something else kills a paused deployment (a sibling failure, an alarm), the hook entry collapses to a bare status with the hook id, expiry and timeout action gone (the multi-service post).

Ledgers that mislead

• The breaker’s failure reason reads “tasks failed to start” for every kind of failure, including tasks that started fine and failed health checks for minutes. The deployment record’s statusReason is the accurate one (the circuit breaker post).
• Alarm-triggered rollbacks never name the firing alarm. Diagnosis means checking alarm histories (same post).
• In a multi-service failure, the stack events name the real reason once, at best. Every other failing resource reads “Resource update cancelled”. The ECS deployment records hold the truth (the multi-service post).

None of these is a bug. Every one is documented behaviour or a documented gap, and each is invisible at exactly the moment you’d want it loud. The rest of this post is the configuration that closes them.

The reference configuration

Three service types, three configurations, trimmed below to the deployment safety machinery. The common rule across all three: a circuit breaker with Rollback: true and a container health check on every service, no exceptions, because ECS owns rollbacks better than CloudFormation does (proven for one service in the circuit breaker post and for every combined failure in the multi-service post).

The web service, blue/green behind an ALB, carries the full set:

 1WebService:
 2  Type: AWS::ECS::Service
 3  Properties:
 4    # ... cluster, task definition, networking
 5    DesiredCount: 2
 6    DeploymentController:
 7      Type: ECS
 8    DeploymentConfiguration:
 9      Strategy: BLUE_GREEN
10      # At least five minutes, ten if the deployment can afford it:
11      # the bake exists to give the roughly three-minute alarm
12      # detection latency room to fire (part five)
13      BakeTimeInMinutes: 10
14      MaximumPercent: 200
15      MinimumHealthyPercent: 100
16      Alarms:
17        Enable: true
18        Rollback: true
19        # The per-target-group pair only. The ALB-wide 5xx alarm stays
20        # on the ops dashboard and off the deployment, so another tier
21        # sharing this ALB can't roll our deployment back
22        AlarmNames: [!Ref TgAFiveXxAlarm, !Ref TgBFiveXxAlarm]
23      DeploymentCircuitBreaker:
24        Enable: true
25        Rollback: true
26        ThresholdConfiguration:
27          Type: COUNT
28          Value: 3
29      LifecycleHooks:
30        # The smoke test phase: green serves the test listener while
31        # production stays on blue, until the pipeline answers CONTINUE
32        # or ROLLBACK. The timeout is the abandoned-deployment failsafe
33        - TargetType: PAUSE
34          LifecycleStages: [POST_TEST_TRAFFIC_SHIFT]
35          TimeoutConfiguration:
36            Action: ROLLBACK
37            TimeoutInMinutes: 15
38    LoadBalancers:
39      - ContainerName: web
40        ContainerPort: 8080
41        TargetGroupArn: !Ref TgA
42        AdvancedConfiguration:
43          AlternateTargetGroupArn: !Ref TgB
44          ProductionListenerRule: !Ref ProdRule
45          TestListenerRule: !Ref TestRule
46          RoleArn: !GetAtt EcsBgRole.Arn

The alarm pair’s full definition, FILL(raw, 0) metric math included so the alarms clear themselves after an incident, is in the alarms post. The listener wiring (two target groups, each rule a weighted forward over both) is in the first post.

The worker fleet, rolling with no load balancer, needs only the breaker on top of its health check:

 1WorkerService:
 2  Type: AWS::ECS::Service
 3  Properties:
 4    # ... cluster, task definition, networking
 5    DesiredCount: 2
 6    DeploymentConfiguration:
 7      Strategy: ROLLING
 8      # Two replacement slots, and the old tasks keep working
 9      # until their replacements count as healthy
10      MaximumPercent: 200
11      MinimumHealthyPercent: 100
12      DeploymentCircuitBreaker:
13        Enable: true
14        Rollback: true
15        ThresholdConfiguration:
16          Type: COUNT
17          Value: 3

And the singleton, the configuration that produced the series’ worst finding, is the one place we treat the breaker and the health check as non-negotiable review items:

 1SingletonService:
 2  Type: AWS::ECS::Service
 3  Properties:
 4    # ... cluster, task definition, networking
 5    DesiredCount: 1
 6    DeploymentConfiguration:
 7      Strategy: ROLLING
 8      # Never two copies: the stop-then-start gap this forces
 9      # is the price of a hard singleton (part six)
10      MaximumPercent: 100
11      MinimumHealthyPercent: 0
12      DeploymentCircuitBreaker:
13        Enable: true
14        Rollback: true
15        ThresholdConfiguration:
16          Type: COUNT
17          # One slot accrues failures slowly, so trip earlier
18          Value: 2

Every task definition, on all three, carries a container health check scoped to liveness only: the container’s own work loop, never a dependency round-trip, for the reasons (and with the example command) in the health checks post.

What the template can’t express: the pipeline rules

The template closes the false successes and the invisible hangs. The silent disarms live in the machinery around the deployment, so they’re closed by the pipeline instead:

1. Wait for every deployment alarm to read OK before issuing the stack update. Nothing else covers this case: it’s the only guarantee that a deployment never starts with alarm monitoring silently off.
2. Answer pause hooks during deployments and during stack rollbacks. A rollback of a completed blue/green runs the full choreography, hook included, and will sit at it until the timeout. Auto-continue when the active deployment is a CloudFormation-initiated restore. (Hooks also emit EventBridge events when they pause, so you can subscribe rather than poll.)
3. Let the hook wait survive surprises. The hook entry can lose every field except a bare status when something else kills the deployment, and a continue call sent after a rollback has started fails harmlessly. The smoke-wait loop has to tolerate both, then read the deployment record’s statusReason for what actually happened.
4. Verify singletons after “success”. A completed stack update is not proof a one-task service is alive. Check that a task from the new revision has stayed running for a couple of minutes with no essential-container exits before calling the release done.
5. Pin the CLI and SDK versions your deployment tooling runs on, at or above a version that knows the hook API. A too-old client fails invisibly, not loudly.
6. Diagnose failures from ECS deployment records, not stack events. The stack ledger is for sequencing. Attribution lives in describe-service-deployments and CloudTrail.

Two of these harden into template review rules for us: a DesiredCount: 1 service with MinimumHealthyPercent: 0 and no breaker or health check doesn’t pass review, and a lifecycle hook on a ROLLING service doesn’t either (it would be silently ignored).

One last measurement: creation runs the whole lifecycle

One question none of the failure tests answered: does any of this machinery run when a service is first created, rather than updated? The documentation doesn’t say. So we created the blue/green web service from scratch, hook attached, and watched (July 2026, eu-west-1, a single verification run).

It does. Creation is a full service deployment in the new engine, not a bare scale-up. The initial tasks came up, the test listener pointed at them, and the deployment paused at the hook and waited for an answer, exactly as on an update. A brand-new service gets a smoke test phase on day one.

The failure mode on creation is tidy too, once you know what it looks like. A creation whose tasks never come good ends with the service in ROLLBACK_FAILED and the reason:

Error occurred during operation 'No rollback candidate was found
to run the rollback.'

That reads alarming and is expected: ECS attempted its own rollback, found no previous revision to roll back to (there is none on a first deployment), and recorded exactly that. CloudFormation then rolled the stack back and deleted the service. Nothing to recover manually. Put the wording in your runbook as the normal signature of a failed creation, not an incident.

This is also what makes new services cheap to add. Standing up a new blue/green service, or moving an existing workload onto ECS behind a load balancer, can be a single stack update with a built-in checkpoint: the creation pauses while the new service serves only the test listener, your smoke checks run against it, and only a CONTINUE lets it take production traffic. And because CloudFormation deletes the resources an update removes only in its cleanup step, after every creation in the update has succeeded, whatever the new service replaces is still running if the creation fails.

Where the series ends

We started with “how does ECS native blue/green actually work under CloudFormation” and ended with a short answer and a long appendix. The short answer: the two get along far better than we feared, as long as exactly one of them owns rollbacks, and that one is ECS. The long appendix is everything above: the defaults fail quietly, and the configuration that makes failure loud, automatic and cheap costs a few dozen lines of template and six pipeline rules.

What we didn’t test is on the record too: the LINEAR and CANARY strategies (CloudFormation accepts them, our stack never ran them), deployment alarms during a reverse deployment, and the rollback-of-a-rollback path behind an unanswered hook. If we measure those one day, the series gets another part.

Deployment machinery that behaves under failure, and not just in the ideal scenario, is part of our DevOps and infrastructure as code work. If your services still run on the defaults, the catalogue above is a reasonable worry list, and working through it is exactly the kind of thing we do with clients.