ECS deployment alarms: the fine print

Deployment alarms are the easiest of the ECS failure detectors to describe. You name some CloudWatch alarms on the service, and if one goes into ALARM during a deployment, ECS fails that deployment and rolls it back. We ran that path repeatedly against both a blue/green service and a rolling one, and the mechanism was never the interesting part. What matters is how late the alarm arrives, what a quiet metric does to it afterwards, and the documented behaviour that switches monitoring off at the worst (or best?) possible moment. Every number below comes from a measured run (July 2026, eu-west-1).

The test stack

Everything here was measured on one small CloudFormation stack of three Fargate services running the same tiny test container, described in full in the first post in this series. Two of them matter for alarms: web, two tasks with Strategy: BLUE_GREEN behind an ALB with two target groups (web-a and web-b), and worker, two tasks with Strategy: ROLLING and no load balancer. The container’s error5xx mode fails about half of ordinary requests while its health endpoint keeps answering 200. A curl loop at one request per second against a freshly deployed error5xx revision is what breached the alarms below.

The alarms attach in DeploymentConfiguration, next to the strategy and the bake time:

1DeploymentConfiguration:
2  Strategy: BLUE_GREEN
3  BakeTimeInMinutes: 10
4  Alarms:
5    Enable: true
6    Rollback: true
7    # a static list of alarm names, evaluated for the whole deployment
8    AlarmNames: [!Ref FiveXxAlarm, !Ref TgAFiveXxAlarm, !Ref TgBFiveXxAlarm]

Each of those alarms is HTTPCode_Target_5XX_Count, summed over a 60-second period, one evaluation period, threshold 5.

Detection latency is the number that matters

On paper that alarm should fire about a minute after things go wrong. It doesn’t. Across four alarm-triggered rollbacks, the gap between the first failing request and the alarm reaching ALARM was consistently around three minutes, even though the 5xx rate ran at roughly 50 per minute against a threshold of 5, ten times over from the first minute.

Neither cause is ECS. The ALB publishes its request metrics once a minute, so the data point is already up to a minute old when CloudWatch sees it, and CloudWatch adds evaluation lag on top, worse on a metric that only reports when something is wrong. ECS itself was quick, acting 2 to 46 seconds after the alarm changed state.

Here is one of those runs as a timeline, with a 10-minute bake. Times are elapsed from the stack update:

Two consequences.

1. Your bake time has to cover the alarm latency, or the deployment completes before the alarm can fire: five minutes is the floor, ten is comfortable.
2. Look at the window between 02:01 and 05:35. Users saw the bad release for three and a half minutes, and across the three runs where the alarm fired after the production shift that exposure was 2m59s, 3m15s and 3m33s.

A deployment alarm is damage limitation, not prevention. If you need a release checked before any client touches it, that’s a smoke test phase in front of the production shift, and it’s the second post’s subject.

Why CloudFormation fails the resource only after ECS has already put things back, and what its own rollback then does, is covered in the circuit breaker post.

The sparse-metric trap, and the metric-math fix

HTTPCode_Target_5XX_Count emits nothing at all on a minute with no 5xx. That’s fine while things are broken and awkward once they’re fixed, because an alarm on a metric that has stopped reporting doesn’t clear promptly. In the first run it sat in ALARM for exactly ten minutes, clearing 6m45s after the stack was already back in a state where we could deploy again. TreatMissingData: notBreaching helps less than it may seem, because missing-data transitions are evaluated lazily.

The fix is a metric-math alarm that fills the gaps, with the same name and threshold:

 1FiveXxAlarm:
 2  Type: AWS::CloudWatch::Alarm
 3  Properties:
 4    AlarmName: web-5xx
 5    Metrics:
 6      - Id: raw
 7        ReturnData: false
 8        MetricStat:
 9          Metric:
10            Namespace: AWS/ApplicationELB
11            MetricName: HTTPCode_Target_5XX_Count
12            Dimensions:
13              - Name: LoadBalancer
14                Value: !GetAtt Alb.LoadBalancerFullName
15          Period: 60
16          Stat: Sum
17      - Id: fivexx
18        # quiet minutes become an explicit 0, so the alarm clears on its own quickly
19        Label: target 5xx per minute, gaps filled with 0
20        ReturnData: true
21        Expression: FILL(raw, 0)
22    EvaluationPeriods: 1
23    Threshold: 5
24    ComparisonOperator: GreaterThanOrEqualToThreshold
25    TreatMissingData: notBreaching

Re-running the same failure with that configuration, the alarm spent exactly five minutes in ALARM and returned to OK 37 seconds after the stack became deployable again. Detection latency was unchanged (3m03s against the plain alarm’s 2m52s), so at a sustained failure rate FILL doesn’t mask late-arriving data points the way we worried it might. Freshly created alarms also settle quickly with it, going from INSUFFICIENT_DATA to OK within a minute.

There’s a bonus finding here we could not find documented anywhere. ECS deployment alarms accept metric-math alarms. The alarms page recommends plain metrics and says nothing either way about metric math or composite alarms. It works, and the rollback in that run was triggered by the metric-math alarm. That’s an observation from July 2026 in eu-west-1 rather than a documented guarantee, but it’s important, because everything below depends on it.

An alarm already in ALARM turns monitoring off

This behaviour is documented. It’s one bullet near the end of a long considerations list on the deployment alarms page, and it is easy to read past:

If an alarm is in the ALARM state at the beginning of a deployment, Amazon ECS will not monitor alarms for the duration of that deployment (Amazon ECS ignores the alarm configuration). This behavior addresses the case where you want to start a new deployment to fix an initial deployment failure.

Taken on its own that’s sensible. Nobody wants a fix-forward deployment blocked by the alarm the previous deployment set off. Read it next to the previous section, though, and it gets uncomfortable.

We confirmed it directly. We held the ALB-wide alarm in ALARM with a set-alarm-state loop, then started a clean deployment with nothing wrong with the new image. The alarm went into ALARM 29 seconds before the update was issued, and flipped in and out of ALARM twelve more times while the deployment ran (the FILL expression kept resetting it to OK, so the loop had to keep re-asserting). The deployment ignored all of it. Production shifted at 02:19, the blue tasks stopped at 12:36 after the bake, and the stack reached UPDATE_COMPLETE at 15:19, with all 1,727 witnessed requests answered 200. Nothing told us monitoring was off. No service event mentioned it, and the deployment record came back SUCCESSFUL with reason: null, indistinguishable from a protected deployment that had nothing to report.

The operational consequence is ours rather than the documentation’s. The realistic sequence after an alarm rollback is that a developer fixes the bug and redeploys a few minutes later. With a sparse-metric alarm still stuck in ALARM (ten minutes, in our measurement) that retry runs with alarm protection silently switched off. Nobody decided that, and nothing reports it. So you want both mitigations: FILL, so alarms clear as soon as the underlying problem does, and a pipeline that waits for every one of the service’s deployment alarms to read OK before it issues the update. That check isn’t a nicety, it’s the only thing that guarantees a deployment starts protected.

Blue/green: scoping alarms when the target groups swap

Blue/green makes alarm scoping awkward, because the two target groups swap roles on every deployment (part one covers that swap). A single target-group-scoped alarm is therefore structurally broken, since half of your deployments would be watching the old revision. An ALB-wide alarm does work, but it couples every service behind that ALB, so one tier’s deployment gets rolled back because a different tier started returning 5xx.

The configuration that works is a pair of per-target-group alarms per service, both named in AlarmNames. Ours are web-5xx-a and web-5xx-b, dimensioned on web-a and web-b, both over the same FILL expression. Attaching them was a quiet in-place configuration update: 38 seconds on the service resource, 49 seconds for the whole stack update, no deployment and no task replacement. The failing deployment then behaved exactly as hoped. The green group’s alarm and the ALB-wide alarm fired in the same second, at 05:30 elapsed, and ECS declared the failure two seconds later. The blue group’s alarm never left OK, and both firing alarms cleared together four minutes afterwards.

What you cannot do is scope an alarm to “whichever group is green”. DeploymentAlarms is a static list of names, and ECS never works out which target group is currently carrying the new revision. Even AWS’s own CodeDeploy blue/green examples attach static per-group pairs. The workarounds (rewriting the alarm list on every deployment, or a lifecycle hook running custom checks against green) cost more than the pair does.

Two caveats to write into your runbook:

• The pair isolates the service, not the new revision. Before the production shift the old revision is still serving every production request, so a genuine production incident during a deployment window will roll that deployment back. Defensible behaviour, but on-call needs to know a rollback can be caused by the version you were moving away from.
• ECS never names the alarm that fired. The deployment record says Service deployment rolled back because of one or more active alarms., the service event says deployment failed: alarm detected, and CloudFormation says Error occurred during operation 'ECS Deployment Alarm Detected'. Three layers, no alarm name anywhere. Diagnosis means reading alarm histories.

Rolling deployment: alarms on a service with no load balancer

Deployment alarms have nothing to do with load balancers. Any alarm you can express will do, which matters for services with no ALB metrics of their own. To test that we attached the same 5xx alarm to the rolling worker service and forced it into ALARM by hand mid-roll. Forcing it has a side benefit: because the state change is instant, it separates ECS’s own reaction time from CloudWatch’s latency. Attaching it was another quiet configuration update, 37 seconds on the service resource, no deployment. The failure run then looked like this:

Three things to take from it.

1. Monitoring is active during scale-up: the alarm fired 45 seconds in, before either old task had been stopped, and ECS reacted 14 seconds later. That confirms the documented starting point, stated most clearly not in the developer guide but in an AWS blog post: monitoring starts “as soon as one or more tasks of the updated service … are in a running state”.
2. The rollback ran on the same deployment record rather than creating a new one, and the record’s lifecycleStage (SCALE_UP mid-flight) goes null the moment the rollback starts.
3. How many tasks the recovery relaunches is timing-dependent. Here only the two new tasks were stopped and the old pair was never touched, because the alarm caught the roll while all four were up. An alarm firing after the old tasks are gone has to start replacements for them instead.

The bake time contradiction

The same page contradicts itself about rolling bake times, and both halves are worth seeing. In the considerations list:

The duration when both blue and green service revisions are running simultaneously after the production traffic has shifted. Amazon ECS computes this time period based on the alarm configuration associated with the deployment. You can’t set this value.

And in the “Bake time” section of the same page:

You can configure the bake time for a rolling deployment. When you use CloudWatch alarms to detect failure, if you change the bake time, and then decide you want the Amazon ECS default, you must manually set the bake time.

The CloudFormation reference sides with the second one. It documents BakeTimeInMinutes for rolling deployments (“For rolling deployments, there is no bake time set by default”) and warns of a side effect: “If you provide a bake time for a rolling deployment, the CloudFormation handler timeout is increased to the maximum of 36 hours.”

We did not measure the computed bake duration, because the forced alarm fired before the roll could finish. The practical consequence is clear enough, though. Attaching alarms to a rolling service adds a bake period you did not ask for, derived from the alarms’ period and evaluation periods, and it slows every clean roll afterwards. That’s why we detached the alarm once the test was done. If you do adopt alarms on a rolling service, set BakeTimeInMinutes explicitly so the number is yours rather than inferred.

Alarms during a pause hook

Worth one line, because it surprises people. Alarms stay live while a deployment is paused at a lifecycle hook, and they will end an unanswered pause. In one run the alarm fired 4m30s into a paused deployment and ECS ended it six seconds later, the fastest reaction we measured, and production never moved. The details are in the pause hooks post.

The two strategies side by side

Where that leaves the configuration

The setup we ended up recommending is short. A pair of per-target-group alarms per service on an ALB, both attached, both metric-math alarms over FILL(raw, 0) so they clear on their own. A bake time of at least five minutes, ten if the deployment can afford it. And a pipeline that waits for every deployment alarm to read OK before it issues the stack update.

You also want a clear idea of what alarms don’t cover. They only see failures that reach a client. A new revision that never passes its health checks produces no 5xx at all, because ALB metrics exclude health-check requests and no client traffic ever reaches it. A task that exits on startup gives you nothing to alarm on either. Those need a different detector, either the circuit breaker or a container health check. Two later posts cover the cases where even those aren’t enough: “The deployment that lied: ECS singletons and false success”, and “ECS container health checks: catching alive-but-sick services”.

Getting deployment safety machinery to behave predictably is part of our DevOps and infrastructure as code work - if you’re not sure what your ECS deployments would actually do on a bad release, we can help you find out before your users do.