Notes from the work

This post is about the question that started the whole series: when an ECS deployment fails while CloudFormation is driving, who puts things back - ECS, CloudFormation, or both, one after the other? We had once watched a service roll back twice in a row on another stack, every task replaced once by ECS and then again by CloudFormation, and that fear is why we built the test stack. Nothing in the AWS documentation says what happens to a service resource when ECS rolls a deployment back in the middle of a stack update. So we measured it: deployments where the new tasks never come good, first with no failure detector at all, then with the circuit breaker in both of its modes, on both deployment strategies. Every number below comes from a measured run (July 2026, eu-west-1).

Rolling is the ECS deployment strategy most people start with, and the one everyone assumes they already understand. Old tasks out, new tasks in, two percentages to control the pace. The percentages are documented properly. What the documentation never actually defines is when a rolling deployment is done. Three separate layers each declare completion at their own moment (the deployment record, the service’s own events, and the CloudFormation resource), and on an update they disagree by well over a minute. We measured where each one lands, on two very different rolling services, and this post is what came out of it.

When we measured a full ECS blue/green deployment in the first post of this series, one number stood out: 28 seconds. That’s how long the new revision sits on the test listener before production traffic shifts to it, and it’s the only period in which a smoke test could stop a bad release before users see it. No realistic test suite fits in that time. Pause hooks are the fix. Announced in May 2026, they make the deployment stop at a stage you choose and wait for you, turning those 28 seconds into a smoke test phase of whatever length you need. Third-party coverage of the feature is still close to zero, so we measured the whole workflow on our test stack: the continue path, the rollback path, an alarm firing mid-pause, and one case where the hook is accepted and then does nothing at all. Every number below comes from a measured run (July 2026, eu-west-1).

In July 2025, ECS introduced a native blue/green deployment type comprising two target groups, a test listener, a bake period and managed rollback, all of which are run by ECS itself with no involvement from CodeDeploy. The documentation covers the ideal scenario. However, it says much less about how the feature behaves with CloudFormation on top, such as where UPDATE_COMPLETE lands relative to the bake, how long each stage really takes and whether stack updates end up conflicting with the listener rules that the feature rewrites. We built a small test stack and ran the machinery repeatedly, documenting everything. This post opens a series on how ECS deployment strategies behave when driven by CloudFormation - every number in it comes from a measured run.

We hit a frustrating issue recently while building PHP images in AWS CodeBuild: composer install started failing, often, with an HTTP/2 400 from codeload.github.com while downloading a package archive. The package that failed changed from run to run, and the same Dockerfile and composer.lock that broke in one pipeline built cleanly in another. The short version is that this is GitHub wobbling rather than anything wrong in your project, that the same failure has come and gone for years, and that the usual environment-variable fixes only take the edge off. If your deployments run composer install, here is what is actually going on and how to stop it taking your builds down.