Blog on Mysterious Code - Senior AWS, DevOps & security engineering

The deployment that lied: ECS singletons and false success

Mon, 10 Aug 2026 08:00:00 +0000

We deployed a container that exits with code 1 five seconds after it starts. Not a subtle failure - the process dies almost immediately, every time. ECS marked the deployment SUCCESSFUL. The service events said “reached a steady state”, twice. CloudFormation took the stack to UPDATE_COMPLETE in 4 minutes 36 seconds, a perfectly ordinary time for this service. Every layer a pipeline could check said the release went out cleanly. Meanwhile the service was down before the first new task launched, and it stayed down, relaunching a dead container roughly every 50 seconds, indefinitely. Nothing was going to notice. This post is the story of that deployment: why it’s in-contract behaviour rather than a bug, and the configuration that closes the gap.

ECS deployment alarms: the fine print

Fri, 07 Aug 2026 08:00:00 +0000

Deployment alarms are the easiest of the ECS failure detectors to describe. You name some CloudWatch alarms on the service, and if one goes into ALARM during a deployment, ECS fails that deployment and rolls it back. We ran that path repeatedly against both a blue/green service and a rolling one, and the mechanism was never the interesting part. What matters is how late the alarm arrives, what a quiet metric does to it afterwards, and the documented behaviour that switches monitoring off at the worst (or best?) possible moment. Every number below comes from a measured run (July 2026, eu-west-1).

ECS circuit breaker and CloudFormation: who owns the rollback

Wed, 05 Aug 2026 09:00:00 +0000

This post is about the question that started the whole series: when an ECS deployment fails while CloudFormation is driving, who puts things back - ECS, CloudFormation, or both, one after the other? We had once watched a service roll back twice in a row on another stack, every task replaced once by ECS and then again by CloudFormation, and that fear is why we built the test stack. Nothing in the AWS documentation says what happens to a service resource when ECS rolls a deployment back in the middle of a stack update. So we measured it: deployments where the new tasks never come good, first with no failure detector at all, then with the circuit breaker in both of its modes, on both deployment strategies. Every number below comes from a measured run (July 2026, eu-west-1).

ECS rolling deployments: what "done" actually means

Mon, 03 Aug 2026 08:00:00 +0000

Rolling is the ECS deployment strategy most people start with, and the one everyone assumes they already understand. Old tasks out, new tasks in, two percentages to control the pace. The percentages are documented properly. What the documentation never actually defines is when a rolling deployment is done. Three separate layers each declare completion at their own moment (the deployment record, the service’s own events, and the CloudFormation resource), and on an update they disagree by well over a minute. We measured where each one lands, on two very different rolling services, and this post is what came out of it.

ECS pause hooks: a real smoke test phase, and where they do nothing

Fri, 31 Jul 2026 09:00:00 +0000

When we measured a full ECS blue/green deployment in the first post of this series, one number stood out: 28 seconds. That’s how long the new revision sits on the test listener before production traffic shifts to it, and it’s the only period in which a smoke test could stop a bad release before users see it. No realistic test suite fits in that time. Pause hooks are the fix. Announced in May 2026, they make the deployment stop at a stage you choose and wait for you, turning those 28 seconds into a smoke test phase of whatever length you need. Third-party coverage of the feature is still close to zero, so we measured the whole workflow on our test stack: the continue path, the rollback path, an alarm firing mid-pause, and one case where the hook is accepted and then does nothing at all. Every number below comes from a measured run (July 2026, eu-west-1).

ECS native blue/green deployments with CloudFormation: how it actually works

Mon, 27 Jul 2026 08:00:00 +0000

In July 2025, ECS introduced a native blue/green deployment type comprising two target groups, a test listener, a bake period and managed rollback, all of which are run by ECS itself with no involvement from CodeDeploy. The documentation covers the ideal scenario. However, it says much less about how the feature behaves with CloudFormation on top, such as where UPDATE_COMPLETE lands relative to the bake, how long each stage really takes and whether stack updates end up conflicting with the listener rules that the feature rewrites. We built a small test stack and ran the machinery repeatedly, documenting everything. This post opens a series on how ECS deployment strategies behave when driven by CloudFormation - every number in it comes from a measured run.

Composer's HTTP/2 400 failures from codeload.github.com in CI

Mon, 29 Jun 2026 09:00:00 +0000

We hit a frustrating issue recently while building PHP images in AWS CodeBuild: composer install started failing, often, with an HTTP/2 400 from codeload.github.com while downloading a package archive. The package that failed changed from run to run, and the same Dockerfile and composer.lock that broke in one pipeline built cleanly in another. The short version is that this is GitHub wobbling rather than anything wrong in your project, that the same failure has come and gone for years, and that the usual environment-variable fixes only take the edge off. If your deployments run composer install, here is what is actually going on and how to stop it taking your builds down.

How to validate a CloudFormation template (CLI and cfn-lint)

Sun, 28 Jun 2026 09:00:00 +0000

Validating an IaC template before you hand it to CloudFormation is one of the cheapest ways to avoid the dreaded UPDATE_ROLLBACK_IN_PROGRESS. You can catch a lot without ever creating a stack - but only if you understand what the tool you use actually checks. Here are the two we usually use, and where one stops and the other takes over.

Multiplatform docker images for cfn-lint (and a v1.52 schema fix)

Sat, 27 Jun 2026 09:00:00 +0000

A while back we wrote about our automatically updated docker image for cfn-lint - a public, daily-rebuilt image for cfn-lint that fills the gap left by the lack of an official one. Two changes have just landed in that build, and both are worth a few words: the images are now multiplatform, and we have fixed a problem that, from cfn-lint v1.52.0 onwards, left the image unable to recognise any resources.

Mysterious Code at Cloud & AI Infrastructure - meet us at Tech Show London 2025

Tue, 18 Feb 2025 11:00:00 +0000

We’re excited to announce that Mysterious Code will be part of the Cloud & AI Infrastructure event at Tech Show London 2025!

Mysterious Code Ltd is now part of AWS Public Sector Program

Fri, 19 Apr 2024 15:00:00 +0000

As a member of the AWS Public Sector Partner Program, Mysterious Code Ltd will leverage its expertise to empower government, education and nonprofit entities with secure, scalable, and compliant cloud solutions built on AWS infrastructure. By combining AWS’s industry-leading cloud platform with Mysterious Code’s specialized knowledge and experience, public sector customers can expect enhanced capabilities in areas such as data security, compliance, and cost optimization.

Automatically updated docker image for cfn-lint

Thu, 18 Apr 2024 15:00:00 +0000

If you’re using CloudFormation, you probably know about cfn-lint - a linting tool created by the CloudFormation team to validate templates against the schema and best practices. Validating each template before deployment is in itself actually considered a best practice by AWS. However, simply using validate-template in the Console or CLI only validates the basic syntax of the template, not the actual contents and resource specification. That’s where using a linter like cfn-lint can be helpful to make sure you’re not making any obvious mistakes or going against best practices in your resources.

You can use cfn-lint in a number of ways during development, including simply within command-line, using git pre-commit hooks or as a plugin to your IDE. All those options, while helpful in day-to-day work, do not establish code quality standards for your overall codebase. To do that, it’s ideal to include linting as part of CI/CD pipeline and/or pull/merge-requests approval process.

That is where you can come across a hurdle: cfn-lint does not have an official, up-to-date docker image

Deploying AWS GuardDuty with CloudFormation for Master and Member accounts

Tue, 01 May 2018 15:21:00 +0000

AWS GuardDuty analyses various events happening on your AWS account and can notify you when suspicious activity takes place. Right now, GuardDuty is specific to a region and needs to be enabled in each region you want to monitor (though AWS recommends you enable it in all regions to ensure global actions are monitored). Going through GuardDuty console in every AWS region can be a daunting task, and quite time consuming if you have multiple AWS accounts which you’d like to connect into Master-Member setup. Luckily, CloudFormation supports enabling and setting up GuardDuty detectors, so you can use it to make it a little bit less painful.

Don't panic! False-positives from GuardDuty and Network Load Balancer (NLB)

Tue, 20 Mar 2018 09:00:12 +0000

If you’re keeping in touch with new services provided by AWS, you probably heard about new security monitoring tool: GuardDuty. You probably also noticed a whole new family of Elastic Load Balancers (v2), which includes Network Load Balancers (NLB). Deploying those two new services may generate some unexpected results - and here’s why.

Complete code: automated EC2 snapshots and retention management

Sun, 11 Mar 2018 23:22:07 +0000

Creating snapshots from EBS drives attached to your EC2 instances is the most basic way of backing up your data. While you have to be cautious when snapshotting running EC2 instances without restart, doing it regularly is a base of many disaster recovery plans. In the latest update to aws-maintenance repo on GitHub you’ll find a complete code and CloudFormation template that will make this as painless a process as possible.

Cross-region RDS recovery: encryption and Aurora support

Sat, 03 Feb 2018 00:45:22 +0000

After my previous post about a complete code for automated RDS cross-region backup copy, some issues and new feature requests have been raised on GitHub (thanks for that!) - and with your help, support for encrypted RDS instances and Aurora Clusters have now been added!

Complete code: cross-region RDS recovery

Thu, 28 Dec 2017 00:12:46 +0000

After posting the previous post on this topic (Copying RDS snapshot to another region for cross-region recovery) , I noticed a lot of people being interested in using the code I provided as an example. Many were not sure how to make use of it, and after a couple of pull requests it became obvious that a complete, fully-working code and CloudFormation template would be a good idea. So, yesterday, I pushed an update to aws-maintenance repository with a fully working code, which you can easily customize via CloudFormation parameters to match your needs.

Sharing encrypted AMIs between AWS accounts (using Python and boto3)

Thu, 02 Nov 2017 18:00:08 +0000

Each Amazon Machine Image (AMI) holds information of the volumes and snapshots of those volumes that should be attached to instances created from that AMI. To protect the data on those snapshots, you can choose to encrypt them using KMS. Encrypting your data at rest is generally a good idea, though many companies choose to avoid encrypting their snapshots, because sharing such snapshots between different AWS accounts (for example, Test and Production accounts) can be difficult. That’s why I’m sharing details on how to make this as easy as possible (and automated!).

Enabling global API Gateway stage logging using CloudFormation

Tue, 18 Jul 2017 09:42:49 +0000

Enabling logging in API Gateway for your stage is fairly easy. You go into the Console, setup a role for API Gateway to use for logging, find the stage and enable logs. It will enable logging for all methods within that stage. Doing the same configuration using CloudFormation is not completely obvious though, as the stage object’s MethodSettings property seems to allow you to only do that for a specific resource and method.

Using Magical Dictionaries to manage long CloudFormation templates with Troposphere

Wed, 05 Jul 2017 15:09:03 +0000

When deploying infrastructure with CloudFormation, at some point you will reach a moment when your CloudFormation JSON or YAML file is just too big. It will be too long to get a good overview of what’s in it, manage parameters and all dependencies between resources within the template. Nested stacks may be a solution, but if sometimes you can’t/won' t/don’t use them for whatever reason (for example, it will get to complicated to manage tested stacks or their contents are not reusable between other stacks).

Even if you’re using troposphere to generate your templates, you’ll face the same issue - a very long Python file. Luckily, if you are using troposphere, you’re inherently using Python - which means you can take advantage of it.

Want to get in touch? Meet us at events this month!

Fri, 09 Jun 2017 10:00:30 +0000

Fancy getting in touch with us? Have a question or want to know a bit more about our work? June is the month of conferences in London, and we will be attending them! Here are the details where and when to meet us:

Controlling access to AWS ElasticSearch Service clusters with IAM

Mon, 05 Jun 2017 10:00:52 +0000

When using AWS ElasticSearch Service, you have multiple ways of controlling access to your cluster. AWS Console gives you some pre-set policies you can use, but the ways of access can be confusing. Using the IP-based access, you can allow access from your EC2 instances. But what if you want to access your cluster from a Lambda?

S3 bucket permissions: what does "everyone" mean?

Wed, 10 May 2017 20:47:05 +0000

On more than one occasion I have seen S3 bucket policies set for the predefined users groups: “Everyone” and “Any authenticated AWS user”, but rarely has it been done with understanding of what those groups actually mean. So, if you’ve ever set (or thought of setting) permissions for those, please read on.

Intrusion detection and prevention with AWS Lambda and DynamoDB streams

Wed, 05 Apr 2017 10:00:07 +0000

Intrusion detection system (IDS) and intrusion prevention system (IPS) tend to be expensive and complicated. In AWS, you can go for much simpler solution - WAF. But that requires you to use Application Load Balancer or CloudFront. But even with WAF, you have to manage a list IP addresses of attackers that should be blocked. Or, if you only ever need to block single IPs for short periods of time, NACLs may be a much easier option! Here’s a walkthrough on how you can implement a terribly simple (yet very powerful) intrusion detection and prevention in AWS with Lambda and DynamoDB Streams for a web application.

AWS Certified Solutions Architect vs DevOps exams - my subjective comparison

Mon, 27 Mar 2017 10:00:55 +0000

Last week, I finally had the time to schedule my AWS Certified Solutions Architect - Professional exam, which I passed on Saturday. It’s been a while since I did the other AWS Professional level (DevOps) certificate, but I thought I’d share my take on how the exams compare.

Bug in KMS: encryption key that cannot be deleted

Mon, 06 Mar 2017 10:00:24 +0000

Recently, I noticed a weird KMS key on an AWS account - what was weird about it, was the fact that it wasn’t marked as AWS-managed key, but no-one (not even root) could delete or modify the key!

Creating Elasticsearch Domain did not stabilize - can't create AWS Elasticsearch 5.1 with CloudFormation

Fri, 17 Feb 2017 12:19:59 +0000

Recently AWS announced support for Elasticsearch 5.1 in their Elasticsearch Service. Today, I tried to upgrade an existing CloudFormation stack, previously using Elasticsearch 2.3, to the new version and, after a very long wait, CloudFormation rolled back the stack with the following error: “Creating Elasticsearch Domain did not stabilize”. Here’s what I did to solve it.

How to tag Redshift clusters (and others) in one place in AWS Console

Fri, 03 Feb 2017 10:00:18 +0000

Management page of Redshift in AWS Console does not currently allow you to add or modify tags on your clusters. That doesn’t mean that Redshift resources cannot be tagged - almost everything from clusters to parameters groups can be tagged through the Tag Editor.

Bypassing CloudTrail and CloudWatch monitoring of AWS API

Mon, 23 Jan 2017 21:13:14 +0000

Since AWS released CloudWatch Logs and allowed for automated streaming of CloudTrail events into CloudWatch, monitoring API usage with CloudWatch metrics and alerts has become increasingly popular. It’s incredibly easy to set up and add custom alerts on almost any API event. But is it actually going to notify you when an unauthorized person gains access?

Adding more nodes to Redshift cluster doesn't always mean you get more space

Fri, 13 Jan 2017 17:33:03 +0000

AWS Redshift is a big data storage (“data warehousing”) solution for analytics. Based on PostgreSQL 8, it can combine up to 128 largest nodes, giving you 2 petabytes for your data. Well, almost. You don’t get ALL that for your data.

AWS S3 storage classes pricing is not what you think

Mon, 02 Jan 2017 10:00:45 +0000

Since publishing this post, AWS reached out to us about the RR pricing. See bottom of the post for the update.

When storing data in S3, at some point everyone asks the question about storage classes. At first glance it looks more or less simple - if you’re accessing your data less often, or it’s not very important, you can pay less. Until you look into details, find all the asterisks and compare the actual numbers.

First thoughts after AWS Certified Security - Specialty BETA exam

Mon, 19 Dec 2016 13:31:58 +0000

Last week I made an attempt at passing one of the new AWS Specialty exams - Security.

DHCP implementation in... PHP

Mon, 12 Dec 2016 09:00:29 +0000

PHP is not usually used to implement network services - and generally for good reasons. Not because it can’t be done, but rather because it’s not what it was meant to be used for. Although, if you think that PHP simply can’t be used for anything other than serving your blog, think again. Because my DHCP implementation in PHP seems to work quite well!

Using PGP keys from YubiKey as SSH RSA keys on MacOS

Mon, 05 Dec 2016 10:00:14 +0000

Recently, I became a fan of YubiKeys - hardware encryptions keys used for storage of PGP keys, FIDO U2F (two-factor authentication) and more. The PGP keys stored on the device can also be used as RSA keys for SSH authentication. The setup is quite simple on MacOS.

Deploying API Gateway and Lambda with CloudFormation

Mon, 21 Nov 2016 10:00:32 +0000

Combination of AWS API Gateway and Lambda functions is a flag example of every “serverless infrastructure”. When deploying API Gateway with CloudFormation there are two different ways you can define your API: via Swagger template or by directly defining your methods in CloudFormation template.

Invalid permissions on Lambda function: fixing API Gateway in CloudFormation

Mon, 14 Nov 2016 19:22:45 +0000

AWS Lambda and API Gateway are becoming synonymous with “serverless infrastructure” and getting more and more popular. To deploy them in repeatable way, one of the tools I recommend is CloudFormation - see deploying API Gateway and Lambda with CloudFormation for the basic setup. There are many ways you can define your API and your Lambda, but when connecting the two with CloudFormation there’s usually something that many people miss, and only notice when {"message": "Internal server error"} is thrown from their API Gateway endpoint.

Packer, Ansible, CentOS and requiretty

Fri, 11 Nov 2016 10:00:42 +0000

If you’ve tried running Packer with CentOS on AWS, you probably noticed this:

 sudo: sorry, you must have a tty to run sudo

How to deploy Wordpress in AWS

Fri, 04 Nov 2016 10:00:02 +0000

I’m not talking about launching an EC2 instance, uploading the zip with WordPress and going through the installer. I’m talking about immutable infrastructure, scalable, self-healing setup of WordPress within AWS.

Copying RDS snapshot to another region for cross-region recovery

Fri, 21 Oct 2016 10:00:17 +0000

For an updated ready-to-use CloudFormation template of this code, see newer post: Complete code: cross-region RDS recovery .

Amazon RDS is a great database-as-a-service, which takes care of almost all database-related maintenance tasks for you - everything from automated backups and patching to replication and fail-overs into another availability zones.

Unfortunately all of this fails if the region where your RDS is hosted fails. Region-wide failures are very rare, but they do happen! RDS does not support cross-region replication at the moment, so you cannot simply create a replica of your database in another region (unless you host the database on an EC2 instance and set up the replication yourself). The second-best option, to make sure you can restore your service quickly in another region, is to always have a copy of your latest database backup in that region. In case of RDS, that can mean copying automated snapshots. There is no option for AWS to do it automatically, but it can be easily scripted with AWS Lambda functions.

How I passed Red Hat Certificate of Expertise in Ansible Automation exam (EX407)

Fri, 14 Oct 2016 10:00:38 +0000

I have recently passed Red Hat’s EX407 (Red Hat Certificate of Expertise in Ansible Automation exam) and I wanted to share my experience for anyone else who might be looking into getting that certificate. At the time of my exam, I was only a second (publicly visible) person in the UK who had that certificate. Probably because it was only made available in summer this year.

Deployment pipeline, part 3: deployment onto an environment

Fri, 07 Oct 2016 10:00:01 +0000

The purpose of every deployment pipeline is… a deployment. So this final part of the series, will focus on just that. If you missed the intro, check out the video where I describe a typical pipeline here. The other parts of this series are part 1: test phase and part 2: build phase.

Once we create an AMI that we’d like to deploy, performing a rolling update on existing instances is fairly easy. Usage of Auto scaling groups and CloudFormation makes it even easier - since AWS Auto scaling groups support the rolling updates out of the box.

Or, if your applications requires blue-green deployments, using CloudFormation is almost essential.

Deployment pipeline, part 2: build phase

Mon, 03 Oct 2016 09:00:58 +0000

Welcome to the second part of my series on deployment pipelines. If you missed the intro, check out the video where I describe a typical pipeline here. The other parts of this series are part 1: test phase and part 3: deployment onto an environment.

After the test phase of the pipeline, once the quality of the code has been checked, we must build a deployable artefact for this version (commit) of the code. In case of AWS, this would be an AMI (Amazon Machine Image), which can then be deployed as a new instance in our environment.

Deployment pipeline, part 1: test phase

Fri, 30 Sep 2016 10:00:10 +0000

Welcome to the first part of my series on deployment pipelines. If you missed the intro, check out the video where I describe a typical pipeline here.

First step of a deployment pipeline, is usually a series of tests. After a commit is made, the code is checked out from the source code repository and tested. Those are usually code style tests and unit tests.

ELB ProxyProtocol with CloudFormation

Mon, 26 Sep 2016 10:12:42 +0000

When using AWS Elastic Load Balancer with TCP listeners (not HTTP or HTTPS), the biggest problem faced by many people is the lack of client’s IP address. Since it’s TCP which works on a lower layer, the ELB does not add the X-Forwaded-For header (like it does for HTTP and HTTPS). For some time, this meant that if you used those listeners, you had no way of getting the original client’s IP address.

Typical CI deployment pipeline - overview

Fri, 23 Sep 2016 00:02:37 +0000

More details on each deployment phase with examples, will be posted at our blog over the coming weeks, so check back soon.

Using Troposphere to create CloudFormation stack template

Fri, 09 Sep 2016 15:12:49 +0000

If you’ve ever wrote AWS CloudFormation template, you probably know that it can be a daunting task. Luckily, it can be much easier, if you use Python’s library “Troposphere”.

Troposphere lets you create Python objects in place of CloudFormation elements, does some basic validation of your input and generates the JSON template for CloudFormation for you. It is much easier and cleaner to use that writing JSON templates manually.

2a0++ misunderstanding

Fri, 08 Nov 2013 11:42:27 +0000

The following eval’ed code was caused a lot of misunderstanding: https://eval.in/61309. The code looks like this:

1$a = "2a0";
2
3for($i = 0; $i < 50; $i++) {
4 print "$a\n";
5 $a++;
6}

Node.js: using socket.io with cluster module

Sun, 26 Aug 2012 23:15:27 +0000

I’ve been recently working on a project that uses Node.js with cluster module and socket.io and I noticed that a lot of people that try to use Node.js with socket.io have problems with how to make it work with cluster module.

Accessing Gmail with oAuth authorization in PHP

Sun, 18 Dec 2011 15:00:22 +0000

Recently, I’ve joined a project that required accessing Gmail accounts using oAuth authentication with PHP. It turns out, that’s not such a very simple thing to do, especially with PHP - Zend has a very nice implementation for it, but the wiki page is terribly old and outdated… I’ve lost a few hours looking for a nice and clean solution and found a “handmadeimap” project done by Pete Warder some time ago. Bad thing is, the latest commit was done in 2010. The good thing: not much has changed since then, so the code works perfectly. It’s even better to use that php-imap extension, as you need to use raw IMAP commands, which gives you the chance to use, for example, gmail’s special X-GM-RAW extension to SEARCH command.