Creating Elasticsearch Domain did not stabilize - can't create AWS Elasticsearch 5.1 with CloudFormation

Recently AWS announced support for Elasticsearch 5.1 in their Elasticsearch Service. Today, I tried to upgrade an existing CloudFormation stack, previously using Elasticsearch 2.3, to the new version and, after a very long wait, CloudFormation rolled back the stack with the following error: “Creating Elasticsearch Domain did not stabilize”. Here’s what I did to solve it.

First, I made sure it is actually possible to even create a working Elasticsearch 5.1 Domain in AWS. It is. You can create a domain using AWS Console and that domain will reach an “Active” state.

Then, I used AWS CLI describe-elasticsearch-domain call to compare the failed domain created by CloudFormation (_ Rollback on failure_ set to No helps a lot with debugging). From the comparison it looked like the domain created by AWS Console had the AdvancedOptions set to default values:

1
2
3
4

"AdvancedOptions": {
    "rest.action.multi.allow_explicit_index": "true", 
    "indices.fielddata.cache.size": ""
},

while the domain from CloudFormation had that option empty (which was correct, since our template did not set the values):

"AdvancedOptions": {}, 

Since this was the only difference between both domains, I added the AdvancedOptions definition to our CloudFormation template, setting them to the default values, as above. With that change, the stack and the domain was created successfully!

Example of how to set this using troposphere:

1
2
3
4
5
6
7
8

elasticsearch.Domain(
    "Elasticsearch",
    ...
    AdvancedOptions={  
        "rest.action.multi.allow_explicit_index": "true",
        "indices.fielddata.cache.size": ""
    }
)

Or directly in CloudFormation’s JSON template:

1
2
3
4
5
6
7
8

"Elasticsearch": {
    "Properties": {
        ...
        "AdvancedOptions": {
            "indices.fielddata.cache.size": "",
             "rest.action.multi.allow_explicit_index": "true"
        },
}

Even though those are default values, and CloudFormation documentation specifies this property is not required, it looks like it is not possible to create Elasticsearch 5.1 domain without those being set.

Update: Bug reported

Since this post was published, I have been in contact with AWS support and reported the above issue as a bug. You can follow the progress of the report on AWS Forums: https://forums.aws.amazon.com/thread.jspa?messageID=768527