Using Magical Dictionaries to manage long CloudFormation templates with Troposphere

When deploying infrastructure with CloudFormation, at some point you will reach a moment when your CloudFormation JSON or YAML file is just too big. It will be too long to get a good overview of what’s in it, manage parameters and all dependencies between resources within the template. Nested stacks may be a solution, but if sometimes you can’t/won' t/don’t use them for whatever reason (for example, it will get to complicated to manage tested stacks or their contents are not reusable between other stacks).

Even if you’re using troposphere to generate your templates, you’ll face the same issue - a very long Python file. Luckily, if you are using troposphere, you’re inherently using Python - which means you can take advantage of it.

Pawel have already written about magical dictionaries in Python before. In essence, they allow you to iterate over properties of a Python object as if it was a dictionary. That means, you can use the object within your code and get all the goodies that come with it - code completion within your IDE and simple referencing, refactoring, etc., and then iterate over that object to add all its properties to your troposphere template.

As a reminder, an example MagicDict class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
class MagicDict(object):
    def __init__(self):
        super(MagicDict, self).__setattr__('internal', {})

    def __setattr__(self, key, value):
        self.internal[key] = value

    def __getattr__(self, key):
        return self.internal[key]

    def __iter__(self):
        return iter(self.internal)

    def values(self):
        return self.internal.values()

I include that as magicdict.py in a directory. To use it, create a file next to it with the part of your infrastructure, for example let’s say you want to create a VPC in vpc.py:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
from troposphere import Ref, Tags
from troposphere import ec2

from magicdict import MagicDict


class VPC(MagicDict):
    def __init__(self):
        super(VPC, self).__init__()

        self.vpc = ec2.VPC(
            "VPC",
            CidrBlock="172.1.0.0/16",
            InstanceTenancy="default",
            EnableDnsSupport=True,
            EnableDnsHostnames=True,
            Tags=Tags(
                Name=Ref("AWS::StackName")
            ),
        )

        # some other VPC stuff will go here as well

I tend to create __main__.py with contents like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
from troposphere import GetAtt, Output, Template

from vpc import VPC


def main():
    template = Template()
    template.add_description("Example stack")

    vpc = VPC()
    for res in vpc.values():
        template.add_resource(res)

    print(template.to_json())


if __name__ == "__main__":
    main()

That’s it! Run python your_directory_name where “your_directory_name” is the directory holding all those 3 files. You’ll get your CloudFormation template in return.

Obviously, this example is very short, so on its own it doesn’t help too much. In general, your vpc.py would be quite long - including all subnets, route tables, NACLs, etc. So, let’s assume you’d like to add an ELB to that stack as well. Instead of adding it to the already long vpc.py, create loadbalancer.py with this content:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
from troposphere import GetAtt, Ref
from troposphere import ec2, elasticloadbalancing

from magicdict import MagicDict
from vpc import VPC


class LoadBalancer(MagicDict):
    def __init__(self, vpc):
        """
        :type vpc VPC
        """
        super(LoadBalancer, self).__init__()

        self.load_balancer_security_group = ec2.SecurityGroup(
            "LoadBalancerSecurityGroup",
            GroupDescription="Loadbalancer security group",
            SecurityGroupIngress=[
                ec2.SecurityGroupRule(
                    IpProtocol="tcp",
                    FromPort=80,
                    ToPort=80,
                    CidrIp="0.0.0.0/0",
                ),
                ec2.SecurityGroupRule(
                    IpProtocol="tcp",
                    FromPort=443,
                    ToPort=443,
                    CidrIp="0.0.0.0/0",
                ),
                ec2.SecurityGroupRule(
                    IpProtocol="icmp",
                    FromPort="-1",
                    ToPort="-1",
                    CidrIp='0.0.0.0/0',
                ),
            ],
            SecurityGroupEgress=[
                ec2.SecurityGroupRule(
                    CidrIp="172.1.0.0/16",
                    FromPort=0,
                    IpProtocol="-1",
                    ToPort=65535
                )
            ],
            VpcId=Ref(vpc.vpc)
        )

        self.load_balancer = elasticloadbalancing.LoadBalancer(
            "LoadBalancer",
            Subnets=[Ref(vpc.public_subnet_1), Ref(vpc.public_subnet_2)],          
            CrossZone=True,
            Listeners=[
                elasticloadbalancing.Listener(
                    LoadBalancerPort="80",
                    InstancePort="80",
                    Protocol="HTTP",
                    InstanceProtocol="HTTP"
                ),
            ],
            HealthCheck=elasticloadbalancing.HealthCheck(
                Target="HTTP:80/",
                HealthyThreshold="2",
                UnhealthyThreshold="3",
                Interval="30",
                Timeout="10",
            ),
            SecurityGroups=[
                GetAtt(self.load_balancer_security_group, "GroupId"),
            ],
            DependsOn=vpc.internet_gateway_attachment.title
        )

and your __main__.py changes to this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
from troposphere import GetAtt, Output, Template

from vpc import VPC
from loadbalancer import LoadBalancer

def main():
    template = Template()
    template.add_description("Example stack")

    vpc = VPC()
    for res in vpc.values():
        template.add_resource(res)

    elb = LoadBalancer(vpc=vpc)
    for res in elb.values():
        template.add_resource(res)

    template.add_output(Output(
        "LoadBalancerDNSName",
        Value=GetAtt(elb.load_balancer, "DNSName")
    ))

    print(template.to_json())


if __name__ == "__main__":
    main()

The load balancer relies on the VPC resources - which is clearly visible now in the __main__.py file. Inside the loadbalancer.py you can very easily use those resources, like when specifying the subnets: Subnets=[Ref(vpc.public_subnet_1), Ref(vpc.public_subnet_2)]. Any reasonable Python IDE will be able to code-complete the vpc resources for you inside the load balancer file, so you can easily reference them where needed. You can obviously add more files and more resources to the template!

For example, almost every CloudFormation stack needs parameters. Create parameters.py:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
from troposphere import Parameter

from magicdict import MagicDict


class Parameters(MagicDict):
    def __init__(self):
        super(Parameters, self).__init__()

        self.key_pair = Parameter(
            "KeyPair",
            Type="AWS::EC2::KeyPair::KeyName",
            Description="Key pair to use to login to your instance"
        )

and add this to __main__.py under the template description:

1
2
3
parameters = Parameters()
for param in parameters.values():
    template.add_parameter(param)

Simple. Your parameters are now in the template and you can pass in the parameters object into other objects (like vpc or load balancer). How to reference the parameter? The same as usual:

    Ref(parameters.key_pair)

One note: in the above example you’ll see this where the load balancer is defined:

    DependsOn=vpc.internet_gateway_attachment.title

the .title suffix is needed here so that troposphere inserts the name of the internet gateway attachment object and not the whole object definition. You don’t need to use .title with Ref() because it does that automatically for you. But for DependsOn we specifically need to insert the name of the resource - which is exposed under title property.

If you want to see this in action, check out our cloudformation-examples repository on GitHub with a full example , which creates a VPC with Auto scaling group and an RDS instance.

Using Magical Dictionaries to manage long CloudFormation templates with Troposphere

Mysterious Code - Your AWS & DevOps consultants