When deploying infrastructure with CloudFormation, at some point you will reach a moment when your CloudFormation JSON or YAML file is just too big. It will be too long to get a good overview of what’s in it, manage parameters and all dependencies between resources within the template. Nested stacks may be a solution, but if sometimes you can’t/won' t/don’t use them for whatever reason (for example, it will get to complicated to manage tested stacks or their contents are not reusable between other stacks).
Even if you’re using troposphere to generate your templates, you’ll face the same issue - a very long Python file. Luckily, if you are using troposphere, you’re inherently using Python - which means you can take advantage of it.
Pawel have already written about magical dictionaries in Python before. In essence, they allow you to iterate over properties of a Python object as if it was a dictionary. That means, you can use the object within your code and get all the goodies that come with it - code completion within your IDE and simple referencing, refactoring, etc., and then iterate over that object to add all its properties to your troposphere template.
As a reminder, an example MagicDict class:
1class MagicDict(object):
2 def __init__(self):
3 super(MagicDict, self).__setattr__('internal', {})
4
5 def __setattr__(self, key, value):
6 self.internal[key] = value
7
8 def __getattr__(self, key):
9 return self.internal[key]
10
11 def __iter__(self):
12 return iter(self.internal)
13
14 def values(self):
15 return self.internal.values()I include that as magicdict.py in a directory. To use it, create a file next to it with the part of your
infrastructure, for example let’s say you want to create a VPC in vpc.py:
1from troposphere import Ref, Tags
2from troposphere import ec2
3
4from magicdict import MagicDict
5
6
7class VPC(MagicDict):
8 def __init__(self):
9 super(VPC, self).__init__()
10
11 self.vpc = ec2.VPC(
12 "VPC",
13 CidrBlock="172.1.0.0/16",
14 InstanceTenancy="default",
15 EnableDnsSupport=True,
16 EnableDnsHostnames=True,
17 Tags=Tags(
18 Name=Ref("AWS::StackName")
19 ),
20 )
21
22 # some other VPC stuff will go here as wellI tend to create __main__.py with contents like this:
1from troposphere import GetAtt, Output, Template
2
3from vpc import VPC
4
5
6def main():
7 template = Template()
8 template.add_description("Example stack")
9
10 vpc = VPC()
11 for res in vpc.values():
12 template.add_resource(res)
13
14 print(template.to_json())
15
16
17if __name__ == "__main__":
18 main()That’s it! Run python your_directory_name where “your_directory_name” is the directory holding all those 3 files.
You’ll get your CloudFormation template in return.
Obviously, this example is very short, so on its own it doesn’t help too much. In general, your vpc.py would be quite
long - including all subnets, route tables, NACLs, etc. So, let’s assume you’d like to add an ELB to that stack as well.
Instead of adding it to the already long vpc.py, create loadbalancer.py with this content:
1from troposphere import GetAtt, Ref
2from troposphere import ec2, elasticloadbalancing
3
4from magicdict import MagicDict
5from vpc import VPC
6
7
8class LoadBalancer(MagicDict):
9 def __init__(self, vpc):
10 """
11 :type vpc VPC
12 """
13 super(LoadBalancer, self).__init__()
14
15 self.load_balancer_security_group = ec2.SecurityGroup(
16 "LoadBalancerSecurityGroup",
17 GroupDescription="Loadbalancer security group",
18 SecurityGroupIngress=[
19 ec2.SecurityGroupRule(
20 IpProtocol="tcp",
21 FromPort=80,
22 ToPort=80,
23 CidrIp="0.0.0.0/0",
24 ),
25 ec2.SecurityGroupRule(
26 IpProtocol="tcp",
27 FromPort=443,
28 ToPort=443,
29 CidrIp="0.0.0.0/0",
30 ),
31 ec2.SecurityGroupRule(
32 IpProtocol="icmp",
33 FromPort="-1",
34 ToPort="-1",
35 CidrIp='0.0.0.0/0',
36 ),
37 ],
38 SecurityGroupEgress=[
39 ec2.SecurityGroupRule(
40 CidrIp="172.1.0.0/16",
41 FromPort=0,
42 IpProtocol="-1",
43 ToPort=65535
44 )
45 ],
46 VpcId=Ref(vpc.vpc)
47 )
48
49 self.load_balancer = elasticloadbalancing.LoadBalancer(
50 "LoadBalancer",
51 Subnets=[Ref(vpc.public_subnet_1), Ref(vpc.public_subnet_2)],
52 CrossZone=True,
53 Listeners=[
54 elasticloadbalancing.Listener(
55 LoadBalancerPort="80",
56 InstancePort="80",
57 Protocol="HTTP",
58 InstanceProtocol="HTTP"
59 ),
60 ],
61 HealthCheck=elasticloadbalancing.HealthCheck(
62 Target="HTTP:80/",
63 HealthyThreshold="2",
64 UnhealthyThreshold="3",
65 Interval="30",
66 Timeout="10",
67 ),
68 SecurityGroups=[
69 GetAtt(self.load_balancer_security_group, "GroupId"),
70 ],
71 DependsOn=vpc.internet_gateway_attachment.title
72 )and your __main__.py changes to this:
1from troposphere import GetAtt, Output, Template
2
3from vpc import VPC
4from loadbalancer import LoadBalancer
5
6def main():
7 template = Template()
8 template.add_description("Example stack")
9
10 vpc = VPC()
11 for res in vpc.values():
12 template.add_resource(res)
13
14 elb = LoadBalancer(vpc=vpc)
15 for res in elb.values():
16 template.add_resource(res)
17
18 template.add_output(Output(
19 "LoadBalancerDNSName",
20 Value=GetAtt(elb.load_balancer, "DNSName")
21 ))
22
23 print(template.to_json())
24
25
26if __name__ == "__main__":
27 main()The load balancer relies on the VPC resources - which is clearly visible now in the __main__.py file. Inside
the loadbalancer.py you can very easily use those resources, like when specifying the
subnets: Subnets=[Ref(vpc.public_subnet_1), Ref(vpc.public_subnet_2)]. Any reasonable Python IDE will be able to
code-complete the vpc resources for you inside the load balancer file, so you can easily reference them where needed.
You can obviously add more files and more resources to the template!
For example, almost every CloudFormation stack needs parameters. Create parameters.py:
1from troposphere import Parameter
2
3from magicdict import MagicDict
4
5
6class Parameters(MagicDict):
7 def __init__(self):
8 super(Parameters, self).__init__()
9
10 self.key_pair = Parameter(
11 "KeyPair",
12 Type="AWS::EC2::KeyPair::KeyName",
13 Description="Key pair to use to login to your instance"
14 )and add this to __main__.py under the template description:
1parameters = Parameters()
2for param in parameters.values():
3 template.add_parameter(param)Simple. Your parameters are now in the template and you can pass in the parameters object into other objects (like vpc
or load balancer). How to reference the parameter? The same as usual:
Ref(parameters.key_pair)One note: in the above example you’ll see this where the load balancer is defined:
DependsOn=vpc.internet_gateway_attachment.titlethe .title suffix is needed here so that troposphere inserts the name of the internet gateway attachment object and
not the whole object definition. You don’t need to use .title with Ref() because it does that automatically for you.
But for DependsOn we specifically need to insert the name of the resource - which is exposed under title property.
**
If you want to see this in action, check out our cloudformation-examples repository on GitHub with a full example , which creates a VPC with Auto scaling group and an RDS instance.
**
