Nested Stacks Pattern

Like terraform, CloudFormation is a declaration language. For complex architect, it is very easy to have a huge, complex template. To manage complex CloudFormation template and reuse codes, you would like to break it into smaller files. It is called Nested Stacks Pattern.

HOWEVER, there are a lots of trivial and manual works to do:

  1. For a Parameter that used in all of the stacks, you have to define it MULTIPLES times and MANUALLY pass it from the master to the nested stacks using the AWS::Cloudformation::Stack.Properties.Parameters property.
  2. To reference a resource’s returns-value from a nested stack, you need to MANUALLY create a Output and using the GetAtt("TheNestedStackName", "Outputs.TheOutputLogicId").

these things obviously breaks the DIY - DO NOT REPEAT YOURSELF philosophy in Python.

Best Practice

In this example, a three layers nested stack structure example is provided. For the sample template code, take a look at https://github.com/MacHu-GWU/troposphere_mate-project/tree/master/troposphere_mate/examples/nested_stack.

They are:

  1. Iam Policy Tier
  2. Iam Role Tier
  3. Iam Instance Profile Tier

#2 depends on #1, #3 depends on #2. And #3 is the master tier.

  1. IAM Policy Tier:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
# -*- coding: utf-8 -*-

"""
This iam policy tier is deeply nested.
"""

from troposphere_mate import Template, Parameter, Output, Export, iam, helper_fn_sub

template = Template()

param_env_name = Parameter(
    "EnvironmentName",
    Type="String",
)

template.add_parameter(param_env_name)

iam_ec2_instance_policy = iam.ManagedPolicy(
    "IamPolicy",
    template=template,
    ManagedPolicyName=helper_fn_sub("{}-web-server", param_env_name),
    PolicyDocument={
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "VisualEditor0",
                "Effect": "Allow",
                "Action": [
                    "s3:Get*",
                    "s3:List*",
                    "s3:Describe*",
                ],
                "Resource": "*"
            }
        ]
    },
)

# allow cross reference from other stack
output_iam_ec2_instance_policy_name = Output(
    "IamInstancePolicyArn",
    Value=iam_ec2_instance_policy.iam_managed_policy_arn,
    Export=Export(helper_fn_sub("{}-iam-ec2-instance-policy-arn", param_env_name)),
    DependsOn=iam_ec2_instance_policy,
)
template.add_output(output_iam_ec2_instance_policy_name)
  1. IAM Role Tier:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
# -*- coding: utf-8 -*-

"""
This iam role tier is nested under the iam instance profile tier.
"""

from troposphere_mate import (
    Template, Parameter, Output, canned, helper_fn_sub, Ref, GetAtt, Export,
    iam, cloudformation,
    link_stack_template,
)
# IMPORTANT!
# import the nested stack python module,
# allows to cross reference parameter or output and
# bind "AWS::Cloudformation::Stack" with nested stack template
from . import tier_1_1_iam_policy

template = Template()

param_env_name = Parameter(
    "EnvironmentName",
    Type="String",
)

template.add_parameter(param_env_name)

iam_policy_stack = cloudformation.Stack(
    "IamPolicyStack",
    template=template,
    TemplateURL="",
    # cross reference parameter
    Parameters={
        tier_1_1_iam_policy.param_env_name.title: Ref(param_env_name),
    },
)
# bind nested stack with a template
link_stack_template(stack=iam_policy_stack, template=tier_1_1_iam_policy.template)

iam_ec_instance_role = iam.Role(
    "IamRoleWebServer",
    template=template,
    RoleName=helper_fn_sub("{}-web-server", param_env_name),
    AssumeRolePolicyDocument=canned.iam.create_assume_role_policy_document([
        canned.iam.AWSServiceName.amazon_Elastic_Compute_Cloud_Amazon_EC2,
    ]),
    # cross reference output
    ManagedPolicyArns=[
        GetAtt(iam_policy_stack, f"Outputs.{tier_1_1_iam_policy.output_iam_ec2_instance_policy_name.title}"),
    ],
    DependsOn=iam_policy_stack,
)

# allow cross reference from other stack
output_iam_ec2_instance_role_name = Output(
    "IamInstanceRoleName",
    Value=iam_ec_instance_role.iam_role_name,
    Export=Export(helper_fn_sub("{}-iam-ec2-instance-role-name", param_env_name)),
    DependsOn=iam_ec_instance_role,
)
template.add_output(output_iam_ec2_instance_role_name)

output_iam_ec2_instance_role_arn = Output(
    "IamInstanceRoleArn",
    Value=iam_ec_instance_role.iam_role_arn,
    Export=Export(helper_fn_sub("{}-iam-ec2-instance-role-arn", param_env_name)),
    DependsOn=iam_ec_instance_role,
)
template.add_output(output_iam_ec2_instance_role_arn)
  1. IAM Instance Profile Tier:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
# -*- coding: utf-8 -*-

"""
This iam instance profile tier is the master stack.
"""

from troposphere_mate import (
    Template, Parameter, Output, GetAtt, helper_fn_sub, Ref, Export,
    iam, cloudformation,
    link_stack_template,
)

# IMPORTANT!
# import the nested stack python module,
# allows to cross reference parameter or output and
# bind "AWS::Cloudformation::Stack" with nested stack template
from . import tier_1_iam_role

template = Template()

param_env_name = Parameter(
    "EnvironmentName",
    Type="String",
)

template.add_parameter(param_env_name)

iam_role_stack = cloudformation.Stack(
    "IamRoleStack",
    template=template,
    TemplateURL="",
    # cross reference parameter
    Parameters={
        tier_1_iam_role.param_env_name.title: Ref(param_env_name),
    },
)
# bind nested stack with a template
link_stack_template(stack=iam_role_stack, template=tier_1_iam_role.template)

iam_instance_profile = iam.InstanceProfile(
    "IamInstanceProfileWebServer",
    template=template,
    InstanceProfileName=helper_fn_sub("{}-web-server", param_env_name),
    # cross reference output
    Roles=[
        GetAtt(iam_role_stack, f"Outputs.{tier_1_iam_role.output_iam_ec2_instance_role_name.title}"),
    ],
    DependsOn=iam_role_stack,
)

# allow cross reference from other stack
output_iam_ec2_instance_profile_name = Output(
    "IamInstanceProfileName",
    Value=iam_instance_profile.iam_instance_profile_name,
    Export=Export(helper_fn_sub("{}-iam-ec2-instance-profile-name", param_env_name)),
)
template.add_output(output_iam_ec2_instance_profile_name)

output_iam_ec2_instance_profile_arn = Output(
    "IamInstanceProfileArn",
    Value=iam_instance_profile.iam_instance_profile_arn,
    Export=Export(helper_fn_sub("{}-iam-ec2-instance-profile-arn", param_env_name)),
)
template.add_output(output_iam_ec2_instance_profile_arn)
  1. The deploy script:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# -*- coding: utf-8 -*-

import boto3

from troposphere_mate import StackManager
from troposphere_mate.examples.nested_stack import tier_master_iam_inst_profile as cft

aws_profile = "eq_sanhe"
aws_region = "us-east-1"
cft_bucket = "eq-sanhe-for-everything"
env_name = "tropo-mate-examples-nested-stack-dev"

boto_ses = boto3.session.Session(profile_name=aws_profile, region_name=aws_region)

sm = StackManager(boto_ses=boto_ses, cft_bucket=cft_bucket)

sm.deploy(
    template=cft.template,
    stack_name=env_name,
    stack_parameters={
        cft.param_env_name.title: env_name
    },
    include_iam=True,
)

Why this is good?

  1. Deploy nested stacks made easy via Python. Manipulating parameters and tags is hard in shell, it’s not easy to implement the deployment script right and integrate with your parameter stores and CI/CD system. See the deploy script.
  2. Parameter and Output reference made easy. For raw cloudformation template, no IDE will tell you the exact logic id of a parameter or output. Even in terraform, it just gives you syntax highlight but not prevent you from typing wrong. Since troposphere_mate is python, everything is an object, you can easily import and reference.
  3. Manage complex project made easy. Since everything is just python object, you can easily break it done to different module and class. And create any custom pre processing / post processing logic.