Partial Deployment Pattern

Usually, for big projects, there are many AWS resources and depends on each other like a chain. Here’s some tips for writing super big and complex template.

  1. DO NOT write a super big template and then deploy, most likely its gonna fail and it is hard to debug a big template. A better approach is to start from a template with only a few resources, and deploy. Then gradually add more resources.
  2. If you already have a big template and it works, it could be challenge to update a resource in the middle of the dependency chain. I recommend to disable all dependent resources and try deploy. If it works, then continue to deploy other dependent resources.
  3. To maximize reuse of your code, I recommend to parameterize your template, and partialize deploy the infrastructure. For example, you may have a template that deploys a public VPC, a private VPC, a bunch of IAM role, some EC2 instance, a auto-scaling-group, a ECS cluster, some lambda function, several S3 buckets and a RDS database, and a code pipeline for your web app. It is a nicely designed an mature architect in your organization. For other projects, you can reuse this template and choose to partially deploy part of the architect.

troposphere_mate provides an api allow you to easily partially deploy your template. The idea is simple, it just automatically or manually label your Resource and remove them using label.

Here’s an example template with an IAM Policy, an IAM Role and an IAM Instance Profile. The Role depends on the policy, the instance profile depends on the role. For the sample template code, take a look at https://github.com/MacHu-GWU/troposphere_mate-project/tree/master/troposphere_mate/examples/partial_deploy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
# -*- coding: utf-8 -*-

from troposphere_mate import (
    Template, Parameter, helper_fn_sub, canned, Ref,
    iam, DEFAULT_LABELS_FIELD,
)

template = Template()

param_env_name = Parameter(
    "EnvironmentName",
    Type="String",
)

template.add_parameter(param_env_name)

LABEL_IAM_POLICY_TIER = "iam_policy_tier"
iam_ec2_instance_policy = iam.ManagedPolicy(
    "IamPolicy",
    template=template,
    Metadata={DEFAULT_LABELS_FIELD: [LABEL_IAM_POLICY_TIER,]},
    ManagedPolicyName=helper_fn_sub("{}-web-server", param_env_name),
    PolicyDocument={
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "VisualEditor0",
                "Effect": "Allow",
                "Action": [
                    "s3:Get*",
                    "s3:List*",
                    "s3:Describe*",
                ],
                "Resource": "*"
            }
        ]
    },
)

LABEL_IAM_ROLE_TIER = "iam_role_tier"
iam_ec_instance_role = iam.Role(
    "IamRoleWebServer",
    template=template,
    Metadata={DEFAULT_LABELS_FIELD: [LABEL_IAM_ROLE_TIER,]},
    RoleName=helper_fn_sub("{}-web-server", param_env_name),
    AssumeRolePolicyDocument=canned.iam.create_assume_role_policy_document([
        canned.iam.AWSServiceName.amazon_Elastic_Compute_Cloud_Amazon_EC2,
    ]),
    # cross reference output
    ManagedPolicyArns=[
        iam_ec2_instance_policy.iam_managed_policy_arn
    ],
    DependsOn=iam_ec2_instance_policy,
)

LABEL_IAM_INSTANCE_PROFILE_TIER = "iam_instance_profile_tier"
iam_instance_profile = iam.InstanceProfile(
    "IamInstanceProfileWebServer",
    template=template,
    Metadata={DEFAULT_LABELS_FIELD: [LABEL_IAM_INSTANCE_PROFILE_TIER,]},
    InstanceProfileName=helper_fn_sub("{}-web-server", param_env_name),
    # cross reference output
    Roles=[
        iam_ec_instance_role.iam_role_name
    ],
    DependsOn=iam_ec_instance_role,
)

# Apply common tags to all resources. It doesn't overwrite the custom resource
# level tags.
common_tags = {
    "EnvironmentName": Ref(param_env_name)
}
template.update_tags(tags_dct=common_tags)

# Apply AWS Resource Type identifier into the label, so you can use label to
# filters resources.
template.create_resource_type_label()

Here’s an example of how you choose to partially deploy it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# -*- coding: utf-8 -*-

import boto3

from troposphere_mate import iam, StackManager
from troposphere_mate.examples.partial_deploy import cft

aws_profile = "eq_sanhe"
aws_region = "us-east-1"
cft_bucket = "eq-sanhe-for-everything"
env_name = "tropo-mate-examples-partial-deploy-dev"

#--- only uncomment 1 line at a time to play with it ---
# cft.template.remove_resource_by_label(iam.InstanceProfile.resource_type)
# cft.template.remove_resource_by_label(iam.Role.resource_type)
# cft.template.remove_resource_by_label(iam.Policy.resource_type)

# cft.template.remove_resource_by_label(cft.LABEL_IAM_INSTANCE_PROFILE_TIER)
# cft.template.remove_resource_by_label(cft.LABEL_IAM_ROLE_TIER)
# cft.template.remove_resource_by_label(cft.LABEL_IAM_POLICY_TIER)

cft.template.to_file("master.json")
boto_ses = boto3.session.Session(profile_name=aws_profile, region_name=aws_region)

sm = StackManager(boto_ses=boto_ses, cft_bucket=cft_bucket)
sm.deploy(
    template=cft.template,
    stack_name=env_name,
    stack_parameters={
        cft.param_env_name.title: env_name
    },
    include_iam=True,
)