Optional Parameters For Pre-existing Resources in AWS CloudFormation Templates

stack creates new AWS resources unless user specifies pre-existing


I like to design CloudFormation templates that create all of the resources necessary to implement the desired functionality without requiring a lot of separate, advanced setup. For example, the AWS Git-backed Static Website creates all of the interesting pieces including a CodeCommit Git repository, S3 buckets for web site content and logging, and even the Route 53 hosted zone.

Creating all of these resources is great if you were starting from scratch on a new project. However, you may sometimes want to use a CloudFormation template to enhance an existing account where one or more of the AWS resources already exist.

For example, consider the case where the user already has a CodeCommit Git repository and a Route 53 hosted zone for their domain. They still want all of the enhanced functionality provided in the Git-backed static website CloudFormation stack, but would rather not have to fork and edit the template code just to fit it in to the existing environment.

What if we could use the same CloudFormation template for different types of situations, sometimes pluging in pre-existing AWS resources, and other times letting the stack create the resources for us?


Alestic.com Blog Infrastructure Upgrade

publishing new blog posts with “git push”

For the curious, the Alestic.com blog has been running for a while on the Git-backed Static Website Cloudformation stack using the AWS Lambda Static Site Generator Plugin for Hugo.

Not much has changed in the design because I had been using Hugo before. However, Hugo is now automatically run inside of an AWS Lambda function triggered by updates to a CodeCommit Git repository.

It has been a pleasure writing with transparent review and publication processes enabled by Hugo and AWS:

  • When I save a blog post change in my editor (written using Markdown), a local Hugo process on my laptop automatically detects the file change, regenerates static pages, and refreshes the view in my browser.

  • When I commit and push blog post changes to my CodeCommit Git repository, the Git-backed Static Website stack automatically regenerates the static blog site using Hugo and deploys to the live website served by AWS.

Running aws-cli Commands Inside An AWS Lambda Function

even though aws-cli is not available by default in AWS Lambda

The AWS Lambda environments for each programming language (e.g., Python, Node, Java) already have the AWS client SDK packages pre-installed for those languages. For example, the Python AWS Lambda environment has boto3 available, which is ideal for connecting to and using AWS services in your function.

This makes it easy to use AWS Lambda as the glue for AWS. A function can be triggered by many different service events, and can respond by reading from, storing to, and triggering other services in the AWS ecosystem.

However, there are a few things that aws-cli currently does better than the AWS SDKs alone. For example, the following command is an efficient way to take the files in a local directory and recursively update a website bucket, uploading (in parallel) files that have changed, while setting important object attributes including MIME types guessing:

aws s3 sync --delete --acl public-read LOCALDIR/ s3://BUCKET/

The aws-cli software is not currently pre-installed in the AWS Lambda environment, but we can fix that with a little effort.

AWS Lambda Static Site Generator Plugins

starting with Hugo!

A week ago, I presented a CloudFormation template for an AWS Git-backed Static Website stack. If you are not familiar with it, please click through and review the features of this complete Git + static website CloudFormation stack.

This weekend, I extended the stack to support a plugin architecture to run the static site generator of your choosing against your CodeCommit Git repository content. You specify the AWS Lambda function at stack launch time using CloudFormation parameters (ZIP location in S3).

The first serious static site generator plugin is for Hugo, but others can be added with or without my involvement and used with the same unmodified CloudFormation template.

The Git-backed static website stack automatically invokes the static site generator whenever the site source is updated in the CodeCommit Git repository. It then syncs the generated static website content to the S3 bucket where the stack serves it over a CDN using https with DNS served by Route 53.

I have written three AWS Lambda static site generator plugins to demonstrate the concept and to serve as templates for new plugins:

AWS Git-backed Static Website

with automatic updates on changes in CodeCommit Git repository

A number of CloudFormation templates have been published that generate AWS infrastructure to support a static website. I’ll toss another one into the ring with a feature I haven’t seen yet.

In this stack, changes to the CodeCommit Git repository automatically trigger an update to the content served by the static website. This automatic update is performed using CodePipeline and AWS Lambda.

This stack also includes features like HTTPS (with a free certificate), www redirect, email notification of Git updates, complete DNS support, web site access logs, infinite scaling, zero maintenance, and low cost.

One of the most exciting features is the launch-time ability to specify an AWS Lambda function plugin (ZIP file) that defines a static site generator to run on the Git repository site source before deploying to the static website. A sample plugin is provided for the popular Hugo static site generator.

Here is an architecture diagram outlining the various AWS services used in this stack. The arrows indicate the major direction of data flow. The heavy arrows indicate the flow of website content.

CloudFormation stack architecture diagram

Deleting a Route 53 Hosted Zone And All DNS Records Using aws-cli

fast, easy, and slightly dangerous recursive deletion of a domain’s DNS

Amazon Route 53 currently charges $0.50/month per hosted zone for your first 25 domains, and $0.10/month for additional hosted zones, even if they are not getting any DNS requests. I recently stopped using Route 53 to serve DNS for 25 domains and wanted to save on the $150/year these were costing.

Amazon’s instructions for using the Route 53 Console to delete Record Sets and a Hosted Zone make it look simple. I started in the Route 53 Console clicking into a hosted zone, selecting each DNS record set (but not the NS or SOA ones), clicking delete, clicking confirm, going back a level, selecting the next domain, and so on. This got old quickly.

Being lazy, I decided to spend a lot more effort figuring out how to automate this process with the aws-cli, and pass the savings on to you.

Steps with aws-cli

Developing CloudStatus, an Alexa Skill to Query AWS Service Status -- an interview with Kira Hammond by Eric Hammond

Interview conducted in writing July-August 2016.

[Eric] Good morning, Kira. It is a pleasure to interview you today and to help you introduce your recently launched Alexa skill, “CloudStatus”. Can you provide a brief overview about what the skill does?

[Kira] Good morning, Papa! Thank you for inviting me.

CloudStatus allows users to check the service availability of any AWS region. On opening the skill, Alexa says which (if any) regions are experiencing service issues or were recently having problems. Then the user can inquire about the services in specific regions.

This skill was made at my dad’s request. He wanted to quickly see how AWS services were operating, without needing to open his laptop. As well as summarizing service issues for him, my dad thought CloudStatus would be a good opportunity for me to learn about retrieving and parsing web pages in Python.

All the data can be found in more detail at status.aws.amazon.com. But with CloudStatus, developers can hear AWS statuses with their Amazon Echo. Instead of scrolling through dozens of green checkmarks to find errors, users of CloudStatus listen to which services are having problems, as well as how many services are operating satisfactorily.

CloudStatus is intended for anyone who uses Amazon Web Services and wants to know about current (and recent) AWS problems. Eventually it might be expanded to talk about other clouds as well.

[Eric] Assuming I have an Amazon Echo, how do I install and use the CloudStatus Alexa skill?

[Kira] Just say “Alexa, enable CloudStatus skill”! Ask Alexa to “open CloudStatus” and she will give you a summary of regions with problems. An example of what she might say on the worst of days is:

Using AWS CodeCommit With Git Repositories In Multiple AWS Accounts

set up each local CodeCommit repository clone to use a specific cross-account IAM role with git clone --config and aws codecommit credentials-helper

When I started testing AWS CodeCommit, I used the Git ssh protocol with uploaded ssh keys to provide access, because this is the Git access mode I’m most familiar with. However, using ssh keys requires each person to have an IAM user in the same AWS account as the CodeCommit Git repository.

In my personal and work AWS usage, each individual has a single IAM user in a master AWS account, and those users are granted permission to assume cross-account IAM roles to perform operations in other AWS accounts. We cannot use the ssh method to access Git repositories in other AWS accounts, as there are no IAM users in those accounts.

AWS CodeCommit comes to our rescue with an alternative https access method that supports Git Smart HTTP, and the aws-cli offers a credential-helper feature that integrates with the git client to authenticate Git requests to the CodeCommit service.

In my tests, this works perfectly with cross-account IAM roles. After the initial git clone command, there is no difference in how git is used compared to the ssh access method.

Most of the aws codecommit credential-helper examples I’ve seen suggest you set up a git config --global setting before cloning a CodeCommit repository. A couple even show how to restrict the config to AWS CodeCommit repositories only so as to not interfere with GitHub and other repositories. (See “Resoures” below)

I prefer to have the configuration associated with the specific Git repositories that need it, not in the global setting file. This is possible by passing in a couple --config parameters to the git clone command.

Create/Get CodeCommit Repository

Creating An Amazon API Gateway With aws-cli For Domain Redirect

Ten commands to launch a minimal, functioning API Gateway

As of this publication date, the Amazon API Gateway is pretty new and the aws-cli interface for it is even newer. The API and aws-cli documentation at the moment is a bit rough, but this article outlines steps to create a functioning API Gateway with the aws-cli. Hopefully, this can help others who are trying to get it to work.


I regularly have a need to redirect browsers from one domain to another, whether it’s a vanity domain, legacy domain, “www” to base domain, misspelling, or other reasons.

I usually do this with an S3 bucket in website mode with a CloudFront distribution in front to support https. This works, performs well, and costs next to nothing.

Now that the Amazon API Gateway has aws-cli support, I was looking for simple projects to test out so I worked to reproduce the domain redirect. I found I can create an API Gateway that will redirect a hostname to a target URL, without any back end for the API (not even a Lambda function).

I’m not saying the API Gateway method is better than using S3 plus CloudFront for simple hostname redirection. In fact, it costs more (though still cheap), takes more commands to set up, and isn’t quite as flexible in what URL paths get redirected from the source domain. It does, however, work and may be useful as an API Gateway aws-cli example.


Pause/Resume AWS Lambda Reading Kinesis Stream

use the aws-cli to suspend an AWS Lambda function processing an Amazon Kinesis stream, then resume it again

At Campus Explorer we are using AWS Lambda extensively, with sources including Kinesis, DyanmoDB, S3, SNS, CloudFormation, API Gateway, custom events, and schedules.

This week, Steve Caldwell (CTO and prolific developer) encountered a situation which required pausing an AWS Lambda function with a Kinesis stream source, and later resuming it, preferably from the same point at which it had been reading in each Kinesis shard.

We brainstormed a half dozen different ways to accomplish this with varying levels of difficulty, varying levels of cost, and varying levels of not-quite-what-we-wanted-ness.

A few hours later, Steve shared that he had discovered the answer (and suggested I pass on the answer to you).