Rewriting TimerCheck.io In Python 3.6 On AWS Lambda With Chalice

If you are using and depending on the TimerCheck.io service, please be aware that the entire code base will be swapped out and replaced with new code before the end of May, 2017.

Ideally, consumers of the TimerCheck.io API will notice no changes, but if you are concerned, you can test out the new implementation using this temporary endpoint: https://new.timercheck.io/

For example:

https://new.timercheck.io/YOURTIMERNAME/60

and

https://new.timercheck.io/YOURTIMERNAME

This new endpoint uses the same timer database, so all timers can be queried and set using either endpoint.

At some point before the end of May, the new code will be activated by the standard https://timercheck.io endpoint.

How Much Does It Cost To Run A Serverless API on AWS?

Serving 2.1 million API requests for $11

Folks tend to be curious about how much real projects cost to run on AWS, so here’s a real example with breakdowns by AWS service and feature.

This article walks through the AWS invoice for charges accrued in November 2016 by the TimerCheck.io API service which runs in the us-east-1 (Northern Virginia) region and uses the following AWS services:

  • API Gateway
  • AWS Lambda
  • DynamoDB
  • Route 53
  • CloudFront
  • SNS (Simple Notification Service)
  • CloudWatch Logs
  • CloudWatch Metrics
  • CloudTrail
  • S3
  • Network data transfer
  • CloudWatch Alarms

During this month, TimerCheck.io service processed over 2 million API requests. Every request ran an AWS Lambda function that read from and/or wrote to a DynamoDB table.

This AWS account is older than 12 months, so any first year free tier specials are no longer applicable.

Total Cost Overview

At the very top of the AWS invoice, we can see that the total AWS charges for the month of November add up to $11.12. This is the total bill for processing the 2.1 million API requests and all of the infrastructure necessary to support them.

Invoice: Summary

Alestic.com Blog Infrastructure Upgrade

publishing new blog posts with “git push”

For the curious, the Alestic.com blog has been running for a while on the Git-backed Static Website Cloudformation stack using the AWS Lambda Static Site Generator Plugin for Hugo.

Not much has changed in the design because I had been using Hugo before. However, Hugo is now automatically run inside of an AWS Lambda function triggered by updates to a CodeCommit Git repository.

It has been a pleasure writing with transparent review and publication processes enabled by Hugo and AWS:

  • When I save a blog post change in my editor (written using Markdown), a local Hugo process on my laptop automatically detects the file change, regenerates static pages, and refreshes the view in my browser.

  • When I commit and push blog post changes to my CodeCommit Git repository, the Git-backed Static Website stack automatically regenerates the static blog site using Hugo and deploys to the live website served by AWS.

Running aws-cli Commands Inside An AWS Lambda Function

even though aws-cli is not available by default in AWS Lambda

The AWS Lambda environments for each programming language (e.g., Python, Node, Java) already have the AWS client SDK packages pre-installed for those languages. For example, the Python AWS Lambda environment has boto3 available, which is ideal for connecting to and using AWS services in your function.

This makes it easy to use AWS Lambda as the glue for AWS. A function can be triggered by many different service events, and can respond by reading from, storing to, and triggering other services in the AWS ecosystem.

However, there are a few things that aws-cli currently does better than the AWS SDKs alone. For example, the following command is an efficient way to take the files in a local directory and recursively update a website bucket, uploading (in parallel) files that have changed, while setting important object attributes including MIME types guessing:

aws s3 sync --delete --acl public-read LOCALDIR/ s3://BUCKET/

The aws-cli software is not currently pre-installed in the AWS Lambda environment, but we can fix that with a little effort.

AWS Lambda Static Site Generator Plugins

starting with Hugo!

A week ago, I presented a CloudFormation template for an AWS Git-backed Static Website stack. If you are not familiar with it, please click through and review the features of this complete Git + static website CloudFormation stack.

This weekend, I extended the stack to support a plugin architecture to run the static site generator of your choosing against your CodeCommit Git repository content. You specify the AWS Lambda function at stack launch time using CloudFormation parameters (ZIP location in S3).

The first serious static site generator plugin is for Hugo, but others can be added with or without my involvement and used with the same unmodified CloudFormation template.

The Git-backed static website stack automatically invokes the static site generator whenever the site source is updated in the CodeCommit Git repository. It then syncs the generated static website content to the S3 bucket where the stack serves it over a CDN using https with DNS served by Route 53.

I have written three AWS Lambda static site generator plugins to demonstrate the concept and to serve as templates for new plugins:

Developing CloudStatus, an Alexa Skill to Query AWS Service Status -- an interview with Kira Hammond by Eric Hammond

Interview conducted in writing July-August 2016.

[Eric] Good morning, Kira. It is a pleasure to interview you today and to help you introduce your recently launched Alexa skill, “CloudStatus”. Can you provide a brief overview about what the skill does?

[Kira] Good morning, Papa! Thank you for inviting me.

CloudStatus allows users to check the service availability of any AWS region. On opening the skill, Alexa says which (if any) regions are experiencing service issues or were recently having problems. Then the user can inquire about the services in specific regions.

This skill was made at my dad’s request. He wanted to quickly see how AWS services were operating, without needing to open his laptop. As well as summarizing service issues for him, my dad thought CloudStatus would be a good opportunity for me to learn about retrieving and parsing web pages in Python.

All the data can be found in more detail at status.aws.amazon.com. But with CloudStatus, developers can hear AWS statuses with their Amazon Echo. Instead of scrolling through dozens of green checkmarks to find errors, users of CloudStatus listen to which services are having problems, as well as how many services are operating satisfactorily.

CloudStatus is intended for anyone who uses Amazon Web Services and wants to know about current (and recent) AWS problems. Eventually it might be expanded to talk about other clouds as well.

[Eric] Assuming I have an Amazon Echo, how do I install and use the CloudStatus Alexa skill?

[Kira] Just say “Alexa, enable CloudStatus skill”! Ask Alexa to “open CloudStatus” and she will give you a summary of regions with problems. An example of what she might say on the worst of days is:

Pause/Resume AWS Lambda Reading Kinesis Stream

use the aws-cli to suspend an AWS Lambda function processing an Amazon Kinesis stream, then resume it again

At Campus Explorer we are using AWS Lambda extensively, with sources including Kinesis, DyanmoDB, S3, SNS, CloudFormation, API Gateway, custom events, and schedules.

This week, Steve Caldwell (CTO and prolific developer) encountered a situation which required pausing an AWS Lambda function with a Kinesis stream source, and later resuming it, preferably from the same point at which it had been reading in each Kinesis shard.

We brainstormed a half dozen different ways to accomplish this with varying levels of difficulty, varying levels of cost, and varying levels of not-quite-what-we-wanted-ness.

A few hours later, Steve shared that he had discovered the answer (and suggested I pass on the answer to you).

Unreliable Town Clock Is Now Using AWS Lambda Scheduled Functions

Today at AWS re:Invent, Amazon has announced the ability to schedule AWS Lambda function invocations using cron syntax. Yay!

I’m happy to announce that the The Unreliable Town Clock is now using this functionality behind the scenes to send the chime messages to the public SNS topic every quarter hour, in both us-east-1 and us-west-1.

No significant changes should be perceived by the hundreds of subscribers to the Unreliable Town Clock public SNS topic.

If you are already using the Unreliable Town Clock, what should you do?

TimerCheck.io - Countdown Timer Microservice Built On Amazon API Gateway and AWS Lambda

deceptively simple web service with super powers

TimerCheck.io is a fully functional, fully scalable microservice built on the just-released Amazon API Gateway and increasingly popular AWS Lambda platforms.

TimerCheck.io is a public web service that maintains a practically unlimited number of countdown timers with one second resolution and no practical limit to the number of seconds each timer can run.

New timers can be created on a whim and each timer can be reset at any time to any number of seconds desired, whether it is still running or has already expired.

Synopsis

Let’s begin with an example to demonstrate the elegant simplicity of the TimerCheck.io interface.

1. Set timer - Any request of the following URL sets a timer named “YOURTIMERNAME” to start counting down immediately from 60 seconds:

https://timercheck.io/YOURTIMERNAME/60

You may click on that link now, or hit a URL of the same format with your own timer name and your chosen number of seconds. You may use a browser, a command like curl, or your favorite programming language.

2. Poll timer - The following URL requests the status of the above timer. Note that the only difference in the URL is that we have dropped the seconds count.

https://timercheck.io/YOURTIMERNAME

If the named timer is still running, TimerCheck.io will return HTTP Status code 200 OK, along with a JSON structure containing information like how many seconds are left.

If the timer has expired, TimerCheck.io will return an HTTP status code 504 Timeout.

That’s it!

No, really. That’s the entire API.

Simple New Web Service: Testers Requested

Interested in adding scheduled job monitoring (dead man’s switch) to the existing monitoring and alerting framework you are already using (Nagios, Sensu, Zenoss, Zabbix, Monit, Pingdom, Montastic, Ruxit, and the like)?

Last month I wrote about how I use Cronitor.io to monitor scheduled events with an example using an SNS Topic and AWS Lambda.

This week I spent a few hours building a simple web service that enables any polling-based monitor software or service to automatically support alerting when a target event has not occurred in a desired timeframe.

The new web service is built on infrastructure technologies that are reliably maintained and scaled by Amazon:

lambdash: AWS Lambda Shell Hack: New And Improved!

easier, simpler, faster, better

Seven months ago I published the lambdash AWS Lambda Shell Hack that lets you run shell commands to explore the environment in which AWS Lambda functions are executed.

I also posted samples of command output that show fascinating properties of the AWS Lambda runtime environment.

In the last seven months, Amazon has released new features and enhancements that have made a completely new version of lambdash possible, with many benefits including:

Monitor an SNS Topic with AWS Lambda and Cronitor.io

get alerted when an expected event does NOT happen

Last week I announced the availability of a public SNS Topic that may be used to run AWS Lambda functions on a recurring schedule. To encourage folks to realize the implications of a free community service maintained by an individual, I named it the “Unreliable Town Clock”.

Even with this understanding, some folks in the AWS community have (again) placed their faith in me and are already starting to depend on the Unreliable Town Clock public SNS Topic to drive their own AWS Lambda functions and SQS queues, and I want to make sure this service is as reliable as I can reasonably make it.

Here are some of the steps I have taken to increase the reliability of the Unreliable Town Clock:

Schedule Recurring AWS Lambda Invocations With The Unreliable Town Clock (UTC)

public SNS Topic with a trigger event every quarter hour


Update 2015-10-08: Amazon has released AWS Lambda Scheduled Functions. I recommend using that feature to schedule AWS Lambda functions. In fact, the Unreliable Town Clock switched to use this feature behind the scenes, the day it was announced.


Scheduled executions of AWS Lambda functions on an hourly/daily/etc basis is a frequently requested feature, ever since the day Amazon introduced the service at AWS re:Invent 2014.

Until Amazon releases a reliable, premium cron feature for AWS Lambda, I’m offering a community-built alternative which may be useful for some non-critical applications.

us-east-1:

arn:aws:sns:us-east-1:522480313337:unreliable-town-clock-topic-178F1OQACHTYF

us-west-2:

arn:aws:sns:us-west-2:522480313337:unreliable-town-clock-topic-N4N94CWNOMTH

Background

Beyond its event-driven convenience, the primary attraction of AWS Lambda is eliminating the need to maintain infrastructure to run and scale code. The AWS Lambda function code is simply uploaded to AWS and Amazon takes care of providing systems to run on, keeping it available, scaling to meet demand, recovering from infrastructure failures, monitoring, logging, and more.

The available methods to trigger AWS Lambda functions already include some powerful and convenient events like S3 object creation, DynamoDB changes, Kinesis stream processing, and my favorite: the all-purpose SNS Topic subscription.

Even so, there is a glaring need for code that wants to run at regular intervals: time-triggered, recurring, scheduled event support for AWS Lambda. Attempts to to do this yourself generally ends up with having to maintain your own supporting infrastructure, when your original goal was to eliminate the infrastructure worries.

Unreliable Town Clock (UTC)

The Unreliable Town Clock (UTC) is a new, free, public SNS Topic (Amazon Simple Notification Service) that broadcasts a “chime” message every quarter hour to all subscribers. It can send the chimes to AWS Lambda functions, SQS queues, and email addresses.

Debugging AWS Lambda Invocations With An Echo Function

As I create architectures that include AWS Lambda functions, I find there are situations where I just want to know that the AWS Lambda function is getting invoked and to review the exact event data structure that is being passed in to it.

I found that a simple “echo” function can be dropped in to copy the AWS Lambda event to the console log (CloudWatch Logs). It’s easy to review this output to make sure the function is getting invoked at the right times and with the right data.

Subscribing AWS Lambda Function To SNS Topic With aws-cli

The aws-cli documentation and command line help text have not been updated yet to include the syntax for subscribing an AWS Lambda function to an SNS topic, but it does work!

Here’s the format:

aws sns subscribe \
  --topic-arn arn:aws:sns:REGION:ACCOUNT:SNSTOPIC \
  --protocol lambda \
  --notification-endpoint arn:aws:lambda:REGION:ACCOUNT:function:LAMBDAFUNCTION

where REGION, ACCOUNT, SNSTOPIC, and LAMBDAFUNCTION are substituted with appropriate values for your account.

For example: