Schedule Recurring AWS Lambda Invocations With The Unreliable Town Clock (UTC)

public SNS Topic with a trigger event every quarter hour


Update 2015-10-08: Amazon has released AWS Lambda Scheduled Functions. I recommend using that feature to schedule AWS Lambda functions. In fact, the Unreliable Town Clock switched to use this feature behind the scenes, the day it was announced.


Scheduled executions of AWS Lambda functions on an hourly/daily/etc basis is a frequently requested feature, ever since the day Amazon introduced the service at AWS re:Invent 2014.

Until Amazon releases a reliable, premium cron feature for AWS Lambda, I’m offering a community-built alternative which may be useful for some non-critical applications.

us-east-1:

arn:aws:sns:us-east-1:522480313337:unreliable-town-clock-topic-178F1OQACHTYF

us-west-2:

arn:aws:sns:us-west-2:522480313337:unreliable-town-clock-topic-N4N94CWNOMTH

Background

Beyond its event-driven convenience, the primary attraction of AWS Lambda is eliminating the need to maintain infrastructure to run and scale code. The AWS Lambda function code is simply uploaded to AWS and Amazon takes care of providing systems to run on, keeping it available, scaling to meet demand, recovering from infrastructure failures, monitoring, logging, and more.

The available methods to trigger AWS Lambda functions already include some powerful and convenient events like S3 object creation, DynamoDB changes, Kinesis stream processing, and my favorite: the all-purpose SNS Topic subscription.

Even so, there is a glaring need for code that wants to run at regular intervals: time-triggered, recurring, scheduled event support for AWS Lambda. Attempts to to do this yourself generally ends up with having to maintain your own supporting infrastructure, when your original goal was to eliminate the infrastructure worries.

Unreliable Town Clock (UTC)

The Unreliable Town Clock (UTC) is a new, free, public SNS Topic (Amazon Simple Notification Service) that broadcasts a “chime” message every quarter hour to all subscribers. It can send the chimes to AWS Lambda functions, SQS queues, and email addresses.

You can use the chime attributes to run your code every fifteen minutes, or only run your code once an hour (e.g., when minute == "00") or once a day (e.g., when hour == "00" and minute == "00") or any other series of intervals.

You can even subscribe a function you only want to run only once at a specific time in the future: Have the function ignore all invocations until it’s after the time it wants. When it is time, it can perform its job, then unsubscribe itself from the SNS Topic.

Connecting your code to the Unreliable Town Clock is fast and easy. No application process or account creation is required:

Example: AWS Lambda Function

These commands subscribe an AWS Lambda function to the Unreliable Town Clock:

# AWS Lambda function
lambda_function_name=YOURLAMBDAFUNCTION
lambda_function_region=us-east-1
account=YOURACCOUNTID
lambda_function_arn="arn:aws:lambda:$lambda_function_region:$account:function:$lambda_function_name"

# Unreliable Town Clock public SNS Topic
sns_topic_arn=arn:aws:sns:us-east-1:522480313337:unreliable-town-clock-topic-178F1OQACHTYF
sns_region=us-east-1

# Allow the SNS Topic to invoke the AWS Lambda function
aws lambda add-permission \
  --region "$lambda_function_region" \
  --function-name "$lambda_function_name"  \
  --action lambda:InvokeFunction \
  --principal sns.amazonaws.com \
  --source-arn "$sns_topic_arn" \
  --statement-id $(uuidgen)

# Subscribe the AWS Lambda function to the SNS Topic
aws sns subscribe \
  --region "$sns_region" \
  --topic-arn "$sns_topic_arn" \
  --protocol lambda \
  --notification-endpoint "$lambda_function_arn"

Example: Email Address

These commands subscribe an email address to the Unreliable Town Clock (useful for getting the feel, testing, and debugging):

# Email address
email=YOUREMAIL@YOURDOMAIN

# Unreliable Town Clock public SNS Topic
sns_topic_arn=arn:aws:sns:us-east-1:522480313337:unreliable-town-clock-topic-178F1OQACHTYF
sns_region=us-east-1

# Subscribe the email address to the SNS Topic
aws sns subscribe \
  --region "$sns_region" \
  --topic-arn "$sns_topic_arn" \
  --protocol email \
  --notification-endpoint "$email"

Example: SQS Queue

These commands subscribe an SQS queue to the Unreliable Town Clock:

# SQS Queue
sqs_queue_name=YOURQUEUE
account=YOURACCOUNTID
sqs_queue_arn="arn:aws:sqs:us-east-1:$account:$sqs_queue_name"
sqs_queue_url="https://queue.amazonaws.com/$account/$sqs_queue_name"
sqs_region=us-east-1

# Unreliable Town Clock public SNS Topic
sns_topic_arn=arn:aws:sns:us-east-1:522480313337:unreliable-town-clock-topic-178F1OQACHTYF
sns_region=us-east-1

# Allow the SNS Topic to post to the SQS queue
sqs_policy='{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "AWS": "*" },
    "Action": "sqs:SendMessage",
    "Resource": "'$sqs_queue_arn'",
    "Condition": {
      "ArnEquals": {
        "aws:SourceArn": "'$sns_topic_arn'"
}}}]}'
sqs_policy_escaped=$(echo $sqs_policy | perl -pe 's/"/\\"/g')
aws sqs set-queue-attributes \
  --region "$sqs_region" \
  --queue-url "$sqs_queue_url" \
  --attributes '{"Policy":"'"$sqs_policy_escaped"'"}'

# Subscribe the SQS queue to the SNS Topic
aws sns subscribe \
  --region "$sns_region" \
  --topic-arn "$sns_topic_arn" \
  --protocol sqs \
  --notification-endpoint "$sqs_queue_arn"

Chime message

The chime message includes convenient attributes like the following:

{
  "type" : "chime",
  "timestamp": "2015-05-26 02:15 UTC",
  "year": "2015",
  "month": "05",
  "day": "26",
  "hour": "02",
  "minute": "15",
  "day_of_week": "Tue",
  "unique_id": "2d135bf9-31ba-4751-b46d-1db6a822ac88",
  "region": "us-east-1",
  "sns_topic_arn": "arn:aws:sns:...",
  "reference": "...",
  "support": "...",
  "disclaimer": "UNRELIABLE SERVICE {ACCURACY,CONSISTENCY,UPTIME,LONGEVITY}"
}

You should only run your code’s primary function when the message type == "chime"

Other values are reserved for other message types which may include things like service notifications or alerts. Those message types may have different attributes.

It might make sense to forward non-chime messages to a human (e.g., post to an SNS Topic where you have an email address subscribed).

Regions

The Unreliable Town Clock is currently available in the following AWS Regions:

  • us-east-1
  • us-west-2

You may create AWS Lambda functions in any AWS accounts in any AWS regions and subscribe them to these SNS Topics.

Problems in one region will not affect the Unreliable Town Clock functionality in the other region. You may subscribe to both topics for additional reliability. [There was an AWS SNS us-east-1 outage on 2015-07-31 that caused the Unreliable Town Clock in that region to not broadcast chimes for almost 3 hours.]

Cost

The Unreliable Town Clock is free for unlimited “lambda” and “sqs” subscriptions.

Yes. Unlimited. Amazon takes care of the scaling and does not charge for sending to these endpoints through SNS.

You may currently add “email” subscriptions, especially to test and see the message format, but if there are too many email subscribers, new subscriptions may be disabled, as it costs the sending account $0.70/year for each address at the current chime frequency.

You are naturally responsible for any charges that occur in your own accounts.

Running an AWS Lambda function four times an hour for a year results in 35,000 invocations, which is negligible if not free, but you need to take care what your functions do and what resources they consume as they are running in your AWS account.

Source

The source code for the infrastructure of the Unreliable Town Clock is available on GitHub

https://github.com/alestic/alestic-unreliable-town-clock

You are welcome to run your own copy, but note that the current code marks the SNS Topic as public so that anybody can subscribe.

Support

The following Google Group mailing list can be used for discussion, questions, enhancement requests, and alerts about problems.

http://groups.google.com/d/forum/unreliable-town-clock

If you plan to use the Unreliable Town Clock, you should subscribe to this mailing list so that you receive service notifications (e.g., if the public SNS Topic ARN is going to change).

Disclaimer

The Unreliable Town Clock service is intended but not guaranteed to be useful. As the name explicitly states, you should consider it unreliable and should not use it for anything you consider important.

Here are some, but not all, of the dimensions in which it is unreliable:

  • Accuracy: The times messages are sent may not be the true times they indicate. Messages may be delayed, get sent early, or be duplicated.

  • Uptime: Chime messages may be skipped for short or long periods of time.

  • Consistency: The formats or contents of the messages may change without warning.

  • Longevity: The service may disappear without warning at any time.

There is no big company behind this service, just a human being. I have experience building and supporting public services used by individuals, companies, and other organizations around the world, but I’m still just one fellow, and this is just an experimental service for the time being.

Comments

What are you thinking of using recurring AWS Lambda invocations for?

Any other features you would like to see?

[Update 2015-07-19: Ok to subscribe across AWS regions]

[Update 2015-07-31: Added second public SNS topic in us-west-2 after AWS SNS outage in us-east-1.]

*[Update 2017-11-22: Added –region options to sample commmands.]