Using AWS Step Functions To Schedule Or Delay SNS Message Publication

with no AWS Lambda function required

A co-worker at Archer asked if there was a way to schedule messages published to an Amazon SNS topic.

I know that scheduling messages to SQS queues is possible to some extent using the DelaySeconds message timer, which allows postponing visibility in the queue up to 15 minutes, but SNS does not currently have native support for delays.

However, since AWS Step Functions has built-in integration with SNS, and since it also has a Wait state that can schedule or delay execution, we can implement a fairly simple Step Functions state machine that puts a delay in front of publishing a message to an SNS topic, without any AWS Lambda code.

Overview

This article uses an AWS CloudFormation template to create a sample AWS stack with one SNS topic and one Step Functions state machine with two states.

AWS architecture diagram

This is the CloudFormation template, if you’d like to review it:

CloudFormation template: aws-sns-delayed

Here is the Step Functions state machine definition from the above CloudFormation template:

{
  "StartAt": "Delay",
  "Comment": "Publish to SNS with delay",
  "States": {
    "Delay": {
      "Type": "Wait",
      "SecondsPath": "$.delay_seconds",
      "Next": "Publish to SNS"
    },
    "Publish to SNS": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": {
        "TopicArn": "${SNSTopic}",
        "Subject.$": "$.subject",
        "Message.$": "$.message"
      },
      "End": true
    }
  }
}

The “Delay” state waits for “delay_seconds” provided in the input to the state machine execution (as we’ll see below).

The “Publish to SNS” task uses the Step Functions integration with SNS to call the publish API directly with the parameters listed, some of which are also passed in to the state machine execution.

Now let’s take it for a spin!

Preparation

Clone the demo repo from GitHub to get the CloudFormation template:

git clone git@github.com:alestic/aws-sns-delayed.git
cd aws-sns-delayed

Select your email address and a stack name for this demo:

email=YOUREMAIL@example.com
stack_name=sns-delayed-demo

Launch Stack

Deploy the CloudFormation stack:

stack_parameters="NotificationEmail=$email"
aws cloudformation deploy \
  --stack-name "$stack_name" \
  --template-file template.yaml \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides "$stack_parameters"

The deploy command can be re-run after you edit template.html and want to deploy the changes.

Get the Step Function State Machine and SNS topic from the running stack outputs:

state_machine=$(aws cloudformation describe-stacks \
  --stack-name "$stack_name" \
  --output text \
  --query 'Stacks[][Outputs][][?OutputKey==`StepFunctionsStateMachine`][OutputValue]')
echo state_machine=$state_machine

sns_topic=$(aws cloudformation describe-stacks \
  --stack-name "$stack_name" \
  --output text \
  --query 'Stacks[][Outputs][][?OutputKey==`SNSTopic`][OutputValue]')
echo sns_topic=$sns_topic

Publish Messages

Before you go any further, make sure you go to your email and confirm your subscription to the SNS topic. If you don’t do this, you won’t get to see any of the test messages published to the topic below.

Test 1: Post directly to the SNS Topic, circumventing the Step Functions state machine:

aws sns publish \
  --topic-arn "$sns_topic" \
  --subject "Test message published directly to SNS topic" \
  --message "hello, world"

Test 2: Post through the Step Functions state machine, with a delay specified in seconds:

execution_input='{
  "delay_seconds": 60,
  "subject": "Test message delayed through AWS Step Functions",
  "message": "hello, world"
}'

aws stepfunctions start-execution \
  --state-machine-arn "$state_machine" \
  --input "$execution_input"

That’s it! Check your email for the test messages.

You can also go to the AWS Step Functions Console and watch your state machjine executing in real time. Here is a graphic representation of the state machine as displayed in the AWS Console, shown while an execution is in the Delay state:

AWS Step Functions state machine

Cleanup

Once you have finished playing with this sample stack, you can delete it:

aws cloudformation delete-stack \
  --stack-name "$stack_name"

This cleans up everything… eventually.

If you have created Step Functions state machine executions that have very long delays, then the state machine will be put in a “DELETING” status, and won’t actually disappear until every execution has entered its next state transition.

If this doesn’t feel clean to you, it is possible to manually stop each running execution, allowing the state machine to finally be deleted.

Adaptations

Absolute Timestamps

The above example uses the SecondsPath field in the Wait state, which delays for the specified number of seconds. To instead schedule the message for publishing at a specific date/time in the future, replace the line:

"SecondsPath": "$.delay_seconds",

with:

"TimestampPath": "$.delay_timestamp",

and change the execution_input to include the absolute timestamp when you want the SNS message to be published, like so:

execution_input='{
  "delay_timestamp": "2020-01-01T00:00:00Z",
  "subject": "Happy New Year!",
  "message": "Peace to you and yours in this new year."
}'

I considered adding extra Step Function states to detect whether a relative or absolute delay was specified in the input, then branching to the appropriate type of Wait state, but adding 50% in Step Functions cost didn’t seem worth it.

Other SNS Message Formats

This example publishes a simple text message to the SNS topic along with a Subject value to make emails look prettier.

SNS publish supports other message types including structured messages where different formatted values are sent to different types of subscribers. You can modify the “Publish to SNS” task parameters in the state machine definition to fit your application’s needs.

Other SNS Topics

This sample CloudFormation template creates its own SNS topic, but you could remove it and parameterize the ARN in the template to put a delay in front of another, pre-existing SNS topic.

You could even add an SNS topic ARN parameter to the Step Functions state machine, so that you can schedule or delay messages to arbitrary SNS topics. In order for this to work, though, you would need to change the IAM role so that the state machine can publish to those topics.

Delaying Other AWS Activities

SNS is not the only resource with built-in AWS Step Functions integration support. You can use this same approach to schedule or delay operations with DynamoDB, AWS Batch, Amazon ECS, Fargate, SQS, AWS Glue, SageMaker, and of course, AWS Lambda.

CAVEATS

Size - The message limit for SNS is currently 256 KB. The message limit for Step Functions is currently 32 KB, so you can’t use this approach for all SNS applications.

Cost - A million SNS publishes currently costs about $0.25 depending on the region, plus delivery charges, if any. A million Step Function state transitions currently costs $25, but this state machine uses 2 state transitions per post, so the resulting cost will be around $50 for a million delayed/scheduled SNS messages.

Time - Step Function state machine executions must complete within 1 year, including any Wait steps, so this method cannot be used to schedule SNS posts more than a year in advance.

Limits tend to increase and costs tend to decrease over time with AWS, so look up the current values before making decisions.

AWS Console “Task Timer” Sample Project

The AWS Step Functions Developer Guide describes a Task Timer project that is available in the AWS Console Step Functions Sample Projects.

The Task Timer sample state machine uses an AWS Lambda function, so it’s a great example for how to drive AWS Lambda with Step Functions.

However, since Step Functions recently announced the ability to integrate directly with SNS, we don’t need to complicate this functionality with Lambda code. Step Functions can do all the work with no function to write or maintain, as demonstrated above.