A fantastic new and oft-requested AWS feature was released during AWS re:Invent, but has gotten lost in all the hype about AWS Lambda functions being triggered when objects are added to S3 buckets. AWS Lambda is currently in limited Preview mode and you have to request access, but this related feature is already available and ready to use.
I’m talking about automatic S3 bucket notifications to SNS topics and SQS queues when new S3 objects are added.
Unlike AWS Lambda, with S3 bucket notifications you do need to maintain the infrastructure to run your code, but you’re already running EC2 instances for application servers and job processing, so this will fit right in.
To detect and respond to S3 object creation in the past, you needed to either have every process that uploaded to S3 subsequently trigger your back end code in some way, or you needed to poll the S3 bucket to see if new objects had been added. The former adds code complexity and tight coupling dependencies. The latter can be costly in performance and latency, especially as the number of objects in the bucket grows.
With the new S3 bucket notification configuration options, the addition of an object to a bucket can send a message to an SNS topic or to an SQS queue, triggering your code quickly and effortlessly.
Here’s a working example of how to set up and use S3 bucket notification configurations to send messages to SNS on object creation and update.
Setup
Replace parameter values with your preferred names.
region=us-east-1
s3_bucket_name=BUCKETNAMEHERE
email_address=YOURADDRESS@EXAMPLE.COM
sns_topic_name=s3-object-created-$(echo $s3_bucket_name | tr '.' '-')
sqs_queue_name=$sns_topic_name
Create the test bucket.
aws s3 mb \
--region "$region" \
s3://$s3_bucket_name
Create an SNS topic.
sns_topic_arn=$(aws sns create-topic \
--region "$region" \
--name "$sns_topic_name" \
--output text \
--query 'TopicArn')
echo sns_topic_arn=$sns_topic_arn
Allow S3 to publish to the SNS topic for activity in the specific S3 bucket.
aws sns set-topic-attributes \
--topic-arn "$sns_topic_arn" \
--attribute-name Policy \
--attribute-value '{
"Version": "2008-10-17",
"Id": "s3-publish-to-sns",
"Statement": [{
"Effect": "Allow",
"Principal": { "AWS" : "*" },
"Action": [ "SNS:Publish" ],
"Resource": "'$sns_topic_arn'",
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:s3:*:*:'$s3_bucket_name'"
}
}
}]
}'
Add a notification to the S3 bucket so that it sends messages to the SNS topic when objects are created (or updated).
aws s3api put-bucket-notification \
--region "$region" \
--bucket "$s3_bucket_name" \
--notification-configuration '{
"TopicConfiguration": {
"Events": [ "s3:ObjectCreated:*" ],
"Topic": "'$sns_topic_arn'"
}
}'
Test
You now have an S3 bucket that is going to post a message to an SNS topic when objects are added. Let’s give it a try by connecting an email address listener to the SNS topic.
Subscribe an email address to the SNS topic.
aws sns subscribe \
--topic-arn "$sns_topic_arn" \
--protocol email \
--notification-endpoint "$email_address"
IMPORTANT! Go to your email inbox now and click the link to confirm that you want to subscribe that email address to the SNS topic.
Upload one or more files to the S3 bucket to trigger the SNS topic messages.
aws s3 cp [SOMEFILE] s3://$s3_bucket_name/testfile-01
Check your email for the notification emails in JSON format, containing attributes like:
{ "Records":[
{ "eventTime":"2014-11-27T00:57:44.387Z",
"eventName":"ObjectCreated:Put", ...
"s3":{
"bucket":{ "name":"BUCKETNAMEHERE", ... },
"object":{ "key":"testfile-01", "size":5195, ... }
}}]}
Notification to SQS
The above example connects an SNS topic to the S3 bucket notification configuration. Amazon also supports having the bucket notifications go directly to an SQS queue, but I do not recommend it.
Instead, send the S3 bucket notification to SNS and have SNS forward it to SQS. This way, you can easily add other listeners to the SNS topic as desired. You can even have multiple SQS queues subscribed, which is not possible when using a direct notification configuration.
Here are some sample commands that create an SQS queue and connect it to the SNS topic.
Create the SQS queue and get the ARN (Amazon Resource Name). Some APIs need the SQS URL and some need the SQS ARN. I don’t know why.
sqs_queue_url=$(aws sqs create-queue \
--queue-name $sqs_queue_name \
--attributes 'ReceiveMessageWaitTimeSeconds=20,VisibilityTimeout=300' \
--output text \
--query 'QueueUrl')
echo sqs_queue_url=$sqs_queue_url
sqs_queue_arn=$(aws sqs get-queue-attributes \
--queue-url "$sqs_queue_url" \
--attribute-names QueueArn \
--output text \
--query 'Attributes.QueueArn')
echo sqs_queue_arn=$sqs_queue_arn
Give the SNS topic permission to post to the SQS queue.
sqs_policy='{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Principal": { "AWS": "*" },
"Action":"sqs:SendMessage",
"Resource":"'$sqs_queue_arn'",
"Condition":{
"ArnEquals":{
"aws:SourceArn":"'$sns_topic_arn'"
}
}
}
]
}'
sqs_policy_escaped=$(echo $sqs_policy | perl -pe 's/"/\\"/g')
sqs_attributes='{"Policy":"'$sqs_policy_escaped'"}'
aws sqs set-queue-attributes \
--queue-url "$sqs_queue_url" \
--attributes "$sqs_attributes"
Subscribe the SQS queue to the SNS topic.
aws sns subscribe \
--topic-arn "$sns_topic_arn" \
--protocol sqs \
--notification-endpoint "$sqs_queue_arn"
You can upload another test file to the S3 bucket, which will now generate both the email and a message to the SQS queue.
aws s3 cp [SOMEFILE] s3://$s3_bucket_name/testfile-02
Read the S3 bucket notification message from the SQS queue:
aws sqs receive-message \
--queue-url $sqs_queue_url
The output of that command is not quite human readable as it has quoted JSON inside quoted JSON inside JSON, but your queue processing software should be able to decode it and take appropriate actions.
You can tell the SQS queue that you have “processed” the message by grabbing the “ReceiptHandle” value from the above output and deleting the message.
sqs_receipt_handle=...
aws sqs delete-message \
--queue-url "$sqs_queue_url" \
--receipt-handle "$sqs_receipt_handle"
You only have a limited amount of time to process the message and delete it before SQS tosses it back in the queue for somebody else to process. This test queue gives you 5 minutes (VisibilityTimeout=300). If you go past this timeout, simply read the message from the queue and try again.
Cleanup
Delete the SQS queue:
aws sqs delete-queue \
--queue-url "$sqs_queue_url"
Delete the SNS topic (and all subscriptions).
aws sns delete-topic \
--region "$region" \
--topic-arn "$sns_topic_arn"
Delete test objects in the bucket:
aws s3 rm s3://$s3_bucket_name/testfile-01
aws s3 rm s3://$s3_bucket_name/testfile-02
Remove the S3 bucket notification configuration:
aws s3api put-bucket-notification \
--region "$region" \
--bucket "$s3_bucket \
--notification-configuration '{}'
Delete the bucket, but only if it was created for this test!
aws s3 rb s3://$s3_bucket_name
History / Future
If the concept of an S3 bucket notification sounds a bit familiar, it’s because AWS S3 has had it for years, but the only supported event type was “s3:ReducedRedundancyLostObject”, triggered when S3 lost an RRS object. Given the way that this feature was designed, we all assumed that Amazon would eventually add more useful events like “S3 object created”, which indeed they released a couple weeks ago.
I would continue to assume/hope that Amazon will eventually support an “S3 object deleted” event because it just makes too much sense for applications that need to keep track of the keys in a bucket.
[Update 2015-04-06: Add code to remove S3 bucket notification, which Amazon just added to aws-cli in release 18]