I was in the audience when Amazon announced the AWS Secrets Manager at the AWS Summit San Francisco. My first thought was that we already have a way to store secrets in SSM Parameter Store. In fact, I tweeted:
“Just as we were all working out the details of using SSM Parameter Store to manage our secrets…
Another tool to help build securely (looking forward to learning about it).
#AWSsummit San Francisco” – @esh 2018-04-04 11:10
When I learned the AWS Secrets Manager pricing plan, my questions deepened.
So, I started poring over the AWS Secrets Manager documentation and slowly strarted to gain possible enlightenment.
I have archived below three stream of consciousness threads that I originally posted to Twitter.
After reading the new AWS Secrets Manager docs, it looks like there is a lot of value in the work Amazon has invested into the design of rotating secrets.
There are a number of different ways systems support secrets, and various failure scenarios that must be accounted for.
Though RDS secret rotation support is built in to AWS Secrets Manager, customers are going to find more value in the ability to plug in custom code to rotate secrets in any service–using AWS Lambda, naturally.
Customers write the code that performs the proper steps, and AWS Secrets Manager will drive the steps.
It almost looks like we could take the secret rotation framework design and develop AWS Step Functions and CloudWatch Events Schedule to drive rotation for secrets in SSM Parameter Store,
but for such a critical piece of security infrastructure execution, it makes sense to lean on Amazon to maintain this part and drive the rotation steps reliably.
There are ways to create IAM policies that are fine-tuned with just the AWS Secrets Manager permissions needed, including conditions on custom tags on each secret.
When designing AWS Secrets Manager, I suspect there were discussions inside Amazon about whether ASM itself should perform the steps to move the current/pending/previous version labels around during secret rotation, to reduce the risk of customer code doing this incorrectly.
I think this may have required giving the ASM service too much permission to manipulate the customer’s secrets, so the decision seems right to keep this with the customer’s AWS Lambda function, even though there is some added complexity in development.
The AWS Secrets Manager documentation is impressively clear for creating custom AWS Lambda functions for secret rotation, especially for how complex the various scenarios can be.
Here’s a link to the AWS Secrets Manager Use Guide. https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html
Thoughts 2: Rotating Secrets Frequently Is Important
I’m starting to understand that it’s “AWS Secrets Manager” not “AWS Secrets Store”, and that the biggest part of that management seem to be the automated, transparent, reliable secret rotation.
Now that I can see a smooth path to regular secret rotation with AWS Secrets Manager, I’m starting to feel like I’ve been living in primitive times letting my database passwords sit at the same value for months (ok, years).
AWS Secrets Manager documentation shows an example of rotating a password every 10 days, but now that I’ve seen the light, I’m thinking, “Why not every hour?!”
In fact, why not rotate the secret using AWS Secret Manager shortly after the longest time we expect it to take to use that secret?
If a database client is likely to take less than 30 seconds to connect to the server after requesting the password, we could rotate that secret every minute (using the two-user method described in the AWS Secret Manager docs).
This moves in the direction of single-use secrets that are handed out when a client has been authorized with the current IAM permissions.
I can imagine that some third party services might not appreciate credential/token rotation requests at super-frequent rates.
The ideal would be to have every service authenticate clients/requests live with IAM directly, but until then, frequent secret rotation with AWS Security Manager seems worth investigating.
Writing AWS Lambda “Rotation Functions” to perform secret rotation with AWS Secrets Manager is straight-forward, but not trivial, and any code related to security needs to be clean, reliable, and trustworthy.
Each integration with AWS Secrets Manager will need to deal with the rotation “scenario” (as described in the docs), the method of changing secrets (database connection, API, web session), and the peculiarities of how the secrets are used with that product or service.
There are going to be a lot of different products and services that need these written.
There are a lot of AWS Secrets Manager customers who will need these integrations.
There doesn’t need to be one implementation per AWS Secrets Manager customer per product/service.
A well written AWS Lambda Rotation Function for AWS Secrets Manager should be usable by any AWS customer who uses the same product or service.
No secrets or account information should be stored in the AWS Lambda Rotation Functions themselves, but would reside in the AWS Secrets Manager storage.
We need a way for developers to publish high quality AWS Secrets Manager Lambda Rotation Functions, and for AWS Secrets Manager customers to find and use them.
Could the AWS Serverless Application Repository be used to host and share AWS Lambda Rotation Functions for the AWS Secrets Manager?
I have concerns about the visibility/auditability of the AWS Lambda Rotation Function code in the SAR, especially when it comes to security code that will be provided customer secrets, but perhaps these can be addressed somehow.
I have not used the AWS Secrets Manager yet, so all of these thoughts should be taken as musings by somebody who is not familiar with the product except through reading the documentation. Feedback and corrections are welcomed in the comments here or on Twitter.