How to safely use sensitive data at EC2 launch with Secret Manager?

Photo by Nicolas HIPPERT on Unsplash

Originally published at https://filip5114.github.io on August 6, 2021.

Recently, I had a task to automate the startup of custom systemd services when a new EC2 is launched. The user data script is a great way to make any necessary changes in the system at EC2 launch. User data script is run as root, gives you pretty much all permissions you need.

The only problem I encountered was with one of the systemd services, which required a password to be started. Since user data script can be retrieved from launched EC2 and the result of the script is stored in logs, it’s against security best practices to put password as plain text in the script. The problem can be solved with AWS Secret Manager and AWS CLI command to retrieve secrets.

Agenda

1. Create Secret in Secret Manger
2. Create EC2 Instance Role with Secret Read Access policy
3. Create EC2 instance and retrieve secret value at launch
4. Make it useful
5. Summary

Create a secret in Secret Manager

Start with creating a secret which we will later on for EC2 instance at launch. Secret Manager -> Store a new secret Then choose type Other type of secrets and specify key/value for new secret.

New Secret

Click Next, then specify the name for secret and click Next again. Now you can configure automatic rotation. It's not necessary for this scenario, therefore leave default and click Next. Review details and click Store. You will be taken to a screen showing all secrets, in my case, it is just the one created now.

List of stored secrets

Create EC2 Instance Role with Secret Read Access policy

We already have secret safely stored within AWS Secret Manager. It can’t be simply read by any resource. EC2 instance will not be able to read secret unless you will add read permissions to the instance role.

Go to IAM -> Roles -> Create Role. Choose EC2 and click Next: Permissions. Then click Create Policy, it will open in a separate web browser tab. Click on the tab JSON and paste the policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "secretsmanager:GetSecretValue"
            ],
            "Resource": "arn:aws:secretsmanager:us-east-1:215538436894:secret:MySecret-u7eM2x",
            "Effect": "Allow"
        }    
    ]
}

Remember to replace resource’s ARN with your secret’s ARN.

Click Next: Tags -> Next: Review, name the policy SecretReadAccess and hit Create Policy.

Go back to the web browser tab with role creation process. Hit refresh arrows, search for SecretReadAccess, mark it and click Next: Tags -> Next: Review, name it EC2InstanceRole and hit Create role.

Choose newly create role

Create EC2 instance and retrieve secret value at launch

Alright! We have everything prepared to launch the instance and safely retrieve secret at launch using user-data.

Go to EC2 -> Instances -> Launch instances. Choose Amazon Linux 2 AMI, then t2.micro instance type and go to Configuration Instance Details.

Make sure that Auto-assign Public IP is enabled and choose the newly created role in IAM role field.

Auto-assign Public IP and IAM role configuration

Scroll to the bottom where Advanced Details can be found and locate User data field. User data is a script which is executed at launch of EC2. It's commonly used to install any necessary software and sometimes you may want to use passwords, keys, or any other sensitive data in user data. You should never put sensitive data directly in user data since it can be visible in logs. The solution is to store sensitive data in Secret Manager and retrieve the secret value in user data.
Copy the script into user data. It will retrieve key/values from the secret.

#!/bin/bash
aws secretsmanager get-secret-value --region us-east-1 --secret-id MySecret

Remember to replace region and secret-id parameters with your values.

Click Review and Launch -> Launch (add new key pair if you don't have an existing one). Go to the EC2 -> Instances and wait until the instance will be in Running state. Now login to the instance. Click on instance, Connect -> EC2 Instant Connect -> Connect. The logs of user data script can be found in /var/log/cloud-init-output.log

Secret retrieved

As you can see, we have successfully retrieved secret from Secret Manager. We have only printed the secret to prove that it can be done. Real world scenario would rather only use the secret, so it wouldn’t be printed in logs.

Make it useful

Although we have successfully retrieved secret, it’s not really useful in raw format. The most interesting part of the secret is value from key/value pair. I’ll show you few user data scripts which will make more out of the secret retrieval.

Get only secret string as text

#!/bin/bash
aws secretsmanager get-secret-value --region us-east-1 --secret-id MySecret --query SecretString --output text

Additional options --query and --output allows to only retrieve SecretString instead of all data returned by AWS CLI.

Secret string

Get value of key/value pair

#!/bin/bash
yum update -y
yum install -y jq
aws secretsmanager get-secret-value --region us-east-1 --secret-id MySecret --query SecretString --output text | jq -r .mySecretKey

Unfortunately, there is no easy way to extract value from key/value pairs in pure bash. You can always write a script which will achieve it, but it is much simpler to use . We use yum to update repositories and install jq. Then jq can be used on the secret to retrieve value from key/value pair by specifying the key.

Value of key/value pair

Get value of key/value pair and store it in environment variable

#!/bin/bash
yum update -y
yum install -y jq
myValue=$(aws secretsmanager get-secret-value --region us-east-1 --secret-id MySecret --query SecretString --output text | jq -r .mySecretKey)
echo $myValue > /var/log/echoSecret.txt

Until now, we have simply printed the retrieved secret. This time, the secret is stored in an environment variable and will not be visible in the script’s log. To check that in fact environment variable has the desired value, I added command to echo the variable into /var/log/echoSecret.txt.

Logs without secret printed

Seceret stored in file

Summary

Today we have successfully created a new secret in AWS Secret Manager, granted secrets read access for EC2 instance, and read the secret at EC2 launch with user data script. The approach can be useful for using sensitive data at EC2 launch, for example — password/key for Linux systemd services.

Thanks for reading!

More content at plainenglish.io