AWS has recently rolled out Secrets Manager in April 2018. It comes with a web console for you to easily CRUD the secrets, and it works with IAM to control who and what can access them. If you run one or more Rails apps in EC2, you can use IAM roles for EC2 to implement access control for each of the secrets.

However, I don’t find it easy and straightforward to fetch the secrets in Ruby on Rails. While it is nice for them to provide the aws-sdk-secretsmanager gem, it would be nicer if they actually put a simple tutorial online for us Rails devs.

This post will cover:

  1. How to organize your secrets
  2. Setting IAM permissions for the secrets
  3. Load the secrets into ENV in Rails

Note: The approach used in this post entails revealing the existence, but not the value of secrets used to other applications under the same AWS account. For example, if you are setting secrets for app_2/production/stripe, your app_1/production app will “know” app_2/production/stripe exists, just that it cannot find out the actual API keys used there.

Blame AWS for not letting us ListSecrets on ARNs of certain secret prefixes. If you are not comfortable with this approach, the more troublesome alternative is sekreto, which involves changing every pieces of code that call ENV[] for secrets previously to Sekreto.get_value.

Our stack

Rails 5.2. Deployed with Docker Swarm on AWS EC2. RDS with Postgres, Sidekiq with ElastiCache Redis.

Before moving to AWS Secrets Manager, we have stored secrets with Docker Swarm. Being able to make our application nodes stateless (by not keeping any secrets on it, except some non-secret configs like environment and secret prefixes), having a web console to get & set secrets rather than through SSH made us think this switch is worthwhile.

Setting your secrets

It costs $0.40/month to store a secret. Each secret stores up to 4096 Unicode characters. Technically, you can just pay only $0.40/month to store all your existing key-value secret pairs, provided you can fit them all in the chars limit. You can also pay a little more to store them in multiple JSON secrets to have them better organized.

For example, this is how I would set the secrets in AWS:

app_1/production/database: # generated by AWS from RDS
app_1/production/smtp: |
  {
    "hostname": "x",
    "username": "x",
    "password": "x"
  }
app_1/production/others: |
  {
    "stripe_api_key": "x",
    "sentry_api_key": "x",
    "secret_key_base": "abc123"
  }
# this one is plain text secret
app_1/production/ssh_key: |
  ----- BEGIN OPENSSH PRIVATE KEY -----
  redacted
  ----- END OPENSSH PRIVATE KEY -----

There are no rules saying you must name the secret keys the way I did (application_name/environment/key). But in later section you will see how dividing it up by slashes is pretty handy for IAM and telling your application which specific secrets to fetch.

Set secret read permissions with IAM

If you host your Rails app on EC2, you probably are already using IAM roles to grant access rights to S3 and other AWS services.

These are the JSON policy statements you can use to grant application permissions:

{
    "Effect": "Allow",
    "Action": [
        "secretsmanager:DescribeSecret",
        "secretsmanager:Get*"
    ],
    "Resource": [
        "arn:aws:secretsmanager:ap-southeast-1:1234567:secret:app_1/production/*"
    ]
},
{
    "Effect": "Allow",
    "Action": [
        "secretsmanager:ListSecrets"
    ],
    "Resource": [
        "*"
    ]
}

Remember to change the AWS region, AWS account ID, and secret prefix into your own in the Resource ARNs.

Load the secrets in Rails

Put this in Gemfile and run bundle:

gem 'aws-sdk-secretsmanager', '~> 1', require: false

We will need to load the secrets before the other gems are even loaded, including Rails. That is because ActiveRecord will discover your credentials from config/database.yml and ENV as soon as it’s loaded. The best place for you to fetch secrets from AWS is inside config/application.rb, just right before Bundler is ordered to load the gems.

You can copy and paste this code snippet into config/application.rb:

# config/application.rb

require_relative 'boot'

# Load env vars before Rails is loaded
require 'aws-sdk-secretsmanager'

if ENV['AWS_REGION'] && !ENV['DISABLE_AWS_SECRETS']
  secrets_prefix = ENV['AWS_SECRETS_PREFIX'] || "app_1/#{ENV['RAILS_ENV']}"
  client = Aws::SecretsManager::Client.new(region: ENV['AWS_REGION'])

  # Fetch a list of all secrets stored under this AWS account.
  # Requires action "secretsmanager:ListSecrets" for "*" in IAM.
  secrets = client.list_secrets(max_results: 100).secret_list.select do |x|
    /^#{secrets_prefix}/.match(x.name)
  end.map do |x|
    [
      /^#{secrets_prefix}\/(.*)/.match(x.name).captures[0],
      client.get_secret_value(secret_id: x.name).secret_string
    ]
  end.to_h

  # This is a hack, it assumes there are no unsafe characters in the username,
  # password and just naively concatenate the attribute values together.
  #
  # Feel free to change the database scheme.
  database_info = secrets['database'] || ENV['DATABASE_URL']
  if database_info
    begin
      db = JSON.parse(database_info)
      ENV['DATABASE_URL'] = "postgres://#{db['username']}:#{db['password']}@#{db['host']}:#{db['port']}/#{db['dbname']}"
    rescue JSON::ParserError => e
      ENV['DATABASE_URL'] = database_info
    end
  end

  # `others` is always a json of key-value pairs to be loaded to top-level in ENV
  JSON.parse(secrets['others']).each_pair do |k, v|
    ENV["#{k}".underscore.upcase] = v
    puts "Loaded env var #{k.underscore.upcase} from `others`"
  end

  # Go through other secret kv pairs in the list. Only allow 1 layer nesting.
  # Assumes most secrets are stored in proper JSON format, if they aren't,
  # fetch as multiline strings.
  secrets.except('database', 'database_url', 'others').each_pair do |k, v|
    begin
      subsecrets = JSON.parse(v)
      subsecrets.each_pair do |kk, vv|
        ENV["#{k}_#{kk}".underscore.upcase] = vv
        puts "Loaded env var #{"#{k}_#{kk}".underscore.upcase}"
      end
    rescue JSON::ParserError => e
      ENV[k.underscore.upcase] = v
      puts "Loaded env var #{k.underscore.upcase}"
    end
  end

elsif ENV['DISABLE_AWS_SECRETS']
  puts "DISABLE_AWS_SECRETS has been set. Secrets will not be loaded from AWS."

elsif !ENV['AWS_REGION']
  puts "AWS_REGION not set. Secrets will not be loaded from AWS."

end

require 'rails/all'

# Require the gems listed in Gemfile, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(*Rails.groups)

# ...redacted

As you can see, there are certain environment variables that you will need to set outside AWS Secrets Manager. As every project has different deployment scenario (we use Docker Swarm), I am not going through the details on how to set them.

If you copy and paste my code snippet, you will need to set the following ENV vars:

  • AWS_REGION - for example: ap-southeast-1

Optionally:

  • AWS_SECRETS_PREFIX - this can be set to override the default “app_name/environment”
  • DISABLE_AWS_SECRETS - if this is set, it will skip the entire code block and not fetch secrets from AWS. Great for development.

Closing

I hope this post has helped you out. If you got any questions or have found any issues with this post, please get in touch with me on Telegram.