Support `rain deploy` sourcing parameters & tags from environment variables

[rtkjbeverly]

Summary

This issue proposes the implementation of a feature for adding support for RAIN_VAR_ and rain_TAG_ environment variable prefixes to be leveraged for auto-populating arguments for the command line, similar to how TF_VAR_ and TF_AWS_DEFAULT_TAGS_ are supported in Terraform and the AWS Provider.

Motivation

In cloud-first environments, there is a strong emphasis on resource tagging and per-user or per-environment configurations. Terraform's standardized environment variable prefixes (TF_VAR_ for variables, TF_AWS_DEFAULT_TAGS_ for default tags) make it easier for users to inject configuration and tags by cross-cutting concerns, like these values being configured by default within a .devcontainer, or the tags populated by some automated tooling.

Rain could provide a similar user experience for CloudFormation deployments by supporting RAIN_VAR_ (for parameters) and RAIN_TAG_ (for resource tags) environment variables.

Note

This doesn't have to be the prefix RAIN_VAR_, could be CF_VAR_ or anything else.

Feature

• RAIN_VAR_: Any environment variable prefixed with RAIN_VAR_ would be automatically interpreted as a CloudFormation parameter. Example: RAIN_VAR_MyParam=foo would set the parameter MyParam to foo.
• RAIN_TAG_: Any environment variable prefixed with RAIN_TAG_ would be automatically collected and attached as a tag to deployed resources. Example: RAIN_TAG_Environment=dev would result in a resource tag Environment=dev on all applicable resources.
• Any arguments passed to the command line (--params, --tags) would override the environment variable value
• The environment variable behaviour could be default & opt-out (--no-source-env), opt-in (--env-from) or baseline

Users would then be able to have the following work to pass parameters:

# Set CloudFormation parameters 'MyParam1' and 'MyParam2'
export RAIN_VAR_MyParam1=value1
export RAIN_VAR_MyParam2=value2

# Set CloudFormation tags 'Environment' and 'Owner'
export RAIN_TAG_Environment=dev
export RAIN_TAG_Owner=$USER

# Deploy with Rain (environment variables are automatically picked up)
rain deploy my-stack template.yaml

# Equivalent command:
# rain deploy my-stack template.yaml \
#   --params MyParam1=value1 MyParam2=value2 \
#   --tags Environment=dev Owner=alice

Use Cases

• CI/CD pipelines that standardize tagging and configuration through environment variables, allowing you to deploy any template and just set the environment variables to pass along tags
• Developers deploying resources in shared environments who need to inject per-user or per-project tags and parameters (ex. Owner, Custodian, Developer, Timezone, etc)
• Scripts abstractions don't require explicit handling of all tagging & parameters, and can instead provide things as needed

References

• Terraform: Input Variables via Environment Variables
• Terraform AWS Provider: Default Tags

[rtkjbeverly]

I wanted to expand on the use cases to demonstrate how this might be applied, especially when working with services/apps that are Cloud-First (e.g. built in AWS for each commit basically), just to help understand what I'm thinking here.

Devcontainers

So a number of the repositories in our organization use devcontainers for managing the developer build environments, which means its possible for us to auto-configure Terraform IaC execution to always have the default tags set. So any terraform apply that someone is running in these environments, will have the proper tagging setup as we'll have those as just environment variables in their development tools.

The same applies with this. As it'd allow setting tags within the devcontainer that would apply to all deployments. A simple example is included below that allows an Owner tag to be sourced from the $USER variable (typically firstname.lastname), and a Timezone tag to assist with tools that scale down systems outside of office hours:

{
  "name": "MyWorkspace",
  "image": "mcr.microsoft.com/devcontainers/base:ubuntu",
  "postCreateCommand": "./.devcontainer/on-create.sh",
  "remoteEnv": {
      "RAIN_TAGS_Owner": "${localEnv:USER}",
      "RAIN_TAGS_Timezone": "${localEnv:TZ}"
    },
}

Or this may be setup by some on startup command in the container that pulls the metadata at start to support ensuring everything has the tags:

#!/usr/bin/env bash

echo "export RAIN_VAR_Project=rain-demo"   >> ~/.bashrc
echo "export RAIN_TAG_Environment=dev"     >> ~/.bashrc

Organization/Grouping Parameters

Another case that it can be super helpful with is when you are needing to generally default values from some set of known "preset" values. Almost like an enumeration but something that can't be represented as a static list (e.g. list of scopes, list of instance types for profile sizes, etc).

As an example you want to align your deployment with a specific scope (e.g. shared instance/resource, EKS Cluster, etc). For these cases at present you basically write scripts that write out these parameters (e.g. --params Environment=... or --params Target=....), possibly with a --ignore-unknown-params to avoid breaking if not present, then have a bunch of glue scripts for your deployments.

However if the parameters are set by some tool that just calculates based on your context, or what you are trying to do, then you just inherit all of the things you need, and can entirely focus on passing parameters that may alter the behaviour of your system. Like a system flag or tuning parameters for how it behaves in AWS.

For this case, the benefit really comes down to removing the layer of a lot of "glue scripts" in a repository that exist in the developer workflow (e.g. [build => glue script => deploy]), allowing it to instead be [ build => deploy ] as the [glue script] just ran at start.

Example of this implemented as helper

This is just a simple example demonstrating how one can make this work as a helper script, that can pass the resultant json (after writing to file) as the --config param.

#!/usr/bin/env bash
set -Eeuo pipefail

tags=$(env | (grep '^CFN_TAG_' || true) | awk -F= '{print substr($1,9) "=\"" $2 "\""}' | sort)
params=$(env | (grep '^CFN_VAR_' || true) | awk -F= '{print substr($1,9) "=\"" $2 "\""}' | sort)

if [[ -z "$tags" ]]; then
  tags_json='{}'
else
  tags_json=$(printf '%s\n' "$tags" | jq -Rn '
    def parse_lines:
      [inputs
        | capture("^(?<key>[^=]+)=\"(?<val>.*)\"$")
        | {(.key): .val}
      ] | add;
    parse_lines'
  )
fi

if [[ -z "$params" ]]; then
  params_json='{}'
else
  params_json=$(printf '%s\n' "$params" | jq -Rn '
    def parse_lines:
      [inputs
        | capture("^(?<key>[^=]+)=\"(?<val>.*)\"$")
        | {(.key): .val}
      ] | add;
    parse_lines'
  )
fi
jq -n --argjson params "$params_json" --argjson tags "$tags_json" '{Parameters: $params, Tags: $tags}'

Which when run looks something like this:

➜ bash CFN_VAR_
{
  "Parameters": {},
  "Tags": {}
}
➜ export CFN_VAR_MyThing="hello"
➜ export CFN_TAG_Environment="world"
➜ bash CFN_VAR_                           
{
  "Parameters": {
    "MyThing": "hello"
  },
  "Tags": {
    "Environment": "world"
  }
}

[rtkjbeverly]

Finally to close out after that info-dump, this is something that I'd be more than happy to contribute a submission to support the feature. Along with any docs/tests/etc that'd be needed for such a feature.