Release notes

2024.07

Upgrade instructions:

  1. substrate upgrade to upgrade your local copy of the Substrate binary.
  2. substrate setup to upgrade AWS Organizations and IAM, your networks, and your Intranet.
  3. Have everyone on your team run substrate upgrade, too.

2024.06

The theme of Substrate 2024.06 is to begin preparing organizations to optionally/additionally use AWS IAM Identity Center. This support will land fully over the next couple of releases.

Upgrade instructions:

  1. substrate upgrade to upgrade your local copy of the Substrate binary.
  2. substrate setup to upgrade AWS Organizations and IAM, your networks, and your Intranet.
  3. Have everyone on your team run substrate upgrade, too.

2024.05

Upgrade instructions:

  1. substrate upgrade to upgrade your local copy of the Substrate binary.
  2. substrate setup to upgrade AWS Organizations and IAM, your networks, and your Intranet.
  3. Have everyone on your team run substrate upgrade, too.

2024.04

Upgrade instructions:

  1. substrate upgrade to upgrade your local copy of the Substrate binary.
  2. substrate setup to upgrade AWS Organizations and IAM, your networks, and your Intranet.
  3. Have everyone on your team run substrate upgrade, too.

2024.03

Note well: Upgrading to Substrate 2024.03 is only supported from Substrate 2024.02.

Upgrade instructions:

  1. substrate upgrade to upgrade your local copy of the Substrate binary.
  2. substrate setup to upgrade AWS Organizations and IAM, your networks, and your Intranet.
  3. Have everyone on your team run substrate upgrade, too.

2024.02

Note about the future: You will not be able to upgrade to Substrate 2024.03 without upgrading to Substrate 2024.02 first.

Upgrade instructions:

  1. substrate upgrade to upgrade your local copy of the Substrate binary.
  2. substrate setup to upgrade AWS Organizations and IAM, your networks, and your Intranet.
  3. Have everyone on your team run substrate upgrade, too.

2024.01

Upgrade instructions:

  1. substrate upgrade to upgrade your local copy of the Substrate binary.
  2. substrate setup to upgrade your Intranet and the basic IAM roles Substrate manages.
  3. Have everyone on your team run substrate upgrade, too.

2023.12

If you added your own internal tools to your Intranet, be sure to follow the updated documentation on protecting internal tools before upgrading to avoid an outage to your internal tools. You can remove the old version after upgrading.

Upgrade Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, it will be upgraded in place. Then run substrate setup to upgrade your Intranet.

2023.11

Upgrade Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.

The 2023.11 release contains a preview of the next generation of the Substrate-managed Intranet. It’s powered by AWS API Gateway v2 which, among other advantages over v1, makes it much, much more straightfoward to wrap and proxy other arbitrary services while transparently and comprehensively authenticating and authorizing traffic using your identity provider. After you run substrate setup, the preview will be available on https://preview.example.com (replacing “example.com” with your Intranet DNS domain name).

If you’ve added any of your internal tools to your Intranet, you’ll need to migrate them to the new style of integration before upgrading to Substrate 2023.12. See the forward-looking half of the documentation on protecting internal tools for the details.

Substrate 2023.12 will swap the new Intranet into place on your Intranet DNS domain name and remove both the old Intranet and the preview subdomain.

2023.10

Upgrade Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.

The 2023.10 release contains an early preview of the next generation of the Substrate-managed Intranet. It’s powered by AWS API Gateway v2 which, among other advantages over v1, makes it much, much more straightfoward to wrap and proxy other arbitrary services while transparently and comprehensively authenticating and authorizing traffic using your identity provider. Opt into the preview and see what’s coming (on a separate hostname so that there’s no impact on your existing Intranet) like this:

SUBSTRATE_FEATURES=APIGatewayV2 substrate setup

The transition schedule from the original implementation of the Intranet to this new one is as follows:

  1. Substrate 2023.11 will serve the new Intranet on https://preview.example.com (replacing “example.com” with your Intranet DNS domain name).
  2. Substrate 2023.12 will swap the new Intranet into place on your Intranet DNS domain name and remove the old one.

If you wish to begin transitioning any custom internal tools you’ve attached to your Intranet, you may add route keys to the Substrate API with the Substrate authorizer. Full documentation of all the expanded possibilities for proxying to internal tools is forthcoming.

2023.09

The only supported upgrade path to Substrate 2023.09 is directly from 2023.08. If you need to make a catch-up upgrade, please upgrade to 2023.08 first and then upgrade to 2023.09 immediately after.

Upgrade Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.

After upgrading Substrate:

  1. Upgrade Terraform to version 1.5.6 by running substrate terraform, downloading from Hashicorp, or some other means of your choice.
  2. Run substrate setup to update your Intranet.
  3. Run substrate setup-cloudtrail to create the new AuditAdministrator role in the audit account.

In August, Hashicorp announced a license change for future releases of their products, including Terraform. Since we’ve gotten questions from a few customers, we thought it best to address this change in these release notes, especially given we’re pushing an upgrade to a Terraform version covered by this new license. To the best of our knowledge, every Substrate customer’s use of Terraform falls comfortably within the Additional Use Grant in Hashicorp’s Business Source License that covers production use that does not compete with Hashicorp’s products. Therefore no action is necessary for any Substrate customer.

2023.08

Substrate 2023.08 is a major release. The marquee feature is the introduction of 12-hour AWS Console sessions from the Intranet’s Accounts page but there’s much, much more happening behind the scenes. Because of the complexity of the changes in this release, we’re declaring ahead of time that you may not skip over this release in a catch-up upgrade. Next month’s release will rely on the fact that all upgrades to 2023.09 will be coming from 2023.08.

Upgrade your local copy of Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, it will be upgraded in place.

After upgrading your local copy of Substrate, run substrate setup. This will turn your admin account into your Substrate account and ensure the entire matrix of IAM policies, roles, and users are up-to-date. It will run Terraform against your deploy, network, and Substrate accounts; you may add -auto-approve to make these non-interactive.

It is still good practice to run sh <(substrate accounts -format shell) or sh <(substrate accounts -format shell -no-apply) followed by sh <(substrate accounts -auto-approve -format shell) to ensure that not only are all your service accounts in good standing in terms of the Substrate-managed IAM policies and roles but that all your Terraform code is working, too. However, as this is often a very onerous part of upgrading Substrate, we will endeavor from now on not to require you to apply Terraform changes in service accounts as part of Substrate upgrades.

2023.07

Upgrade Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.

After upgrading Substrate, run substrate create-admin-account -quality <quality> to upgrade the Intranet. This is all that’s required but, for good measure, you should run sh <(substrate accounts -format shell -no-apply), review what Terraform plans to do, and then run sh <(substrate accounts -auto-approve -format shell) to apply the changes.

2023.06

Upgrade Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.

After upgrading Substrate, run substrate create-admin-account -quality <quality> to upgrade the Intranet. This is all that’s required but, for good measure, you should run sh <(substrate accounts -format shell -no-apply), review what Terraform plans to do, and then run sh <(substrate accounts -auto-approve -format shell) to apply the changes.

2023.05

Before upgrading Substrate, if you use Okta as your identity provider, add the okta.users.read.self scope to your Intranet application and the AWS_RoleName attribute to each of your users by integrating your Okta identity provider again. If you use a different identity provider, no action is necessary.

Upgrade Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.

After upgrading Substrate, run substrate create-admin-account -quality <quality> to upgrade the Intranet. This is all that’s required but, for good measure, you should run sh <(substrate accounts -format shell -no-apply), review what Terraform plans to do, and then run sh <(substrate accounts -auto-approve -format shell) to apply the changes.

2023.04

Upgrade Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.

After upgrading Substrate, run substrate create-admin-account -quality <quality> to upgrade the Intranet. This is all that’s required but, for good measure, you should run sh <(substrate accounts -format shell -no-apply), review what Terraform plans to do, and then run sh <(substrate accounts -auto-approve -format shell) to apply the changes.

2023.03

Upgrade Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.

After upgrading Substrate, you should run sh <(substrate accounts -format shell -no-apply). This Substrate upgrade does not require any Terraform changes to be applied so whether and when to do so is left to you.

This month, Substrate’s documentation and release notes are moving to https://docs.src-bin.com/substrate/. Old URLs will redirect to their equivalent on the new subdomain. And, for your troubles, we’re pleased to now offer search over the Substrate documentation! Going forward, the new canonical URL for the release notes is http://docs.src-bin.com/substrate/releases.

2023.02

Upgrade Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.

After upgrading Substrate, you should run sh <(substrate accounts -format shell -no-apply), review what Terraform plans to do, and then run sh <(substrate accounts -auto-approve -format shell) to apply the changes. Commit the new substrate.enforce-.htmlsv2 and substrate.manage-cloudtrail files to version control.

2023.01

Upgrade Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.

After upgrading Substrate:

  1. Upgrade Terraform to version 1.3.6 by running substrate terraform, downloading from Hashicorp, or some other means of your choice.
  2. Run sh <(substrate accounts -format shell -no-apply) to review what Terraform plans to do.
  3. Run sh <(substrate accounts -auto-approve -format shell) to apply the changes.

2022.12

Upgrade Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.

After upgrading Substrate, you should run sh <(substrate accounts -format shell -no-apply), review what Terraform plans to do, and then run sh <(substrate accounts -auto-approve -format shell) to apply the changes.

2022.11

Upgrade Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.

After upgrading Substrate, you should run sh <(substrate accounts -format shell -no-apply), review what Terraform plans to do, and then run sh <(substrate accounts -auto-approve -format shell) to apply the changes.

2022.10

If you wish to instantiate the new common modules in your existing service accounts, take the following steps for each domain:

  1. Add the following block to modules/domain/global/main.tf:

     module "common" {
       providers = {
         aws           = aws
         aws.us-east-1 = aws.us-east-1
       }
       source = "../../common/global"
     }
    
  2. Add the following block to modules/domain/regional/main.tf:

     module "common" {
       providers = {
         aws         = aws
         aws.network = aws.network
       }
       source = "../../common/regional"
     }
    
  3. Run substrate create-account -domain`` ``domain``-environment``environment``-quality``quality for each domain service account.

These modules aren’t being instantiated in existing service accounts automatically because Substrate can’t guarantee that’s safe.

Upgrade Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.

After upgrading Substrate, you should run sh <(substrate accounts -format shell -no-apply), review what Terraform plans to do, and then run sh <(substrate accounts -auto-approve -format shell) to apply the changes.

2022.09

Get the 2022.09 release by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.

After upgrading Substrate, you at least need to run substrate create-admin-account -quality`` ``quality to update your Intranet. Even better would be to run sh <(substrate accounts -format shell -no-apply), review what Terraform plans to do, and then run sh <(substrate accounts -auto-approve -format shell) to apply the changes.

2022.08

Upgrade Substrate as in the updated installation manual:

tar xf substrate-version-commit-OS-ARCH.tar.gz -C ~/bin --strip-components 2 substrate-version-commit-OS-ARCH/bin/substrate

Each released version and commit is offered in four binary formats; choose the appropriate one for your system. OS is one of “darwin” or “linux” and ARCH is one of “.html64” or “arm64”.

You can install Substrate wherever you like. If ~/bin doesn’t suit you, just ensure the directory where you install it is on your PATH.

After upgrading Substrate, the best idea is to run sh <(substrate accounts -format shell -no-apply), review what Terraform proposes to do, and then run sh <(substrate accounts -auto-approve -format shell) to ensure your code and your AWS organization don’t diverge. If you need a minimal upgrade process, it’s substrate create-admin-account -quality`` ``quality to update your Intranet.

Advance notice of an upcoming change: Next month’s release will delete an old EC2 security group that was used by the Instance Factory until late 2021. Beware that, if you have any Instance Factory instances from 2021 or earlier, you’ll have to change their security group or terminate them before upgrading next month.

2022.07

Upgrade Substrate as in the updated installation manual:

tar xf substrate-version-commit-OS-ARCH.tar.gz -C ~/bin --strip-components 2 substrate-version-commit-OS-ARCH/bin/substrate

Each released version and commit is offered in four binary formats; choose the appropriate one for your system. OS is one of “darwin” or “linux” and ARCH is one of “.html64” or “arm64”.

You can install Substrate wherever you like. If ~/bin doesn’t suit you, just ensure the directory where you install it is on your PATH.

After upgrading Substrate:

  1. substrate bootstrap-management-account
  2. substrate bootstrap-network-account
  3. substrate bootstrap-deploy-account
  4. substrate create-admin-account -quality`` ``quality for each of your admin accounts
  5. substrate create-account -domain`` ``domain``-environment``environment``-quality``quality for each of your service accounts

If your shell supports process substitution, you can run sh <(substrate accounts -format shell) to run all of these, in the proper order, in one command. As of this release you can make this non-interactive as sh <(substrate accounts -auto-approve -format shell) but this is not recommended as it forgoes your opportunity to object before Terraform applies changes.

2022.06

After upgrading Substrate:

  1. Upgrade to Terraform 1.2.3
  2. substrate bootstrap-management-account
  3. substrate bootstrap-network-account
  4. substrate bootstrap-deploy-account
  5. substrate create-admin-account -quality`` ``quality for each of your admin accounts
  6. substrate create-account -domain`` ``domain``-environment``environment``-quality``quality for each of your service accounts

If your shell supports process substitution, you can upgrade Terraform and then run sh <(substrate accounts -format shell) to run all of these, in the proper order, in one command.

2022.05

After upgrading Substrate:

  1. substrate bootstrap-management-account
  2. substrate create-admin-account -quality`` ``quality for each of your admin accounts

2022.04

After upgrading Substrate:

  1. Configure Substrate shell completion
  2. substrate bootstrap-management-account
  3. substrate create-admin-account -quality`` ``quality for each of your admin accounts

2022.03

After upgrading Substrate:

  1. substrate bootstrap-deploy-account
  2. substrate-create-admin-account -quality="..." for each of your admin accounts

2022.02

You must upgrade to Terraform 1.1.6 in order to use Substrate 2022.02. Terraform 1.1.6 may be found here:

After upgrading Substrate, do the following to land the Terraform upgrade and remove the SubstrateVersion tags:

  1. substrate-bootstrap-network-account
  2. substrate-bootstrap-deploy-account
  3. substrate-create-admin-account -quality="..." for each of your admin accounts
  4. substrate-create-account -domain="..." -environment="..." -quality="..." for each of your service accounts

2022.01

After upgrading Substrate:

  1. substrate-create-admin-account -quality="..."
  2. Upgrade Substrate in your Instance Factory instances, if you install it there

2021.12

The upgrade process this month is much more involved that most. As such, we’ll talk in Slack about when you’re going to perform the upgrade to ensure support’s available in the moment.

Before upgrading Substrate, audit your Terraform modules for resources in global modules that aren’t from global AWS serivces by copying the following program to audit.sh in your Substrate repository and running sh audit.sh.

set -e

substrate root-modules |
grep "/global\$" |
while read DIRNAME
do
    echo "$DIRNAME" >&2
    terraform -chdir="$DIRNAME" state pull >"$DIRNAME/audit.tfstate"
    grep -F ":us-east-1:" "$DIRNAME/audit.tfstate" || :
    rm -f "$DIRNAME/audit.tfstate"
    echo >&2
done

Every resource this program identifies needs to be modified before proceeding. The most likely modification is to add provider = aws.us-east-1 to resources in the Terraform code that manages them.

Block all your coworkers from making Terraform changes however you usually do (announcing in Slack, deactivating CI/CD jobs, taking state file locks, etc.) and move your global state files from us-east-1 to your default region by copying the following program to mv-state.sh in your Substrate repository and running sh mv-state.sh.

set -e

DEFAULT_REGION="$(cat "substrate.default-region")"
PREFIX="$(cat "substrate.prefix")"

if [ "$DEFAULT_REGION" = "us-east-1" ]
then exit # nothing to do
fi

eval $(substrate-assume-role -role="DeployAdministrator" -special="deploy")

substrate root-modules |
grep "/global\$" |
while read DIRNAME
do
    echo "$DIRNAME" >&2
    aws s3 cp "s3://$PREFIX-terraform-state-us-east-1/$DIRNAME/terraform.tfstate" "s3://$PREFIX-terraform-state-$DEFAULT_REGION/$DIRNAME/terraform.tfstate"
    aws s3 rm "s3://$PREFIX-terraform-state-us-east-1/$DIRNAME/terraform.tfstate"
    echo >&2
done

Once you’ve run this program, there’s a provider to thread through the tree of Terraform modules before you can upgrade to Substrate 2021.12.

(I regret not being able to provide a patch(1) file for these operations but the contents of versions.tf post-Terraform 1.0 are too unpredictable to do so safely.)

Now you can upgrade Substrate. Don’t release your block just yet, though.

After upgrading Substrate:

  1. substrate-bootstrap-deploy-account
  2. substrate-create-admin-account -quality="..." for each of your admin accounts
  3. substrate-create-account -domain="..." -environment="..." -quality="..." for each of your service accounts

Once all of these have run successfully, ensure all your coworkers upgrade Substrate and unblock Terraform changes.

I regret the complexity of this upgrade process but feel it is, on balance, less risky than attempting to hide all this motion behind automation. Thanks for your patience.

2021.11

Before upgrading Substrate, if you’re using Google as your IdP:

  1. Add an additional custom attribute as follows:
    1. Visit https://admin.google.com/ac/customschema in a browser (or visit https://admin.google.com, click Users, click More, and click Manage custom attributes)
    2. Click the AWS section
    3. In the blank bottom row, enter “RoleName” for Name, select “Text” for Info type, select “Visible to user and admin” for Visibility, select “Single Value” for No. of values
    4. Click SAVE
  2. Visit https://admin.google.com/ac/users and set the RoleName attribute in the AWS category to “Administrator” for every user authorized to use AWS.
  3. Visit https://console.cloud.google.com/apis/library/admin.googleapis.com, confirm the selected project is the one that contains your Intranet’s OAuth OIDC configuration (its name will be listed next to “Google Cloud Platform” in the header), and click ENABLE.

After upgrading Substrate:

  1. Run substrate-create-admin-account -quality="..." to upgrade your Intranet.

2021.10

After upgrading Substrate:

  1. Run substrate-bootstrap-deploy-account to fix the bucket policy so that all authorized principals in the organization can upload to the deploy artifact bucket(s).
  2. Run substrate-create-admin-account -quality="..." to upgrade your Intranet and Auditor roles. Note well this will produce a fair number of new resources; this is step one in a multi-month process of brining some naming consistency to Substrate-managed resources in IAM, Lambda, and other AWS services.

2021.09.28

If you’re upgrading from 2021.08, follow the upgrade instructions from 2021.09. If you already upgraded to 2021.09, there are no further upgrade steps.

2021.09

This release changes the interactive interface to substrate-bootstrap-network-account and substrate-create-admin-account to make them easier to run in CI. If you are automating these commands by providing yes and no answers on standard input, this release will break your automation; you should run these commands interactively first to see what’s changed. The details of what’s changed are listed in the usual format below.

After upgrading Substrate:

  1. Run substrate-bootstrap-management-account to update your organization’s Service Control Policy.
  2. Run substrate-bootstrap-deploy-account to reconfigure the deploy buckets in S3 and generate the global root module.
  3. Run substrate-create-admin-account -quality="..." to add the e-mail address column to your Intranet’s /accounts page.

2021.08

After upgrading Substrate:

  1. Run substrate-bootstrap-management-account to grant substrate-whoami the permissions it needs.
  2. Run substrate-bootstrap-network-account to remove coarse-grained organization-wide VPC sharing.
  3. Run substrate-create-admin-account -quality="..." to upgrade your Intranet.

2021.07

You must upgrade to Terraform 1.0.2 in order to use Substrate 2021.07. Terraform 1.0.2 may be found here:

After upgrading Terraform and Substrate:

  1. Run substrate-bootstrap-network-account and substrate-bootstrap-deploy-account to complete the Terraform 1.0.2 upgrade there. Note well that tags and tags_all output will be somewhat confusing but will ultimately do the right thing.
  2. Run substrate-create-admin-account and substrate-create-account to complete the Terraform 1.0.2 upgrade for each of your admin and service accounts. Here, too, note well that tags and tags_all output will be somewhat confusing but will ultimately do the right thing.

2021.06

You must upgrade to Terraform 0.15.5 in order to use Substrate 2021.06. Terraform 0.15.5 may be found here:

After upgrading Terraform and Substrate:

  1. Run substrate-bootstrap-network-account and substrate-bootstrap-deploy-account to complete the Terraform 0.15.5 upgrade there.
  2. Run substrate-create-admin-account -quality="..." to update your Intranet.
  3. Run substrate-create-account -domain="..." -environment="..." -quality="..." for all your service accounts to tag your shared VPCs.

If you’ve added any stub provider blocks to your modules, leave them in place for now and accept the deprecation warning. Terraform only allows one required_providers block and that is now managed by Substrate. A future release will accommodate these additional providers.

2021.05

After upgrading:

2021.04

After upgrading, run substrate-create-admin-account -quality=<quality> to add /accounts to your Intranet.

2021.03

You must upgrade to Terraform 0.14.7 in order to use Substrate 2021.03. Terraform 0.14.7 may be found here:

After upgrading:

  1. rm -f -r root-modules/network/*/peering and remove these files from version control.
  2. substrate-bootstrap-network-account to peer all your VPCs that should be peered.
  3. substrate-create-admin-account -quality="..." to fix Instance Factory IAM roles, following the Google SAML setup guide if Google is your IdP to also get 12-hour AWS Console sessions.

2021.02

You must upgrade to Terraform 0.13.6 in order to use Substrate 2021.02. Terraform 0.13.6 may be found here:

2021.01

You must run substrate-create-admin-account for each of your admin accounts before you’ll be able to use eval $(substrate-credentials) to streamline your use of the Credential Factory.

2020.12

You must run substrate-bootstrap-management-account in order to re-tag your former master account as your management account. (This rename follows AWS’ own renaming.)

2020.11 and prior releases

Contact hello@src-bin.com for prior release notes.