Last updated May 03, 2023

This article describes how to use AWS Key Management Service (KMS) to create a Customer Master Key (CMK) to encrypt Apache Kafka on Heroku in Private and Shield Spaces. This process involves three high-level steps:

1. Create a Customer Master Key (CMK) in your AWS KMS
2. Apply an IAM policy to that CMK to permit Heroku Data to use the key on your behalf
3. Create a Kafka cluster with the encryption key

AWS Prerequisites

You perform the steps in this section from your Amazon KMS dashboard. Alternatively, you can use the AWS CLI as shown below.

Step 1: Create a Customer Master Key

When logged into the AWS web console, navigate to Key Management Service and click Create Key.

Step 2: Select Symmetric Key

Select Symmetric Key and click Next.

We use S3 to store backups and it only supports symmetric CMKs.

Step 3: Add Details and Set Permissions

Add an alias (e.g.: heroku-data) and press Next. There’s no requirement to further configure key administrative permissions. Click Next.

When defining key usage permissions, scroll to the bottom to Add another AWS account. Enter the Heroku Data AWS Account ID (021876802972) in the box and press Next.

Step 4: Review and Complete

Review the key policy and complete creation. Note the Amazon Resource Name (ARN) of your CMK.

Step 5: Enable Automatic Key Rotation

You can enable automatic key rotation by opening the info page for your key. At the bottom of the page, select the Key Rotation tab, check the box, and press Save.

AWS automatically handles key rotations every year without any intervention. This rotation for symmetric keys generated within AWS KMS doesn’t require re-encrypting your data. AWS KMS manages the previous versions of keys used for the decryption of data encrypted under an old version of a key. All new encryption requests use the new version of the key. Your Apache Kafka on Heroku cluster doesn’t experience downtime when the key is rotated.

Alternative: Use the AWS CLI

You can use the AWS CLI to create your Customer Master Key (CMK) with the appropriate key policy.

$ export AWS_ACCOUNT_ID=`aws sts get-caller-identity --output text --query 'Account'`
$ export HEROKU_DATA_ACCOUNT_ID=021876802972
$ curl -s -o key-policy.json https://gist.githubusercontent.com/jdowning/8d146cd238de828141e81b458dc546f0/raw/fc2a69603dc1364f1bc2fd2b5beb0af210150444/key-policy.json
$ aws kms create-key --description 'heroku-data-test' --policy $(envsubst < key-policy.json)

The output of the create-key command includes the key’s ARN which you need during provisioning. This is referred to as CMK_ARN in later steps.

We recommend you enable automatic key rotation on your CMK:

$ aws enable-key-rotation --key-id CMK_ARN

Create an Apache Kafka on Heroku Cluster with an Encryption Key

Now that you have a Customer Master Key with the appropriate permissions configured, you can use that key to encrypt your data managed by Heroku Data. You need the full Amazon Resource Name (ARN) of your CMK. You can use the --encryption-key provisioning flag when creating your database.

$ heroku addons:create heroku-kafka:private-extended-0 --encryption-key CMK_ARN --app your-app-name

Disabling Your Encryption Key

As part of the lifecycle of your encryption key, you may need to disable the key. When performing this action, all data encrypted with the key will be rendered inaccessible.

You can disable your encryption key via the AWS web console or the AWS CLI.

$ aws kms disable-key --key-id CMK_ARN

When we receive notification of the key disablement, there’s a 10 minute waiting period before action is taken. The waiting period helps make sure accidental changes to the key status don’t unnecessarily alter resources that depend on the key.

After the 10 minute waiting period, we shutdown all services that use the encryption key. Next, we stop all servers that run those services. Finally, we send an email notification to application administrators notifying them of the disabled action. Expect approximately 20 minutes to elapse between key disablement and service shutdown.

Enabling Your Encryption Key

If you need to regain access to data that is encrypted with your encryption key, you can enable that key to restore. You can enable your encryption key via the AWS web console or the AWS CLI.

$ aws kms enable-key --key-id CMK_ARN

After we receive notification of the key enablement, we start previously stopped servers. When they’re running, we start affected services. Finally, we send an email notification to application administrators notifying them of the enabled action. Expect approximately five minutes to elapse between key enablement and service restart.

Limitations

There are limitations that apply to databases encrypted with a customer encryption key.

• This feature is only available for Apache Kafka on Heroku add-ons in Private or Shield plans.