Encrypting Apache Kafka on Heroku with Your Encryption Key
Last updated May 03, 2023
Table of Contents
This article describes how to use AWS Key Management Service (KMS) to create a Customer Master Key (CMK) to encrypt Apache Kafka on Heroku in Private and Shield Spaces. This process involves three high-level steps:
- Create a Customer Master Key (CMK) in your AWS KMS
- Apply an IAM policy to that CMK to permit Heroku Data to use the key on your behalf
- Create a Kafka cluster with the encryption key
You perform the steps in this section from your Amazon KMS dashboard. Alternatively, you can use the AWS CLI as shown below.
Step 1: Create a Customer Master Key
When logged into the AWS web console, navigate to Key Management Service and click Create Key.
Step 2: Select Symmetric Key
Select Symmetric Key and click Next.
We use S3 to store backups and it only supports symmetric CMKs.
Step 3: Add Details and Set Permissions
Add an alias (e.g.:
heroku-data) and press Next. There’s no requirement to further configure key administrative permissions. Click Next.
When defining key usage permissions, scroll to the bottom to Add another AWS account. Enter the Heroku Data AWS Account ID (
021876802972) in the box and press Next.
Step 4: Review and Complete
Review the key policy and complete creation. Note the Amazon Resource Name (ARN) of your CMK.
Step 5: Enable Automatic Key Rotation
You can enable automatic key rotation by opening the info page for your key. At the bottom of the page, select the Key Rotation tab, check the box, and press Save.
AWS automatically handles key rotations every year without any intervention. This rotation for symmetric keys generated within AWS KMS doesn’t require re-encrypting your data. AWS KMS manages the previous versions of keys used for the decryption of data encrypted under an old version of a key. All new encryption requests use the new version of the key. Your Apache Kafka on Heroku cluster doesn’t experience downtime when the key is rotated.
Alternative: Use the AWS CLI
You can use the AWS CLI to create your Customer Master Key (CMK) with the appropriate key policy.
$ export AWS_ACCOUNT_ID=`aws sts get-caller-identity --output text --query 'Account'`
$ export HEROKU_DATA_ACCOUNT_ID=021876802972
$ curl -s -o key-policy.json https://gist.githubusercontent.com/jdowning/8d146cd238de828141e81b458dc546f0/raw/fc2a69603dc1364f1bc2fd2b5beb0af210150444/key-policy.json
$ aws kms create-key --description 'heroku-data-test' --policy $(envsubst < key-policy.json)
The output of the
create-key command includes the key’s ARN which you need during provisioning. This is referred to as
CMK_ARN in later steps.
We recommend you enable automatic key rotation on your CMK:
$ aws enable-key-rotation --key-id CMK_ARN
Create an Apache Kafka on Heroku Cluster with an Encryption Key
Now that you have a Customer Master Key with the appropriate permissions configured, you can use that key to encrypt your data managed by Heroku Data. You need the full Amazon Resource Name (ARN) of your CMK. You can use the
--encryption-key provisioning flag when creating your database.
Encrypting Apache Kafka on Heroku with your encryption key requires a Private or Shield Spaces plan
$ heroku addons:create heroku-kafka:private-extended-0 --encryption-key CMK_ARN --app your-app-name
Disabling Your Encryption Key
As part of the lifecycle of your encryption key, you may need to disable the key. When performing this action, all data encrypted with the key will be rendered inaccessible.
Disabling an encryption key triggers a shutdown of all services and servers that use that key. Use extreme caution when taking this action. We recommend you notify and coordinate with Support when performing this action.
You can disable your encryption key via the AWS web console or the AWS CLI.
$ aws kms disable-key --key-id CMK_ARN
When we receive notification of the key disablement, there’s a 10 minute waiting period before action is taken. The waiting period helps make sure accidental changes to the key status don’t unnecessarily alter resources that depend on the key.
After the 10 minute waiting period, we shutdown all services that use the encryption key. Next, we stop all servers that run those services. Finally, we send an email notification to application administrators notifying them of the disabled action. Expect approximately 20 minutes to elapse between key disablement and service shutdown.
Enabling Your Encryption Key
If you need to regain access to data that is encrypted with your encryption key, you can enable that key to restore. You can enable your encryption key via the AWS web console or the AWS CLI.
$ aws kms enable-key --key-id CMK_ARN
After we receive notification of the key enablement, we start previously stopped servers. When they’re running, we start affected services. Finally, we send an email notification to application administrators notifying them of the enabled action. Expect approximately five minutes to elapse between key enablement and service restart.
There are limitations that apply to databases encrypted with a customer encryption key.
- This feature is only available for Apache Kafka on Heroku add-ons in Private or Shield plans.
For your security, there’s no automated way to migrate from using a customer-supplied key to a Heroku-managed key lifecycle. If you would like to stop using this feature on one or more data services, contact Heroku support.