Connecting to a Private or Shield Kafka Cluster from an External Resource
Last updated November 21, 2022
Use Mutual TLS to create a secure and mutually authenticated channel between an external resource and an Apache Kafka on Heroku cluster running in a Private Space or a Shield Private Space. External resources can include any mTLS-enabled application or system running in private data centers or public clouds. To use this feature, you must allowlist the external IP used to connect to your cluster.
This feature is only available for Private and Shield Apache Kafka on Heroku add-ons.
All Kafka clusters running on Heroku have mTLS enabled by default.
When provisioning the Kafka add-on, Heroku automatically creates mTLS certificates for its use. Connections to Kafka clusters require authenticating with these certificates. They’re available in the
KAFKA_CLIENT_CERT_KEY config vars for your application.
Private and Shield Kafka clusters exist in an isolated data resource network associated with their Private or Shield space. By default, only resources within the Private or Shield space can connect to the cluster. Use the mTLS feature to configure IP addresses for your external resources and allow connections from those sources.
Mutual TLS setup requires the following Heroku resources:
- A Private Space or Shield Private Space
- A Heroku app running in the Space with an attached Private or Shield Apache Kafka on Heroku add-on
Step 1: Install the Mutual TLS CLI Plugin
$ heroku plugins:install mtls
Find the different commands available and further details about this plugin at Heroku mTLS Plugin CLI commands.
Step 2: Allowlist external IPs
You can allowlist up to a hard limit of 60 IP blocks per Kafka cluster. Allowlist an IP block to access your Kafka cluster using the following Heroku CLI command:
$ heroku data:mtls:ip-rules:create KAFKA_ADDON_NAME --app APP_NAME \ --cidr CIDR_BLOCK \ --description DESCRIPTION
Provide the following parameters:
KAFKA_ADDON_NAME: The name of your Kafka cluster (for example,
APP_NAME: The name of your application
CIDR_BLOCK: The CIDR block to add to the allowlist. Define individual IP addresses as
DESCRIPTION: The description of the specified CIDR block (for example,
$ heroku data:mtls:ip-rules:create kafka-sushi-12345 --app example-app \ --cidr "126.96.36.199/32" \ --description "My Office IP" Creating IP Rule for kafka-sushi-12345... done cidr: 188.8.131.52/32 created_at: 2021-10-27T15:44:22.901+00:00 description: My Office IP id: 38c466b6-dcfb-4869-b5ac-40420b786fb4 status: Authorizing updated_at: 2021-10-27T15:44:22.901+00:00
The new IP rule has an id that you can use to check the rule details or to delete it.
It usually takes 10-15 minutes to allowlist an external IP. Track the progress and check the status of the configured IP ranges with the
data:mtls:ip-rules:get command. Provide the IP rule id with the
$ heroku data:mtls:ip-rules:get KAFKA_ADDON_NAME --id IP_RULE_ID --app APP_NAME
$ heroku data:mtls:ip-rules:get kafka-sushi-12345 --id 38c466b6-dcfb-4869-b5ac-40420b786fb4 --app example-app cidr: 184.108.40.206/32 created_at: 2021-10-27T15:44:22.901+00:00 description: My Office IP id: 38c466b6-dcfb-4869-b5ac-40420b786fb4 status: Authorized updated_at: 2021-10-27T15:44:22.901+00:00
You can view all the IP addresses allowlisted for a cluster with
heroku data:mtls:ip-rules KAFKA_ADDON_NAME --app APP_NAME.