Skip Navigation
Show nav
Heroku Dev Center
  • Get Started
  • Documentation
  • Changelog
  • Search
  • Get Started
    • Node.js
    • Ruby on Rails
    • Ruby
    • Python
    • Java
    • PHP
    • Go
    • Scala
    • Clojure
  • Documentation
  • Changelog
  • More
    Additional Resources
    • Home
    • Elements
    • Products
    • Pricing
    • Careers
    • Help
    • Status
    • Events
    • Podcasts
    • Compliance Center
    Heroku Blog

    Heroku Blog

    Find out what's new with Heroku on our blog.

    Visit Blog
  • Log inorSign up
View categories

Categories

  • Heroku Architecture
    • Dynos (app containers)
    • Stacks (operating system images)
    • Networking & DNS
    • Platform Policies
    • Platform Principles
  • Command Line
  • Deployment
    • Deploying with Git
    • Deploying with Docker
    • Deployment Integrations
  • Continuous Delivery
    • Continuous Integration
  • Language Support
    • Node.js
    • Ruby
      • Rails Support
      • Working with Bundler
    • Python
      • Background Jobs in Python
      • Working with Django
    • Java
      • Working with Maven
      • Java Database Operations
      • Working with the Play Framework
      • Working with Spring Boot
      • Java Advanced Topics
    • PHP
    • Go
      • Go Dependency Management
    • Scala
    • Clojure
  • Databases & Data Management
    • Heroku Postgres
      • Postgres Basics
      • Postgres Getting Started
      • Postgres Performance
      • Postgres Data Transfer & Preservation
      • Postgres Availability
      • Postgres Special Topics
    • Heroku Data For Redis
    • Apache Kafka on Heroku
    • Other Data Stores
  • Monitoring & Metrics
    • Logging
  • App Performance
  • Add-ons
    • All Add-ons
  • Collaboration
  • Security
    • App Security
    • Identities & Authentication
    • Compliance
  • Heroku Enterprise
    • Private Spaces
      • Infrastructure Networking
    • Enterprise Accounts
    • Enterprise Teams
    • Heroku Connect (Salesforce sync)
      • Heroku Connect Administration
      • Heroku Connect Reference
      • Heroku Connect Troubleshooting
    • Single Sign-on (SSO)
  • Patterns & Best Practices
  • Extending Heroku
    • Platform API
    • App Webhooks
    • Heroku Labs
    • Building Add-ons
      • Add-on Development Tasks
      • Add-on APIs
      • Add-on Guidelines & Requirements
    • Building CLI Plugins
    • Developing Buildpacks
    • Dev Center
  • Accounts & Billing
  • Troubleshooting & Support
  • Integrating with Salesforce
  • Databases & Data Management
  • Apache Kafka on Heroku
  • Connecting to Apache Kafka on Heroku in a Private or Shield Space via PrivateLink

Connecting to Apache Kafka on Heroku in a Private or Shield Space via PrivateLink

English — 日本語に切り替える

Last updated December 12, 2022

Table of Contents

  • Heroku Prerequisites
  • Provisioning the Heroku Endpoint
  • Provisioning the Amazon VPC endpoint
  • Connecting the Heroku and Amazon VPC endpoints
  • Limitations

This article describes how to use AWS PrivateLink to create a secure connection between an AWS VPC and an Apache Kafka on Heroku add-on provisioned in a Private Space or a Shield Private Space. This process involves three high-level steps:

  1. Creating an Endpoint Service on your Apache Kafka on Heroku add-on running in a Private Space or a Shield Private Space
  2. Creating an Endpoint Network Interface in your AWS VPC
  3. Establishing the secure connection between the two endpoints

As part of setting up the connection, you can specify a list of approved accounts to limit access to your Kafka add-on from the VPC.

To use this feature, the Amazon VPC Endpoint you create must be provisioned in a subnet that is in the same region as your Apache Kafka on Heroku add-on.

Heroku Prerequisites

The following Heroku resources are required to set up a PrivateLink endpoint:

  • A Private Space. This article describes how to create a Private Space using either the Heroku Dashboard or the Heroku CLI.

  • A Shield Private Space. This article describes how to create a Shield Private Space using either the Heroku Dashboard or the Heroku CLI.

  • A Heroku app running in the Private Space or Shield Private Space with an attached Apache Kafka on Heroku add-on. All Apache Kafka on Heroku instances running in a Private Space or Shield Private Space use one of the private or shield plan types, respectively.

Provisioning the Heroku Endpoint

Step 1: Install the Heroku Data via PrivateLink CLI Plugin

$ heroku plugins:install data-privatelink

Step 2: Obtain Your AWS Account ID

You can obtain your AWS account ID with the AWS CLI like so:

$ aws sts get-caller-identity --output text --query 'Account'

123456789101

The example command returns an account ID of 123456789101.

You can also obtain your account ID from the My Account page of your AWS account. The Account ID is shown in the Account Settings section:

A screen capture of the My Account window of a sample AWS account

Step 3: Create a PrivateLink Endpoint

Create a PrivateLink endpoint using the following Heroku CLI command (note the values to substitute):

$ heroku data:privatelink:create KAKFA_ADDON_NAME --aws-account-id ACCOUNT_ID --app APP_NAME
  • Replace KAKFA_ADDON_NAME with the name of your Kafka add-on (for example, kafka-sushi-12345).
  • Replace APP_NAME with your app’s name.
  • Replace ACCOUNT_ID with the AWS account that receives access to your Kafka add-on. This ID can match any of the following patterns:
    • account-id
    • account-id:user/username
    • account-id:role/rolename

You can specify the --aws-account-id flag multiple times to include multiple accounts.

Here’s an example command with accompanying output:

$ heroku data:privatelink:create kafka-sushi-12345 --aws-account-id 123456789101:user/abc.xyz --app privatelink-vpc-endpoint-demo
Creating privatelink... done

Service Name: Provisioning
Status:       Provisioning

The privatelink is now being provisioned for kafka-sushi-12345.
Run heroku data:privatelink:wait KAFKA_URL -a APP to check the creation process.

New PrivateLink endpoints typically take between 5–10 minutes to become available. You can track your progress with heroku data:privatelink:wait KAFKA_URL --app APP_NAME.

Step 4: Obtain Your Endpoint’s Service Name

When the PrivateLink endpoint finishes provisioning, use the following command to view its details:

$ heroku data:privatelink KAKFA_ADDON_NAME --app APP_NAME

Replace KAFKA_ADDON_NAME with the name of your Kafka add-on, and replace APP_NAME with your app’s name.

Here’s an example command with accompanying output:

$ heroku data:privatelink kafka-sushi-12345 --app privatelink-vpc-endpoint-demo
=== privatelinks for kafka-sushi-12345
Service Name:         com.amazonaws.vpce.us-east-1.vpce-svc-0410a2e25933fe8ec
Status:               Operational

=== Allowed Accounts
ARN                                    Status
arn:aws:iam::123456789101:user/abc.xyz Active

Your privatelink is now operational.
You must now copy the Service Name and follow the rest of the steps listed in https://devcenter.heroku.com/articles/heroku-kafka-via-privatelink.

Copy the value of the Service Name field from the command’s output (in the example above, the value is com.amazonaws.vpce.us-east-1.vpce-svc-0410a2e25933fe8ec). You need this value to provision the Amazon VPC endpoint.

Provisioning the Amazon VPC endpoint

You perform the steps in this section from your Amazon VPC dashboard.

Step 1: Create and Configure a Security Group

Your endpoint requires a security group with appropriate ingress security rules. Click Create security group in the Security Groups tab of your VPC dashboard:

A screen capture showing the Create security group window on the AWS Console

Specify an appropriate security group name and description and select your desired VPC before clicking Create:

A screen capture showing the security group name and description being edited on the AWS Console

Select your newly created security group from the list and click Actions > Edit inbound rules:

A screen capture showing the security group being selected

Enable TCP access to ports 10000–11000 from any valid IP address and click Save rules.

Step 2: Create the endpoint

Navigate to the Endpoints tab of your VPC dashboard and click Create Endpoint:

A screen capture showing the Create Endpoint window on the AWS Console

In the Create Endpoint form that appears, select the Find service by name option and paste the Service Name value you obtained earlier.

Then click Verify to display the list of available subnets:

A screen capture showing the endpoint being created

Attach the security group you created earlier to the VPC Endpoint and click Create endpoint:

Security group being attached

The endpoint is created with an initial status of pending acceptance, which transitions to available after 5-10 minutes:

VPC Endpoint in the available state

Connecting the Heroku and Amazon VPC endpoints

After the Amazon VPC endpoint becomes available, you can obtain the URL that allows your VPC to communicate with your Apache Kafka on Heroku add-on.

First, obtain your PrivateLink endpoint’s Endpoint ID and extract the 17-character string that appears at the end of it. Convert that string to upper case and use it in the following command.

For example, if the Endpoint ID is vpce-01c87ae3c05563935, the Endpoint ID is 01C87AE3C05563935.

Run the following command, substituting the obtained string where indicated:

$ heroku config --app your_app_name | grep ENDPOINT_ID_HERE

This command displays the AWS VPC Endpoint connection URL and the corresponding connection string for your Kafka add-on. The connection string has the following format:

kafka+ssl://vpc-endpoint-dns-name:10001,kafka+ssl://vpc-endpoint-dns-name:10002,kafka+ssl://vpc-endpoint-dns-name:10003

You can now use this connection string to connect the applications in your AWS VPC to your Apache Kafka on Heroku add-on. Here’s an example command with accompanying output:

$ heroku config --app privatelink-vpc-endpoint-demo | grep 01C87AE3C05563935

KAFKA_ENDPOINT_01C87AE3C05563935_URL:  kafka+ssl://vpc-endpoint-dns-name:10001,kafka+ssl://vpc-endpoint-dns-name:10002,kafka+ssl://vpc-endpoint-dns-name:10003

For any issues or concerns with using this feature, open a support ticket.

Limitations

  • The Amazon VPC Endpoint you create must be provisioned in a subnet that is in the same region as your Apache Kafka on Heroku add-on.
  • You can only connect to your Kafka add-on in Availability Zones that are common between your own VPC and the Heroku Data VPC.
  • It is your responsibility to verify the security of your VPC to ensure fully secure access to your Apache Kafka on Heroku add-on.

Keep reading

  • Apache Kafka on Heroku

Feedback

Log in to submit feedback.

Robust Usage of Apache Kafka on Heroku Connecting to a Private or Shield Kafka Cluster from an External Resource

Information & Support

  • Getting Started
  • Documentation
  • Changelog
  • Compliance Center
  • Training & Education
  • Blog
  • Podcasts
  • Support Channels
  • Status

Language Reference

  • Node.js
  • Ruby
  • Java
  • PHP
  • Python
  • Go
  • Scala
  • Clojure

Other Resources

  • Careers
  • Elements
  • Products
  • Pricing

Subscribe to our monthly newsletter

Your email address:

  • RSS
    • Dev Center Articles
    • Dev Center Changelog
    • Heroku Blog
    • Heroku News Blog
    • Heroku Engineering Blog
  • Heroku Podcasts
  • Twitter
    • Dev Center Articles
    • Dev Center Changelog
    • Heroku
    • Heroku Status
  • Facebook
  • Instagram
  • Github
  • LinkedIn
  • YouTube
Heroku is acompany

 © Salesforce.com

  • heroku.com
  • Terms of Service
  • Privacy
  • Cookies
  • Cookie Preferences