Skip Navigation
Show nav
Heroku Dev Center
  • Get Started
  • Documentation
  • Changelog
  • Search
  • Get Started
    • Node.js
    • Ruby on Rails
    • Ruby
    • Python
    • Java
    • PHP
    • Go
    • Scala
    • Clojure
  • Documentation
  • Changelog
  • More
    Additional Resources
    • Home
    • Elements
    • Products
    • Pricing
    • Careers
    • Help
    • Status
    • Events
    • Podcasts
    • Compliance Center
    Heroku Blog

    Heroku Blog

    Find out what's new with Heroku on our blog.

    Visit Blog
  • Log inorSign up
View categories

Categories

  • Heroku Architecture
    • Dynos (app containers)
    • Stacks (operating system images)
    • Networking & DNS
    • Platform Policies
    • Platform Principles
  • Command Line
  • Deployment
    • Deploying with Git
    • Deploying with Docker
    • Deployment Integrations
  • Continuous Delivery
    • Continuous Integration
  • Language Support
    • Node.js
    • Ruby
      • Working with Bundler
      • Rails Support
    • Python
      • Background Jobs in Python
      • Working with Django
    • Java
      • Working with Maven
      • Java Database Operations
      • Working with the Play Framework
      • Working with Spring Boot
      • Java Advanced Topics
    • PHP
    • Go
      • Go Dependency Management
    • Scala
    • Clojure
  • Databases & Data Management
    • Heroku Postgres
      • Postgres Basics
      • Postgres Performance
      • Postgres Data Transfer & Preservation
      • Postgres Availability
      • Postgres Special Topics
    • Heroku Redis
    • Apache Kafka on Heroku
    • Other Data Stores
  • Monitoring & Metrics
    • Logging
  • App Performance
  • Add-ons
    • All Add-ons
  • Collaboration
  • Security
    • App Security
    • Identities & Authentication
    • Compliance
  • Heroku Enterprise
    • Private Spaces
      • Infrastructure Networking
    • Enterprise Accounts
    • Enterprise Teams
    • Heroku Connect (Salesforce sync)
    • Single Sign-on (SSO)
  • Patterns & Best Practices
  • Extending Heroku
    • Platform API
    • App Webhooks
    • Heroku Labs
    • Building Add-ons
      • Add-on Development Tasks
      • Add-on APIs
      • Add-on Guidelines & Requirements
    • Building CLI Plugins
    • Developing Buildpacks
    • Dev Center
  • Accounts & Billing
  • Troubleshooting & Support
  • Databases & Data Management
  • Heroku Streaming Data Connectors

Heroku Streaming Data Connectors

English — 日本語に切り替える

Last updated 08 October 2020

Table of Contents

  • Heroku App Setup
  • Heroku Add-ons Setup
  • Heroku’s Streaming Data Connector Setup
  • Managing a Connector
  • Destroying a Connector

This article describes how to configure Change Data Capture (CDC) for Heroku Postgres events and stream them to your Apache Kafka on Heroku add-on provisioned in a Private Space or a Shield Private Space. This process involves three high-level steps:

  1. Creating an app in Private Space or Shield Private Space.
  2. Provisioning a Heroku Postgres add-on and a Apache Kafka on Heroku add-on on your new app
  3. Creating a streaming data connector to enable CDC events from your Postgres to your Kafka

Heroku App Setup

To begin, you will need to create a Private or Shield Private Space. Once your Space is available, you can create an app in your Space.

$ heroku spaces:create --region virginia --team my-team-name --space myspace
$ heroku spaces:wait --space myspace
$ heroku apps:create --space myspace my-cdc-app

Heroku Add-ons Setup

Next, you will need two Private or Shield data add-ons attached to your app.

Your Postgres add-on will need to be version 10 or higher. Your Kafka add-on will need to be version 2.3 or higher.

$ heroku addons:create heroku-postgresql:private-7 --as DATABASE --app my-cdc-app
$ heroku addons:create heroku-kafka:private-extended-2 --as KAFKA --app my-cdc-app

You can monitor the add-on provisioning progress:

$ heroku addons:wait --app my-cdc-app

Once your add-ons are available, you will need to import your schema and/or data into your Postgres database.

Heroku’s Streaming Data Connector Setup

Once you have a Private or Shield Private Space App with Heroku Postgres and Apache Kafka on Heroku add-ons configured, you can provision a connector.

First, you will need to install the CLI plugin:

$ heroku plugins:install @heroku-cli/plugin-data-connectors

To create a connector, you will need to gather several pieces of information.

  1. The name of the Kafka add-on
  2. The name of the Postgres add-on
  3. The name(s) of the Postgres tables from which you want to capture events
  4. (optionally) The name(s) of the columns you wish to exclude from capture events

In order to capture events in your Postgres database, a few requirements must be met:

  • The database encoding must be UTF-8
  • The table(s) must currently exist
  • The table(s) must have a primary key
  • The table name(s) must only contain the characters [a-z,A-Z,0-9,\_]
  • The Kafka Formation needs to have direct Zookeeper access disabled

You will want to take care in choosing the what tables to capture. A single connector may not be able to keep up with a high volume of events from many tables.

Next, you can create the connector. You will need the names of your Postgres and Kafka add-ons, as well as a list of fully qualified tables you want to include in your database capture events:

$ heroku data:connectors:create \
    --source postgresql-neato-98765 \
    --store kafka-lovely-12345 \
    --table public.posts --table public.users

Provisioning can take approximately 15-20 minutes to complete. You can monitor the connector provisioning progress:

$ heroku data:connectors:wait gentle-connector-1234

Once your connector is available, you can view the details including newly created Kafka topics:

$ heroku data:connectors:info gentle-connector-1234
=== Data Connector status for gentle_connector_1234
Name:   gentle_connector_1234
Status: available

=== Configuration
Table Name   Topic Name
public.posts gentle_connector_1234.public.posts
public.users gentle_connector_1234.public.users

Managing a Connector

Once you have created your connector, there a few options available for managing it.

Pause or Resume

You can pause processing of new events. While a connector is paused, it will stop polling for additional records until you resume it. Alternating between these two states is simple:

# to pause
$ heroku data:connectors:pause gentle-connector-1234

# to resume
$ heroku data:connectors:resume gentle-connector-1234

Under normal operation, the connector will not lose change events that occur while a connector is paused. The connector uses a replication slot on the Postgres database to track progress, and will pick up where it left off without losing data when resumed.

 

If the Postgres database fails while a connector is paused, the replication status will be lost. Any change events created between the time the connector is paused and the time of the failure will be lost.

 

If the connector is paused for a very long time on a busy database, the replication slot will prevent Postgres from deleting unread write-ahead logs (WAL). As a result, the WAL drive can fill up, which will cause the database to shut down. Our automation will generally detect these situations ahead of time, but in a worst-case scenario, we might need to drop the replication slot to protect the database. In that rare case, change events would not make it to Kafka.

Update Configuration

You can modify certain properties associated with your connector via the CLI. These include:

property possible values default value details
decimal.handling.mode precise, double, string precise docs
hstore.handling.mode map, json map docs
time.precision.mode adaptive, adaptive_time_microseconds, connect adaptive docs
interval.handling.mode numeric, string numeric docs
tombstones.on.delete true, false true docs
binary.handling.mode bytes, base64, hex bytes docs

For example, you can update the tombstones.on.delete to false:

$ heroku data:connectors:update gentle-connector-1234 \
  --setting tombstones.on.delete=false

It is recommended that you familiarize yourself with our recommended Best Practices when working with connectors.

Update Tables and Excluded Columns

You can also modify the connector’s Postgres tables, as well as excluded columns.

For example, you can add the table public.parcels and remove the table public.posts:

$ heroku data:connectors:update gentle-connector-1234 \
  --add-table public.parcels \
  --remove-table public.posts

New tables must adhere to the same requirements as outlined in the Setup.

Likewise, you can add and remove excluded columns:

$ heroku data:connectors:update gentle-connector-1234 \
  --exclude-column public.parcels.address \
  --remove-excluded-column public.posts.keys

Destroying a Connector

You can destroy a connector via the CLI.

This will not destroy the Kafka topics used to produce events. You will need to manage their lifecycle independently.

$ heroku data:connectors:destroy gentle-connector-1234

Keep reading

  • Databases & Data Management

Feedback

Log in to submit feedback.

PostgreSQL, libpq5.12.1, and Breaking Changes Impacting Connection Behavior PostgreSQL, libpq5.12.1, and Breaking Changes Impacting Connection Behavior

Information & Support

  • Getting Started
  • Documentation
  • Changelog
  • Compliance Center
  • Training & Education
  • Blog
  • Podcasts
  • Support Channels
  • Status

Language Reference

  • Node.js
  • Ruby
  • Java
  • PHP
  • Python
  • Go
  • Scala
  • Clojure

Other Resources

  • Careers
  • Elements
  • Products
  • Pricing

Subscribe to our monthly newsletter

Your email address:

  • RSS
    • Dev Center Articles
    • Dev Center Changelog
    • Heroku Blog
    • Heroku News Blog
    • Heroku Engineering Blog
  • Heroku Podcasts
  • Twitter
    • Dev Center Articles
    • Dev Center Changelog
    • Heroku
    • Heroku Status
  • Facebook
  • Instagram
  • Github
  • LinkedIn
  • YouTube
Heroku is acompany

 © Salesforce.com

  • heroku.com
  • Terms of Service
  • Privacy
  • Cookies