This add-on is operated by Softtrends LLC

Synchronize Marketing data to Postgres, Amazon Redshift, or Kafka

Last updated November 09, 2022

Softtrends Marketing Connector is an add-on that synchronizes data between a marketing environment (such as Salesforce Marketing Cloud), and a Postgres database, AWS Redshift database, or a Kafka topic.

Use Marketing Connector to:

• Sync data into Postgres for reporting
• Stream data to Kafka to perform predictive analytics
• Sync data into Marketing Cloud for marketing campaigns
• Create consumer services and marketing-driven applications
• Sync data into an AWS Redshift data warehouse

Marketing Connector supports all the languages and frameworks supported by Heroku.

Before Getting Started

Ensure the following setup tasks are complete before:

1. The marketing environment’s configuration allows API access from external applications. For Salesforce Marketing Cloud, reference this article for guidance on proper configuration.

   Marketing Connector only supports Salesforce Marketing Cloud editions that have API access enabled. Some plan types, including trial versions, don’t have API access by default and can’t be used with Marketing Connector.

2. A provisioned Postgres or Kafka instance and its configuration details. Marketing Connector requires an app to have a Postgres database and or Kafka topic available to sync or publish data.

3. Open Marketing Connector and complete its configuration. Access Marketing Connector from the app’s Resources tab on the Heroku Dashboard, or via the CLI:

   $ heroku addons: open mcconnect
   

Full instructions for configuring Marketing Connector are described in Navigating the Add-on Dashboard.

Supported Data Services

Heroku Postgres

Marketing Connector supports all standard, premium, and private Heroku Postgres plan types. Marketing Connector syncs data between a Salesforce Marketing Cloud instance and a Heroku Postgres database in or out of a Heroku Private Space.

Apache Kafka on Heroku

Marketing Connector supports all Standard, Private, and Extended Apache Kafka on Heroku plan types. Marketing Connector can sync Data Extensions between a Salesforce Marketing Cloud instance and an Apache Kafka on Heroku Topic in or out of a Heroku Private Space.

AWS Redshift

Marketing Connector supports AWS Redshift. Marketing Connector can be used to sync data between a Salesforce Marketing Cloud instance and an AWS Redshift database.

Provisioning the Marketing Connector Add-on

If an app doesn’t exist to use with Marketing Connector, this Heroku button creates an app that automatically provisions Marketing Connector and its dependencies.

To add Marketing Connector to a new or an existing app, provision it from the app’s Resources tab in the Heroku Dashboard or via the CLI:

$ heroku addons:create mcconnect
Creating mcconnect on sharp-mountain-4005... free
Created mcconnect-infinite-31247

Steps for creating the app manually are described in the next section.

Manual Provisioning

If an app needs provisioning for use with Marketing Connector, create a Heroku app and add the Marketing Connector add-on via the Heroku CLI or Dashboard.

Step 1

Create a Heroku app from the Heroku Dashboard.

Step 2

After the app is created, select the Resources tab from the Heroku App Dashboard.

Step 3a (Kafka)

If pairing Marketing Connector with Apache Kafka on Heroku, provision the Apache Kafka on Heroku addon, too, via the dashboard or share an existing Apache Kafka on Heroku add-on with the new Heroku app.

$ heroku addons:attach my-existing-kafka-appname::KAFKA -a app-name
Attaching kafka-cubed-12345 to ⬢ app-name... done
Setting KAFKA config vars and restarting ⬢ app-name... done, v3

Step 3b (Postgres)

If pairing Marketing Connector with Heroku Postgres, provision the Heroku Postgres add-on, too, via the dashboard or share an existing Heroku Postgres add-on with the new Heroku app.

$ heroku addons:attach my-existing-postgres-appname::DATABASE --app app-name
Attaching postgresql-metric-12345 to ⬢ app-name... done
Setting DATABASE config vars and restarting ⬢ app-name... done, v5

Step 4

A new Heroku app with Apache Kafka on Heroku or Heroku Postgres newly provisioned or shared is available on the Dashboard:

Advanced Provisioning Options

Provisioning Multiple Add-ons Per App

Marketing Connector supports provisioning multiple instances of itself on a single app. For example, to synchronize data from multiple Salesforce Marketing Cloud instances, each instance of Marketing Connector can authorize and connect to independent Marketing Cloud instances or Data Extensions.

When multiple Marketing Connector add-on instances exist, the globally unique identifiers are required when running commands from the CLI to prevent an error about add-on name ambiguity. For example, to open the Marketing Connector dashboard from the CLI with multiple Marketing Connector add-on instances installed:

$ heroku addons:open mcconnect-cavernous-77445 -a app-name

Upgrading the Add-on Plan

Select the Edit plan option for the Marketing Connector add-on on the Heroku Dashboard or using the Heroku CLI to upgrade a Marketing Connector plan. The free test or dev-edition plans are upgradable to the paid mc-edition, hub-edition, or end-edition plans.

$ heroku addons:upgrade mcconnect:ent-edition -a app-name
Changing mcconnect-cavernous-77445 on app-name from mcconnect:test to mcconnect:ent-edition... done, $3500/month

Using the Heroku CLI, upgrading one of multiple instances of Marketing Connector attached to a single app requires defining the globally unique identifier, too:

$ heroku addons:upgrade mcconnect-cavernous-77445 mcconnect:ent-edition
Changing mcconnect-cavernous-77445 on app-name from mcconnect:test to mcconnect:ent-edition... done, $3500/month

Removing the Add-on

Remove the Marketing Connector add-on using the Heroku Dashboard or the Heroku CLI.

$ heroku addons:destroy mcconnect -a app-name
▸    WARNING: Destructive Action
▸    This command will affect the app app-name
▸    To proceed, type app-name or rerun this command with --confirm app-name

$heroku addons:destroy mcconnect -a app-name --confirm app-name
Destroying mcconnect-cavernous-77445 on ⬢ app-name... done

Navigating the Add-on Dashboard

Configure, monitor, and troubleshoot data mappings using the Marketing Connector dashboard. Users with add-on access on an app have access to the Marketing Connector dashboard. Refer to Collaborating with Other Developers on Your App for managing users’ access to an application.

Administration

The Administration menu allows users to:

• Access a summary view of all existing data connectors
• Create a data connector
• Access the Postgres Data Explorer
• Access the Kafka Data Explorer
• View Data Extension data
• View exposed external objects in Postgres
• View and set your schedules and timers

Create a Connector.

View the current sync status of a Connector.

View synced marketing data after a successful Heroku Postgres synchronization.

View synced marketing data after a successful Apache Kafka on Heroku synchronization.

View marketing data synchronized to the Data Extension.

Review and set data synchronization schedules and timers.

Add-on Settings

The Add-on Settings menu allows users to:

• View and set Heroku Postgres, Apache Kafka on Heroku, AWS Redshift, and Salesforce Marketing Cloud instance configurations
• View the Heroku host application’s settings
• View and edit email IDs to receive alert notifications, and retrieve an API key for the Marketing Connector API

View and set Heroku Postgres connection configuration.

View and set AWS Redshift connection configuration.

View and set Apache Kafka on Heroku connection configuration.

View and set Marketing Cloud connection configuration.

View the current Heroku app configuration values including the app name, app ID, and attached add-ons.

Configure the email address for alerts and retrieve the API Key for the Marketing Connector API.

Reports & Statistics

The Reports & Statistics Menu allows users to:

• View summary information about configured Marketing Connectors
• View events and trends occurring within marketing data
• View Marketing Connector logs
• View the logs for the Heroku app Marketing Connector is attached to

View logs for the configured Marketing Cloud connectors.

View the logs for the Heroku app Marketing connector is attached to.

Configuring the Add-on

When setting up a new data connector in the add-on, Marketing Connector can sync Salesforce Marketing Cloud data from a Data Extension to Heroku Postgres, AWS Postgres, Azure Postgres, AWS Redshift, or Apache Kafka on Heroku. Marketing Connector can sync data from any Postgres or AWS Redshift instance to Salesforce Marketing Cloud. The next sections cover configuring the add-on to select them as a source or destination when creating a Connector.

From the Resources tab of your Heroku application, click the Marketing Connector add-on to open its administrative dashboard. Select Add-on Settings in the left-hand menu to expose the related Add-on setting options.

Configuring the Add-on for Postgres

Select Add-On Settings tab, then Database Config to begin configuring the add-on for your Postgres instances. Enter an instance a name and settings for them. The name entered when creating a connector appears in a dropdown menu and are an option when selecting any of the Postgres instances as a source or destination.

No Existing Postgres Database Settings

When configuring a Postgres database for the first time, the Default Postgres Database URL field will be empty:

Next to the Default Postgres Database URL field, “Get DB URL from App” appears if a Heroku Postgres database and Marketing Connector are attached to the same app. Click this link and Marketing Connector auto-detects the Heroku Postgres instance and fill in the Default Postgres Database URL field for confirmation.

Enter the connection string manually for Heroku Postgres databases attached to another app or AWS or Azure Postgres databases. Use the following format:

postgres://username:password@hostname:5432/databasename

Existing Postgres Database Settings

The Postgres Configuration page displays the named Postgres configurations already created. The Used Count shows how many connectors are using the configuration and hovering over that will show which connectors are using it. If you Edit a URL, all connectors using the URL will start using the modified URL.

Click Add new URL button at the bottom of the page to enter additional named configurations.

Configure Marketing Connector for Marketing Cloud

Click the Add-On Settings menu, then Marketing Config to provide Salesforce Marketing Cloud settings. The available Marketing Cloud settings vary depending on the current Marketing Connector plan.

Configuration when accessing Add-on Settings for the First time

Click on Add New Marketing Config button and create the Configuration.

Add-on Plan Allows One Configuration

The configuration page shows a single configuration that you created. You can Edit the configuration but the Configuration Name can’t be changed.

There will be No option to Add New Marketing Config.

Add-on Plan Allows Multiple Configurations

Configuration page shows a list UI of the configurations you have added, and the option to add new marketing configurations.

Clicking Edit on any of the listed configurations will let you edit the configuration or clicking Add New Marketing Config will allow you to create a new configuration.

Configure the Add-on for Redshift

Click the Add-On Settings tab, then Redshift Config. For the first-time configuration, enter the Default AWS Redshift Database URL and click Save.

If a configuration exists, click Edit, update the URL as necessary and click Save.

Multiple Redshift Configuration Support

Supported Marketing Connector plans allow multiple database URLs. Click Add New URL** to add additional URLs with custom names to the configuration.

When creating a new connector that uses Redshift, a drop-down menu displays the available configured database URLs.

Configuring the Add-on for Kafka

Click the Add-On Settings tab, then Kafka Config. If Apache Kafka on Heroku is attached to the same app as Marketing Connector, click Get Kafka URLs from App to auto-detect the cluster URLs.

Configuring the Add-on for Alert Notification

Click the Add-On Settings tab, then Additional Config to configure Marketing Connector to send alert notifications via Email. This page also displays the API Key used to access the Marketing connector API.

Create a Marketing Cloud to Heroku Postgres Sync Connector

Step 1: New Connector

Select New Connector from the Administration Menu.

• Connector Name: A custom value that describes the connector (for example, Subscriber Records).
• Data Source: Select Salesforce Marketing Cloud.
• Marketing Cloud API Configuration Name: Select a configuration name (name you have used for the configuration in Marketing config menu). The parameters from the selected API Configuration name will be automatically filled in. If you have provisioned any of the Enterprise plans, you can also select Custom and enter the configuration here.
• Data Source Folder: Dropdown will show the Data Extension folders for the Marketing cloud instance you have selected above. Select one
• Data Source Object to Sync: Select the Salesforce Marketing Cloud Data Extension for this connector.
• Data Source Object Columns: The columns to include in the sync to Postgres.

• Destination : Select Heroku Postgres.
• Database URL Name: Select a configuration name (name you have used for the configuration in Database config menu). If you have provisioned any of the Enterprise plans, you can also select New Custom and enter the Postgres URL here.
• Postgres Schema (Destination): Schema where you want the destination Postgres table to be created.
• Postgres table : Name of destination Postgres table.

• Sync Process: Dropdown in first row can be used to select Created date filter field and it timezone, and the dropdown in second row can be used to select Updated date filter field and its timezone
• Sync Schedule: Defines the frequency of synchronizations.
• Clear Synced Records in Destination before each Sync: Default is Do not clear. If you set it to clear then all records in the destination Postgres table will be deleted and all records from the Source will be synced again.

Step 2: Initiate the Sync

After saving, Marketing Connector will return to the Dashboard, displaying the now-configured Connector.

Select Sync to initiate the first synchronization from the defined marketing Data Extension to Postgres.

After the initial sync completes, the Status updates to Completed. View or modify the connector if needed, create new connectors, or select Postgres Data Explorer from the administration menu to view the data replicated into Heroku Postgres.

Step 3: View Postgres Explorer

Select View for the Postgres Connector. The Postgres Data Explorer displays a table of synchronized data from this connector.

Create a Heroku Postgres to Marketing Cloud Sync Connector

Step 1: New Connector

Select New Connector from the Administration menu.

• Connector Name: A custom value that describes the connector (for example, Subscribers_Data_Extension)
• Data Source: Select Heroku Postgres
• Database URL Name: Select a previously configured Database connection string or define it in the field below.
• Data Source Schema: Select a schema from the pre-populated list of detected schemas.
• Data Source Object Name: Select an object from the pre-populated list of detected objects.
• Data Source Object Columns: Choose the columns Marketing Connector will sync, or choose Select All
• Use Data field transformation This feature is described in Data Science feature support.

• Destination: Select Salesforce Marketing Cloud
• Marketing Cloud API Configuration name: Select a pre-defined configuration created within Marketing Config.
• It is a Child Business unit: Choose this option to create the Data Extension under a Business Unit. Enter the Business Unit name and MID.
• Data Extension Folder: Select the Data Extension folder from a pre-defined list of detected folders.
• Data Extension table: Enter the name of the Data Extension table for this connector. If the table doesn’t exist, Marketing Connector creates it.
• Mark as Sendable: Allows the Data Extension to be sendable after sync.
• Mark as Testable: Allows the Data Extension to be testable after sync.

• Data Field to check Date-time of New record insertion to Data source with Timezone: Define the timestamp field used for created records for incremental sync.
• Data Field to check Date-time of Update to existing records in Data source with Timezone: Define the timestamp field for updated records for incremental sync.
• Moment-in-time Sync for Updated records using database trigger: Selectable only if a timestamp for updated records is selected. Selecting Configure will create a trigger in the Heroku Postgres database. When a record is updated, the trigger will update the timestamp, allowing the next sync to pick up the changes.
• Sync Schedule: Defines the sync schedule.
• Clear Synced Records in Destination before each Sync?: Defines the connector’s behavior for clearing all data within the Data Extension before syncing records from Heroku Postgres. Refer to the Truncate and Sync section for additional detail on this behavior.

Create a Marketing Cloud to Apache Kafka on Heroku Sync Connector

Step 1: New Connector

Select New Connector from the Administration menu.

• Connector Name: A custom value that describes the connector (for example, Subscribers_Data_Extension)
• Connector Type: Select Kafka
• Data Source: Select Salesforce Marketing Cloud
• Marketing Object to Sync: Select the Salesforce Marketing Cloud Data Extension for this connector.
• Data Extension Columns: The columns to include in the sync to the Kafka topic.
• Sync Schedule: Defines the frequency of synchronizations.
• Apache Kafka on Heroku Topic: Define the topic name or select Get Existing Topics from Kafka.

Step 2: Initiate the Sync

After saving, Marketing Connector will return to the Dashboard, displaying the now-configured Connector.

Select Sync to initiate the first synchronization from the defined marketing Data Extension to the Kafka topic.

After the initial sync completes, the Status updates to Completed. View or modify the connector if needed, create new connectors, or select Kafka Data Explorer from the administration menu to view the Kafka topic.

Step 3: View Kafka Explorer

Select View for the Kafka Connector. The Kafka Data Explorer displays a table of synchronized data from this connector.

Sync Configuration Details

Scheduled Sync

Sync event timing is configured based on pre-defined frequencies or a custom interval. The pre-defined intervals of Once Daily, Once Weekly, Twice Weekly and Twice Daily, have the option to specify a time of day. The default timezones are GMT and the time for Daily is 00:00 UTC, and Weekly is Monday at 00:00 UTC. If you enter a custom time while accessing Marketing Connector in US EST, then the value saved will be in US EST time expressed in UTC.

Streaming Sync

Streaming sync isn’t directly supported in Marketing Connector. However, a Custom sync time is configurable for scheduled sync with a minimum of 1 minute and a maximum of 9,999 minutes.

Incremental Sync

Incremental Sync allows only new records or updates to existing records to be synchronized at the next sync interval, provided: - a connector is set up to sync a Data Extension to Heroku Postgres or vice versa, and - the connector has synced all records in the source to destination after setup.

A timestamp field in the source data is required for Marketing Connector to sync newly created or updated records. Specify the timestamp field and its timezone in the Sync Process settings section.

Truncate and Sync

Use Marketing Connector’s truncate and sync capability when the source table is truncated and updated with new records before the records are synced with the destination. An example of this is a marketing campaign using an email list. The email list is defined every time the campaign is run. The destination should be cleared before the source is synced, again.

Support

Submit all support and runtime inquiries for Softtrends Marketing Connector via the Heroku Support portal. Direct non-support-related topics or product feedback to heroku@softtrends.com.