
This add-on is operated by Softtrends LLC
Synchronize Marketing data to Postgres, Amazon Redshift, or Kafka
Marketing & CDP Connect
Last updated November 09, 2022
Table of Contents
- Before Getting Started
- Supported Data Services
- Provisioning the Marketing Connector Add-on
- Advanced Provisioning Options
- Navigating the Add-on Dashboard
- Configuring the Add-on
- Configure Marketing Connector for Marketing Cloud
- Configure the Add-on for Redshift
- Configuring the Add-on for Kafka
- Configuring the Add-on for Alert Notification
- Create a Marketing Cloud to Heroku Postgres Sync Connector
- Create a Heroku Postgres to Marketing Cloud Sync Connector
- Create a Marketing Cloud to Apache Kafka on Heroku Sync Connector
- Sync Configuration Details
- Truncate and Sync
- Support
Softtrends Marketing Connector is an add-on that synchronizes data between a marketing environment (such as Salesforce Marketing Cloud), and a Postgres database, AWS Redshift database, or a Kafka topic.
Use Marketing Connector to:
- Sync data into Postgres for reporting
- Stream data to Kafka to perform predictive analytics
- Sync data into Marketing Cloud for marketing campaigns
- Create consumer services and marketing-driven applications
- Sync data into an AWS Redshift data warehouse
Marketing Connector supports all the languages and frameworks supported by Heroku.
If you’re just getting started with Heroku or Heroku add-ons, see the Heroku Getting Started Guides or the add-ons overview.
Before Getting Started
Ensure the following setup tasks are complete before:
The marketing environment’s configuration allows API access from external applications. For Salesforce Marketing Cloud, reference this article for guidance on proper configuration.
Marketing Connector only supports Salesforce Marketing Cloud editions that have API access enabled. Some plan types, including trial versions, don’t have API access by default and can’t be used with Marketing Connector.
A provisioned Postgres or Kafka instance and its configuration details. Marketing Connector requires an app to have a Postgres database and or Kafka topic available to sync or publish data.
Open Marketing Connector and complete its configuration. Access Marketing Connector from the app’s Resources tab on the Heroku Dashboard, or via the CLI:
$ heroku addons: open mcconnect
Full instructions for configuring Marketing Connector are described in Navigating the Add-on Dashboard.
Supported Data Services
Heroku Postgres
Marketing Connector supports all standard, premium, and private Heroku Postgres plan types. Marketing Connector syncs data between a Salesforce Marketing Cloud instance and a Heroku Postgres database in or out of a Heroku Private Space.
Essential-tier Postgres database Heroku Postgres plans are usable with Marketing Connector, but not recommended. Essential-tier databases have limited row and connection allowances that Marketing Connector consumes too quickly if it’s configured with multiple data connectors.
Apache Kafka on Heroku
Marketing Connector supports all Standard, Private, and Extended Apache Kafka on Heroku plan types. Marketing Connector can sync Data Extensions between a Salesforce Marketing Cloud instance and an Apache Kafka on Heroku Topic in or out of a Heroku Private Space.
AWS Redshift
Marketing Connector supports AWS Redshift. Marketing Connector can be used to sync data between a Salesforce Marketing Cloud instance and an AWS Redshift database.
Network latency can become a factor of service performance. Consider the overall architecture of any solution that employs a Marketing Connector, Heroku Postgres/Kafka, and Salesforce Marketing Cloud instance distributed across divergent geographic locations.
Provisioning the Marketing Connector Add-on
The available plans for the Marketing Connector add-on are listed here.
If an app doesn’t exist to use with Marketing Connector, this Heroku button creates an app that automatically provisions Marketing Connector and its dependencies.
To add Marketing Connector to a new or an existing app, provision it from the app’s Resources tab in the Heroku Dashboard or via the CLI:
$ heroku addons:create mcconnect
Creating mcconnect on sharp-mountain-4005... free
Created mcconnect-infinite-31247
Steps for creating the app manually are described in the next section.
Manual Provisioning
If an app needs provisioning for use with Marketing Connector, create a Heroku app and add the Marketing Connector add-on via the Heroku CLI or Dashboard.
Step 1
Create a Heroku app from the Heroku Dashboard.
Step 2
After the app is created, select the Resources tab from the Heroku App Dashboard.
Step 3a (Kafka)
If pairing Marketing Connector with Apache Kafka on Heroku, provision the Apache Kafka on Heroku addon, too, via the dashboard or share an existing Apache Kafka on Heroku add-on with the new Heroku app.
$ heroku addons:attach my-existing-kafka-appname::KAFKA -a app-name
Attaching kafka-cubed-12345 to ⬢ app-name... done
Setting KAFKA config vars and restarting ⬢ app-name... done, v3
Step 3b (Postgres)
If pairing Marketing Connector with Heroku Postgres, provision the Heroku Postgres add-on, too, via the dashboard or share an existing Heroku Postgres add-on with the new Heroku app.
$ heroku addons:attach my-existing-postgres-appname::DATABASE --app app-name
Attaching postgresql-metric-12345 to ⬢ app-name... done
Setting DATABASE config vars and restarting ⬢ app-name... done, v5
Step 4
A new Heroku app with Apache Kafka on Heroku or Heroku Postgres newly provisioned or shared is available on the Dashboard:
Advanced Provisioning Options
Provisioning Multiple Add-ons Per App
Marketing Connector supports provisioning multiple instances of itself on a single app. For example, to synchronize data from multiple Salesforce Marketing Cloud instances, each instance of Marketing Connector can authorize and connect to independent Marketing Cloud instances or Data Extensions.
Use a different Postgres database schema name or Kafka topic name for each marketing instance to allow Marketing Connector to use a single Heroku Postgres database or Kafka cluster with multiple instances of the Marketing Connector add-on.
When multiple Marketing Connector add-on instances exist, the globally unique identifiers are required when running commands from the CLI to prevent an error about add-on name ambiguity. For example, to open the Marketing Connector dashboard from the CLI with multiple Marketing Connector add-on instances installed:
$ heroku addons:open mcconnect-cavernous-77445 -a app-name
When using multiple Marketing Connector add-ons, use a distinct integration user per add-on. Each Marketing Connector add-on uses a different access token for Salesforce Marketing Cloud API calls. Using the same integration user for multiple add-ons can exceed the limit permitted by Salesforce Marketing Cloud, potentially leading to connectors failing or requiring reauthorization. Both scenarios lead to failure of data synchronizations.
Upgrading the Add-on Plan
Select the Edit plan option for the Marketing Connector add-on on the Heroku Dashboard or using the Heroku CLI to upgrade a Marketing Connector plan. The free test
or dev-edition
plans are upgradable to the paid mc-edition
, hub-edition
, or end-edition
plans.
$ heroku addons:upgrade mcconnect:ent-edition -a app-name
Changing mcconnect-cavernous-77445 on app-name from mcconnect:test to mcconnect:ent-edition... done, $3500/month
For enterprise users, the upgrade options assume your organization has a paid add-on plan entitlement already in place.
Using the Heroku CLI, upgrading one of multiple instances of Marketing Connector attached to a single app requires defining the globally unique identifier, too:
$ heroku addons:upgrade mcconnect-cavernous-77445 mcconnect:ent-edition
Changing mcconnect-cavernous-77445 on app-name from mcconnect:test to mcconnect:ent-edition... done, $3500/month
Removing the Add-on
Remove the Marketing Connector add-on using the Heroku Dashboard or the Heroku CLI.
Detaching Marketing Connector from an app drops any related tables from the associated Heroku Postgres database. Ensure necessary database backups exist before proceeding.
$ heroku addons:destroy mcconnect -a app-name
▸ WARNING: Destructive Action
▸ This command will affect the app app-name
▸ To proceed, type app-name or rerun this command with --confirm app-name
$heroku addons:destroy mcconnect -a app-name --confirm app-name
Destroying mcconnect-cavernous-77445 on ⬢ app-name... done
If the app has multiple Marketing Connector add-ons attached, specify the globally unique identified assigned to the add-on being removed using the Heroku CLI.
Navigating the Add-on Dashboard
Configure, monitor, and troubleshoot data mappings using the Marketing Connector dashboard. Users with add-on access on an app have access to the Marketing Connector dashboard. Refer to Collaborating with Other Developers on Your App for managing users’ access to an application.
Administration
The Administration menu allows users to:
- Access a summary view of all existing data connectors
- Create a data connector
- Access the Postgres Data Explorer
- Access the Kafka Data Explorer
- View Data Extension data
- View exposed external objects in Postgres
- View and set your schedules and timers
Create a Connector.
View the current sync status of a Connector.
View synced marketing data after a successful Heroku Postgres synchronization.
View synced marketing data after a successful Apache Kafka on Heroku synchronization.
View marketing data synchronized to the Data Extension.
Review and set data synchronization schedules and timers.
Add-on Settings
The Add-on Settings menu allows users to:
- View and set Heroku Postgres, Apache Kafka on Heroku, AWS Redshift, and Salesforce Marketing Cloud instance configurations
- View the Heroku host application’s settings
- View and edit email IDs to receive alert notifications, and retrieve an API key for the Marketing Connector API
View and set Heroku Postgres connection configuration.
View and set AWS Redshift connection configuration.
View and set Apache Kafka on Heroku connection configuration.
View and set Marketing Cloud connection configuration.
View the current Heroku app configuration values including the app name, app ID, and attached add-ons.
Configure the email address for alerts and retrieve the API Key for the Marketing Connector API.
Reports & Statistics
The Reports & Statistics Menu allows users to:
- View summary information about configured Marketing Connectors
- View events and trends occurring within marketing data
- View Marketing Connector logs
- View the logs for the Heroku app Marketing Connector is attached to
View logs for the configured Marketing Cloud connectors.
View the logs for the Heroku app Marketing connector is attached to.
Configuring the Add-on
When setting up a new data connector in the add-on, Marketing Connector can sync Salesforce Marketing Cloud data from a Data Extension to Heroku Postgres, AWS Postgres, Azure Postgres, AWS Redshift, or Apache Kafka on Heroku. Marketing Connector can sync data from any Postgres or AWS Redshift instance to Salesforce Marketing Cloud. The next sections cover configuring the add-on to select them as a source or destination when creating a Connector.
From the Resources tab of your Heroku application, click the Marketing Connector add-on to open its administrative dashboard. Select Add-on Settings in the left-hand menu to expose the related Add-on setting options.
Configuring the Add-on for Postgres
Select Add-On Settings tab, then Database Config to begin configuring the add-on for your Postgres instances. Enter an instance a name and settings for them. The name entered when creating a connector appears in a dropdown menu and are an option when selecting any of the Postgres instances as a source or destination.
No Existing Postgres Database Settings
When configuring a Postgres database for the first time, the Default Postgres Database URL field will be empty:
Next to the Default Postgres Database URL field, “Get DB URL from App” appears if a Heroku Postgres database and Marketing Connector are attached to the same app. Click this link and Marketing Connector auto-detects the Heroku Postgres instance and fill in the Default Postgres Database URL field for confirmation.
Enter the connection string manually for Heroku Postgres databases attached to another app or AWS or Azure Postgres databases. Use the following format:
postgres://username:password@hostname:5432/databasename
The option to have more than one Postgres configuration is only available in all Enterprise Edition plans
Existing Postgres Database Settings
The Postgres Configuration page displays the named Postgres configurations already created. The Used Count shows how many connectors are using the configuration and hovering over that will show which connectors are using it. If you Edit a URL, all connectors using the URL will start using the modified URL.
Click Add new URL button at the bottom of the page to enter additional named configurations.
Configure Marketing Connector for Marketing Cloud
All Enterprise Edition Marketing Connector plans support syncing data to and from multiple Marketing Cloud instances.
Click the Add-On Settings menu, then Marketing Config to provide Salesforce Marketing Cloud settings. The available Marketing Cloud settings vary depending on the current Marketing Connector plan.
Configuration when accessing Add-on Settings for the First time
Click on Add New Marketing Config button and create the Configuration.
Add-on Plan Allows One Configuration
The configuration page shows a single configuration that you created. You can Edit the configuration but the Configuration Name can’t be changed.
There will be No option to Add New Marketing Config.
Add-on Plan Allows Multiple Configurations
Configuration page shows a list UI of the configurations you have added, and the option to add new marketing configurations.
Clicking Edit on any of the listed configurations will let you edit the configuration or clicking Add New Marketing Config will allow you to create a new configuration.
Marketing connector can be used to sync data to/from Org’s root
folder or any business unit
. These are adjustable in Config settings
Configure the Add-on for Redshift
Click the Add-On Settings tab, then Redshift Config. For the first-time configuration, enter the Default AWS Redshift Database URL and click Save.
If a configuration exists, click Edit, update the URL as necessary and click Save.
Multiple Redshift Configuration Support
Supported Marketing Connector plans allow multiple database URLs. Click Add New URL** to add additional URLs with custom names to the configuration.
When creating a new connector that uses Redshift, a drop-down menu displays the available configured database URLs.
Configuring the Add-on for Kafka
Click the Add-On Settings tab, then Kafka Config. If Apache Kafka on Heroku is attached to the same app as Marketing Connector, click Get Kafka URLs from App to auto-detect the cluster URLs.
Configuring the Add-on for Alert Notification
Click the Add-On Settings tab, then Additional Config to configure Marketing Connector to send alert notifications via Email. This page also displays the API Key used to access the Marketing connector API.
Create a Marketing Cloud to Heroku Postgres Sync Connector
Step 1: New Connector
Select New Connector from the Administration Menu.
Connector Name
: A custom value that describes the connector (for example, Subscriber Records).Data Source
: Select Salesforce Marketing Cloud.Marketing Cloud API Configuration Name
: Select a configuration name (name you have used for the configuration in Marketing config menu). The parameters from the selected API Configuration name will be automatically filled in. If you have provisioned any of the Enterprise plans, you can also select Custom and enter the configuration here.Data Source Folder
: Dropdown will show the Data Extension folders for the Marketing cloud instance you have selected above. Select oneData Source Object to Sync
: Select the Salesforce Marketing Cloud Data Extension for this connector.Data Source Object Columns
: The columns to include in the sync to Postgres.
Destination
: Select Heroku Postgres.Database URL Name
: Select a configuration name (name you have used for the configuration inDatabase config
menu). If you have provisioned any of the Enterprise plans, you can also selectNew Custom
and enter the Postgres URL here.Postgres Schema (Destination)
: Schema where you want the destination Postgres table to be created.Postgres table
: Name of destination Postgres table.
Sync Process
: Dropdown in first row can be used to selectCreated date filter field and it timezone
, and the dropdown in second row can be used to selectUpdated date filter field and its timezone
Sync Schedule
: Defines the frequency of synchronizations.Clear Synced Records in Destination before each Sync
: Default isDo not clear
. If you set it to clear then all records in the destination Postgres table will be deleted and all records from the Source will be synced again.
Step 2: Initiate the Sync
After saving, Marketing Connector will return to the Dashboard, displaying the now-configured Connector.
Select Sync to initiate the first synchronization from the defined marketing Data Extension to Postgres.
After the initial sync completes, the Status updates to Completed
. View or modify the connector if needed, create new connectors, or select Postgres Data Explorer from the administration menu to view the data replicated into Heroku Postgres.
Step 3: View Postgres Explorer
Select View for the Postgres Connector. The Postgres Data Explorer displays a table of synchronized data from this connector.
Create a Heroku Postgres to Marketing Cloud Sync Connector
Step 1: New Connector
Select New Connector from the Administration menu.
Connector Name
: A custom value that describes the connector (for example,Subscribers_Data_Extension
)Data Source
: Select Heroku PostgresDatabase URL Name
: Select a previously configured Database connection string or define it in the field below.Data Source Schema
: Select a schema from the pre-populated list of detected schemas.Data Source Object Name
: Select an object from the pre-populated list of detected objects.Data Source Object Columns
: Choose the columns Marketing Connector will sync, or chooseSelect All
Use Data field transformation
This feature is described in Data Science feature support.
Destination
: SelectSalesforce Marketing Cloud
Marketing Cloud API Configuration name
: Select a pre-defined configuration created within Marketing Config.It is a Child Business unit
: Choose this option to create the Data Extension under a Business Unit. Enter the Business Unit name and MID.Data Extension Folder
: Select the Data Extension folder from a pre-defined list of detected folders.Data Extension table
: Enter the name of the Data Extension table for this connector. If the table doesn’t exist, Marketing Connector creates it.Mark as Sendable
: Allows the Data Extension to be sendable after sync.Mark as Testable
: Allows the Data Extension to be testable after sync.
Data Field to check Date-time of New record insertion to Data source with Timezone
: Define the timestamp field used for created records for incremental sync.Data Field to check Date-time of Update to existing records in Data source with Timezone
: Define the timestamp field for updated records for incremental sync.Moment-in-time Sync for Updated records using database trigger
: Selectable only if a timestamp for updated records is selected. SelectingConfigure
will create a trigger in the Heroku Postgres database. When a record is updated, the trigger will update the timestamp, allowing the next sync to pick up the changes.Sync Schedule
: Defines the sync schedule.Clear Synced Records in Destination before each Sync?
: Defines the connector’s behavior for clearing all data within the Data Extension before syncing records from Heroku Postgres. Refer to the Truncate and Sync section for additional detail on this behavior.
Create a Marketing Cloud to Apache Kafka on Heroku Sync Connector
Step 1: New Connector
Select New Connector from the Administration menu.
Connector Name
: A custom value that describes the connector (for example,Subscribers_Data_Extension
)Connector Type
: SelectKafka
Data Source
: SelectSalesforce Marketing Cloud
Marketing Object to Sync
: Select the Salesforce Marketing Cloud Data Extension for this connector.Data Extension Columns
: The columns to include in the sync to the Kafka topic.Sync Schedule
: Defines the frequency of synchronizations.Apache Kafka on Heroku Topic
: Define the topic name or select Get Existing Topics from Kafka.
Step 2: Initiate the Sync
After saving, Marketing Connector will return to the Dashboard, displaying the now-configured Connector.
Select Sync to initiate the first synchronization from the defined marketing Data Extension to the Kafka topic.
After the initial sync completes, the Status updates to Completed
. View or modify the connector if needed, create new connectors, or select Kafka Data Explorer from the administration menu to view the Kafka topic.
Step 3: View Kafka Explorer
Select View for the Kafka Connector. The Kafka Data Explorer displays a table of synchronized data from this connector.
Sync Configuration Details
Scheduled Sync
Sync event timing is configured based on pre-defined frequencies or a custom interval. The pre-defined intervals of Once Daily, Once Weekly, Twice Weekly and Twice Daily, have the option to specify a time of day. The default timezones are GMT and the time for Daily is 00:00 UTC, and Weekly is Monday at 00:00 UTC. If you enter a custom time while accessing Marketing Connector in US EST, then the value saved will be in US EST time expressed in UTC.
Streaming Sync
Streaming sync isn’t directly supported in Marketing Connector. However, a Custom
sync time is configurable for scheduled sync with a minimum of 1 minute and a maximum of 9,999 minutes.
Incremental Sync
Incremental Sync allows only new records or updates to existing records to be synchronized at the next sync interval, provided: - a connector is set up to sync a Data Extension to Heroku Postgres or vice versa, and - the connector has synced all records in the source to destination after setup.
Marketing connector maintains the Last Sync time
of the connector and compares the timestamps for Created
and Updated
records as specified in the next section. The timestamps must be in reference to the Heroku system time based on your timezone. If you are creating/updating the records directly in Heroku Postgres then it will not be an issue, but, if you are importing records from an external system using (e.g. an ETL tool), then the timestamps imported to ‘Created’ and ‘Updated’ time fields will be in reference to the system from where records are imported which, you must adjust before you write the record to reflect the Heroku system timezone.
A timestamp field in the source data is required for Marketing Connector to sync newly created or updated records. Specify the timestamp field and its timezone in the Sync Process settings section.
Truncate and Sync
Use Marketing Connector’s truncate and sync capability when the source table is truncated and updated with new records before the records are synced with the destination. An example of this is a marketing campaign using an email list. The email list is defined every time the campaign is run. The destination should be cleared before the source is synced, again.
Support
Submit all support and runtime inquiries for Softtrends Marketing Connector via the Heroku Support portal. Direct non-support-related topics or product feedback to heroku@softtrends.com.