Reading Data from Salesforce with Heroku Connect

Last updated 03 October 2019

Heroku Connect automatically chooses the most efficient method to transfer data between your database and your Salesforce organization by employing a set of best practices that take into account data change volume and the many subtle details of Salesforce API operation. This is true for both reading from and writing to Salesforce.

Salesforce offers a variety of APIs that are best suited to specific integration requirements and Heroku Connect utilizes two of them for data transfer: SOAP and Bulk. In addition the Streaming API is used to consume Push Topic notifications for mappings that use accelerated polling.

In this article we’ll discuss how Heroku Connect uses these APIs to sync (read) data from Salesforce, as well as how to address common read errors.

SOAP API

Heroku Connect primarily uses the SOAP API for interactions with your Salesforce organization. The API has been optimized for operations involving a small number of records and is therefore used by Heroku Connect for the following tasks when there are up to 10,000 records to be processed:

• initially loading data from your Salesforce organization for a new mapping
• reloading all data into an existing mapping
• reading changes from your Salesforce organization

The SOAP API is also used for administrative tasks such as counting records and querying for mapping fields.

Lastly, the SOAP API is also used when writing data to Salesforce, though the threshold of records to be processed is lower. You can read more about this here.

Bulk API

The Bulk API is optimized for loading large sets of data making use of asynchronous processing to retrieve batches of records. Heroku Connect makes use of the Bulk API for the following tasks when there are more than 10,000 records to be read from Salesforce:

• initially loading data from your Salesforce organization for a new mapping
• reloading all data into an existing mapping
• reading changes from your Salesforce organization

The Bulk API can also be used when writing changes to Salesforce on read-write mappings, although the threshold of records to be processed is lower. You can read more about how the Bulk API is used when writing data to Salesforce here.

Streaming API

When you enable accelerated polling for a mapping, Heroku Connect uses the Streaming API to essentially listen for changes in Salesforce, rather than having to wait for changes to be seen in the next polling period. It does so by creating a Push Topic that will send notifications to trigger polling when data is changed in Salesforce.

The following API limit needs to be considered when configuring a mapping to use accelerated polling:

• Push Topics created by Heroku Connect count towards the ‘Maximum number of topics (PushTopic records) per organization’.

Heroku Connect’s Streaming API event consumption will not be included when you check your current Streaming API event usage in Salesforce:

• Log in to your Salesforce organization.
• Go to Setup and type Company Information in the Quick Find box.
• Click on Company Information.
• Streaming API Events, Last 24 Hours shows the number of notifications received and the maximum allowed. It does not include notifications received by Heroku Connect.

API call usage

The number of API calls used by Heroku Connect will depend on a number of factors:

• the total number of mappings
• the directionality of each mapping (read only or read/write)
• the synchronization mode of each mapping (polling or streaming) (see synchronization mode)

Depending on the API used and the directionality of the mapping, Heroku Connect will batch changes differently.

For changes to records being read from Salesforce, Heroku Connect will use:

• the SOAP API in batches of up to 2000 records OR
• the Bulk API in batches of up to 150,000 records

For changes to records being written to Salesforce, Heroku Connect will use:

• the SOAP API in batches of up to 200 records OR
• the Bulk API in batches of 2,000 to 10,000 records

None of these calls will count towards your daily SOAP or Bulk API limits.

Resolving read errors

Read errors can occur from either insufficient view permissions on the objects being mapped, unsupported object types, or problems with an object’s attributes.

View permissions

Heroku Connect requires “View All” permissions, which is scoped at the individual object level, for optimal operation. In certain cases where an object doesn’t have the option to set “View All” permissions, “View All Data” can be used instead. For Salesforce organizations with large record sets, some operations may not be possible without those permissions. The system will notify members or collaborators on your application if an operation requiring those permissions fails without them.

Unsupported objects

Heroku Connect supports read of all standard and custom objects. You can find a full list of supported objects here Knowledge Base objects, and standard or custom objects requiring special API handling or without required attributes such as SystemModStamp are not mappable. Heroku Connect attempts to detect and suppress these unmappable objects from presentation in the UI.

Common read errors

• INVALID_TYPE_FOR_OPERATION... : This or a similar error will be displayed in the logs when Heroku Connect cannot query for a mapped object.

• MALFORMED_QUERY... : This or a similar error will be displayed in the logs when Heroku Connect cannot map an object because it requires object specific filters.

• Cannot poll changes for table '>fieldname<' because it contains neither... : This or a similar error will be displayed in the logs when Heroku Connect cannot map an object because it lacks a required attribute.

• Skipped fields on >mapping name< not present in Salesforce: >fieldname<, >fieldname<... : This error will display in the logs when Heroku Connect cannot import a field in an exported configuration.

Remove the ineligible object or field from the mapping to resolve these errors.

Resolving Synchronization Issues

There are a number of issues that mappings on a connection can encounter that will require user intervention. This covers some of the more common issues a user might see.

BAD_CONFIG

Cause: After updating a mapping configuration, Connect performs validations to ensure that configuration is correct. We validate the following:

• The object is not a Big Object or a Platform Event.
• The object does not require a filter to query (e.g., CollaborationGroupRecord, ContentDocumentLink, ContentFolderItem, ContentFolderMember, FeedPollVote, IdeaComment, KnowledgeArticleVersion, Vote, etc.)
• The object is listed as “queryable” by the Salesforce API
• The object doesn’t have implementation restrictions (e.g., KnowledgeArticleVersion objects)
• The object has a timestamp field that can be used for synchronization (e.g., SystemModstamp)
• The object has an Id field
• If the mapping is set to read-write, the object must be updateable
• The fields of the mapping have known types that can be mapped into Postgres types

Resolution: If you’re seeing this error, you can diagnose this list and then edit and save your mapping to attempt to recover from this error.

ABORTED

Cause: Someone has pressed the “Abort” button for the mapping.

Resolution: To get your mapping to restart synchronization, you must reload your mapping

BULK_LOAD_ERROR

Cause: An error occurred while loading bulk data from Salesforce into your database that was not recoverable. Please see your logs to understand the error and determine if it’s transient or something requiring your intervention.

Some examples of cases that will require your intervention - you’ve defined a field with a specific length in Salesforce but Salesforce contains data larger than that. Connect mirrors the schema into PostgreSQL so if we try to insert data that is too large for the column definition, we will be unable to proceed. You’ll either need to correct the data in Salesforce or change the field definition in Salesforce to fix it.

Resolution: After resolving the root cause of the error, reload your mapping.