Reading Data from Salesforce with Heroku Connect

Last updated 14 May 2018

Heroku Connect employs a set of best practices that take into account data change volume and the many subtle details of Salesforce API operation to automatically choose the most efficient method to transfer data between your database and your Salesforce organization.

Salesforce offers a variety of APIs that are best suited to specific integration requirements and Heroku Connect utilizes two of them for data transfer: SOAP and Bulk. In addition the Streaming API is used to configure Push Topic notifications for mappings that use accelerated polling.

SOAP API

Heroku Connect primarily uses the SOAP API for interactions with your Salesforce organization. The API has been optimized for operations involving a small number of records and is therefore used by Heroku Connect for the following tasks when there are up to 10,000 records to be processed:

• initially loading data from your Salesforce organization for a new mapping
• reloading all data into an existing mapping
• reading changes from your Salesforce organization

The SOAP API is also used for:

• writing changes to your Salesforce organization regardless of the number of records involved
• administrative tasks such as counting records and querying for mapping fields

Bulk API

The Bulk API is optimized for loading large sets of data making use of asynchronous processing to retrieve batches of records in parallel. Heroku Connect makes use of the Bulk API for the following tasks when there are more than 10,000 records to be processed:

• initially loading data from your Salesforce organization for a new mapping
• reloading all data into an existing mapping
• reading changes from your Salesforce organization

Streaming API

When you enable accelerated polling for a mapping, Heroku Connect uses the Streaming API to create a Push Topic that will send notifications to trigger polling when data is changed in Salesforce.

Push Topics created by Heroku Connect have a hc_ naming prefix, for example hc_123, and will automatically be deleted either when you switch to polling mode or delete the mapping.

The following API limit needs to be considered when configuring a mapping to use accelerated polling:

• Push Topics created by Heroku Connect count towards the ‘Maximum number of topics (PushTopic records) per organization’.

Heroku Connect’s Streaming API event consumption will not be included when you check your current Streaming API event usage in Salesforce:

• Login to your Salesforce organization
• Go to Setup and type Company Information in the Quick Find box
• Click on Company Information
• Streaming API Events, Last 24 Hours shows the number of notifications received and the maximum allowed. It does not include notifications received by Heroku Connect.

API call usage

The number of API calls used by Heroku Connect will depend on a number of factors:

• the total number of mappings
• the directionality of each mapping (read only or read/write)
• the synchronization mode of each mapping (polling or streaming) (see synchronization mode)

Heroku Connect will primarily make use of the SOAP API, only switching to the Bulk API for read operations involving over 10,000 records. Bulk API calls make use of multiple asynchronously processed batches of 150,000 records. As an example, Heroku Connect will use the following API calls to perform a load of one million records:

• 1x SOAP API call to retrieve the number of records
• 7x Bulk API calls to retrieve the records in batches of 150,000

None of these calls will count towards your daily SOAP or Bulk API limits.

Heroku Connect will use the SOAP API for writing to Salesforce in batches of up to 200 records: for example writing 10,000 records will require at least 50 SOAP API calls. If you are using Ordered Writes and Heroku Connect detects a bulk batch sized INSERT, UPDATE or DELETE, it will write to Salesforce using the Bulk API in batches of between 2,000 and 10,000 records (see the Writing SOAP vs Bulk API documentation for more details).

Resolving read errors

Heroku Connect requires either “View All Data” permissions, or “View All”, which is scoped at the individual object level, for optimal operation. For Salesforce organizations with large record sets, some operations may not be possible without those permissions. The system will notify members or collaborators on your application if an operation requiring those permissions fails without them.

Heroku Connect supports read of all standard and custom objects. Knowledge Base objects, and standard or custom objects requiring special API handling or without required attributes such as SystemModStamp are not mappable. Heroku Connect attempts to detect and suppress these unmappable objects from presentation in the UI.

1. If Heroku Connect cannot query for a mapped object, this error or similar will display in the logs: 'INVALID_TYPE_FOR_OPERATION…’

2. If Heroku Connect cannot map an object because it requires object specific filters, this error or similar will display in the logs: 'MALFORMED_QUERY…’

3. If Heroku Connect cannot map an object because it lacks a required attribute, this error or similar will display in the logs: 'Cannot poll changes for table ’>fieldname<’ because it contains neither…’

Remove the ineligible object or field from the mapping to resolve these errors.