Reading Data from Salesforce with Heroku Connect

Last updated 08 August 2017

Heroku Connect employs a set of best practices that take into account data change volume and the many subtle details of Salesforce API operation to automatically choose the most efficient method to transfer data between your database and your Salesforce organization.

Salesforce offers a variety of APIs that are best suited to specific integration requirements and Heroku Connect utilizes two of them for data transfer: SOAP and Bulk. In addition the Streaming API is used to configure Push Topic notifications for mappings that use streaming mode.

Heroku Connect can only be used with Salesforce editions that have API access. Some plan types, including trial versions, do not have API access by default and cannot be used with Heroku Connect.

SOAP API

Heroku Connect primarily uses the SOAP API for interactions with your Salesforce organization. The API has been optimized for operations involving a small number of records and is therefore used by Heroku Connect for the following tasks when there are up to 20,000 records to be processed:

  • initially loading data from your Salesforce organization for a new mapping
  • reloading all data into an existing mapping
  • reading changes from your Salesforce organization

The SOAP API is also used for:

  • writing changes to your Salesforce organization regardless of the number of records involved
  • administrative tasks such as counting records and querying for mapping fields

SOAP API calls made by Heroku Connect do not count towards your API request limits. They are counted in your Salesforce org, but they do not count towards API license utilization.

Bulk API

The Bulk API is optimized for loading large sets of data making use of asynchronous processing to retrieve batches of records in parallel. Heroku Connect makes use of the Bulk API for the following tasks when there are more than 20,000 records to be processed:

  • initially loading data from your Salesforce organization for a new mapping
  • reloading all data into an existing mapping
  • reading changes from your Salesforce organization

Bulk API calls do not count towards your Bulk API limits. Bulk data load jobs created by Heroku Connect will appear in your list of jobs in Salesforce, but will not increment your count of batches processed in the last 24 hours.

Streaming API

When you choose streaming mode for a mapping, Heroku Connect uses the Streaming API to create a Push Topic that will send notifications to trigger polling when data is changed in Salesforce.

Push Topics created by Heroku Connect have a hc_ naming prefix, for example hc_123, and will automatically be deleted either when you switch to polling mode or delete the mapping.

Streaming API calls do count towards the Streaming API limits that apply to your Salesforce organization.

The following API limits need to be considered when configuring a mapping to use streaming mode:

  • Push Topics created by Heroku Connect count towards the ‘Maximum number of topics (PushTopic records) per organization’.
  • Notifcations received by Heroku Connect count towards the 'Maximum number of events per day (24–hour period)’.

You can check your current notification usage in Salesforce:

  • Login to your Salesforce organization
  • Go to Setup and type Company Information in the Quick Find box
  • Click on Company Information
  • Streaming API Events, Last 24 Hours shows the number of notifications received and the maximum allowed

The usage information includes all notifications sent from your organization and not just those used by Heroku Connect.

API call usage

The number of API calls used by Heroku Connect will depend on a number of factors:

  • the total number of mappings
  • the directionality of each mapping (read only or read/write)
  • the synchronization mode of each mapping (polling or streaming) (see synchronization mode)

Heroku Connect will primarily make use of the SOAP API, only switching to the Bulk API for read operations involving over 20,000 records. Bulk API calls make use of multiple asynchronously processed batches of 150,000 records. As an example, Heroku Connect will use the following API calls to perform a load of one million records:

  • 1x SOAP API call to retrieve the number of records
  • 7x Bulk API calls to retrieve the records in batches of 150,000

None of these calls will count towards your daily SOAP or Bulk API limits.

Heroku Connect will exclusively use the SOAP API for writing to Salesforce in batches of 200 records: for example writing 10,000 records will require 50 SOAP API calls.

Resolving read errors

Heroku Connect requires “View All Data” permissions, or “View all Data” scoped to the mapped objects, for optimal operation. For Salesforce organizations with large record sets, some operations may not be possible without those permissions. The system will notify members or collaborators on your application if an operation requiring those permissions fails without them.

Heroku Connect supports read of all standard and custom objects. Knowledge Base objects, and standard or custom objects requiring special API handling or without required attributes such as SystemModStamp are not mappable. Heroku Connect attempts to detect and suppress these unmappable objects from presentation in the UI.

  1. If Heroku Connect cannot query for a mapped object, this error or similar will display in the logs: 'INVALID_TYPE_FOR_OPERATION…’

  2. If Heroku Connect cannot map an object because it requires object specific filters, this error or similar will display in the logs: 'MALFORMED_QUERY…’

  3. If Heroku Connect cannot map an object because it lacks a required attribute, this error or similar will display in the logs: 'Cannot poll changes for table ’>fieldname<’ because it contains neither…’

Remove the ineligible object or field from the mapping to resolve these errors.

Feedback

Log in to submit feedback.