Reading Data from Salesforce with Heroku Connect
Last updated 02 October 2018
Table of Contents
Heroku Connect employs a set of best practices that take into account data change volume and the many subtle details of Salesforce API operation to automatically choose the most efficient method to transfer data between your database and your Salesforce organization.
Salesforce offers a variety of APIs that are best suited to specific integration requirements and Heroku Connect utilizes two of them for data transfer: SOAP and Bulk. In addition the Streaming API is used to configure Push Topic notifications for mappings that use accelerated polling.
Heroku Connect can only be used with Salesforce editions that have API access. Some plan types, including trial versions, do not have API access by default and cannot be used with Heroku Connect.
SOAP API
Heroku Connect primarily uses the SOAP API for interactions with your Salesforce organization. The API has been optimized for operations involving a small number of records and is therefore used by Heroku Connect for the following tasks when there are up to 10,000 records to be processed:
- initially loading data from your Salesforce organization for a new mapping
- reloading all data into an existing mapping
- reading changes from your Salesforce organization
The SOAP API is also used for:
- writing changes to your Salesforce organization regardless of the number of records involved
- administrative tasks such as counting records and querying for mapping fields
SOAP API calls made by Heroku Connect do not count towards your API request limits. They are counted in your Salesforce org, but they do not count towards API license utilization.
Bulk API
The Bulk API is optimized for loading large sets of data making use of asynchronous processing to retrieve batches of records in parallel. Heroku Connect makes use of the Bulk API for the following tasks when there are more than 10,000 records to be processed:
- initially loading data from your Salesforce organization for a new mapping
- reloading all data into an existing mapping
- reading changes from your Salesforce organization
Bulk API calls do not count towards your Bulk API limits. Bulk data load jobs created by Heroku Connect will appear in your list of jobs in Salesforce, but will not increment your count of batches processed in the last 24 hours.
Streaming API
When you enable accelerated polling for a mapping, Heroku Connect uses the Streaming API to create a Push Topic that will send notifications to trigger polling when data is changed in Salesforce.
Push Topics created by Heroku Connect have a hc_ naming prefix, for example hc_123, and will automatically be deleted either when you switch to polling mode or delete the mapping.
Streaming API calls do not count towards the daily Streaming API event limit that applies to your Salesforce organization, provided that your connection has been authorized on April 5, 2017 or later.
The following API limit needs to be considered when configuring a mapping to use accelerated polling:
- Push Topics created by Heroku Connect count towards the ‘Maximum number of topics (PushTopic records) per organization’.
Heroku Connect’s Streaming API event consumption will not be included when you check your current Streaming API event usage in Salesforce:
- Login to your Salesforce organization
- Go to Setup and type Company Information in the Quick Find box
- Click on Company Information
- Streaming API Events, Last 24 Hours shows the number of notifications received and the maximum allowed. It does not include notifications received by Heroku Connect.
API call usage
The number of API calls used by Heroku Connect will depend on a number of factors:
- the total number of mappings
- the directionality of each mapping (read only or read/write)
- the synchronization mode of each mapping (polling or streaming) (see synchronization mode)
Heroku Connect will primarily make use of the SOAP API, only switching to the Bulk API for read operations involving over 10,000 records. Bulk API calls make use of multiple asynchronously processed batches of 150,000 records. As an example, Heroku Connect will use the following API calls to perform a load of one million records:
- 1x SOAP API call to retrieve the number of records
- 7x Bulk API calls to retrieve the records in batches of 150,000
None of these calls will count towards your daily SOAP or Bulk API limits.
Heroku Connect will use the SOAP API for writing to Salesforce in batches of up to 200 records: for example writing 10,000 records will require at least 50 SOAP API calls. If you are using Ordered Writes and Heroku Connect detects a bulk batch sized INSERT, UPDATE or DELETE, it will write to Salesforce using the Bulk API in batches of between 2,000 and 10,000 records (see the Writing SOAP vs Bulk API documentation for more details).
Resolving read errors
Heroku Connect requires either “View All Data” permissions, or “View All”, which is scoped at the individual object level, for optimal operation. For Salesforce organizations with large record sets, some operations may not be possible without those permissions. The system will notify members or collaborators on your application if an operation requiring those permissions fails without them.
Heroku Connect supports read of all standard and custom objects. Knowledge Base objects, and standard or custom objects requiring special API handling or without required attributes such as SystemModStamp are not mappable. Heroku Connect attempts to detect and suppress these unmappable objects from presentation in the UI.
If Heroku Connect cannot query for a mapped object, this error or similar will display in the logs:
INVALID_TYPE_FOR_OPERATION...If Heroku Connect cannot map an object because it requires object specific filters, this error or similar will display in the logs:
MALFORMED_QUERY...If Heroku Connect cannot map an object because it lacks a required attribute, this error or similar will display in the logs:
Cannot poll changes for table '>fieldname<' because it contains neither...If Heroku Connect cannot import a field in an exported configuration, this error will display in the logs:
Skipped fields on >mapping name< not present in Salesforce: >fieldname<, >fieldname<...
Remove the ineligible object or field from the mapping to resolve these errors.