Heroku Connect Log Messages and Common Errors

Last updated 12 February 2020

Heroku Connect produces logs of its sync activity, which you can access from your log drain (see also, Heroku Connect Application Logging). You can also view the logs as they are emitted by clicking the Logs tab. This article documents some of the errors and messages you may see in those logs. For information about Heroku External Objects logging, please visit the Logging documentation for Heroku External Objects.

Log Messages

Summary for Opportunity status=DATA_SYNCED db=31 sf=31 pending=0 errors=0

If you’ve enabled application logging for your connection, you’ll see a variety of log lines related to Connect activity, including this one. This status line is logged periodically for each mapping. It contains these fields:

  • status: DATA_SYNCED means the mapping is fully synchronized. You may occasionally see other statuses relating to polling or bulk updates. If you see a status of SYSTEM_ERROR, you should visit your Connect Dashboard to investigate the error and resume your connection.
  • db: The current number of rows in the database table.
  • sf: The current number of rows in Salesforce. Please note that this count can occasionally be delayed.
  • pending: The count of records in the trigger log that are waiting to be written to Salesforce.
  • errors: The number of records in the trigger log that failed to write to Salesforce. If this number is not 0, you should query the trigger log for records where state = 'FAILED' and possibly attempt to retry writing them to Salesforce.

[2/34] Completed Bulk Batch … from Bulk Job …

This message is logged when synchronizing from Salesforce to PostgreSQL using the Bulk API. This will come after messages that indicate we’ve inserted or updated a number of records into your database. It is intended to help you understand how many batches are left before normal synchronization continues.

[31/267] Completed 31 Empty Bulk Batches from Bulk Job …

This message is logged when synchronizing from Salesforce to PostgreSQL using the Bulk API. When we can, Connect uses Native PK Chunking and this can generate empty bulk batches in Salesforce. To save time when processing the bulk update to your data, we skip any batches that Salesforce tells us has no data. To provide context and a complete picture of progress, we log this message.

Common Log Messages

If you’ve enabled application logging for your connection, you may see a variety of messages that match a pattern similar to:

$number $object_name $verb $arrow $salesforce_or_db $total_number ($timing secs) [$api]

Below are some examples that are not exhaustive along with explanations:

1 Custom_Contact__c UPDATE ↑SALESFORCE 1 rows total (0.26 secs) [SOAP]

1 Custom_Contact__c record was updated in PostgreSQL and written to Salesforce. It took Salesforce 0.26 seconds to process the write sent via the SOAP API.

1 Custom_Account__c DELETE ↑SALESFORCE 1 rows total (0.44 secs) [SOAP]

1 Custom_Contact__c record was deleted in PostgreSQL and deleted from Salesforce. It took Salesforce 0.44 seconds to process the delete sent via the SOAP API.

Lead QUERY ↷SALESFORCE 1 rows (0.16 secs)

A Lead record was updated via a SOAP query. Applying the update took 0.16 seconds.

Update_Table_Bulk__c UPDATE ↓DATABASE 5000 rows (4.15 secs)

A bulk number of records were changed in the Update_Table_Bulk__c object. 5000 of those changes were applied in bulk to PostgreSQL and the bulk update took 4.15 seconds.

Custom_Account__c, Streaming poll found 1 changes

The Custom_Account__c object has Accelerated Polling enabled. An accelerated poll happened as a result and updated 1 record prior to the next scheduled poll.

Custom_Account__c DELETE ↓DATABASE 1 rows (0.01 secs)

A record in Custom_Account__c was deleted in Salesforce. It took 0.01 seconds to delete that record from PostgreSQL.

6 Custom_Account_c, 5 Custom_Contact_c INSERT ↑SALESFORCE 11 rows total (0.64 secs) [SOAP]

6 new Custom_Account_c records and 5 new Custom_Contact_c records were created in Salesforce via the SOAP API in a single request. It took Salesforce 0.64 seconds to process the newly created records.

Account QUERY ↷SALESFORCE 2300 rows (1.80 secs)

2300 records were found by a soap query from Salesforce in 1.8 seconds. This does not indicate that this message from Salesforce contained 2300 records. The maximum number of records that can be in a message is 2000. This message will be followed by a QUERY MORE message like: Account QUERY MORE ↷SALESFORCE 300 rows (0.70 secs) which indicates that another data page was fetched from Salesforce and a log message like Account UPDATE ↓DATABASE 2000 rows (2.15 secs) which indicates that the a page of data was written to your PostgreSQL database.

Account QUERY MORE ↷SALESFORCE 300 rows (0.70 secs)

300 Account records were fetched from Salesforce and will be written into your PostgreSQL database. This is a continuation of a prior SOAP query to retrieve updates. When you see QUERY MORE, this indicates that a QueryLocator cursor is in use.

This will be preceded by a similar log message that looks like: Account QUERY ↷SALESFORCE 2300 rows (1.80 secs).

See also:

Log Errors

Attempted to Open More Than 10 Queries to SFDC

See Operation delayed, no available query cursors

Operation delayed, no available query cursors

Salesforce enforces a limit of 10 open queries per connected user. Heroku Connect synchronizes each mapping independently, and it will often synchronize many at the same time. This message indicates Connect throttled itself to avoid exceeding the limit and further delaying synchronization,

If you are consistently seeing this message it may indicate there is an issue worth investigating. Some possibilities are:

  1. You are using the same user credentials for multiple Connect instances. Authenticate with Salesforce with a different user for each Connection.
  2. You have a large number of active mappings. Consider reducing the number of mappings you’ve created.
  3. You have suddenly started seeing this message on all your Mappings and no records are being synchronized. Please file a support ticket for our team to investigate.

SYSTEM_ERROR (Unexpected error)

If your connection is reporting a SYSTEM_ERROR state, this indicates an unexpected situation occurred that interrupted synchronization. If the situation was temporary, you may be able to recover the connection using the Connect Dashboard. Click the Settings link, then click Manage Connection. From there, you can click the Recover button to clear out the error state and attempt to resume synchronization.

If the underlying problem persists, the connection will end up in the same state, and will require further investigation and manual repair. Check your connection’s logs for any further details, and if you’re unable to resolve the issue, please contact Heroku support for investigation.

(psycopg2.IntegrityError) duplicate key value violates unique constraint “lead_idx_rpm_account_id__c”

This problem can have several different causes.

Insert Round-Trip Issues

If the conflict is on an SFID field, such as "contact_idx_sfid", it might be due to a race condition related to inserting records into your database. When you insert a new record into your database and Connect pushes that record to Salesforce, Connect will wait to receive the success response from Salesforce and update the database record with the newly created row’s SFID. Connect will sometimes also get that row back as part of its regular polling while the initial insert is still in process. This results in a race condition where both processes are attempting to write a record with the same SFID. This can cause unique index conflicts, but it can also cause duplicate records in the database where one record is missing the SFID.

The workaround is to add an external ID to any object you’ll be creating from the database side, and make sure to populate each new record with a unique ID (such as a GUID) when you create it. Then, add a unique index on that external ID. That way, if Connect attempts to write a duplicate record, the unique index will cause the write to fail. You’ll still see index violations in your logs, but if it’s on the external ID field, which means a duplicate record was prevented.

If you add an External ID field to an object, it’s important to not change that ID once it is set.

Error during sync: (psycopg2.IntegrityError) update or delete on table “mycustomobject__c” violates foreign key constraint…

You should not specify foreign key constraints on mapped tables or referencing mapped tables. Heroku Connect sometimes needs to reload mapped tables, and foreign key constraints will prevent deletion/truncation of records in the Postgres table to clean up before reloading. This failure to delete will generally lead to the error:

Too many sync errors. Sync operations may be disabled for up to 15 minute(s).

With Heroku Connect, the system of record is your Salesforce org, not the Postgres database. Relationships should be enforced in your Salesforce org. Heroku Connect does not support Postgres foreign key constraints.

Error during sync: This Connection is closed

If you see this error upon reloading a mapping, it’s likely that the database table is being referenced in a foreign key constraint. When Heroku Connect reloads a mapping, it attempts to truncate the database table, which isn’t possible if the table is part of a foreign key constraint. You’ll need to remove the foreign key constraint first, and then reload the mapping.

Unable to Reload Mapping with ID: abcdef12…

If you’ve recently changed a mapping or imported a configuration, you may see this error. Check the status of the mapping in your Connect Dashboard. If you see BULK_LOADING, this means that Connect is in the middle of a bulk loading operation that can’t be interrupted. You’ll need to wait until the operation is complete to reload the mapping.

QUERY_TIMEOUT: Your query request was running for too long. Trying again later.

This error usually results in multiple error messages that look something like this:

Batch 751m0000001AoAtAAK of job 750m0000001BfWOAA0 failed: InternalServerError : Retried more than 15 times; InvalidBatch : Failed to process query: QUERY_TIMEOUT: Your query request was running for too long. Trying again later.; InvalidBatch : Failed to process query: QUERY_TIMEOUT: Your query request was running for too long. Trying again later.; InvalidBatch : Failed to process query: QUERY_TIMEOUT: Your query request was running for too long. Trying again later.; (...)

This error comes directly from Salesforce. Connect uses the bulk API to load large amounts of data. Occasionally, a bulk query will time out, and Connect will automatically retry the query. If the query continues to time out after several retries, you’ll need to open a ticket with Salesforce to investigate why it’s happening.

Batch 54321000007DvJXYZ0 of job 54321000005lfIZXYZ failed: InvalidBatch : Failed to process query: INVALID_FIELD

This error typically indicates there has been some amount of schema drift between a mapping’s configuration and Salesforce. Most often a field has been removed or renamed in Salesforce, but that field remains in the mapping’s configuration. To correct the issue, update the mapping configuration removing the invalid field.

In other instances this error may indicate a permission issue, where a particular field is included in a mapping’s configuration, but is no longer visible to the Salesforce user on the connection. To resolve the issue, try re-authenticating with different Salesforce user credentials or modifying the permissions of the current user. Once the permission issue is corrected, you can either update the mapping configuration by removing the field(s) and then adding them back to force a field level resync, or you can reload the entire mapping.

Salesforce Configuration Saved for This Object no Longer Maps to the Current Salesforce Data Model

If you make changes to mapped objects in Salesforce, such as changing the data type of a field, Heroku Connect will not automatically update your database schema. You may see errors such as this one as a result. To resolve the issue, edit the mapping in your Connect Dashboard.

“Server Raised Fault: ‘REQUEST_LIMIT_EXCEEDED: ConcurrentPerOrgApex Limit exceeded” or “System.LimitException: Apex CPU time limit exceeded”

When you write records into a table configured to send updates to Salesforce, Connect will periodically poll your database and group many records—up to 200—into a single request to the Salesforce API.

If your object has Apex code that gets triggered when new records are created, Salesforce imposes limits on the amount of computing resources it dedicates to those triggers. As a result, you may see one of the above errors if you insert more records than usual. This Salesforce guide describes some ways to optimize your Apex code to improve performance.

If this continues to happen, contact Heroku support about lowering your write batch size. We can limit your insert batches to 100 or 50 records, but this may also increase the latency of inserting new records.

DB is Missing Connect Metadata [DatabaseValidationFailed]

Indicates the Database Connect is connecting to is not the same as the one it was initialized against. This usually occurs when the Database Upgrade procedure has not been followed. If you are unable to recover into a working state a quick solution could be to Recreate the connection.

This could also be encountered if a process has deleted the _hc_meta table that Heroku Connect uses to validate that it is talking to the right database for the right connection. Again, recreating the connection would be the quickest solution to getting back into a working state.

“Expired Transition Aborted by Health_Check” or “Expired transition aborted by sweep”

Both of these messages appear in your logs when a sync worker dyno has hung. Every 15 minutes, we run a health check on your mappings from a different dyno, and if the health check notices a hung process, it will kill it and reset the state of the mapping so it starts polling again. This can happen for a variety of reasons, such as temporary network outages or dynos restarting.

Query_Wait_Timeout: Server Closed the Connection Unexpectedly

This can happen if the database suddenly disconnects during a sync operation. Connect should recover automatically, but if it doesn’t, you can resume your connection from the Connect Dashboard.

Streaming Error: 403::Organization Total Events Daily Limit Exceeded

This error means that you’ve used up your available PushTopic events from the Salesforce Streaming API. You can avoid this error by taking advantage of free unlimited Salesforce Streaming API events.

Note that when you get the error, sync will continue, but it will fall back to the polling interval you’ve configured for each mapping. Events will resume automatically when the rolling 24-hour window resets.

Function get_xmlbinary() Does Not Exist

Connect creates a function called get_xmlbinary in the public schema of your database, which it uses as part of the sync process. If you remove the public schema from your search_path, Connect won’t be able to find that function, and sync will fail.

Too Many Polling Errors. Polling will be Disabled for Up to 15 minute(s)

This error indicates that polling the database or Salesforce has resulted in consecutive and unexpected failures. As a result, Connect will back off polling for the specified duration to allow time for the underlying problem to be resolved. You can look for a notification at the top of your Connect Dashboard that might explain the problem, or go to the Logs tab and filter for Error log lines to investigate. These issues are often related to your database schema being out of date, which can happen if you change your Salesforce object fields without editing your Connect mapping to update it.

Too Many Bulk Errors, Bulk Usage will be Disabled for up to 2 hour(s)

This error indicates that bulk polling requests to Salesforce have resulted in consecutive and unexpected failures. As a result, Connect will back off bulk polling for the specified duration to allow time for the underlying problem to be resolved. You can look for a notification at the top of your Connect Dashboard that might explain the problem, or go to the Logs tab and filter for Error log lines to investigate. These issues are often related to exceeding bulk usage limits in Salesforce, which you should be able to see on your Connect Dashboard.

Too many sync errors. Sync operations may be disabled for up to 15 minute(s)

This error indicates that there were too many unexpected failures trying to synchronize data in the connection or mapping. As a result, Connect will back off in trying to execute synchronization. After the specified period of time, Connect will try again to synchronize data.

SERVER_UNAVAILABLE: Request Timed out waiting for Connection

You may see something similar to this message in your logs:

faultcode = "sf:SERVER_UNAVAILABLE" faultstring = "SERVER_UNAVAILABLE: Request timed out waiting for connection: [config 1,000ms, actual 1,001ms]" detail = (detail){ UnexpectedErrorFault = (UnexpectedErrorFault){ exceptionCode = "SERVER_UNAVAILABLE" exceptionMessage = "Request timed out waiting for connection: [config 1,000ms, actual 1,001ms]" } } }

This error happens when Salesforce has problems connecting to an internal database. It should be a transient issue that resolves itself in a few minutes. If it happens repeatedly, please contact Heroku support.

Server Raised Fault: 'OPERATION_TOO_LARGE: exceeded 100000 distinct who/what’s’

OPERATION_TOO_LARGE errors are the result of not granting the View All permission to your authenticated Salesforce user. Without this permission, Salesforce has to run per-row checks to determine if your user has permission to view that row. For larger mappings, the overhead of evaluating permissions may result in OPERATION_TOO_LARGE errors. The only way to resolve this error is to remove the need for permissions queries by granting the View All permission for this object, and if necessary, related objects.

Parsing SOAP Response Failed: SAXParseException: <unknown>:1:0: syntax error

This error indicates that Heroku Connect made a SOAP request to Salesforce but the response received was not valid SOAP XML. This is a transient condition that may occur if Salesforce experiences an internal error. Heroku Connect will automatically retry the SOAP request.

SOAP Transport Error: BadStatusLine

This error indicates that the response from Salesforce was malformed. This error is transient in nature. As such, subsequent attempts to sync the record(s) which caused this error should succeed. This error does not represent any serious issues with your Connect mappings.

(psycopg2.OperationalError) FATAL: Too Many Connections for Role

This error occurs when Connect tries to open a connection to your database but you’ve already reached your connection limit. For very active connections, Connect may use 10 or more database connections. For this reason, we strongly advise against using hobby tier database plans with Connect.

CREATE TEMPORARY TABLE _tmp_sf…

You may see statements related to a temporary table appear in your log stream:

CREATE TEMPORARY TABLE _tmp_sf_contact AS (...)
CREATE UNIQUE INDEX ON
         _tmp_sf_contact (sfid)
CREATE INDEX ON
         _tmp_sf_contact (__operation, sfid)

These temporary tables are created by the Connect sync process in order to more efficiently sync inserts and updates from Salesforce to your database. They do not cause any performance impact to your database, and they are safe to ignore.

Value does not exist or does not match filter criteria

This error can happen if you attempt to insert an invalid value into a picklist field. Connect supports the picklist data type, but does not provide any data validation.

Missed Updates Due to Salesforce Transactions

It is possible that a small set of records may be locked in a transaction at the time of a poll and therefore not available to be synced. Heroku Connect handles this by performing a poll for Salesforce IDs that should have been included 2 minutes after successful poll completion. Any missed records will automatically be synced when detected. This “insurance” poll reconciles records in flight at time of last sync operation.

urlopen error [Errno -3] Temporary failure in name resolution

This is an issue with DNS resolution within the Heroku Connect service. It generally occurs due to transient networking errors. Heroku Connect is resilient to intermittent issues like this and will continue to synchronize data when the DNS resolution issue is resolved.

Errors with Associated Notifications

When errors occur that require user intervention, Heroku Connect sends notifications to all users that have chosen to receive them. To view a list of users that will receive notifications, navigate to the Settings tab, Manage Connection in the dashboard: the Access section lists users who have access to the Connect dashboard and is where the Notifications Enabled setting can be configured.

You will receive notifications for the errors summarized below. The notification message describes the error that has occurred and includes helpful information so that you can understand and resolve the problem. You will also receive notifications when an error that occurred previously has been resolved.

Bad Configuration

A bad configuration error can occur when changes to the Salesforce object or Heroku Postgres table used by a mapping are changed. The mapping will not be synchronized until the error is resolved however other mappings should continue to synchronize as normal.

Change User Permissions

This error occurs when the user credentials you use to authenticate to Salesforce don’t have the correct permissions to allow Heroku Connect to efficiently synchronize a large number of records. Re-authenticating with different Salesforce user credentials or modifying the permissions of the current user will resolve the error.

Connection Error

A connection error can occur for a number of reasons and may prevent synchronization from occurring on one or more of your mappings. You will be asked to check the logs in the Heroku Connect dashboard in order to identify the exact nature of the error.

Database Unavailable

If the Heroku Postgres database cannot be reached then Heroku Connect will be unable to synchronize any mappings. You will need to review your database configuration in order to resolve this problem.

Reauthorization Required

If the user credentials used to authorize Heroku Connect with Salesforce expire or change then synchronization will stop for all mappings and you will need to reauthorize with Salesforce in order for it to resume.

Salesforce Bulk Load Error

This error occurs when there was a problem during the initial bulk load or reload of a mapping. You can investigate the cause of the error in the Heroku Connect dashboard by viewing the Logs tab and filtering for Error log lines. In most cases these errors are due to bulk batch failures. A failure during a bulk load operation can leave a mapping in an inconsistent state, and cannot be automatically recovered from. After identifying and correcting the cause of the error, the mapping should be reloaded.

Connection Reset by Peer

This error is caused by intermittent network connectivity issues and can typically be ignored as Connect will detect this condition and retry the operation. If you observe frequent occurrences of this error message, we recommend that you reach out to Heroku support for assistance.

Answers to the most frequently asked questions are available in the Heroku Connect FAQ article.

Delayed Synchronization

This will correlate with the error message: “Too many sync errors. Sync operations may be disabled for up to 15 minute(s)”. This notification alerts you to the fact that synchronization will be delayed by at least 15 minutes. Connect will automatically attempt to retry synchronization after the period of time specified in the log message. This may happen for one or more of your mappings as well as the connection itself.

Audit Messages

Along with the operational log messages listed above, Heroku Connect also keeps a record of administrative changes to your connection’s configuration. These messages indicate who made a change, as well as when that change was made. In addition, a few operations that affect the state of the connection or its data are also included. Some examples include:

  • Updating connection settings
  • Adding, modifying and deleting a mapping
  • Importing a previously-exported configuration file
  • Reloading a mapping’s data from Salesforce
  • Aborting a currently-running operation
  • Pausing and resuming sync operations
  • Enabling and disabling External Objects support

In addition to being visible on the logs page of your dashboard, a CSV download of your connection’s activity can be downloaded from the Manage Connection page.

Feedback

Log in to submit feedback.