Heroku Connect Log Messages and Common Errors

Last updated 28 September 2016

Heroku Connect keeps logs of its sync activity, which you can access from your Connect Dashboard by clicking the Logs tab. This article documents some of the errors and messages you may see in those logs.

Attempted to open more than 10 queries to SFDC

Salesforce enforces a limit of 10 open queries per connected user. Heroku Connect synchronizes each mapping independently, and it will often sync several at the same time. Connect keeps track of how many queries it’s running across all of your mappings, and it won’t try to run more than 10 in parallel for each connection.

However, if you’re using the same user credentials elsewhere (or with another Heroku Connect connection), you might end up attempting to use more than 10 queries at a time. If you’re having this problem and you have multiple connections, try authenticating each connection with a different user account.

SYSTEM_ERROR (Unexpected error)

If your connection is reporting a SYSTEM_ERROR state, you can restart the connection from the Connect Dashboard to resume synchronization. Click the “Settings” link, then click “Manage Connection”. From there, you can click the “Restart” button to restart synchronization.

(psycopg2.IntegrityError) duplicate key value violates unique constraint “lead_idx_rpm_account_id__c”

This problem can have several different causes.

Sandbox Refresh

If you’re refreshed a sandbox reorganization without destroying and recreating the Connect connection, you might get some of these errors. We strongly recommend creating a new connection when you reload a sandbox instead of trying to reuse the existing one.

Insert Round-Trip Issues

If the conflict is on an SFID field, such as "contact_idx_sfid", it might be due to a race condition related to inserting records into your database. When you insert a new record into your database and Connect pushes that record to Salesforce, Connect will wait to receive the success response from Salesforce and update the database record with the newly created row’s SFID. Connect will sometimes also get that row back as part of its regular polling while the initial insert is still in process. This results in a race condition where both processes are attempting to write a record with the same SFID. This can cause unique index conflicts, but it can also cause duplicate records in the database where one record is missing the SFID.

The workaround is to add an external ID to any object you’ll be creating from the database side, and make sure to populate each new record with a unique ID (such as a GUID) when you create it. Then, add a unique index on that external ID. That way, if Connect attempts to write a duplicate record, the unique index will cause the write to fail. You’ll still see index violations in your logs, but if it’s on the external ID field, which means a duplicate record was prevented.

If you add an external ID field to an object, it’s important not to change that ID once it’s set.

Unable to reload mapping with id: abcdef12…

If you’ve recently changed a mapping or imported a configuration, you may see this error. Check the status of the mapping in your Connect Dashboard. If you see BULK_LOADING, this means that Connect is in the middle of a bulk loading operation that can’t be interrupted. You’ll need to wait until the operation is complete to reload the mapping.

QUERY_TIMEOUT: Your query request was running for too long. Trying again later.;

This error usually results in multiple error messages that look something like this:

Batch 751m0000001AoAtAAK of job 750m0000001BfWOAA0 failed: InternalServerError : Retried more than 15 times; InvalidBatch : Failed to process query: QUERY_TIMEOUT: Your query request was running for too long. Trying again later.; InvalidBatch : Failed to process query: QUERY_TIMEOUT: Your query request was running for too long. Trying again later.; InvalidBatch : Failed to process query: QUERY_TIMEOUT: Your query request was running for too long. Trying again later.; (...)

This error comes directly from Salesforce. Connect uses the bulk API to load large amounts of data. Occasionally, a bulk query will time out, and Connect will automatically retry the query. If the query continues to time out after several retries, you’ll need to open a ticket with Salesforce to investigate why it’s happening.

The Salesforce configuration saved for this object no longer maps to the current Salesforce data model

If you make changes to mapped objects in Salesforce, such as changing the data type of a field, Heroku Connect will not automatically update your database schema. You may see errors such as this one as a result. To resolve the issue, edit the mapping in your Connect Dashboard.

“Server raised fault: ‘REQUEST_LIMIT_EXCEEDED: ConcurrentPerOrgApex Limit exceeded” or “System.LimitException: Apex CPU time limit exceeded”

When you write records into a table configured to send updates to Salesforce, Connect will periodically poll your database and group many records—up to 200—into a single request to the Salesforce API.

If your object has Apex code that gets triggered when new records are created, Salesforce imposes limits on the amount of computing resources it dedicates to those triggers. As a result, you may see one of the above errors if you insert more records than usual. This Salesforce guide describes some ways to optimize your Apex code to improve performance.

If this continues to happen, contact Heroku support about lowering your write batch size. We can limit your insert batches to 100 or 50 records, but this may also increase the latency of inserting new records.

duplicate value found: Contact_External_ID__c duplicates value on record with id: a006…

This error indicates that Connect tried to push a record to Salesforce, but Salesforce wasn’t able to save it. In this case, Connect was trying to insert an object with a SampleExternalID that already exists and is supposed to be unique.

In some cases, this can be caused by a race condition in Heroku Connect itself. If you are inserting records into the database, it’s a good idea to add and populate an External ID field for each record.

Expired transition aborted by health_check

This message appears in your logs when a sync worker dyno has hung. Every 15 minutes, we run a health check on your mappings from a different dyno, and if the health check notices a hung process, it will kill it and reset the state of the mapping so it starts polling again. This can happen for a variety of reasons, such as temprorary network outages.

query_wait_timeout: server closed the connection unexpectedly

This can happen if the database suddenly disconnects during a sync operation. Connect should recover automatically, but if it doesn’t, you can restart your connection from the Connect Dashboard.

Streaming error: 403::Organization total events daily limit exceeded

This error means that you’ve used up your available PushTopic events. Sync will continue, but it will fall back to the polling interval you’ve configured for each mapping. Events will resume automatically when the rolling 24-hour window resets.

function get_xmlbinary() does not exist

Connect creates a function called get_xmlbinary in the public schema of your database, which it uses as part of the sync process. If you remove the public schema from your search_path, Connect won’t be able to find that function, and sync will fail.

Too many polling errors. Polling will be disabled for up to 15 minute(s)

This error indicates that polling the database or Salesforce has resulted in consecutive and unexpected failures. As a result, Connect will back off polling for the specified duration to allow time for the underlying problem to be resolved. You can look for a notification at the top of your Connect Dashboard that might explain the problem, or go to the Logs tab and filter for Error log lines to investigate. These issues are often related to your database schema being out of date, which can happen if you change your Salesforce object fields without editing your Connect mapping to update it.

Too many bulk errors, bulk usage will be disabled for up to 2 hour(s)

This error indicates that bulk polling requests to Salesforce have resulted in consecutive and unexpected failures. As a result, Connect will back off bulk polling for the specified duration to allow time for the underlying problem to be resolved. You can look for a notification at the top of your Connect Dashboard that might explain the problem, or go to the Logs tab and filter for Error log lines to investigate. These issues are often related to exceeding bulk usage limits in Salesforce, which you should be able to see on your Connect Dashboard.

SERVER_UNAVAILABLE: Request timed out waiting for connection

You may see something similar to this message in your logs:

faultcode = "sf:SERVER_UNAVAILABLE" faultstring = "SERVER_UNAVAILABLE: Request timed out waiting for connection: [config 1,000ms, actual 1,001ms]" detail = (detail){ UnexpectedErrorFault = (UnexpectedErrorFault){ exceptionCode = "SERVER_UNAVAILABLE" exceptionMessage = "Request timed out waiting for connection: [config 1,000ms, actual 1,001ms]" } } }

This error happens when Salesforce has problems connecting to an internal database. It should be a transient issue that resolves itself in a few minutes. If it happens repeatedly, please contact Heroku support.

Server raised fault: 'OPERATION_TOO_LARGE: exceeded 100000 distinct who/what’s’

OPERATION_TOO_LARGE errors are the result of not granting the View All Data permission to your authenticated Salesforce user. Without this permission, Salesforce has to run additional queries to check if your user has permission to view each row. For larger mappings, those queries can result in OPERATION_TOO_LARGE errors. Unfortunately, the only way to fix this is to remove the need for permissions queries by granting View All Data for this object, and if necessary, related objects.