Last updated March 14, 2023

Heroku Connect produces logs of its sync activity, which you can access from your log drain. You can also view the logs as they’re emitted by clicking the Logs tab. This article documents some of the errors and messages you see in those logs. For information about Heroku External Objects logging, visit the Logging documentation for Heroku External Objects.

Application Logging

Heroku Connect emits log messages to Logplex making them available to your application via the CLI or using logging add-ons.

Log entries follow the standard log format with a source of app, a dyno of herokuconnect, and a message that contains structured data including standard fields such as event and level.

Here’s an example of the Logplex entries for a reload of the broker__c mapping:

$ heroku logs -d herokuconnect
2022-02-28T20:41:10.000000+00:00 app[herokuconnect]: event="RELOAD TABLE broker__c" addon_id=683f0847-bd8e-4011-b77a-89a83a6925cf app_name=dhdev-1101 object_type=mapping object_id=2a2a5511-e1c1-47c5-aa6c-afdc6b135743 object_name=Broker__c state=RELOAD_TABLE level=info
2022-02-28T20:41:11.000000+00:00 app[herokuconnect]: event="Cleared table broker__c for load" addon_id=683f0847-bd8e-4011-b77a-89a83a6925cf app_name=dhdev-1101 object_type=mapping object_id=2a2a5511-e1c1-47c5-aa6c-afdc6b135743 object_name=Broker__c state=RELOAD_TABLE level=info
2022-02-28T20:41:11.000000+00:00 app[herokuconnect]: event="Loading table broker__c via SOAP" addon_id=683f0847-bd8e-4011-b77a-89a83a6925cf app_name=dhdev-1101 object_type=mapping object_id=2a2a5511-e1c1-47c5-aa6c-afdc6b135743 object_name=Broker__c state=RELOAD_TABLE level=info
2022-02-28T20:41:11.000000+00:00 app[herokuconnect]: event="Broker__c QUERY ↷SALESFORCE 8 rows (0.15 secs)" addon_id=683f0847-bd8e-4011-b77a-89a83a6925cf app_name=dhdev-1101 object_type=sync object_id=683f0847-bd8e-4011-b77a-89a83a6925cf object_name=Broker__c state=IDLE level=info
2022-02-28T20:41:11.000000+00:00 app[herokuconnect]: event="Broker__c INSERT ↓DATABASE 8 rows (0.00 secs)" addon_id=683f0847-bd8e-4011-b77a-89a83a6925cf app_name=dhdev-1101 object_type=sync object_id=683f0847-bd8e-4011-b77a-89a83a6925cf object_name=Broker__c state=IDLE level=debug
2022-02-28T20:41:11.000000+00:00 app[herokuconnect]: event="RELOAD COMPLETE" addon_id=683f0847-bd8e-4011-b77a-89a83a6925cf app_name=dhdev-1101 object_type=mapping object_id=2a2a5511-e1c1-47c5-aa6c-afdc6b135743 object_name=Broker__c state=RELOAD_TABLE level=info

Log Messages

[2/34] Completed Bulk Batch … from Bulk Job …

This message is logged when synchronizing from Salesforce to PostgreSQL using the Bulk API. This comes after messages that indicate we’ve inserted or updated a number of records into your database. It’s intended to help you understand how many batches are left before normal synchronization continues.

[31/267] Completed 31 Empty Bulk Batches from Bulk Job …

This message is logged when synchronizing from Salesforce to PostgreSQL using the Bulk API. When we can, Connect uses Native PK Chunking and this can generate empty bulk batches in Salesforce. To save time when processing the bulk update to your data, we skip any batches that Salesforce tells us has no data. To provide context and a complete picture of progress, we log this message.

Common Log Messages

If you’ve enabled application logging for your connection, you see a variety of messages that match a pattern similar to:

$number $object_name $verb $arrow $salesforce_or_db $total_number ($timing secs) [$api]

The following are some examples that aren’t exhaustive along with explanations:

1 Custom_Contact__c UPDATE ↑SALESFORCE 1 rows total (0.26 secs) [SOAP]

1 Custom_Contact__c record was updated in PostgreSQL and written to Salesforce. It took Salesforce 0.26 seconds to process the write sent via SOAP API.

1 Custom_Account__c DELETE ↑SALESFORCE 1 rows total (0.44 secs) [SOAP]

1 Custom_Contact__c record was deleted in PostgreSQL and deleted from Salesforce. It took Salesforce 0.44 seconds to process the delete sent via SOAP API.

Lead QUERY ↷SALESFORCE 1 rows (0.16 secs)

A Lead record was updated via a SOAP query. Applying the update took 0.16 seconds.

Update_Table_Bulk__c UPDATE ↓DATABASE 5000 rows (4.15 secs)

A bulk number of records were changed in the Update_Table_Bulk__c object. 5000 of those changes were applied in bulk to PostgreSQL and the bulk update took 4.15 seconds.

Custom_Account__c, Streaming poll found 1 changes

The Custom_Account__c object has Accelerated Polling enabled. An accelerated poll happened as a result and updated 1 record before the next scheduled poll.

Custom_Account__c DELETE ↓DATABASE 1 rows (0.01 secs)

A record in Custom_Account__c was deleted in Salesforce. It took 0.01 seconds to delete that record from PostgreSQL.

6 Custom_Account_c, 5 Custom_Contact_c INSERT ↑SALESFORCE 11 rows total (0.64 secs) [SOAP]

6 new Custom_Account__c records and 5 new Custom_Contact__c records were created in Salesforce via SOAP API in a single request. It took Salesforce 0.64 seconds to process the newly created records.

Account QUERY ↷SALESFORCE 2300 rows (1.80 secs)

2300 records were found by a soap query from Salesforce in 1.8 seconds. This doesn’t indicate that this message from Salesforce contained 2300 records. The maximum number of records that can be in a message is 2000. This message is followed by a QUERY MORE message like: Account QUERY MORE ↷SALESFORCE 300 rows (0.70 secs), which indicates that another data page was fetched from Salesforce and a log message like Account UPDATE ↓DATABASE 2000 rows (2.15 secs), which indicates that the page of data was written to your PostgreSQL database.

Account QUERY MORE ↷SALESFORCE 300 rows (0.70 secs)

300 Account records were fetched from Salesforce and were written into your PostgreSQL database. This is a continuation of a prior SOAP query to retrieve updates. When you see QUERY MORE, this indicates that a QueryLocator cursor is in use.

This is preceded by a similar log message that looks like: Account QUERY ↷SALESFORCE 2300 rows (1.80 secs).

See also:

• Reading Data from Salesforce
• Operation delayed, no available query cursors

Log Errors

Attempted to Open More Than 10 Queries to SFDC

See Operation delayed, no available query cursors

Operation delayed, no available query cursors

Salesforce enforces a limit of 10 open queries per connected user. Heroku Connect synchronizes each mapping independently, and it often synchronizes many at the same time. This message indicates Connect throttled itself to avoid exceeding the limit and further delaying synchronization,

If you’re consistently seeing this message it indicates there’s an issue worth investigating. Some possibilities are:

1. You’re using the same user credentials for multiple Connect instances. Authenticate with Salesforce with a different user for each Connection.
2. You have a large number of active mappings. Consider reducing the number of mappings you’ve created.
3. You have suddenly started seeing this message on all your Mappings and no records are being synchronized. File a support ticket for our team to investigate.

SYSTEM_ERROR (Unexpected error)

If your connection is reporting a SYSTEM_ERROR state, this error indicates an unexpected situation occurred that interrupted synchronization. If the situation was temporary, you’re able to recover the connection using the Connect Dashboard. Click the Settings link, then click Manage Connection. From there, you can click the Recover button to clear out the error state and attempt to resume synchronization.

If the underlying problem persists, the connection ends up in the same state, and requires further investigation and manual repair. Check your connection’s logs for any further details, and if you’re unable to resolve the issue, contact Heroku Support for investigation.

(psycopg2.IntegrityError) duplicate key value violates unique constraint “lead_idx_rpm_account_id__c”

This problem can have several different causes.

Insert Round-Trip Issues

If the conflict is on an SFID field, such as "contact_idx_sfid", it can be due to a race condition related to inserting records into your database. When you insert a new record into your database and Connect pushes that record to Salesforce, Connect waits to receive the success response from Salesforce and updates the database record with the newly created row’s SFID. Connect sometimes also gets that row back as part of its regular polling while the initial insert is still in process. This results in a race condition where both processes are attempting to write a record with the same SFID. This can cause unique index conflicts, but it can also cause duplicate records in the database where one record is missing the SFID.

The workaround is to add an external ID to any object you’re creating from the database side, and make sure to populate each new record with a unique ID (such as a GUID) when you create it. Then, add a unique index on that external ID. That way, if Connect attempts to write a duplicate record, the unique index causes the write to fail. You still see index violations in your logs, but if it’s on the external ID field, which means a duplicate record was prevented.

Error during sync: (psycopg2.IntegrityError) update or delete on table “mycustomobject__c” violates foreign key constraint…

Don’t specify foreign key constraints on mapped tables or reference mapped tables. Heroku Connect sometimes needs to reload mapped tables, and foreign key constraints prevent deletion/truncation of records in the Postgres table to clean up before reloading. This failure to delete generally leads to the error:

Too many sync errors. Sync operations may be disabled for up to 15 minute(s).

With Heroku Connect, the system of record is your Salesforce org, not the Postgres database. Relationships must be enforced in your Salesforce org. Heroku Connect doesn’t support Postgres foreign key constraints.

Error during sync: This Connection is closed

If you see this error upon reloading a mapping, it’s likely that the database table is being referenced in a foreign key constraint. When Heroku Connect reloads a mapping, it attempts to truncate the database table, which isn’t possible if the table is part of a foreign key constraint. You must remove the foreign key constraint first, and then reload the mapping.

Unable to Reload Mapping with ID: abcdef12…

If you’ve recently changed a mapping or imported a configuration, you can see this error. Check the status of the mapping in your Connect Dashboard. If you see BULK_LOADING, this means that Connect is in the middle of a bulk loading operation that can’t be interrupted. You must wait until the operation is complete to reload the mapping.

QUERY_TIMEOUT: Your query request was running for too long. Trying again later.

This error usually results in multiple error messages that look something like this:

Batch 751m0000001AoAtAAK of job 750m0000001BfWOAA0 failed: InternalServerError : Retried more than 15 times; InvalidBatch : Failed to process query: QUERY_TIMEOUT: Your query request was running for too long. Trying again later.; InvalidBatch : Failed to process query: QUERY_TIMEOUT: Your query request was running for too long. Trying again later.; InvalidBatch : Failed to process query: QUERY_TIMEOUT: Your query request was running for too long. Trying again later.; (...)

This error comes directly from Salesforce. Connect uses the bulk API to load large amounts of data. Occasionally, a bulk query times out, and Connect automatically retries the query. If the query continues to time out after several retries, open a ticket with Salesforce to investigate why it’s happening.

Batch 54321000007DvJXYZ0 of job 54321000005lfIZXYZ failed: InvalidBatch : Failed to process query: INVALID_FIELD

This error typically indicates there has been some amount of schema drift between a mapping’s configuration and Salesforce. Most often a field has been removed or renamed in Salesforce, but that field remains in the mapping’s configuration. To correct the issue, update the mapping configuration removing the invalid field.

In other instances this error can indicate a permission issue, where a particular field is included in a mapping’s configuration, but is no longer visible to the Salesforce user on the connection. To resolve the issue, try reauthenticating with different Salesforce user credentials or modifying the permissions of the current user. When the permission issue is corrected, you can either update the mapping configuration by removing the field(s) and then adding them back to force a field level resync, or you can reload the entire mapping.

Salesforce Configuration Saved for This Object no Longer Maps to the Current Salesforce Data Model

If you change the mapped objects in Salesforce, such as changing the data type of a field, Heroku Connect doesn’t automatically update your database schema. You can see errors such as this one as a result. To resolve the issue, edit the mapping in your Connect Dashboard.

“Server Raised Fault: ‘REQUEST_LIMIT_EXCEEDED: ConcurrentPerOrgApex Limit exceeded” or “System.LimitException: Apex CPU time limit exceeded”

When you write records into a table configured to send updates to Salesforce, Connect periodically polls your database and group many records—up to 200—into a single request to the Salesforce API.

If your object has Apex code that gets triggered when new records are created, Salesforce imposes limits on the amount of computing resources it dedicates to those triggers. As a result, you can see one of the above errors if you insert more records than usual. This Salesforce guide describes some ways to optimize your Apex code to improve performance.

If this continues to happen, contact Heroku Support about lowering your write batch size. We can limit your insert batches to 100 or 50 records, but this can also increase the latency of inserting new records.

DB is Missing Connect Metadata [DatabaseValidationFailed]

Indicates the Database Connect is connecting to isn’t the same as the one it was initialized against. This usually occurs when the Database Upgrade procedure hasn’t been followed. If you’re unable to recover into a working state a quick solution could be to Recreate the connection.

This could also be encountered if a process has deleted the _hc_meta table that Heroku Connect uses to validate that it’s talking to the right database for the right connection. Again, recreating the connection would be the quickest solution to getting back into a working state.

“Expired Transition Aborted by Health_Check” or “Expired transition aborted by sweep”

Both of these messages appear in your logs when a sync worker dyno has hung. Every 15 minutes, we run a health check on your mappings from a different dyno, and if the health check notices a hung process, it kills it and resets the state of the mapping so it starts polling again. This can happen for a variety of reasons, such as temporary network outages or dynos restarting.

Query_Wait_Timeout: Server Closed the Connection Unexpectedly

This can happen if the database suddenly disconnects during a sync operation. Connect recovers automatically, but if it doesn’t, you can resume your connection from the Connect Dashboard.

Streaming Error: 403::Organization Total Events Daily Limit Exceeded

This error means that you’ve used up your available PushTopic events from the Salesforce Streaming API. You can avoid this error by taking advantage of free unlimited Salesforce Streaming API events.

When you get the error, sync continues, but it falls back to the polling interval you’ve configured for each mapping. Events resume automatically when the rolling 24-hour window resets.

Function get_xmlbinary() Does Not Exist

Connect creates a function called get_xmlbinary in the public schema of your database, which it uses as part of the sync process. If you remove the public schema from your search_path, Connect isn’t able to find that function, and sync fails. See How do I resolve “function get_xmlbinary() does not exist” errors in Heroku Connect? for more info.

Too Many Polling Errors. Polling will be Disabled for Up to 15 minute(s)

This error indicates that polling the database or Salesforce has resulted in consecutive and unexpected failures. As a result, Connect backs off polling for the specified duration to allow time for the underlying problem to be resolved. You can look for a notification at the top of your Connect Dashboard that can explain the problem, or go to the Logs tab and filter for Error log lines to investigate. These issues are often related to your database schema being out of date, which can happen if you change your Salesforce object fields without editing your Connect mapping to update it.

Too Many Bulk Errors, Bulk Usage will be Disabled for up to 12 hour(s) for this mapping

This error indicates that bulk polling requests to Salesforce have resulted in consecutive and unexpected failures for a specific mapping. As a result, Connect backs off bulk polling for the specified duration to allow time for the underlying problem to be resolved. You can look for a notification at the top of your Connect Dashboard that can explain the problem, or go to the Logs tab and filter for Error log lines to investigate. These issues are often related to your database schema being out of date, which can happen if you change your Salesforce object fields without editing your Connect mapping to update it.

To force Connect to poll the mapping again, edit the mapping and ensure that the database schema matches the Salesforce object. You can also get polling to resume by recovering the connection.

Too many sync errors. Sync operations may be disabled for up to 15 minute(s)

This error indicates that there were too many unexpected failures trying to synchronize data in the connection or mapping. As a result, Connect backs off in trying to execute synchronization. After the specified period of time, Connect will try again to synchronize data.

SERVER_UNAVAILABLE: Request Timed out waiting for Connection

You can see something similar to this message in your logs:

faultcode = "sf:SERVER_UNAVAILABLE" faultstring = "SERVER_UNAVAILABLE: Request timed out waiting for connection: [config 1,000ms, actual 1,001ms]" detail = (detail){ UnexpectedErrorFault = (UnexpectedErrorFault){ exceptionCode = "SERVER_UNAVAILABLE" exceptionMessage = "Request timed out waiting for connection: [config 1,000ms, actual 1,001ms]" } } }

This error happens when Salesforce has problems connecting to an internal database. It’s a transient issue that resolves itself in a few minutes. If it happens repeatedly, contact Heroku Support.

Server Raised Fault: 'OPERATION_TOO_LARGE: exceeded 100000 distinct who/what’s’

OPERATION_TOO_LARGE errors are the result of not granting the View All permission to your authenticated Salesforce user. Without this permission, Salesforce has to run per-row checks to determine if your user has permission to view that row. For larger mappings, the overhead of evaluating permissions can result in OPERATION_TOO_LARGE errors. The only way to resolve this error is to remove the need for permissions queries by granting the View All permission for this object, and if necessary, related objects.

Parsing SOAP Response Failed: SAXParseException: <unknown>:1:0: syntax error

This error indicates that Heroku Connect made a SOAP request to Salesforce but the response received wasn’t valid SOAP XML. This is a transient condition that can occur if Salesforce experiences an internal error. Heroku Connect automatically retries the SOAP request.

SOAP Transport Error: BadStatusLine

This error indicates that the response from Salesforce was malformed. This error is transient in nature. As such, subsequent attempts to sync the record(s) which caused this error succeeds. This error doesn’t represent any serious issues with your Connect mappings.

(psycopg2.OperationalError) FATAL: Too Many Connections for Role

This error occurs when Connect tries to open a connection to your database but you’ve already reached your connection limit. For very active connections, Connect can use 10 or more database connections. For this reason, we strongly advise against using Essential-tier database plans with Connect.

CREATE TEMPORARY TABLE _tmp_sf…

You can see statements related to a temporary table appear in your log stream:

CREATE TEMPORARY TABLE _tmp_sf_contact AS (...)
CREATE UNIQUE INDEX ON
         _tmp_sf_contact (sfid)
CREATE INDEX ON
         _tmp_sf_contact (__operation, sfid)

These temporary tables are created by the Connect sync process in order to more efficiently sync inserts and updates from Salesforce to your database. They don’t cause any performance impact to your database, and they’re safe to ignore.

Value does not exist or does not match filter criteria

This error can happen if you attempt to insert an invalid value into a picklist field. Connect supports the picklist data type, but doesn’t provide any data validation.

Missed Updates Due to Salesforce Transactions

It’s possible that a small set of records can be locked in a transaction at the time of a poll and therefore not available to be synced. Heroku Connect handles this by performing a poll for Salesforce IDs that have been included 2 minutes after successful poll completion. Any missed records are automatically synced when detected. This “insurance” poll reconciles records in flight at the time of the last sync operation.

urlopen error [Errno -3] Temporary failure in name resolution

This is an issue with DNS resolution within the Heroku Connect service. It generally occurs due to transient networking errors. Heroku Connect is resilient to intermittent issues like this and continues to synchronize data when the DNS resolution issue is resolved.

Connection Reset by Peer

This error is caused by intermittent network connectivity issues and can typically be ignored as Connect detects this condition and retry the operation. If you observe frequent occurrences of this error message, we recommend that you contact Heroku Support for assistance.

Read Operation Timed Out

The read operation timed out error is caused by unstable network connections. These can typically be ignored as Connect detects this condition and retry the operation. If you observe frequent occurrences of this error message, we recommend that you contact Heroku Support for assistance.

Disconnected from database due to: (psycopg2.errors.ProgramLimitExceeded) index row size <field_value_size> exceeds btree version 4 maximum 2704 for index <index_name>

The error message indicates that the column you’ve applied an index to can’t index the value for the row being inserted into a B-tree type index. If you used the mapping editor on the Heroku Connect dashboard to create this index, deselect the Indexed option for the respective column. If you created a custom index, drop it manually. If you need an index on this column, refer to the PostgreSQL documentation for guidance on indexing columns with large values.

Errors with Associated Notifications

When errors occur that require user intervention, Heroku Connect shows those notifications in the Connect Dashboard and sends email notifications to all users that have chosen to receive them. To view a list of users that receive notifications, navigate to the Settings tab, Manage Connection in the dashboard: the Access section lists users who have access to the Connect dashboard and is where the Notifications Enabled setting can be configured.

You receive notifications for the errors summarized below. The notification message describes the error that has occurred and includes helpful information so that you can understand and resolve the problem. You also receive notifications when an error that occurred previously has been resolved.

Bad Configuration

A bad configuration error can occur when changes to the Salesforce object or Heroku Postgres table used by a mapping are changed. The mapping isn’t synchronized until the error is resolved however other mappings continue to synchronize as normal.

Change User Permissions

This error occurs when the user credentials you use to authenticate to Salesforce don’t have the correct permissions to allow Heroku Connect to efficiently synchronize a large number of records. Reauthenticating with different Salesforce user credentials or modifying the permissions of the current user resolves the error.

Connection Error

A connection error can occur for a number of reasons and can prevent synchronization from occurring on one or more of your mappings. You’re asked to check the logs in the Heroku Connect dashboard in order to identify the exact nature of the error.

Database Unavailable

If the Heroku Postgres database can’t be reached then Heroku Connect is unable to synchronize any mappings. You must review your database configuration in order to resolve this problem.

Reauthorization Required

If the user credentials used to authorize Heroku Connect with Salesforce expire or change then synchronization stops for all mappings and you must reauthorize with Salesforce in order for it to resume.

Salesforce Bulk Load Error

This error occurs when there was a problem during the initial bulk load or reload of a mapping. You can investigate the cause of the error in the Heroku Connect dashboard by viewing the Logs tab and filtering for Error log lines. In most cases these errors are due to bulk batch failures. A failure during a bulk load operation can leave a mapping in an inconsistent state, and can’t be automatically recovered from. After identifying and correcting the cause of the error, the mapping is reloaded.

Delayed Synchronization

This correlates with the error message: “Too many sync errors. Sync operations may be disabled for up to 15 minute(s)”. This notification alerts you to the fact that synchronization is delayed by at least 15 minutes. Connect will automatically attempt to retry synchronization after the period of time specified in the log message. This can happen for one or more of your mappings as well as the connection itself.

Audit Messages

Heroku Connect also keeps a record of administrative changes to your connection’s configuration. Audit events are visible on the Logs tab of the Heroku Connect Dashboard. These messages indicate who made a change and when that change was made. In addition, a few operations that affect the state of the connection or its data are also included. Some examples include:

• Updating connection settings
• Adding, modifying, and deleting a mapping
• Importing a previously exported configuration file
• Reloading a mapping’s data from Salesforce
• Aborting a currently running operation
• Pausing and resuming sync operations
• Enabling and disabling External Objects support

A CSV download of your connection’s activity can be downloaded from the Manage Connection page.