PostgreSQL, libpq5.12.1, and Breaking Changes Impacting Connection Behavior
Last updated 12 December 2019
Table of Contents
The client library that is commonly used to communicate with PostgreSQL databases, has a breaking change that may impact your application. This article explains the change, what to do if your application is impacted, and what we’ve done to mitigate this change.
If you are experiencing a database connection error due to libpq5.12.1+, it means that you are trying to set a PostgreSQL setting to an incorrect value. For example, the connect_timeout value is expected to be an integer, but if you pass it a string such as "5 seconds" then the value will trigger an error. To fix your application, you’ll need to determine what value is incorrect and fix it.
Fixing Ruby on Rails connection values
This error is more common on Ruby on Rails applications because the database configuration can be done using YAML. If your application cannot connect using libpq5.12.1 it may see an error that looks like:
PG::ConnectionBad (invalid integer value "15# seconds" for connection option "connect_timeout")
The most common reason for this error is not putting a space between an ERB block and a comment in a config/database.yml file.
For example this YAML:
connect_timeout: <%= ENV['DB_CONNECT_TIMEOUT'] || 5 %># seconds
Will produce a value of connect_timeout: "5 # seconds" and will raise an error. If you put a space after the ERB block, it will interpret the value as a comment. For example:
connect_timeout: <%= ENV['DB_CONNECT_TIMEOUT'] || 5 %> # seconds, with a space after the ERB block
Will produce a value of connect_timeout: 5.
You can view the connection values of your application in a heroku run bash session:
$ rails c
> pp ActiveRecord::Base.connection_config
{:adapter=>"postgresql",
:username=>"<username>",
:password=>"<password>",
:port=>5432,
:database=>"<database>",
:host=>"<host>",
:connect_timeout=>"5# seconds, this value will raise an error"}
If you see a value that is the wrong type, inspect your config/database.yml and correct the value. If your YAML looks correct, it might also be set via an initializer or somewhere else, such as in a Puma after_fork block.
What is libpq, and why is 5.12.1 needed?
Libpq is the client library that understands the PostgreSQL wire protocol. It is used by other languages; for example, in Ruby, ActiveRecord uses the pg gem to power its PostgreSQL support. The pg gem links to the libpq client library (via a shared object). When Active Record tries to connect to Postgres, it does so using the pg gem, which then uses the libpq bindings.
The version of libpq correlates to the version of PostgreSQL. Recently Postgresql version 12 was released, which brought libpq version 5.12. To fully support PostgreSQL 12 and to receive all future security patches, all operating systems on Heroku must be upgraded to libpq5.12.1+ eventually.
What behavior changed between 5.11 and 5.12 of libpq?
In libpq5.11, you could pass invalid values to the database connection configuration, and the library would ignore those values. For example the connect_timeout value is expected to be an integer, but if you pass it a string such as "5 seconds" then the value would be ignored.
This behavior changed in libpq5.12 so that invalid values passed to a database connection will result in an error.
This means that applications that were previously running with no errors now might see an exception as the application tries to connect to the database.
Detection and mitigation by Heroku
This breaking change in libpq5.12 was not originally documented in the libpq changelog. As a result, the library was rolled out as part of a stack image upgrade. Several customers reported problems, and the change was reverted until we could investigate.
As part of the investigation an improvement was suggested and 5.12.1 was released, now with a changelog entry documenting the breaking change.
Rather than attempt another roll out to the stack image, it was decided that a slower roll-out using Heroku’s buildpack system could be used instead. By utilizing a buildpack, the connection error behavior will not trigger until a new version of your application effect. This gives a chance to catch the error in the release phase as well as guarantees that if production does break, that it will happen while a developer is present and active.
The rollout of libpq5.12.1 via buildpack also means that customers who are impacted can rollback to an older working version of their application to avoid production downtime. It also allows developers to “opt-out” by setting the environment variable HEROKU_SKIP_LIBPQ12=1.
In addition to this slow-rollout, we’ve been able to proactively detect and warn a small subset of users who we know are affected.
We believe this libpq, change does not affect a large number of applications, but we value stability and want to proceed with caution.
When will libpq5.12.1 be the default libpq client version?
The rollout of this client via the buildpack is only temporary. The version of libpq will be upgraded to 5.12.1 on or after January 2nd, 2020. After that date, the HEROKU_SKIP_LIBPQ12 will no longer have any effect and rolling back to an older release version of your application will not change the libpq version that your application uses.