Skip Navigation
Show nav
Heroku Dev Center
  • Get Started
  • Documentation
  • Changelog
  • Search
  • Get Started
    • Node.js
    • Ruby on Rails
    • Ruby
    • Python
    • Java
    • PHP
    • Go
    • Scala
    • Clojure
  • Documentation
  • Changelog
  • More
    Additional Resources
    • Home
    • Elements
    • Products
    • Pricing
    • Careers
    • Help
    • Status
    • Events
    • Podcasts
    • Compliance Center
    Heroku Blog

    Heroku Blog

    Find out what's new with Heroku on our blog.

    Visit Blog
  • Log inorSign up
View categories

Categories

  • Heroku Architecture
    • Dynos (app containers)
    • Stacks (operating system images)
    • Networking & DNS
    • Platform Policies
    • Platform Principles
  • Command Line
  • Deployment
    • Deploying with Git
    • Deploying with Docker
    • Deployment Integrations
  • Continuous Delivery
    • Continuous Integration
  • Language Support
    • Node.js
    • Ruby
      • Working with Bundler
      • Rails Support
    • Python
      • Background Jobs in Python
      • Working with Django
    • Java
      • Working with Maven
      • Java Database Operations
      • Working with the Play Framework
      • Working with Spring Boot
      • Java Advanced Topics
    • PHP
    • Go
      • Go Dependency Management
    • Scala
    • Clojure
  • Databases & Data Management
    • Heroku Postgres
      • Postgres Basics
      • Postgres Getting Started
      • Postgres Performance
      • Postgres Data Transfer & Preservation
      • Postgres Availability
      • Postgres Special Topics
    • Heroku Data For Redis
    • Apache Kafka on Heroku
    • Other Data Stores
  • Monitoring & Metrics
    • Logging
  • App Performance
  • Add-ons
    • All Add-ons
  • Collaboration
  • Security
    • App Security
    • Identities & Authentication
    • Compliance
  • Heroku Enterprise
    • Private Spaces
      • Infrastructure Networking
    • Enterprise Accounts
    • Enterprise Teams
    • Heroku Connect (Salesforce sync)
      • Heroku Connect Administration
      • Heroku Connect Reference
      • Heroku Connect Troubleshooting
    • Single Sign-on (SSO)
  • Patterns & Best Practices
  • Extending Heroku
    • Platform API
    • App Webhooks
    • Heroku Labs
    • Building Add-ons
      • Add-on Development Tasks
      • Add-on APIs
      • Add-on Guidelines & Requirements
    • Building CLI Plugins
    • Developing Buildpacks
    • Dev Center
  • Accounts & Billing
  • Troubleshooting & Support
  • Integrating with Salesforce
  • Databases & Data Management
  • PostgreSQL, libpq5.12.1, and Breaking Changes Impacting Connection Behavior

PostgreSQL, libpq5.12.1, and Breaking Changes Impacting Connection Behavior

English — 日本語に切り替える

Last updated December 12, 2019

Table of Contents

  • Fixing Ruby on Rails connection values
  • What is libpq, and why is 5.12.1 needed?
  • What behavior changed between 5.11 and 5.12 of libpq?
  • Detection and mitigation by Heroku
  • When will libpq5.12.1 be the default libpq client version?

The client library that is commonly used to communicate with PostgreSQL databases, has a breaking change that may impact your application. This article explains the change, what to do if your application is impacted, and what we’ve done to mitigate this change.

If you are experiencing a database connection error due to libpq5.12.1+, it means that you are trying to set a PostgreSQL setting to an incorrect value. For example, the connect_timeout value is expected to be an integer, but if you pass it a string such as "5 seconds" then the value will trigger an error. To fix your application, you’ll need to determine what value is incorrect and fix it.

Fixing Ruby on Rails connection values

This error is more common on Ruby on Rails applications because the database configuration can be done using YAML. If your application cannot connect using libpq5.12.1 it may see an error that looks like:

PG::ConnectionBad (invalid integer value "15# seconds" for connection option "connect_timeout")

The most common reason for this error is not putting a space between an ERB block and a comment in a config/database.yml file.

For example this YAML:

connect_timeout: <%= ENV['DB_CONNECT_TIMEOUT'] || 5 %># seconds

Will produce a value of connect_timeout: "5 # seconds" and will raise an error. If you put a space after the ERB block, it will interpret the value as a comment. For example:

connect_timeout: <%= ENV['DB_CONNECT_TIMEOUT'] || 5 %> # seconds, with a space after the ERB block

Will produce a value of connect_timeout: 5.

You can view the connection values of your application in a heroku run bash session:

$ rails c
> pp ActiveRecord::Base.connection_config
{:adapter=>"postgresql",
  :username=>"<username>",
  :password=>"<password>",
  :port=>5432,
  :database=>"<database>",
  :host=>"<host>",
  :connect_timeout=>"5# seconds, this value will raise an error"}

If you see a value that is the wrong type, inspect your config/database.yml and correct the value. If your YAML looks correct, it might also be set via an initializer or somewhere else, such as in a Puma after_fork block.

What is libpq, and why is 5.12.1 needed?

Libpq is the client library that understands the PostgreSQL wire protocol. It is used by other languages; for example, in Ruby, ActiveRecord uses the pg gem to power its PostgreSQL support. The pg gem links to the libpq client library (via a shared object). When Active Record tries to connect to Postgres, it does so using the pg gem, which then uses the libpq bindings.

The version of libpq correlates to the version of PostgreSQL. Recently Postgresql version 12 was released, which brought libpq version 5.12. To fully support PostgreSQL 12 and to receive all future security patches, all operating systems on Heroku must be upgraded to libpq5.12.1+ eventually.

What behavior changed between 5.11 and 5.12 of libpq?

In libpq5.11, you could pass invalid values to the database connection configuration, and the library would ignore those values. For example the connect_timeout value is expected to be an integer, but if you pass it a string such as "5 seconds" then the value would be ignored.

This behavior changed in libpq5.12 so that invalid values passed to a database connection will result in an error.

This means that applications that were previously running with no errors now might see an exception as the application tries to connect to the database.

Detection and mitigation by Heroku

This breaking change in libpq5.12 was not originally documented in the libpq changelog. As a result, the library was rolled out as part of a stack image upgrade. Several customers reported problems, and the change was reverted until we could investigate.

As part of the investigation an improvement was suggested and 5.12.1 was released, now with a changelog entry documenting the breaking change.

Rather than attempt another roll out to the stack image, it was decided that a slower roll-out using Heroku’s buildpack system could be used instead. By utilizing a buildpack, the connection error behavior will not trigger until a new version of your application effect. This gives a chance to catch the error in the release phase as well as guarantees that if production does break, that it will happen while a developer is present and active.

The rollout of libpq5.12.1 via buildpack also means that customers who are impacted can rollback to an older working version of their application to avoid production downtime. It also allows developers to “opt-out” by setting the environment variable HEROKU_SKIP_LIBPQ12=1.

In addition to this slow-rollout, we’ve been able to proactively detect and warn a small subset of users who we know are affected.

We believe this libpq, change does not affect a large number of applications, but we value stability and want to proceed with caution.

When will libpq5.12.1 be the default libpq client version?

The rollout of this client via the buildpack is only temporary. The version of libpq will be upgraded to 5.12.1 on or after January 2nd, 2020. After that date, the HEROKU_SKIP_LIBPQ12 will no longer have any effect and rolling back to an older release version of your application will not change the libpq version that your application uses.

Keep reading

  • Databases & Data Management

Feedback

Log in to submit feedback.

Information & Support

  • Getting Started
  • Documentation
  • Changelog
  • Compliance Center
  • Training & Education
  • Blog
  • Podcasts
  • Support Channels
  • Status

Language Reference

  • Node.js
  • Ruby
  • Java
  • PHP
  • Python
  • Go
  • Scala
  • Clojure

Other Resources

  • Careers
  • Elements
  • Products
  • Pricing

Subscribe to our monthly newsletter

Your email address:

  • RSS
    • Dev Center Articles
    • Dev Center Changelog
    • Heroku Blog
    • Heroku News Blog
    • Heroku Engineering Blog
  • Heroku Podcasts
  • Twitter
    • Dev Center Articles
    • Dev Center Changelog
    • Heroku
    • Heroku Status
  • Facebook
  • Instagram
  • Github
  • LinkedIn
  • YouTube
Heroku is acompany

 © Salesforce.com

  • heroku.com
  • Terms of Service
  • Privacy
  • Cookies
  • Cookie Preferences