Heroku Postgres Maintenance Windows
Last updated 12 May 2020
Table of Contents
From time to time, Heroku needs to take a Heroku Postgres database offline to perform maintenance tasks. Typical tasks include updating the underlying infrastructure of the database (for example, patching the operating system or required libraries) or upgrading Postgres itself. This maintenance is handled automatically by Heroku.
Maintenance windows are available for all production database plans.
Checking if maintenance is required for your database
You can check if maintenance is required on a database by using
$ heroku pg:info -a sushi === DATABASE_URL Plan: Standard 4 Status: Available Data Size: 26.1 MB ... Maintenance: required by 2018-05-04 21:00:00 +0000 Maintenance: scheduled for 2018-05-01 14:30:00 +0000, required by 2018-05-04 21:00:00 +0000, replacement currently being prepared Maintenance window: Tuesdays 14:30 to 18:30 UTC
You can also check with the
$ heroku pg:maintenance DATABASE -a sushi Maintenance scheduled for 2018-05-01 14:30:00 +0000, required by 2018-05-04 21:00:00 +0000, replacement currently being prepared, window is Tuesdays 14:30 to 18:30 UTC
Setting a maintenance window
Some Heroku Postgres plans allow you to specify a maintenance window for your database. You can specify the day-of-week and time (UTC) at which the window begins:
$ heroku pg:maintenance:window DATABASE "Sunday 14:30" -a sushi
Setting a maintenance window allows you to minimize the impact of the maintenance on your application and users. Heroku recommends selecting a time during which maintenance probably has the smallest impact on your business.
Maintenance windows are 4 hours long. Heroku attempts to begin the maintenance as close to the beginning of the specified window as possible. The duration of the maintenance can vary, but usually, your database is only offline for 10 – 60 seconds.
Note that the window you specify must end before the
required by time indicated by the
heroku pg:maintenance command.
You can change your database’s maintenance window by simply running the
pg:maintenance:window command again and providing a different start time for the window.
Running a maintenance manually
If your Heroku Postgres plan supports a maintenance window, it also supports manually running a maintenance with the Heroku CLI.
Before you run a maintenance manually, it is recommended that you first put your database’s associated app in maintenance mode.
With maintenance mode enabled
The following commands demonstrate putting an app in maintenance mode and then running a manual maintenance on its associated database:
$ heroku maintenance:on -a sushi Enabling maintenance mode for ⬢ sushi... done $ heroku pg:maintenance:run DATABASE -a sushi Starting maintenance for postgresql-clean-29349... done $ heroku maintenance:off -a sushi Disabling maintenance mode for ⬢ sushi... done
Without maintenance mode enabled
Running a manual maintenance without first enabling maintenance mode requires the
$ heroku pg:maintenance:run DATABASE --force -a sushi
This runs the maintenance in the next available window.
When performing database maintenance, Heroku uses one of several strategies, such as restart, changeover, or failover. The strategy used depends on the nature of the maintenance and the database’s associated plan.
For example, for high-availability (HA) plans (premium, private, and shield), Heroku uses a failover strategy whenever possible, unless the database is the follower of another database. For non-HA plans (standard) and all follower databases regardless of plan, Heroku usually uses a changeover strategy.
If your database has one or more followers and a failover or changeover maintenance occurs, Heroku attempts to point the followers to the replacement database. Pointing the followers to the replacement requires the follower database to be restarted.
In some cases, if we are be unable to point the follower at the replacement, we will automatically prepare and promote a replacement database.
Monitoring and initiating a changeover maintenance
When Heroku schedules a changeover maintenance for your database, it begins creating a replacement database. Creating a replacement can take time, depending on the size of your database.
While the replacement is still being created, the
pg:maintenance command displays the following:
$ heroku pg:maintenance DATABASE_NAME --app myapp Maintenance scheduled for 2018-05-01 14:30:00 +0000, required by 2018-05-04 21:00:00 +0000, replacement currently being prepared, window is Tuesdays 14:30 to 18:30 UTC
This info is not shown when a particular maintenance uses a failover or restart strategy.
When the replacement is ready and the maintenance can proceed, the
pg:maintenance command shows the following:
$ heroku pg:maintenance DATABASE -a sushi Maintenance scheduled for 2018-05-01 14:30:00 +0000, required by 2018-05-04 21:00:00 +0000, replacement ready, window is Tuesdays 14:30 to 18:30 UTC
Maintenance windows that can be set and manual maintenance runs are only available on production plans (standard, premium, private and shield).
Heroku will make the best effort to honor your maintenance window request but it is not guaranteed. If there is an emergency where the security or integrity of your data is threatened, we may perform the maintenance outside of your regular window at our discretion.
Frequently Asked Questions
Why is this maintenance occurring?
Heroku Postgres is a managed Postgres offering, and one of the largest values provided is providing patches for security and feature updates. Heroku monitors for and proactively patches security vulnerabilities as part of its Heroku Postgres offering.
When is this maintenance happening?
You should have received an email letting you know that the maintenance will occur during your upcoming maintenance window. You can verify when the maintenance will occur with the
heroku pg:maintenance command. This will indicate what time your maintenance is scheduled for as well as if it is ready for maintenance to be performed.
Will my app experience downtime?
Any maintenance requires some downtime. If you are on a high-availability (HA) plan (Premium/Private/Shield tier), you are already aware of this as you have taken steps to protect yourself in the event of an outage. If you are not on an HA plan, Heroku attempts to ensure as little downtime as possible for your application. HA plans experience some time after a maintenance where HA is unavailable while a new HA follower is created and caught up.
During maintenance downtime, you may see app errors such as
sql_error_code = 25006 ERROR: cannot execute INSERT in a read-only transaction, or more general errors that indicate your application cannot connect to the database.
How do I minimize downtime?
Make your app resilient to connection errors
Failover for maintenance often occurs quickly, but apps can have problems reestablishing connections, which leads to additional downtime. If this is an issue, you can proactively restart your app after a database maintenance to guard against connection-specific errors. You can also build that resiliency into the error handling of the app itself with an exponential back-off when reestablishing connections.
Use a read-only follower
You can provision a database follower to provide read-only access to your database. Your app will need to support a “read-only” mode and be configured so that it only requires access to the follower. Depending on the use case, this might enable the app to continue to run in a degraded, read-only mode while maintenance is being run on the main database.
Verify that your database is ready before running migrations
If your app has migrations that run on boot, first confirm that you have a read-write connection to the database, and then run migrations. If you attempt to run migrations with a read-only connection, your app might crash, resulting in more downtime as it recovers.
You can verify that the app is out of read-only mode with this SQL command:
Do I need to do anything to perform the maintenance?
There are three different ways for maintenance to occur. The most common way is to do let Heroku perform the maintenance for you. We use your databases maintenance window to determine the best time to run this maintenance automatically.
If you wish to run your maintenance manually you can do so in one of two ways. Via
heroku pg:maintenance --force. This will immediately fail over to your HA follower or the hidden follower if it’s been caught up. Your application will immediately restart and be pointed to the new follower.
The other method involves putting your application into maintenance mode before performing the PostgreSQL failover. This can have benefits such as providing a clean landing page for your users, but total downtime can be longer as it is a manual process to enable and disable maintenance mode. We have more detailed information on the maintenance mode page.
There are two ways this maintenance can happen: 1) Automatically by us, or 2) manually by you.
The most common way is to do let Heroku perform the maintenance for you. We use your databases maintenance window to determine the best time to run this maintenance automatically. If you want it to happen automatically, you don’t need to do anything. Your app will restart as part of the maintenance process, so some users may see errors or delays for a minute or three.
notice You may see these errors in your logs:
pg is read only, etc. These are artifacts of the changeover process and are safe to ignore as long as everything comes back online.
Manually by you
If you want to have better control over your users’ experience during the maintenance, you can do it yourself. To perform the maintenance, run the
heroku pg:maintenance:run DATABASE --force -a sushi command at any time before your scheduled maintenance.
You may want to use maintenance mode to turn the app off while the maintenance runs to minimize the errors your users see. That might look something like:
heroku maintenance:on heroku pg:maintenance:run --force heroku pg:wait heroku maintenance:off
If you’ve built a ‘read-only’ or 'database-independent’ mode into your app, you could use that as well:
# for example heroku config:set READ_ONLY=true heroku pg:maintenance run --force heroku pg:wait heroku config:set READ_ONLY=false
See Maintenance Types for more information on running the maintenance manually.
How often do maintenances occur?
Heroku Postgres databases are updated regularly to maintain important security patches in Postgres as well as the underlying software and hardware. Maintenances occur at least once every 30 days.
Is a maintenance history available for my database?
Heroku does not provide a history of maintenances running on your applications. Heroku does send an email before every maintenance which you can search for.
Can I change the scheduled maintenance time?
Yes. You can change the scheduled maintenance time to another time before the required by time by changing the maintenance window.
Will my DATABASE_URL or HEROKU_POSTGRESQL_<COLOR>_URL change?
In most cases, no. A new release will be created for your app, which states that certain config vars have been updated, but the values will be the same. Your app will still restart with this change.