• Databases & Data Management
›
• Heroku Postgres
›
• Postgres Availability
›
• Heroku Postgres Maintenance Windows

Heroku Postgres Maintenance Windows

Last updated 23 July 2019

From time to time, Heroku needs to take a Heroku Postgres database offline to perform maintenance tasks. Typical tasks include updating the underlying infrastructure of the database (for example, patching the operating system or required libraries) or upgrading Postgres itself. This maintenance is handled automatically by Heroku.

Checking if maintenance is required for your database

You can check if maintenance is required on a database by using pg:info.

$ heroku pg:info -a sushi
=== DATABASE_URL
Plan:           Standard 4
Status:         Available
Data Size:      26.1 MB
...
Maintenance:    required by 2018-05-04 21:00:00 +0000
Maintenance:    scheduled for 2018-05-01 14:30:00 +0000, required by 2018-05-04 21:00:00 +0000, replacement currently being prepared
Maintenance window:    Tuesdays 14:30 to 18:30 UTC

You can also check with the pg:maintenance command:

$ heroku pg:maintenance DATABASE -a sushi
Maintenance scheduled for 2018-05-01 14:30:00 +0000, required by 2018-05-04 21:00:00 +0000, replacement currently being prepared, window is Tuesdays 14:30 to 18:30 UTC

Setting a maintenance window

Some Heroku Postgres plans allow you to specify a maintenance window for your database. You can specify the day-of-week and time (UTC) at which the window begins:

$ heroku pg:maintenance:window DATABASE "Sunday 14:30" -a sushi

Setting a maintenance window allows you to minimize the impact of the maintenance on your application and users. Heroku recommends selecting a time during which maintenance probably has the smallest impact on your business.

Maintenance windows are 4 hours long. Heroku attempts to begin the maintenance as close to the beginning of the specified window as possible. The duration of the maintenance can vary, but usually, your database is only offline for 10 – 60 seconds.

Note that the window you specify must end before the required by time indicated by the heroku pg:maintenance command.

Rescheduling maintenance

You can change your database’s maintenance window by simply running the pg:maintenance:window command again and providing a different start time for the window.

Running a maintenance manually

If your Heroku Postgres plan supports a maintenance window, it also supports manually running a maintenance with the Heroku CLI.

With maintenance mode enabled

The following commands demonstrate putting an app in maintenance mode and then running a manual maintenance on its associated database:

$ heroku maintenance:on -a sushi
Enabling maintenance mode for ⬢ sushi... done

$ heroku pg:maintenance:run DATABASE -a sushi
Starting maintenance for postgresql-clean-29349... done

$ heroku maintenance:off -a sushi
Disabling maintenance mode for ⬢ sushi... done

Without maintenance mode enabled

Running a manual maintenance without first enabling maintenance mode requires the --force flag:

$ heroku pg:maintenance:run DATABASE --force -a sushi

This runs the maintenance in the next available window.

Maintenance types

When performing database maintenance, Heroku uses one of several strategies, such as restart, changeover, or failover. The strategy used depends on the nature of the maintenance and the database’s associated plan.

For example, for high-availability (HA) plans (premium, private, and shield), Heroku uses a failover strategy whenever possible, unless the database is the follower of another database. For non-HA plans (standard) and all follower databases regardless of plan, Heroku usually uses a changeover strategy.

Monitoring and initiating a changeover maintenance

When Heroku schedules a changeover maintenance for your database, it begins creating a replacement database. Creating a replacement can take time, depending on the size of your database.

While the replacement is still being created, the pg:maintenance command displays the following:

$ heroku pg:maintenance DATABASE_NAME --app myapp
Maintenance scheduled for 2018-05-01 14:30:00 +0000, required by 2018-05-04 21:00:00 +0000, replacement currently being prepared, window is Tuesdays 14:30 to 18:30 UTC

When the replacement is ready and the maintenance can proceed, the pg:maintenance command shows the following:

$ heroku pg:maintenance DATABASE -a sushi
Maintenance scheduled for 2018-05-01 14:30:00 +0000, required by 2018-05-04 21:00:00 +0000, replacement ready, window is Tuesdays 14:30 to 18:30 UTC

Limitations

Maintenance windows that can be set and manual maintenance runs are only available on production plans (standard, premium, private and shield).

Heroku will make the best effort to honor your maintenance window request but it is not guaranteed. If there is an emergency where the security or integrity of your data is threatened, we may perform the maintenance outside of your regular window at our discretion.

Frequently Asked Questions

Why is this maintenance occurring?

Heroku Postgres is a managed Postgres offering, and one of the largest values provided is providing patches for security and feature updates. Heroku monitors for and proactively patches security vulnerabilities as part of its Heroku Postgres offering.

When is this maintenance happening?

You should have received an email letting you know that the maintenance will occur during your upcoming maintenance window. You can verify when the maintenance will occur with the heroku pg:maintenance command. This will indicate what time your maintenance is scheduled for as well as if it is ready for maintenance to be performed.

Will my app experience downtime?

Any maintenance requires some downtime. If you are on a high-availability (HA) plan (Premium/Private/Shield tier), you are already aware of this as you have taken steps to protect yourself in the event of an outage. If you are not on an HA plan, Heroku attempts to ensure as little downtime as possible for your application. HA plans experience some time after a maintenance where HA is unavailable while a new HA follower is created and caught up.

During maintenance downtime, you may see app errors such as sql_error_code = 25006 ERROR: cannot execute INSERT in a read-only transaction, or more general errors that indicate your application cannot connect to the database.

How do I minimize downtime?

Make your app resilient to connection errors

Failover for maintenance often occurs quickly, but apps can have problems reestablishing connections, which leads to additional downtime. If this is an issue, you can proactively restart your app after a database maintenance to guard against connection-specific errors. You can also build that resiliency into the error handling of the app itself with an exponential back-off when reestablishing connections.

Use a read-only follower

You can provision a database follower to provide read-only access to your database. Your app will need to support a “read-only” mode and be configured so that it only requires access to the follower. Depending on the use case, this might enable the app to continue to run in a degraded, read-only mode while maintenance is being run on the main database.

Verify that your database is ready before running migrations

If your app has migrations that run on boot, first confirm that you have a read-write connection to the database, and then run migrations. If you attempt to run migrations with a read-only connection, your app might crash, resulting in more downtime as it recovers.

You can verify that the app is out of read-only mode with this SQL command: SELECT pg_is_in_recovery().

Do I need to do anything to perform the maintenance?

There are three different ways for maintenance to occur. The most common way is to do let Heroku perform the maintenance for you. We use your databases maintenance window to determine the best time to run this maintenance automatically.

If you wish to run your maintenance manually you can do so in one of two ways. Via heroku pg:maintenance --force. This will immediately fail over to your HA follower or the hidden follower if it’s been caught up. Your application will immediately restart and be pointed to the new follower.

The other method involves putting your application into maintenance mode before performing the PostgreSQL failover. This can have benefits such as providing a clean landing page for your users, but total downtime can be longer as it is a manual process to enable and disable maintenance mode. We have more detailed information on the maintenance mode page.

There are two ways this maintenance can happen: 1) Automatically by us, or 2) manually by you.

Automatically

The most common way is to do let Heroku perform the maintenance for you. We use your databases maintenance window to determine the best time to run this maintenance automatically. If you want it to happen automatically, you don’t need to do anything. Your app will restart as part of the maintenance process, so some users may see errors or delays for a minute or three.

notice You may see these errors in your logs: connection error, pg is read only, etc. These are artifacts of the changeover process and are safe to ignore as long as everything comes back online.

Manually by you

If you want to have better control over your users’ experience during the maintenance, you can do it yourself. To perform the maintenance, run the heroku pg:maintenance:run DATABASE --force -a sushi command at any time before your scheduled maintenance.

You may want to use maintenance mode to turn the app off while the maintenance runs to minimize the errors your users see. That might look something like:

heroku maintenance:on
heroku pg:maintenance:run --force
heroku pg:wait
heroku maintenance:off

If you’ve built a ‘read-only’ or 'database-independent’ mode into your app, you could use that as well:

# for example
heroku config:set READ_ONLY=true
heroku pg:maintenance run --force
heroku pg:wait
heroku config:set READ_ONLY=false

See Maintenance Types for more information on running the maintenance manually.

How often do maintenances occur?

Heroku Postgres databases are updated regularly to maintain important security patches in Postgres as well as the underlying software and hardware. Maintenances occur at least once every 30 days.

Is a maintenance history available for my database?

Heroku does not provide a history of maintenances running on your applications. Heroku does send an email before every maintenance which you can search for.

Can I change the scheduled maintenance time?

Yes. You can change the scheduled maintenance time to another time before the required by time by changing the maintenance window.

Will my DATABASE_URL or HEROKU_POSTGRESQL_<COLOR>_URL change?

In most cases, no. A new release will be created for your app, which states that certain config vars have been updated, but the values will be the same. Your app will still restart with this change.