Dyno Restarts
Last updated December 05, 2024
This article describes the restart behavior of Heroku dynos and how to restart them.
Manual Restarts
The dyno manager restarts all your app’s dynos when you run heroku restart. You can also choose to restart a specific dyno, or dynos for a specific process type.
Cedar-Generation Apps
For Cedar apps, in the Common Runtime, if you run ps:stop on dynos that are part of a scaled process, the dyno manager stops and automatically restarts them. In Private Spaces, ps:stop terminates and replaces the dedicated instance running the dyno(s).
Fir-Generation Apps
For Fir-generation apps, dynos are powered by Kubernetes pods. Whenever a call to ps:stop is triggered, the corresponding dyno(s) gets terminated and replaced.
Automatic Restarts
An app’s dynos automatically restart when you:
- create a new release by deploying new code
- change your config vars
- change your add-ons
The dyno manager for Cedar-generation apps also automatically restarts all your app’s dynos at least once per day to help maintain the health of applications running on Heroku.
We delete all changes to the local filesystem when we restart the dyno.
In addition, dynos restart as needed for the overall health of the system and your app. For example, the dyno manager occasionally detects a fault in the underlying hardware and needs to move your dyno to a new physical location. These things happen transparently and automatically on a regular basis and appear in your application logs.
Dynos also restart if the command used to start the dyno, exits. The cases when the command used to start a dyno can exit, are as follows:
- Defect in startup code - If your app is missing a critical dependency, or has any other problem during startup, it exits immediately with a stack trace.
- Transient error on a resource used during startup - If your app accesses a resource during startup, and that resource is offline, it can exit. For example, if you use Amazon RDS as your database and didn’t create a security group ingress for your Heroku app, it generates an error or times out trying to boot.
- Segfault in a binary library - If your app uses a binary library, for example, an XML parser, and it crashes, it can take your entire application with it. Exception handling can’t trap it, so your process dies.
- Interpreter or compiler bug - The rare case of a bug in an interpreter (Ruby, Python) or in the results of compilation (Java, Scala) can take down your process.
Daily Restarts for Cedar-Generation Apps
The dyno manager for Cedar-generation apps automatically restarts all your app’s dynos at least once per day to help maintain the health of applications running on Heroku. The cycling happens once every 24 hours (plus up to 216 random minutes, to prevent every dyno for an application from restarting at the same time). Manual restarts and releases, whether from deploys or changing config vars, reset this 24-hour period. Cycling happens for all dynos, including one-off dynos, so dynos run for a maximum of 24 hours + 216 minutes. If you have multiple dynos, they cycle at different times based on the random 0 to 216 minutes difference. If you continually make changes to your application without a 24-hour gap, there’s no cycling at all. When a dyno cycles, you see a log entry similar to this:
2015-08-18T06:20:13+00:00 heroku[web.1]: Cycling
No Daily Restarts for Fir-Generation Apps
In the Fir generation, dynos don’t automatically restart every 24 hours. You must manually restart dynos.
Dyno Crash Restart Policy
A dyno crash represents any event originating with the process running in the dyno that causes the dyno to stop. That includes the process exiting with an exit code of 0 (or any other exit code).
Common Runtime Restart Policy
The Common Runtime implements an incremental backoff policy for crashing dynos:
- The first time a dyno crashes, we restart it immediately.
- If the dyno crashes again, it’s subject to a cool-off period before another restart attempt.
- The first cool-off period is up to 20 minutes, the next one is up to 40 minutes, then up to 60 minutes, up to 180 minutes, and finally, up to 320 minutes.
- After the 320 minute cool-off period, restart attempts happen every 320 minutes.
The cool-off period resets when:
- The dyno starts correctly.
- You push a new release to your app.
- You restart your app.
- You scale your dynos to 0 and then scale them back up again.
Cedar Private Spaces Restart Policy
There’s no incremental backoff policy for Cedar-generation Private Spaces. When a dyno crashes it continuously restarts with no cool-off period.
Fir Private Spaces Restart Policy
When a dyno is crashing, Fir uses the built-in incremental backoff policy powered by Kubernetes. Read the official Kubernetes docs to learn how pods handle problems.