Dyno Restarts
Last updated October 16, 2024
This article describes the restart behavior of Heroku dynos and how to restart them.
Manual Restarts
The dyno manager restarts all your app’s dynos when you run heroku restart. You can also choose to restart a specific dyno, or dynos for a specific process type.
In the Common Runtime, if you run ps:stop on dynos that are part of a scaled process, the dyno manager stops and automatically restarts them. In Private Spaces, ps:stop terminates and replaces the dedicated instance running the dyno(s).
Automatic Restarts
The dyno manager automatically restarts all your app’s dynos at least once per day to help maintain the health of applications running on Heroku.
In addition, an app’s dynos automatically restart when you:
- create a new release by deploying new code
- change your config vars
- change your add-ons
We delete all changes to the local filesystem when we restart the dyno. The cycling happens once every 24 hours (plus up to 216 random minutes, to prevent every dyno for an application from restarting at the same time). Manual restarts and releases, whether from deploys or changing config vars, reset this 24-hour period. Cycling happens for all dynos, including one-off dynos, so dynos run for a maximum of 24 hours + 216 minutes. If you have multiple dynos, they cycle at different times based on the random 0 to 216 minutes difference. If you continually make changes to your application without a 24-hour gap, there’s no cycling at all. When a dyno cycles, you see a log entry similar to this:
2015-08-18T06:20:13+00:00 heroku[web.1]: Cycling
In addition, dynos restart as needed for the overall health of the system and your app. For example, the dyno manager occasionally detects a fault in the underlying hardware and needs to move your dyno to a new physical location. These things happen transparently and automatically on a regular basis and appear in your application logs.
Dynos also restart if the command used to start the dyno, exits. The cases when the command used to start a dyno can exit, are as follows:
- Defect in startup code - If your app is missing a critical dependency, or has any other problem during startup, it exits immediately with a stack trace.
- Transient error on a resource used during startup - If your app accesses a resource during startup, and that resource is offline, it can exit. For example, if you use Amazon RDS as your database and didn’t create a security group ingress for your Heroku app, it generates an error or times out trying to boot.
- Segfault in a binary library - If your app uses a binary library, for example, an XML parser, and it crashes, it can take your entire application with it. Exception handling can’t trap it, so your process dies.
- Interpreter or compiler bug - The rare case of a bug in an interpreter (Ruby, Python) or in the results of compilation (Java, Scala) can take down your process.
Dyno Crash Restart Policy
A dyno crash represents any event originating with the process running in the dyno that causes the dyno to stop. That includes the process exiting with an exit code of 0 (or any other exit code).
The Common Runtime implements an incremental backoff policy for crashing dynos:
- The first time a dyno crashes, we restart it immediately.
- If the dyno crashes again, it’s subject to a cool-off period before another restart attempt.
- The first cool-off period is up to 20 minutes, the next one is up to 40 minutes, then up to 60 minutes, up to 180 minutes, and finally, up to 320 minutes.
- After the 320 minute cool-off period, restart attempts happen every 320 minutes.
The cool-off period resets when:
- The dyno starts correctly.
- You push a new release to your app.
- You restart your app.
- You scale your dynos to 0 and then scale them back up again.
There’s no incremental backoff policy for Private Spaces. When a dyno crashes it continuously restarts with no cool-off period.