Tuning glibc Memory Behavior
Last updated 08 June 2016
Table of Contents
Application maintainers may tune application performance and memory use by adjusting the MALLOC_ARENA_MAX environment variable. This may be particularly useful if you’re seeing adverse memory-use patterns after migrating apps from the Cedar-10 stack to the Cedar-14 stack.
The recommendations in this article are based on tests performed by Heroku before launching the Cedar-14 stack. See the Testing Cedar-14 memory usage article for details on these tests.
Recent versions of glibc use multiple memory pools that malloc can allocate memory from. Especially for threaded programs, using multiple memory pools can speed up memory allocation (be reducing locking) and improve overall performance. See this article for details on the glibc malloc arena implementation.
With some memory allocation patterns, this performance improvement can come at the cost of somewhat greater app memory consumption. This may especially be the case for programs that create and destroy many threads and allocate lots of memory in those threads.
On Heroku, you’ll get a newer version of glibc when you upgrade from Cedar-10 to Cedar-14. Both versions have support for multiple memory pools, but a difference in the default behavior in the version on Cedar-14 means that apps may be especially susceptible to increased memory use.
We have chosen to not override default number of memory pools for either stack. As a result, many apps will get better performance on Cedar-14 than on Cedar-10, but some apps may also consume more memory and may require additional tuning. Apps that allocate memory from lots of threads and utilize nearly all available memory are the most likely to require a second look.
Heroku users can change the number of memory pools by changing the MALLOC_ARENA_MAX setting for apps. Read on for details on when and how to do this.
When to tune MALLOC_ARENA_MAX
In testing, we’ve so far only seen adverse changes in memory use on Cedar-14, specifically for apps that were migrated from Cedar to Cedar-14. For that reason, we recommend testing code in a non-production Cedar-14 staging app before moving an app from Cedar-10 to Cedar-14. If the Cedar-14 staging app uses more memory that the Cedar production app, consider tuning MALLOC_ARENA_MAX before moving the production app to Cedar-14.
Other things to look out for are R14 errors or increased memory consumption reported in application metrics.
What value to choose for MALLOC_ARENA_MAX
The glibc default number of memory pools on 64bit systems is 8 times the number of CPU cores (the number of CPU cores seen by dynos on Heroku varies with dyno type).
If your app doesn’t perform well with the default glibc setting, you can adjust the behavior by changing the MALLOC_ARENA_MAX configuration variable for the app.
$ heroku config:set MALLOC_ARENA_MAX=2
If you don’t want to apply the setting to the entire app, but only to a particular process type showing adverse memory behavior, you can prepend your Procfile command with the environment setting: sidekiq: env MALLOC_ARENA_MAX=1 bundle exec sidekiq
Choosing a value MALLOC_ARENA_MAX is generally a tradeoff between performance and memory consumption. Not setting MALLOC_ARENA_MAX gives the best performance, but may mean higher memory use. Setting MALLOC_ARENA_MAX to “2” or “1” makes glibc use fewer memory pools and potentially less memory, but this may reduce performance. Based on the testing we’ve done, we recommend a value of “2” if you want to try to reduce app memory use.
In general, if your app appears to be memory-constrained, we recommend either adding more dynos and potentially reducing WEB_CONCURRENCY or opting for a dyno type with more memory.