Tuning glibc Memory Behavior
Last updated 24 September 2019
Table of Contents
Application maintainers may tune application performance and memory use by adjusting the MALLOC_ARENA_MAX environment variable. This may be particularly useful if you’re seeing adverse memory-use patterns after migrating apps from the Cedar-10 stack to the Cedar-14+ stack.
Recent versions of glibc use multiple memory pools that malloc can allocate memory from. Especially for threaded programs, using multiple memory pools can speed up memory allocation (be reducing locking) and improve overall performance. See this article for details on the glibc malloc arena implementation.
With some memory allocation patterns, this performance improvement can come at the cost of somewhat greater app memory consumption. This may especially be the case for programs that create and destroy many threads and allocate lots of memory in those threads.
On Heroku, you’ll get a newer version of glibc when you upgrade from Cedar-10 to Cedar-14+. Both versions have support for multiple memory pools, but a difference in the default behavior in the version on Cedar-14+ means that apps may be especially susceptible to increased memory use.
As of September 24th, 2019 - when you deploy a new application to Heroku we will default your MALLOC_ARENA_MAX=2. You can check if this value is set for your app by running
$ heroku run bash
~$ env | grep MALLOC_ARENA_MAX
MALLOC_ARENA_MAX=2
If this is not set on your app by default, you can manually set it:
$ heroku config:set MALLOC_ARENA_MAX=2
When to tune MALLOC_ARENA_MAX
If your application has MALLOC_ARENA_MAX=2 set but has plenty of memory to spare, you may be able to see small gains in response time by increasing this value. For more information see how we originally tested this setting and what the impacts were. While testing was performed a long time ago, since then there have been numerous reports of the positive effects of this setting compared to it’s a relatively small downside.
If your application does not use MALLOC_ARENA_MAX=2 look out for R14 errors or increased memory consumption reported in application metrics. Setting this value to 2 may help reduce memory consumption for your application.
What value to choose for MALLOC_ARENA_MAX
The glibc default number of memory pools on 64bit systems is 8 times the number of CPU cores (the number of CPU cores seen by dynos on Heroku varies with dyno type).
If your app doesn’t perform well with the default glibc setting, you can adjust the behavior by changing the MALLOC_ARENA_MAX configuration variable for the app.
$ heroku config:set MALLOC_ARENA_MAX=2
If you don’t want to apply the setting to the entire app, but only to a particular process type showing adverse memory behavior, you can prepend your Procfile command with the environment setting: sidekiq: env MALLOC_ARENA_MAX=2 bundle exec sidekiq
Choosing a value MALLOC_ARENA_MAX is generally a tradeoff between performance and memory consumption. Not setting MALLOC_ARENA_MAX gives the best performance, but may mean higher memory use. Setting MALLOC_ARENA_MAX to “2” or “1” makes glibc use fewer memory pools and potentially less memory, but this may reduce performance. Based on the testing we’ve done, we recommend a value of “2” if you want to try to reduce app memory use.
If your application is still memory-constrained please see the article on R14 - Memory Quota Exceeded in Ruby (MRI) which contains ideas for how to reduce your memory usage.