Optimizing Node.js Application Concurrency
Last updated May 15, 2024
Table of Contents
Node has a limited ability to scale to different container sizes. It’s single-threaded, so it can’t automatically take advantage of additional CPU cores.
Node.js apps must fork multiple processes to maximize their available resources. This “clustering” is supported by the Node.js Cluster API. You can invoke the Cluster API directly in your app, or you can use one of many abstractions over the API. Here, we use Throng.
With Cluster API, you can optimize your app’s performance across various dyno types. The Heroku Node.js buildpack provides environment variables to help.
Heroku Enterprise customers with Premier or Signature Success Plans can request in-depth guidance on this topic from the Customer Solutions Architecture (CSA) team. Learn more about Expert Coaching Sessions here or contact your Salesforce account executive.
Enabling Concurrency in Your App
We recommend that all applications support clustering. Even if you don’t anticipate running more than a single process, clustering offers greater control and flexibility over your app’s performance. Let’s take a look at an example.
First, we determine how many processes to cluster.
var WORKERS = process.env.WEB_CONCURRENCY || 1;
Second, we define a start
function as the entry point for each newly clustered process.
function start() {
// ...
}
Third, we use throng
to cluster the app into multiple processes. We specify a lifetime of Infinity
to tell throng to respawn if a worker dies, so we always have WORKERS
processes running.
throng({
workers: WORKERS,
lifetime: Infinity
}, start);
Testing Locally
With concurrency implemented, we can observe the cluster.
$ npm start
> example-concurrency@1.0.0 start example-concurrency
> node server.js
Listening on 3000
$ WEB_CONCURRENCY=4
$ npm start
> example-concurrency@1.0.0 start example-concurrency
> node server.js
Listening on 3000
Listening on 3000
Listening on 3000
Listening on 3000
Tuning the Concurrency Level
Each app has unique memory, CPU, and I/O requirements, so there’s no such thing as a one-size-fits-all scaling solution. The Heroku buildpack provides reasonable defaults through two environment variables: WEB_MEMORY
and WEB_CONCURRENCY
. You can override both to fit your specific application.
WEB_MEMORY
specifies, in MB, the expected memory requirements of your application’s processes. It defaults to 512 MB.WEB_CONCURRENCY
specifies the recommended number of processes to cluster for your application. It’s essentiallyMEMORY_AVAILABLE / WEB_MEMORY
.
Read more about configuring your application’s memory use when clustering.
Defaults
Common Runtime
Dyno Type | Number of Cluster workers |
---|---|
Eco, Basic, Standard-1X | 1 |
Standard-2X | 2 |
Performance-M | 5 |
Performance-L | 28 |
Performance-L-RAM | 15 |
Performance-XL | 31 |
Performance-2XL | 63 |
Private Spaces and Shield Private Spaces
Dyno Type | Number of Cluster workers |
---|---|
Private-S / Shield-S | 2 |
Private-M / Shield-M | 5 |
Private-L / Shield-L | 28 |
Private-L-RAM / Shield-L-RAM | 15 |
Private-XL / Shield-XL | 31 |
Private-2XL / Shield-2XL | 63 |
For Performance-L dynos, applications work well with the 28 workers suggested for its 14 GB of memory. Always test an application to see whether it can support that many concurrent processes.
These defaults are reasonable for most apps. In most cases, clustering more than one worker on a Standard-1x dyno hurts rather than helps performance. However, try any combination of WEB_CONCURRENCY
with any dyno size to see what works best for your workload.
Decreasing the WEB_MEMORY
increases WEB_CONCURRENCY
. Similarly, increasing WEB_MEMORY
reduces concurrency. When the size of your dyno changes, WEB_CONCURRENCY
is recalculated automatically to fill available memory.
You can also set WEB_CONCURRENCY
directly, but it prevents your app from automatically reclustering when you change dyno sizes.
See It in Action
To log Node concurrency settings on startup, set the LOG_CONCURRENCY
config var.
$ heroku config:set LOG_CONCURRENCY=true
After you deploy an app with clustering to Heroku, you can tail its log to observe it scaling into different container sizes.
$ heroku logs --tail
$ heroku scale web=1:standard-1x
heroku[api]: Resize web to standard-1x
heroku[api]: Scale to web=1
heroku[web.1]: State changed from up to starting
heroku[web.1]: State changed from up to starting
heroku[web.1]: Starting process with command `npm start`
app[web.1]: Detected 512 MB available memory, 512 MB limit per process (WEB_MEMORY)
app[web.1]: Recommending WEB_CONCURRENCY=1
heroku[web.1]: Stopping all processes with SIGTERM
app[web.1]:
app[web.1]: > example-concurrency@1.0.0 start /app
app[web.1]: > node server.js
app[web.1]:
app[web.1]: Listening on 51077
heroku[web.1]: State changed from starting to up
$ heroku scale web=1:performance-l
heroku[api]: Resize web to performance-l
heroku[api]: Scale to web=1
heroku[web.1]: State changed from up to starting
heroku[web.1]: Starting process with command `npm start`
app[web.1]: Recommending WEB_CONCURRENCY=12
app[web.1]: Detected 6144 MB available memory, 512 MB limit per process (WEB_MEMORY)
heroku[web.1]: Stopping all processes with SIGTERM
app[web.1]:
app[web.1]: > example-concurrency@1.0.0 start /app
app[web.1]: > node server.js
app[web.1]:
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
heroku[web.1]: Process exited with status 143
heroku[web.1]: State changed from starting to up