Optimizing Node.js Application Concurrency

Last updated 21 October 2019

Node has a limited ability to scale to different container sizes. It’s single-threaded, so it can’t automatically take advantage of additional CPU cores. Further, it’s based on V8 - which has a hard memory limit of about 1.5 GB - so it also cannot automatically take advantage of additional memory.

Instead, Node.js apps must fork multiple processes to maximize their available resources. This is called “clustering,” and is supported by the Node.js Cluster API. You can invoke the Cluster API directly in your app, or you can use one of many abstractions over the API. Here, we’ll use throng.

With Cluster, you can optimize your app’s performance across various dyno types. The Heroku Node.js buildpack provides environment variables to help.

Enabling concurrency in your app

We recommend that all applications support clustering. Concurrency, even if you don’t anticipate running more than a single process, grants you greater control and flexibility over your app’s performance in the future. Let’s take a look at an example.

First, we determine how many processes we should cluster:

var WORKERS = process.env.WEB_CONCURRENCY || 1;

Second, we define a start function that will be the entry point for each newly clustered process:

function start() {
  // ...
}

Finally, we use throng to cluster the app into multiple processes. We specify a lifetime of Infinity to tell throng that, if a worker dies, it should be respawned - so we will always have WORKERS processes running:

throng({
  workers: WORKERS,
  lifetime: Infinity
}, start);

Testing locally

With that implemented, we can observe the cluster:

$ npm start

> example-concurrency@1.0.0 start example-concurrency
> node server.js

Listening on 3000

$ WEB_CONCURRENCY=4

$ npm start

> example-concurrency@1.0.0 start example-concurrency
> node server.js

Listening on 3000
Listening on 3000
Listening on 3000
Listening on 3000

Tuning the concurrency level

Each app has unique memory, CPU, and I/O requirements, so there’s no such thing as a one-size-fits-all scaling solution. The Heroku buildpack provides reasonable defaults through two environment variables: WEB_MEMORY and WEB_CONCURRENCY. Both of these can be overridden to fit your specific application.

• WEB_MEMORY specifies, in MB, the expected memory requirements of your application’s processes. It defaults to 512.

• WEB_CONCURRENCY specifies the recommended number of processes to cluster for your application. It’s essentially MEMORY_AVAILABLE / WEB_MEMORY.

Read more about configuring your application’s memory use when clustering

Defaults:

These are reasonable defaults for most apps. In most cases, clustering more than one worker on a standard-1x dyno will hurt, rather than help performance. However, feel free to try any combination of WEB_CONCURRENCY with any dyno size to see what works best for your workload.

Decreasing the WEB_MEMORY will increase WEB_CONCURRENCY. Increasing WEB_MEMORY will, similarly, reduce concurrency. When the size of your dyno changes, WEB_CONCURRENCY will be recalculated automatically to fill available memory.

See it in action

To log Node concurrency settings on startup, set the LOG_CONCURRENCY config var:

$ heroku config:set LOG_CONCURRENCY=true

Once you’ve deployed a cluster-able app to Heroku, you can tail its log to observe it scaling into different container sizes:

$ heroku logs --tail

$ heroku scale web=1:standard-1x

heroku[api]: Resize web to standard-1x by hunter@heroku.com
heroku[api]: Scale to web=1 by hunter@heroku.com
heroku[web.1]: State changed from up to starting
heroku[web.1]: State changed from up to starting
heroku[web.1]: Starting process with command `npm start`
app[web.1]: Detected 512 MB available memory, 512 MB limit per process (WEB_MEMORY)
app[web.1]: Recommending WEB_CONCURRENCY=1
heroku[web.1]: Stopping all processes with SIGTERM
app[web.1]:
app[web.1]: > example-concurrency@1.0.0 start /app
app[web.1]: > node server.js
app[web.1]:
app[web.1]: Listening on 51077
heroku[web.1]: State changed from starting to up

$ heroku scale web=1:performance-l

heroku[api]: Resize web to performance-l by hunter@heroku.com
heroku[api]: Scale to web=1 by hunter@heroku.com
heroku[web.1]: State changed from up to starting
heroku[web.1]: Starting process with command `npm start`
app[web.1]: Recommending WEB_CONCURRENCY=12
app[web.1]: Detected 6144 MB available memory, 512 MB limit per process (WEB_MEMORY)
heroku[web.1]: Stopping all processes with SIGTERM
app[web.1]:
app[web.1]: > example-concurrency@1.0.0 start /app
app[web.1]: > node server.js
app[web.1]:
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
heroku[web.1]: Process exited with status 143
heroku[web.1]: State changed from starting to up