Getting ideal performance out of any application is a challenging and subtle area of study which is not well-understood by many developers. The biggest challenge is in dividing the problem space and understanding what actually represents ideal. When we say performance do we mean raw speed? Latency? Concurrency? Throughput? But although performance issues can come in many shapes and sizes, almost all of them have the same symptom: your users complain that your site is slow.

Developers building apps on Heroku have the same performance challenges that you would anywhere else. The good news is that we’ve created some tools and some safeguards that help identify performance bottlenecks, and then provide clear paths for ways that you can improve the situation if you so choose.

Definitions

Dyno - A dyno is a single web process running on Heroku. It is capable of serving a single web request (pageview) at a time.

Request time - The length of time spent serving a web request in the dyno. Request times are generally in the range of 5 - 200ms, with the average being around 50ms. Heroku’s logging lists this time as “service.” In the following example the action took 138ms:

2011-06-23T17:43:06+00:00 heroku[router]: POST appname.heroku.com/users/sign_in dyno=web.1 queue=0 wait=0ms service=138ms status=200 bytes=2598

Round-trip time - The time a request takes from the browser or client app. This number includes the request time, but also includes the time spent traversing the internet and the hops in front of your app include Nginx and Varnish. You can test round-trip time from your local machine like this:

$ ab -n 1 http://myapp.heroku.com/
...
Time per request:       207.189 [ms] (mean)

Concurrency - The number of requests that can be processed simultaneously. On Heroku, your concurrency level is exactly equal to the number of dynos you have selected on the My Apps -> Resources page for your app.

Throughput - Throughput is the number of requests that can be served in a given time period. You can calculate your app’s request-per-second throughput by dividing your concurrency (number of dynos) by your average request time. For example, if you’re running four dynos and have an average request time of 100ms, your maximum throughput will be 4 / 0.100 = 40 requests per second.

Backlog - When a web request comes in for your app, the Heroku routing mesh routes it to a randomly-chosen dyno. In most cases, a dyno will be ready to take the request immediately. But if your app is getting more requests than it can handle, all dynos may be busy, in which case the mesh has to hold onto the request and wait for a dyno to become available. The requests being held in this manner are stored in a queue known as a backlog.

A brief backlog is not anything to worry about: a brief spike of traffic may cause requests to sit in the backlog for a few hundred milliseconds, then be forwarded onto a dyno to be processed as usual. But if the traffic spike is not brief but in fact a longer-term increase in traffic, a growing backlog can indicate that you may soon need to take action.

Further reading: Backlogs and Request Time