HTTP Routing

Last Updated: 08 June 2015

routing

Table of Contents

The Heroku platform automatically routes HTTP requests sent to your app’s hostname(s) to your web dynos. The entry point for all applications on the Cedar stack is the herokuapp.com domain which offers a direct routing path to your web dynos.

This article provides a detailed reference of how the router behaves, and how it conforms to the HTTP specification.

Routing

Inbound requests are received by a load balancer that offers SSL termination. From here they are passed directly to a set of routers.

The routers are responsible for determining the location of your application’s web dynos and forwarding the HTTP request to one of these dynos.

A request’s unobfuscated path from the end-client through the Heroku infrastructure to your application allows for full support of HTTP 1.1 features such as chunked responses, long polling, websockets, and using an async webserver to handle multiple responses from a single web process. HTTP 1.0 compatibility is also maintained.

Request distribution

Routers use a random selection algorithm for HTTP request load balancing across web dynos.

Request queueing

Each router maintains an internal per-app request counter. For Cedar apps, routers limit the number of active requests per dyno to 50. There is no coordination between routers however, so this request limit is per router. The request counter on each router has a maximum backlog size of 50n (n = the number of web dynos your app has running). If the request counter on a particular router fills up, subsequent requests to that router will immediately return an H11 (Backlog too deep) response.

Dyno connection behavior

When Heroku receives an HTTP request, a router establishes a new upstream TCP connection to a randomly selected web dyno. If the connection is refused or has not been successfully established after 5 seconds, the dyno will be quarantined and no other connections will be forwarded from that router to the dyno for up to 5 seconds. The quarantine only applies to a single router. Because each router keep its own list of quarantined dynos, other routers may continue to forward connections to the dyno.

When a connection to a dyno is refused or times out, the router processing the request will retry the request on another dyno. A maximum of 10 attempts (or fewer if you don’t have 10 running web dynos) will be made before returning an H19 or H21 error.

If all dynos are quarantined, the router will retry for up to 75 seconds (with an incremental back off), before serving an H99 error.

Timeouts

After a dyno connection has been established, HTTP requests have an initial 30 second window in which the web process must return response data (either the completed response or some amount of response data to indicate that the process is active). Processes that do not send response data within the initial 30-second window will see an H12 error in their logs.

After the initial response, each byte sent (either from the client or from your app process) resets a rolling 55 second window. If no data is sent during this 55 second window then the connection is terminated and a H15 error is logged.

Additional details can be found in the Request Timeout article.

Simultaneous connections

The herokuapp.com routing stack allows many concurrent connections to web dynos. For production apps, you should always choose an embedded webserver that allows multiple concurrent connections to maximize the responsiveness of your app. You can also take advantage of concurrent connections for long-polling requests.

Almost all modern web frameworks and embeddable webservers support multiple concurrent connections. Examples of webservers that allow concurrent request processing in the dyno include Unicorn (Ruby), Goliath (Ruby), Puma (JRuby), Gunicorn (Python), and Jetty (Java).

Request buffering

When processing an incoming request, a router sets up a buffer to receive the entire HTTP request line and request headers. Each of these have further limitations described in HTTP validation and restrictions. The body of a request is transmitted by using a 1024 byte buffer, filled and flushed continuously. This represents a size that encompasses the vast majority of requests in terms of volume. The request will start being dispatched to a dyno once the entire set of HTTP headers has been received.

As a result, each router buffers the header section of all requests, and will deliver this to your dyno as fast as our internal network will run. The dyno is protected from slow clients until the request body needs to be read. If you need protection from clients transmitting the body of a request slowly, you’ll have the request headers available to you in order to make a decision as to when you want to drop the request by closing the connection at the dyno.

Response buffering

The router maintains a 1MB buffer for responses from the dyno per connection. This means that you can send a response up to 1MB in size before the rate at which the client receives the response will affect the dyno - even if the dyno closes the connection, the router will keep sending the response buffer to the client. The transfer rate for responses larger than the 1MB buffer size will be limited by how fast the client can receive data.

Heroku headers

All headers are considered to be case-insensitive, as per HTTP Specification.

  • X-Forwarded-For: the originating IP address of the client connecting to the Heroku router
  • X-Forwarded-Proto: the originating protocol of the HTTP request (example: https)
  • X-Forwarded-Port: the originating port of the HTTP request (example: 443)
  • X-Request-Start: unix timestamp (milliseconds) when the request was received by the router
  • X-Request-Id: the Heroku HTTP Request ID
  • Via: a code name for the Heroku router

Heroku router log format

info logs

2012-10-11T03:47:20+00:00 heroku[router]: at=info method=GET path=/ host=myapp.herokuapp.com fwd="204.204.204.204" dyno=web.1 connect=1ms service=18ms status=200 bytes=13
  • method: HTTP request method
  • path: HTTP request path and query string
  • host: HTTP request Host header value
  • fwd: HTTP request X-Forwarded-For header value
  • dyno: name of the dyno that serviced the request
  • connect: amount of time in milliseconds spent establishing a connection to the backend web process
  • service: amount of time in milliseconds spent proxying data between the backend web process and the client
  • status: HTTP response code
  • bytes: Number of bytes transferred from the backend web process to the client

Error logs

2012-10-11T03:47:20+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path=/ host=myapp.herokuapp.com fwd="204.204.204.204" dyno=web.1 connect= service=30000ms status=503 bytes=0

Caching

Apps serving large amounts of static assets can take advantage of HTTP caching to improve performance and reduce load.

WebSockets

WebSocket functionality is supported for all applications.

Gzipped responses

Since requests to Cedar apps are made directly to the application server – not proxied through an HTTP server like nginx – any compression of responses must be done within your application.

Supported HTTP Methods

The Heroku HTTP stack supports any HTTP method (sometimes called a “verb”), even those not defined in an RFC, except the following: CONNECT.

Commonly used methods include GET, POST, PUT, DELETE, HEAD, OPTIONS, and PATCH. Method names are limited to 127 characters in length.

Expect: 100-continue

The HTTP protocol has a few built-in mechanisms to help clients cooperate with servers in order to get better service overall. One of such mechanisms is the Expect: 100-continue header that can be sent along with requests[1].

This header and value are to be used by friendly clients when they are sending large HTTP requests, and want to know if the server can safely accept it before sending it, in order to prevent denial of service issues, and allow some optimizations. The cURL HTTP client is the most known library using this mechanism, doing it for any content-body larger than 1kb (undocumented behavior).

Whenever asking for the permission, the server is allowed to respond with a 100 Continue HTTP status, which lets the client know that it is ready to accept the request and to let the client proceed. If the server cannot deal with it, it can return any other HTTP response that makes sense for it, such as 413 Request Entity Too Large if it doesn’t want to handle the load, and can optionally close the connection right after the fact.

As such, between any two clients and server, the mechanism that takes place may look a bit like this for a regular call:

[Client]                           [Server]
   |-------- Partial Request -------->|
   |<--------- 100 Continue ----------|
   |--------- Request Body ---------->|
   |<----------- Response ------------|

Or, for a denied request:

[Client]                           [Server]
   |-------- Partial Request -------->|
   |<-- 413 Request Entity Too Large -|

The mechanism needs to be more resilient, however, because not all servers and clients can understand that mechanism. As such, if the client doesn’t know whether the server can accept and honor the Expect: 100-Continue headers, it should be sending the actual body after having waited a period of time:

[Client]                           [Server]
   |-------- Partial Request -------->|
   |                                  |
   |                                  |
   |                                  |
   |--------- Request Body ---------->|
   |<----------- Response ------------|

By default, many web servers do not handle Expect: 100-continue as a mechanism. Therefore, the Heroku HTTP routers will automatically insert a 100 Continue response on behalf of the application it routes to, and will later forward the data.

This more or less disables the mechanism entirely as far as the dyno’s webserver is concerned.

Enabling 100-Continue Support

End-to-end continue support is now available (to members of the Heroku beta program) through a Heroku labs feature:

$ heroku labs:enable http-end-to-end-continue

If the extension is enabled, the general flow of 100-Continue feature is restored, and the router will pass on the expect: 100-continue headers and their associated 100 continue responses transparently.

This feature comes with its own set of corner cases and behaviors that have to be specified, however.

Corner cases

There are a set of specific corner cases that may come with this kind of request.

The original HTTP 1.1 RFC (RFC 2068) allowed use of the 100 Continue partial response to let the server say “the entire request isn’t done parsing, but keep going, I’m not denying it right away”. Later RFCs (RFC 2616 for example) contain the Expect: 100-continue mechanism, and reappropriated the 100 Continue partial response as part of the mechanism defined in the previous section of the text.

For backwards compatibility reasons, the server could possibly send a 100 Continue without having received the related Expect header, but is strongly advised against doing so.

A second corner case is that a server may decide not to send back a 100 Continue response if it has started receiving and processing body data first hand, which the client should be aware of.

Servers also have to be careful when parsing Expect headers, given they could contain more than one value. For example, Expect: 100-continue, auth could be sent when expecting the server to handle a large body, while asking to be authenticated before doing so. Technically, there can be as many values as desired within the Expect header with the behavior being defined purely for the client and the server’s sake.

Other special cases come from unexpected interactions coming from having multiple headers that manage connection flow. For example:

These behaviors are undefined by the original specifications, and the Heroku router has to make a decision regarding them in order to provide consistent behavior.

Proxy requirements

Even though the Expect header is defined to be an End-To-End header (only the client and the server would have to care about them, explaining why any term can be used in it), the 100 Continue mechanism itself (and the general Expect behavior support) requires coordination with proxies and is hop-by-hop. The RFC has special conditions added for them:

  • The proxy should pass the header as-is, whether it knows if the server can handle it or not
  • If the proxy knows that the server to which it’s routing has an HTTP version of 1.0 or lower, it must deny the request with a 417 Expectation Failed status, but it’s possible it doesn’t know about it.
  • An HTTP 1.0 client (or earlier) could send a request without an Expect: 100-Continue header, and the server could still respond to it using the 100 Continue HTTP code (as part of RFC 2068), in which case the proxy should strip the response entirely and wait to relay the final status.

Heroku router 100-continue support

The router takes some liberties regarding the undefined behaviors and corner cases mentioned earlier:

  • 100 Continue will be stripped as a response if the client is a HTTP 1.0 (or earlier) client and the Expect: 100-continue header isn’t part of the request, and will be forwarded otherwise no matter what.
  • The router will not require a 100 Continue response to start sending the request body, but will leave the wait time up to the client (without breaking the regular inactivity rules for connections on Heroku)
  • The router will automatically respond with a 417 Expectation Failed response and close the connection to the dyno if the Expect header contains any value other than 100 Continue (case insensitive)
  • If a WebSocket upgrade is requested, it will be sent as-is to the dyno, and the router will honor whichever response comes in: a 100 Continue status may ignore the WebSocket upgrade and return any code (as usual), and a 101 Switching Protocol will ignore the Expect headers' behavior. Note that in order to respect the HTTPbis Draft, The router will still look for a 101 Switching Protocol following a 100 Continue received from the server.
  • The router will ignore Connection: close on a 100 Continue and only honor it after having received the final response. Given that the RFC specifies that the connection should be closed “after the current request/response is complete”, and that 100 is not a terminal status, the connection will be closed only after having received a terminal status. Note however, that because Connection: close is a hop-by-hop mechanism, the router will not necessarily close the connection to the client, and may not forward it.
  • The router will strip all headers from a 100 Continue response, given no header is prescribed by the RFC and it makes the implementation much simpler.
  • The router will return a 5xx error code if the server returns a 100 Continue following an initial 100 Continue response. The router does not yet support infinite 1xx streams.
  • The router will close the connection to the server following a terminal status code, whether it was preceded by a 100 Continue or not beforehand – connections to dynos aren’t keep-alive.
  • The router will close the connection to the client following a terminal status code that was not preceded by a 100 Continue response. This will avoid having the client need to send the request body anyway before having the server being able to process the next request.

Other mechanisms should be respected as-is by the protocol, and the router should forward requests as specified by the RFC.

HTTP versions supported

Three main versions of HTTP are used in the wild: HTTP/0.9, HTTP/1.0, and HTTP/1.1. Side protocols such as SPDY and drafts for HTTP/2.0 are being worked on, but do not yet see widespread adoption.

The Heroku router only supports HTTP/1.0 and HTTP/1.1 clients. HTTP/0.9 and earlier are no longer supported. SPDY and HTTP/2.0 are not supported at this point.

The router’s behavior is to be as compliant as possible with the HTTP/1.1 specifications. Special exceptions must be made for HTTP/1.0 however:

  • The router will advertise itself as using HTTP/1.1 no matter if the client uses HTTP/1.0 or not.
  • The router will take on itself to do the necessary conversions from a chunked response to a regular HTTP response. In order to do so without accumulating potentially gigabytes of data, the response to the client will be delimited by the termination of the connection (See Point 4.4.5)
  • The router will assume that the client wants to close the connection on each request (no keep-alive).
  • An HTTP/1.0 client may send a request with an explicit connection:keep-alive header. Despite the keep-alive mechanism not being defined back in 1.0 (it was ad-hoc), the router makes the assumption that the behavior requested is similar to the HTTP/1.1 behavior at this point.

HTTP validation and restrictions

Request validation:

  • In the case of chunked encoding and content-length both being present in the request, the router will give precedence to chunked encoding.
  • If multiple content-length fields are present, and that they have the same length, they will be merged into a single content-length header
  • If a content-length header contain multiple values (content-length: 15,24) or a request contains multiple content-length headers with multiple values, the request will be denied with a code 400.
  • Headers are restricted to 8192 bytes per line (and 1000 bytes for the header name)
  • Hop-by-hop headers will be stripped to avoid confusion
  • At most, 1000 headers are allowed per request
  • The request line of the HTTP request is limited to 8192 bytes
  • The request line expects single spaces to separate between the verb, the path, and the HTTP version.

Response validation:

  • Hop-by-hop headers will be stripped to avoid confusion
  • Headers are restricted to 512kb per line
  • Cookies are explicitly restricted to 8192 bytes. This is to protect against common restrictions (for example, imposed by CDNs) that rarely accept larger cookie values. In such cases, a developer could accidentally set large cookies, which would be submitted back to the user, who would then see all of his or her requests denied.
  • The status line (HTTP/1.1 200 OK) is restricted to 8192 bytes in length

Applications that break these limits in responses will see their requests fail with a 502 Bad Gateway response, and an H25 error will be injected into the application log stream. Clients that break these limits in requests will see their request fail with a 400 Bad Request response.

Additionally, while HTTP/1.1 requests and responses are expected to be keep-alive by default, if the initial request had an explicit connection: close header from the router to the dyno, the dyno can send a response delimited by the connection termination, without a specific content-encoding nor an explicit content-length.

Protocol upgrades

Whereas the previous Heroku router restricted HTTP protocol upgrades to WebSockets only, the new router tolerates any upgrade at all.

Specific points related to the implementation:

  • Any HTTP verb can be used with an upgradable connection
  • Even though HEAD HTTP verbs usually do not require having a proper response sent over the line (regarding content-length, for example), HEAD requests are explicitly made to work with 101 Switching Protocols responses. A dyno that doesn’t want to upgrade should send a different status code, and the connection will not be upgraded

Not supported

  • SPDY
  • HTTP/2.x
  • Expect headers with any content other than 100-continue (yields a 417)
  • HTTP Extensions such as WEBDAV
  • A HEAD, 1xx, 204, or 304 response with a content-length or chunked encoding doesn’t have the proxy try to relay a body that will never come
  • Header line endings other than CRLF (\r\n)
  • Caching of HTTP Content
  • Caching the HTTP versions of servers running on dynos
  • Long-standing preallocated idle connections. The limit is set to 1 minute before an idle connection is closed.
  • HTTP/1.0 requests without the Host header, even when the full URL is submitted in the request line.