Using AWS S3 to Store Static Assets and File Uploads

Last Updated: 07 February 2014

file upload s3 static assets

AWS Simple Storage Service, e.g. S3, is a “highly durable and available store” and can be used to reliably store application content such as media files, static assets and user uploads. It allows you to offload your entire storage infrastructure and offers better scalability, reliability, and speed than just storing files on the filesystem.

AWS S3, or similar storage services, are important when architecting applications for scale and are a perfect complement to Heroku’s ephemeral filesystem.

Overview

S3 is a different kind of file service and has different semantics from other file-based services.

All files in S3 are stored in buckets which act as a top-level container much like a directory. All files sent to S3 belong to a bucket and bucket names must be unique across the whole Amazon system.

Access to the S3 API is governed by an Access Key ID and a Secret Access Key. The access key identifies your S3 user account while the secret key is akin to a password and should be kept secret.

S3 setup

Enabling an application to use S3 requires that the application have access to the AWS credentials as well as the name of the bucket to store files.

Credentials

Your S3 credentials can be found on the Security Credentials section of the AWS “My Account/Console” menu

Because of the sensitive nature of the S3 credentials they should never be stored in a file or committed to source control. On Heroku such information is stored as application config vars. Use heroku config:set to set both keys.

$ heroku config:set AWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=yyy
Adding config vars and restarting app... done, v21
  AWS_ACCESS_KEY_ID     => xxx
  AWS_SECRET_ACCESS_KEY => yyy

Bucket

A single bucket typically stores the files, assets and uploads for an application. To create a bucket access the S3 section of the AWS Management Console and create a new bucket in the US Standard region.

The bucket can be named anything as long as the name is unique across all of S3. Store the bucket name in a config var to give your application access to its value.

heroku config:set S3_BUCKET_NAME=appname-assets
Adding config vars and restarting app... done, v22
  S3_BUCKET_NAME     => appname-assets

Static assets

Static assets such as videos, PDFs, Javascript, CSS and image files can be manually uploaded to your S3 account using the command line or a graphical browser like the Amazon S3 Organizer add-on for Firefox or S3Hub app for OS X.

Once uploaded your application can reference its assets by copying their public URLs (such as http://s3.amazonaws.com/bucketname/filename) and pasting them directly into your app’s views or HTML files. These files will now be served directly from S3, freeing up your application to serve only dynamic requests.

File uploads

There are two approaches to processing and storing file uploads from a Heroku app to S3: direct and pass-through.

Direct upload

This is the preferred approach if you’re working with file uploads bigger than 4MB. The idea is to skip the hop to your dyno, making a direct connection from the end user browser to S3. While this reduces the processing required by your application it is a more complex implementation and limits the ability to modify (transform, filter, resize etc…) the file before storing in S3.

Direct upload support is available by POSTing directly to S3, using general-purpose client-side library like SWFUpload or a language-specific solution such as carrierwave_direct for Rails. There are also third-party services such as Transloadit that handle the upload to S3.

Pass-through uploads

Pass-through uploading sends the file from the user to the application on Heroku which then uploads it to S3. Benefits of this approach include the ability to pre-process user-uploads before storing in S3. However, be careful that large files don’t tie up your application dynos during the upload process.

Large files uploads in single-threaded, non-evented environments (such as Rails) block your application's web dynos and can cause request timeouts and H11, H12 errors. EventMachine, Node.js and JVM-based languages are less susceptible to such issues. Please be aware of this constraint and choose the right approach for your language or framework.

There are many language-specific libraries that automate this process including: