Table of Contents
AWS Simple Storage Service, e.g. S3, is a “highly durable and available store” and can be used to reliably store application content such as media files, static assets and user uploads. It allows you to offload your entire storage infrastructure and offers better scalability, reliability, and speed than just storing files on the filesystem.
AWS S3, or similar storage services, are important when architecting applications for scale and are a perfect complement to Heroku’s ephemeral filesystem.
S3 is a different kind of file service and has different semantics from other file-based services.
All files in S3 are stored in buckets which act as a top-level container much like a directory. All files sent to S3 belong to a bucket and bucket names must be unique across the whole Amazon system.
Access to the S3 API is governed by an Access Key ID and a Secret Access Key. The access key identifies your S3 user account while the secret key is akin to a password and should be kept secret.
Enabling an application to use S3 requires that the application have access to the AWS credentials as well as the name of the bucket to store files.
Because of the sensitive nature of the S3 credentials they should never be stored in a file or committed to source control. On Heroku such information is stored as application config vars. Use
heroku config:set to set both keys.
$ heroku config:set AWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=yyy Adding config vars and restarting app... done, v21 AWS_ACCESS_KEY_ID => xxx AWS_SECRET_ACCESS_KEY => yyy
A single bucket typically stores the files, assets and uploads for an application. To create a bucket access the S3 section of the AWS Management Console and create a new bucket in the US Standard region.
The bucket can be named anything as long as the name is unique across all of S3. Store the bucket name in a config var to give your application access to its value.
heroku config:set S3_BUCKET_NAME=appname-assets Adding config vars and restarting app... done, v22 S3_BUCKET_NAME => appname-assets
Once uploaded your application can reference its assets by copying their public URLs (such as
http://s3.amazonaws.com/bucketname/filename) and pasting them directly into your app’s views or HTML files. These files will now be served directly from S3, freeing up your application to serve only dynamic requests.
There are two approaches to processing and storing file uploads from a Heroku app to S3: direct and pass-through.
This is the preferred approach if you’re working with file uploads bigger than 4MB. The idea is to skip the hop to your dyno, making a direct connection from the end user browser to S3. While this reduces the processing required by your application it is a more complex implementation and limits the ability to modify (transform, filter, resize etc…) the file before storing in S3.
Direct upload support is available by POSTing directly to S3, using general-purpose client-side library like SWFUpload or a language-specific solution such as carrierwave_direct for Rails. There are also third-party services such as Transloadit that handle the upload to S3.
Pass-through uploading sends the file from the user to the application on Heroku which then uploads it to S3. Benefits of this approach include the ability to pre-process user-uploads before storing in S3. However, be careful that large files don’t tie up your application dynos during the upload process.
Large files uploads in single-threaded, non-evented environments (such as Rails) block your application's web dynos and can cause request timeouts and H11, H12 errors. EventMachine, Node.js and JVM-based languages are less susceptible to such issues. Please be aware of this constraint and choose the right approach for your language or framework.
There are many language-specific libraries that automate this process including: