This add-on is operated by Crazy Ant Labs

Fully Managed SFTP/FTPS Service with S3 and HTTP file access

Last updated 17 September 2020

SFTP To Go allows you to add fully managed cloud SFTP storage to your Heroku applications. SFTP is a standard network protocol for secure file access, transfer, and management.

Having a managed SFTP service enables you to exchange data with partners and customers securely in a standard protocol, without purchasing and running your own SFTP servers and storage. SFTP To Go is based on AWS offerings, making it highly scalable and highly available with multi-AZ architecture.

With SFTP To Go, you can also use FTPS and S3 to access your files.

Provisioning the add-on

SFTP To Go can be attached to a Heroku application via the CLI:

$ heroku addons:create sftptogo
-----> Adding sftptogo to sharp-mountain-4005... done, v18 (free)

After you provision SFTP To Go, the SFTPTOGO_URL config var is available in your app’s configuration. It contains the URL to your provisioned SFTP service, including the root username, password, and host. You can confirm this via the heroku config:get command:

$ heroku config:get SFTPTOGO_URL
sftp://user:pass@server.sftptogo.com

The root user has read/write access to your entire SFTP service.

Once you provision the SFTP To Go add-on, there is nothing to set up. You and your application can immediately access your SFTP service.

Local setup

Environment setup

After you provision the add-on, it’s necessary to locally replicate its config vars so your development environment can operate against the service.

Use the Heroku Local command-line tool to configure, run and manage process types specified in your app’s Procfile. Heroku Local reads configuration variables from a .env file. To view all of your app’s config vars, type heroku config. Use the following command for each value that you want to add to your .env file:

$ heroku config:get SFTPTOGO_URL -s  >> .env

For more information, see the Heroku Local article.

Configuring DNS for custom subdomain

After provisioning the add-on, you can add a DNS record to use your subdomain instead of SFTP To Go’s host name. To do that, copy your add-on host name from the dashboard and add a CNAME record to point to the host name. For example:

Consult your DNS provider’s documentation for specific instructions on creating CNAME records.

Using with Linux/Mac command line interface

Start an interactive SFTP session using the following command:

$ sftp user@host

You will then be prompted to enter your password.

Some of the simple commands you can use with SFTP:

Using with data integration platforms

Modern data integration platforms allow you to easily read data files from SFTP or write files into SFTP to easily integrate data from a variety of sources for further analysis and BI processes.

For example, to connect from Xplenty, a data integration add-on in Heroku, to SFTP To Go, create an SFTP connection in Xplenty. Go to SFTP To Go dashboard, copy the host, user and password, and paste in Xplenty’s new SFTP Connection page according to the steps here.

Using with graphical client tools

There are many available SFTP and FTPS GUI clients, such as Cyberduck, WinSCP, FileZilla, Transmit and CloudBerry.

To use Cyberduck with SFTP To Go, copy the value of your SFTPTOGO_URL config var to CyberDuck and click Quick Connect.

Using with Ruby

Ruby applications need to add the following entry into their Gemfile specifying the Net-SFTP client library:

gem 'net-sftp'

Update application dependencies with bundler:

$ bundle install

To download a file:

require 'net/sftp'
require 'uri'

sftptogo_url = ENV['SFTPTOGO_URL']

uri = URI.parse(sftptogo_url)

Net::SFTP.start(uri.host, uri.user, :password => uri.password) do |sftp|
  # download a file or directory from the remote host
  sftp.download!("/path/to/remote", "/path/to/local")
end

Using with Node

Node applications need to add the following entry into their package.json specifying the dependency on the ssh2-sftp-client library:

{
  "name": "your-app-name",
  "dependencies": {
    "ssh2-sftp-client": "^5.2.2" ...
    }
}

You can also install it locally:

$ npm install ssh2-sftp-client

To list files:

let Client = require('ssh2-sftp-client');
let sftp = new Client();

sftp.connect({
  host: 'myserver.sftptogo.com',
  port: '22',
  username: 'username',
  password: '******'
}).then(() => {
  return sftp.list('/home');
}).then(data => {
  console.log(data, 'the data info');
}).catch(err => {
  console.log(err, 'catch error');
});

See more examples here.

Using with other languages

To use SFTP To Go with other languages, use language-specific SFTP SDKs (e.g., Python, Java)

Amazon S3 access

In addition to the SFTP protocol, SFTP To Go allows you to use S3 APIs to access the same data. This allows you to share data with some users or applications using SFTP and with others, using S3, out of the box. When the add-on is provisioned, the following environment variables are added to your app to allow programmatic access using S3 APIs:

• SFTPTOGO_AWS_ACCESS_KEY_ID - IAM access key
• SFTPTOGO_AWS_SECRET_ACCESS_KEY - IAM secret
• SFTPTOGO_AWS_BUCKET - bucket name
• SFTPTOGO_AWS_REGION - AWS region

You can also use the values in these environment variables to access your data using Amazon S3 CLI and UI clients.

Note that the values (especially access key and secret) may change, so please refer to the environment variables.

Using with the AWS CLI

If you have Amazon S3 access, you can use the the AWS CLI to interact with your files.

After you install the AWS CLI, you can configure it with the SFTPTOGO_AWS_* settings in your app or on your dashboard:

$ aws configure
AWS Access Key ID [None]: <$SFTPTOGO_AWS_ACCESS_KEY_ID>
AWS Secret Access Key [None]: <$SFTPTOGO_AWS_SECRET_ACCESS_KEY>
Default region name [None]: <$SFTPTOGO_AWS_REGION>

The following lists popular commands using the AWS CLI:

Dashboard

Use The SFTP To Go dashboard to:

1. Access your SFTP server’s information (host, user name, and password),
2. Rotate your password. This generates a new password and changes the URI in the SFTPTOGO_URL config var.

1. Get storage and bandwidth metrics to monitor your add-on usage.
2. Add/remove/modify/activate/deactivate credentials (see below).

You can access the dashboard via the CLI:

$ heroku addons:open sftptogo
Opening sftptogo for sharp-mountain-4005

or by visiting the Heroku Dashboard and selecting the application in question. Select SFTP To Go from the Add-ons menu.

Credentials and permissions

You can create more credentials to use within the same add-on instance using the UI:

1. Click Add credential
2. Optionally choose a nickname for the credential. This only shows in the UI.
3. Optionally select a home directory for the credential. By default, each credential has access to its own home directory and nothing else. You can change the credential’s home directory to have multiple credentials access the same directory.
4. Select the level of permissions for the new user. By default the user has read-only access to her home directory:

1. Click Add credential. The user will be given a random user name and password.
2. You may deactivate a user to revoke access to the user while maintaining all credential information.

You may also import public ssh keys to use with the user name instead of password:

1. Click Import SSH key.
2. Generate a new key pair or copy an existing public key (usually ends with .pub). You can generate a new key pair using ssh-keygen -t rsa on Linux/Mac or using PuTTYgen on Windows. Make sure you generate a new RSA key.
3. Paste the public key. Make sure it begins with ssh-rsa.
4. Click Import SSH key

Webhooks

Webhooks enable you to receive notifications whenever particular events occur within your SFTP To Go add-on. You can subscribe to notifications for the following events:

• File / directory creation
• File / directory deletion

Webhook notifications are sent as HTTPS POST requests to a URL of your choosing. To integrate with webhooks, you need to implement a server endpoint that receives and handles these requests.

To subscribe to webhooks, go to the webhooks interface in the dashboard by Webhooks. Then click the Add webhook but ton.

In the dialog that opens, fill out the following:

• Nickname (optional) - a descriptive name for your webhook.
• Endpoint URL - HTTPS URL of your server endpoint that will receive all webhook notifications.
• Authorization Header (optional) - a custom Authorization header that will be included with all webhook notifications.
• Topics - select the types of notifications you want to be informed about. You must include at least one topic.

And click Add webhook.

Securing Webhooks

When a webhook is created, a signing secret is generated and displayed once.

Each request is signed by SFTP To Go in the X-Hub-Signature header. If you’d like to verify the authenticity of our request, copy the and use it to verify the webhook signature. You can rotate the signing secret at any time and get a new one.

You may also use an authorization header to verify that the request came from SFTP To Go. If set, the authorization header is passed through in the Authorization header in the request. It should be validated using the authorization mechanism of your choice on your server.

Managing Webhooks

After creating a webhook, you may do the following:

• Pause/Resume - temporarily pause webhook notifications or resume them.
• Rotate secret - see Securing Webhooks
• Ping webhook - manually send a test event to your endpoint
• View deliveries - View a log of the notifications that SFTP To Go has enqueued to send. Each notification has a status (one of Succeeded, Failed, ‘Pending’), Created timestamp, ID, Topic (one of job.execution.failed, job.execution.started, job.execution.succeeded, webhook.ping) and Duration. You may also view the Request payload, Response code and Response body.

Receiving Webhooks

When a webhook event that you’ve subscribed to occurs, SFTP To Go sends a POST request to your server endpoint with the details of the event.

You can verify the authenticity of these requests in the following ways:

• The request’s Authorization header matches the value you provided when subscribing to notifications.
• The request’s X-Hub-Signature header contains the HMAC SHA256 signature of the request body (signed with the secret value provided when subscribing).

A resulting webhook notification request resembles the following:

POST https://webhook.site/394f2074-e56f-4110-7bf7-ca14a1f48b7c
Authorization: Bearer 01234567-89ab-cdef-0123-456789abcdef
X-Hub-Signature: cLcN5U5x+jHEkANnVaaRwBw7yE4uv4pXdjcY9Cajc7M=

{
  "Id": "2e579289-4c4a-4085-9b43-74020865cdda",
  "Topic": "webhook.ping",
  "UpdatedAt": 1590911947688,
  "CreatedAt": 1590911947688,
  "Resource": "webhook",
  "PreviousData": null,
  "Data": {
    "Topics": [
      "file.deleted"
    ],
    "State": "enabled",
    "Alias": "Test Webhook",
    "CreatedAt": 1590501031122,
    "Id": "6b1c6f0c-52c1-47a6-9344-57a4579ced68",
    "OrganizationId": "d57060b1-23fe-2d59-afd0-7f56d9e1fc55",
    "UpdatedAt": 1590911941179,
    "Url": "https://webhook.site/394f2074-e56f-4110-7bf7-ca14a1f48b7c"
  },
  "Metadata": {
    "Webhook": {
      "Id": "6b1c6f0c-52c1-47a6-9344-57a4579ced68"
    },
    "Delivery": {
      "Id": "c7ce90e6-5708-43b1-a052-13991f45c771"
    },
    "Attempt": {
      "Id": "2e241efe-d22f-4fe7-aab9-adcde63fca6d"
    },
    "Event": {
      "Id": "2e579289-4c4a-4085-9b43-74020865cdda",
      "Topic": "webhook.ping"
    }
  }
}

You should always respond with a 200-level status code to indicate that you received the notification. SFTP To Go ignores the body of your response, so a 204 status with an empty body is ideal:

204 No Content

file.created Event Format

{
  "Id": "36cd78bc-0662-43b1-a7aa-0cde4876ab2a",
  "Topic": "file.created",
  "CreatedAt": 1591106805970,
  "UpdatedAt": 1591106805970,
  "Resource": "File",
  "PreviousData": null,
  "Data": {
    "Path": "dir/file1.txt",
    "Size": 357464
  },
  "Metadata": {
    "Organization": {
      "Id": "d57060b1-23fe-2d59-afd0-7f56d9e1fc55"
    },
    "Webhook": {
      "Id": "7f3c5b1d-5df4-409f-bbbd-3ac6a72d8b5a"
    },
    "Delivery": {
      "Id": "0a4ed750-fbb8-4718-8a6c-86c35c9b6348"
    },
    "Attempt": {
      "Id": "2cb02803-b3a4-4ccd-bd19-ad769c51b291"
    },
    "Event": {
      "Id": "36cd78bc-0662-43b1-a7aa-0cde4876ab2a",
      "Topic": "file.created"
    }
  }
}

file.deleted Event Format

{
  "Id": "f63ad40e-711d-482b-979a-dca97281d8b7",
  "Topic": "file.deleted",
  "CreatedAt": 1591106814353,
  "UpdatedAt": 1591106814353,
  "Resource": "File",
  "PreviousData": null,
  "Data": {
    "Path": "dir/file1.txt",
    "Size": null
  },
  "Metadata": {
    "Organization": {
      "Id": "d57060b1-23fe-2d59-afd0-7f56d9e1fc55"
    },
    "Webhook": {
      "Id": "7f3c5b1d-5df4-409f-bbbd-3ac6a72d8b5a"
    },
    "Delivery": {
      "Id": "aecd305e-f080-4c87-b40b-6b3d5d0d17db"
    },
    "Attempt": {
      "Id": "7dc9f0dc-06ad-4585-a9fd-5ac36d73f817"
    },
    "Event": {
      "Id": "f63ad40e-711d-482b-979a-dca97281d8b7",
      "Topic": "file.deleted"
    }
  }
}

webhook.ping Event Format

{
  "Id": "2e579289-4c4a-4085-9b43-74020865cdda",
  "Topic": "webhook.ping",
  "CreatedAt": 1590911947688,
  "UpdatedAt": 1590911947688,
  "Resource": "webhook",
  "PreviousData": null,
  "Data": {
    "Topics": [
      "job.execution.succeeded",
      "job.execution.failed",
      "job.execution.started"
    ],
    "State": "paused",
    "Alias": "Test Webhook",
    "CreatedAt": 1590501031122,
    "Id": "6b1c6f0c-52c1-47a6-9344-57a4579ced68",
    "OrganizationId": "d57060b1-23fe-2d59-afd0-7f56d9e1fc55",
    "UpdatedAt": 1590911941179,
    "Url": "https://webhook.site/394f2074-e56f-4110-7bf7-ca14a1f48b7c"
  },
  "Metadata": {
    "Webhook": {
      "Id": "6b1c6f0c-52c1-47a6-9344-57a4579ced68"
    },
    "Delivery": {
      "Id": "c7ce90e6-5708-43b1-a052-13991f45c771"
    },
    "Attempt": {
      "Id": "2e241efe-d22f-4fe7-aab9-adcde63fca6d"
    },
    "Event": {
      "Id": "2e579289-4c4a-4085-9b43-74020865cdda",
      "Topic": "webhook.ping"
    }
  }
}

Inbound rules

Inbound rules control the incoming traffic to your server by defining a safe list of IP addresses or IP address ranges using CIDR notation. By default, access from any IP to use your credentials with any protocol (SFTP or FTPS) is allowed. You can create up to 10 rules per add-on.

To create a new inbound rule:

1. Click Settings.
2. Add new inbound rule.
3. Select the allowed protocols.
4. Fill out the source IP address or CIDR for IP address range.
5. Add an optional description.
6. Click Add inbound rule.

You can also edit, disable, enable and delete inbound rules.

SFTP Connection via IP addresses

Some legacy systems require an IP address to connect to an SFTP server. You can perform an nslookup on your SFTP To Go host to find its underlying IP addresses and connect to one of them.

If you need static IPs, please contact us.

Migrating between plans

You can either use the UI to upgrade or downgrade SFTP To Go, or use the heroku addons:upgrade command to migrate to a new plan.

$ heroku addons:upgrade sftptogo:newplan
-----> Upgrading sftptogo:[[newplan]] to sharp-mountain-4005... done, v18 ($50/mo)
       Your plan has been updated to: sftptogo:newplan

Removing the add-on

You can remove SFTP To Go via the CLI:

$ heroku addons:destroy sftptogo
-----> Removing sftptogo from sharp-mountain-4005... done, v20 (free)

Before removing SFTP To Go, you can download your data using your favorite sftp client.

Known issues and remedies

Troubleshooting slow connection time

If you see that opening a connection to SFTP To Go takes a long time, or get connection timeouts in your client, follow these steps to troubleshoot:

1. Try to connect to SFTP To Go from a different network - there may be issues in your local network that are causing these connectivity issues. If it works well on the new network, resolve issues in your network.

2. Try to connect to SFTP To Go from a different computer - it may be your computer and services (e.g. anti-virus apps) on it that are causing these connectivity issues. If it works well on the new computer, resolve issues in your computer.

3. If after running these two tests, you still experience slow connection time or connection timeouts, use the following command pattern on a Mac console to get a client side log with timestamps. Replace URL with your add-on URL (copied from the add-on dashboard):

sftp -v URL 2>&1 | while read line; do echo "`date -u +"%Y-%m-%dT%H:%M:%SZ"` $line"; done

For example:

sftp -v sftp://a9fe70b82zbb94f7160dfcBfcd7@yellow-rectangle-14793.sftptogo.com 2>&1 | while read line; do echo "`date -u +"%Y-%m-%dT%H:%M:%SZ"` $line"; done

Paste your password when prompted to do it, and hit enter/return.

Wait until you see the text Connected to ... .sftptogo.com. and then type quit and hit enter/return.

Copy the client log and send it to us in addition to your computer’s time zone so we can investigate what happened on the servers at the same time.

How to handle FileZilla connection timeout to SFTP To Go

If you use FileZilla to connect to SFTP To Go and keep getting error messages such as Connection timed out after 20 seconds of inactivity, follow these steps to change client side configuration:

• On Windows, click Edit >> Settings
• On Mac, click FileZilla >> Settings
• Under [Connection], increase the value for timeout in seconds to 600 (default is 20 seconds).
• Click [OK] to save you changes and connect again.

Uploaded files are corrupted or empty

Try to reproduce the issue while running Wireshark to capture network packets and send us Wireshark’s output along with the paths to your corrupted files and the name and version of your SFTP client.

There are known issues that are the result of using clients with Perl modules. If you can’t avoid using Perl modules please make sure to take the following steps:

1. Use the option backend => 'Net_SSH2’. Add this to %attr and remove QueueDepth=1.
2. Make sure that you have access to the CPAN module Net::SFTP::Foreign::Backend::Net_SSH2, located on the metacpan website.
3. Use plink or run “sftp” within your Perl script using batch mode. Batch mode works well in this case because the command itself is interactive.

Supported and unsupported file operations in SFTP

All common commands to create, read, update and delete files and directories are supported.

Files are stored as objects on Amazon S3 and directories are managed as folder objects in S3. Directory rename operations, file append operations, changing ownership, permissions and file timestamps, and use of symbolic and hard links are not supported

Known limitations with FTPS

1. Only Explicit mode is supported. Implicit mode is not supported.
2. Only Passive mode is supported.
3. Only STREAM mode is supported.
4. Only Image/Binary mode is supported.
5. TLS - PROT C (unprotected) TLS for the data connection is the default.

Security

• The data you store on SFTP To Go is encrypted in transit and at rest on the server side.
• Our physical infrastructure is hosted on Amazon Web Services’ (AWS) data centers.
• AWS Security Groups (virtual firewalls) are utilized to restrict access to our network from external networks and between systems internally.
• Backend access to the OS level and AWS console is limited to Crazy Ant Labs staff and requires key authentication and multi factor authentication.

Support

All SFTP To Go support and runtime issues should be submitted via one of the Heroku Support channels. Any non-support related issues or product feedback is welcome at hello@crazyantlabs.com.