SFTP to S3 Backup

Abhimanyu Babbar — Mon, 15 Jul 2019 17:05:18 +0000

In our company, we had a simple SFTP Server where third parties used to supply us with reports, files etc. In order to backup the information onto S3 where we could start processing the files, I wrote a simple utility in Go to achieve the same sftp-s3-backup.

The purpose of this article is to walk through a simple process of backing up the files present in SFTP onto S3.

S3 Login Credentials

In order to backup onto S3 our system needs to be able to login to the platform. Below environment variables needs to be set for the program to automatically pick them up. The variables are as follows:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_REGION
AWS_SESSION_TOKEN // In case you have 2 factor auth enabled.

Simple Backup

package main

import (
  "log",
  backup "github.com/babbarshaer/sftp-s3-backup"
)

func main() {

  config := backup.Config{
    User:              "user",
    Address:           "address",
    Port:              22,
    PublicKeyLocation: "/home/user/.ssh/id_rsa.pub",
  }

  client := backup.Client(config)
  err = client.Init()
  if err != nil {
    log.Fatalf("Unable to initialize the client: %s", err.Error())
  }

  defer client.Close()

  err := client.Backup(
    "/data/user/uploads",
    "aws.bucket",
    backup.DefaultPathTransformer)

  // If we don't want the default path transformer
  // we can override it with our own implementation.

  if err != nil {
    log.Fatalf("Unable to backup, err: %s", err.Error())
  }

We will go through each part of the program step by step:

We start by creating a config object which mainly contains the credentials of the sftp server who's directory we need to backup onto S3.

config := backup.Config{
    User:              "user",
    Address:           "address",
    Port:              22,
    PublicKeyLocation: "/home/user/.ssh/id_rsa.pub",
  }

We then create a new backup client object by supplying in the config. We initialize the client by calling the Init() method.

client := backup.Client(config)
client.Init()

If everything goes smoothly, we should be able to establish a sftp connection after this point. Otherwise, it would return an error. Handling of error depends upon the user at this point.

We should always remember to free up the client connection.

defer client.Close()

Finally, we can now perform a directory backup. This method will backup all the files in the directory to the corresponding bucket in s3. We supply an additional transformer variable which dictates how the path needs to be transformed when storing the object from sftp onto s3.

directoryToBackup := "/data/user/uploads"
awsBucket := "aws.bucket"
transformer := backup.DefaultPathTransformer

err := client.Backup(
    directoryToBackup,
    awsBucket,
    transformer)

A user can supply there own transformer function as follows:

func MyTransformer(path string) string {
  newPath := path
  // computation ...
  return newPath
}

Forem: Abhimanyu Babbar

SFTP to S3 Backup

S3 Login Credentials

Simple Backup