gof3r

README

Command gof3r provides a command-line interface to Amazon S3.

Documentation:
http://godoc.org/github.com/rlmcpherson/s3gof3r/gof3r

Documentation

Overview

Command gof3r is a command-line interface for s3gof3r: fast, concurrent, streaming access to Amazon S3.

Usage:

To stream up to S3:
   $  <input_stream> | gof3r put -b <bucket> -k <s3_path>
To stream down from S3:
   $ gof3r get -b <bucket> -k <s3_path> | <output_stream>
To upload a file to S3:
   $ gof3r  put --path=<local_path> --bucket=<bucket> --key=<s3_path> -m<http_header1> -m<http_header2>...
To download a file from S3:
   $ gof3r  get --bucket=<bucket> --key=<path>

Set AWS keys as environment Variables (required unless using ec2 instance-based credentials):

$ export AWS_ACCESS_KEY_ID=<access_key>
$ export AWS_SECRET_ACCESS_KEY=<secret_key>

Examples:

$ tar -cf - /foo_dir/ | gof3r put -b my_s3_bucket -k bar_dir/s3_object -m x-amz-meta-custom-metadata:abc123 -m x-amz-server-side-encryption:AES256
$ gof3r get -b my_s3_bucket -k bar_dir/s3_object | tar -x

FULL USAGE

get
    download from S3

    get (download) object from S3

    -p, --path
           Path to file. Defaults to standard output for streaming.

    -k, --key
           S3 object key

    -b, --bucket
           S3 bucket

    --no-ssl
           Do not use SSL for endpoint connection.

    --no-md5
           Do not use md5 hash checking to ensure data integrity. By default, the md5 hash of is calculated concurrently during puts, stored at <bucket>.md5/<key>.md5, and verified on gets.

    -c, --concurrency
           Concurrency of transfers

    -s, --partsize
           Initial size of concurrent parts, in bytes

    --endpoint
           Amazon S3 endpoint

    --debug
           Enable debug logging.

    -v, --versionId
           Version ID of the object. Incompatible with md5 check (use --no-md5).

    -h, --help
           Show this help message

put
    upload to S3

    put (upload) data to S3 object

    -p, --path
           Path to file. Defaults to standard input for streaming.

    -k, --key
           S3 object key

    -b, --bucket
           S3 bucket

    --no-ssl
           Do not use SSL for endpoint connection.

    --no-md5
           Do not use md5 hash checking to ensure data integrity. By default, the md5 hash of is calculated concurrently during puts, stored at <bucket>.md5/<key>.md5, and verified on gets.

    -c, --concurrency
           Concurrency of transfers

    -s, --partsize
           Initial size of concurrent parts, in bytes

    --endpoint
           Amazon S3 endpoint

    --debug
           Enable debug logging.

    -m, --header
           HTTP headers

    -h, --help
           Show this help message