s3link

README

s3link

An experiment in using the Storj Network via the AWS SDK for Go

background

In order to increase user adoption, Storj runs an S3-compliant web service called Gateway-MT. Gateway-MT has increased user adoption, but compromises on a core feature of the Storj Network in that it requires centralized HTTP endpoints. Analysis of various Amazon SDKs indicates it would be difficult to code generate "one size fits all" S3-API-compatible bindings across languages. Indeed [and perhaps because they span service types], most Amazon SDKs can be described as having "HTTP as the architecture".

Thus an initial attempt to integrate with the AWS SDK for Go by implementing an interface was abandoned. This more language agnostic solution accepts the overhead of serializing code to HTTP streams, which are then processed without actually sending those streams. Application logic from Storj's Gateway-ST project handles adapting the request back to native libuplink Storj Network requests.

usage

The Gateway-ST / Minio libraries used by this code implement request signing. Thus, the credentials and host names must match between the AWS SDK for Go session variable and the S3Link logic. EG:

	awsSession, _ := session.NewSession(&aws.Config{
		Credentials:      credentials.NewStaticCredentials("placeholder", "placeholder", ""),
		Endpoint:         aws.String("http://s3.amazonaws.com"),
		Region:           aws.String("us-east-1"),
		S3ForcePathStyle: aws.Bool(true),
	})

	client := s3.New(awsSession, aws.NewConfig().WithHTTPClient(s3link.NewHTTPClient()))
	downloader := s3manager.NewDownloader(awsSession, func(d *s3manager.Downloader) { d.S3 = client }),
	uploader := s3manager.NewUploader(awsSession, func(u *s3manager.Uploader) { u.S3 = client }),

Documentation

Index

• Variables
• func NewHTTPClient() *http.Client
• type StorjTransport
  • func NewStorjTransport() StorjTransport
  • func (n StorjTransport) RoundTrip(req *http.Request) (resp *http.Response, err error)

Variables

var GlobalActiveCred auth.Credentials

GlobalActiveCred exposes minio's cmd.globalActiveCred.

nolint: golint

var GlobalServerRegion string

GlobalServerRegion exposes minio's cmd.globalServerRegion.

nolint: golint

Functions

func NewHTTPClient

func NewHTTPClient() *http.Client

NewHTTPClient returns a new http.Client with StorjTransport as the Transport. client := s3.New(awsSession, aws.NewConfig().WithHTTPClient(&s3link.NewHTTPClient()})) downloader := s3manager.NewDownloader(awsSession, func(d *s3manager.Downloader) { d.S3 = client }) uploader := s3manager.NewUploader(awsSession, func(u *s3manager.Uploader) { u.S3 = client })

Types

type StorjTransport

type StorjTransport struct {
	// contains filtered or unexported fields
}

StorjTransport can be used as an http.Transport, but uses the Storj Network directly via DRCP, not HTTP.

func NewStorjTransport

func NewStorjTransport() StorjTransport

NewStorjTransport can be used as an http.Client{Transport: NewStorjTransport()}. A serialized Storj access grant should be passed in via the environment variables "ACCESS".

func (StorjTransport) RoundTrip

func (n StorjTransport) RoundTrip(req *http.Request) (resp *http.Response, err error)

RoundTrip implements the Transport interface.