inflight

README

Inflight

Inflight is a simple package which abstracts away writing to and from S3. It handles retries (with an exponential back off algorithm), and can be configured to write to any S3 bucket.

An Example of where this package could be useful is when the data you're working with is too large to be manually passed through. This provides a simple way to ship references to data between functions in a state machine.

Example usage.

Lambda 1:

package main

var infl *inflight.Infight

func init() {
  bucket := os.Getenv("INFLIGHT_BUCKET")
  path := os.Getenv("INFLIGHT_PATH")
  s3 := //an s3 client
  infl = inflight.NewInflight(Bucket(bucket), KeyPath(path), s3)
}

func handler(event interface{}) *inflight.Ref {
  dataWhichIsOverTheAwsLimit := bytes.NewReader(...)
  return infl.Write(dataWhichIsOverTheAwsLimit)
}

func main() {
  lambda.Start(handler)
}

Lambda 2:

package main

var infl *inflight.Infight

func init() {
  bucket := os.Getenv("INFLIGHT_BUCKET")
  path := os.Getenv("INFLIGHT_PATH")
  s3 := //an s3 client
  infl = inflight.NewInflight(Bucket(bucket), KeyPath(path), s3)
}

func handler(event *inflight.Ref) *inflight.Ref {
  b, err := infl.Get(event.Object)
  // ...
  // Do Some stuff with your data
  // ...

  dataWhichIsOverTheAwsLimit := doSomeStuffWithYourData(b)

  // Write the data back
  return infl.Write(dataWhichIsOverTheAwsLimit)
}

func main() {
  lambda.Start(handler)
}

Documentation

Index

• type Bucket
• type Inflight
  • func NewInflight(bucket Bucket, keypath KeyPath, s3 s3iface.S3API) *Inflight
  • func (i *Inflight) Get(object string) ([]byte, error)
  • func (i *Inflight) Write(data []byte) (ref *Ref, err error)
• type KeyPath
• type ObjectKeyFunc
• type Ref

Types

type Bucket

type Bucket string

Bucket is a string value which represents the s3 bucket

type Inflight

type Inflight struct {
	s3iface.S3API
	Bucket  Bucket
	KeyPath KeyPath

	// ObjectKeyFunc will be called when Inflight#Write(io.ReadSeeker) is invoked.
	// The data will be given the name that this function generates.
	ObjectKeyFunc ObjectKeyFunc
}

Inflight is a structure which provides an interface to retrieving and writing data to s3, it doesn't care about what data you're writing, just provides an easy way to get to it

func NewInflight

func NewInflight(bucket Bucket, keypath KeyPath, s3 s3iface.S3API) *Inflight

NewInflight Creates a reference to an Inflight struct

func (*Inflight) Get

func (i *Inflight) Get(object string) ([]byte, error)

Get will retrieve the Object at the Bucket and KeyPath from S3.Get For instance, if you need the object at `cool-bucket/a/cool/key-path/the-object.json` you would say inflight::Get("the-object.json")

func (*Inflight) Write

func (i *Inflight) Write(data []byte) (ref *Ref, err error)

Write will take the data given and attempt to put it in S3 It then will return the S3 URI back to the caller so that the data may be passed between callers

type KeyPath

type KeyPath string

KeyPath is a string value which represents the s3 name space objects will be written to

type ObjectKeyFunc

type ObjectKeyFunc func([]byte) (string, error)

ObjectKeyFunc is a function which generates the string to be used as the object key.

type Ref

type Ref struct {
	Bucket string `json:"bucket"`
	Path   string `json:"path"`
	Object string `json:"object"`
}

Ref represents the path to an object in S3 broken down by bucket, Path, and Object Bucket = "my-s3-bucket" Path = "some/path/within" Object = "an-object-in-s3.json" s3://my-s3-bucket/some/path/within/an-object-in-s3.json