pathio

README

pathio

-- import "gopkg.in/Clever/pathio.v3"

Package pathio is a package that allows writing to and reading from different types of paths transparently. It supports two types of paths:

1. Local file paths
2. S3 File Paths (s3://bucket/key)

Note that using s3 paths requires setting two environment variables

1. AWS_SECRET_ACCESS_KEY
2. AWS_ACCESS_KEY_ID

Usage

Pathio has a very easy to use interface, with 4 main functions:

import "gopkg.in/Clever/pathio.v3"
var err error

ListFiles

// func ListFiles(path string) ([]string, error)
files, err = pathio.ListFiles("s3://bucket/my/key") // s3
files, err = pathio.ListFiles("/home/me")           // local

Write / WriteReader

// func Write(path string, input []byte) error
toWrite := []byte("hello world\n")
err = pathio.Write("s3://bucket/my/key", toWrite)   // s3
err = pathio.Write("/home/me/hello_world", toWrite) // local

// func WriteReader(path string, input io.ReadSeeker) error
toWriteReader, err := os.Open("test.txt") // this implements Read and Seek
err = pathio.WriteReader("s3://bucket/my/key", toWriteReader)   // s3
err = pathio.WriteReader("/home/me/hello_world", toWriteReader) // local

Read

// func Reader(path string) (rc io.ReadCloser, err error)
reader, err = pathio.Reader("s3://bucket/key/to/read") // s3
reader, err = pathio.Reader("/home/me/file/to/read")   // local

Documentation

Overview

Package pathio is a generated GoMock package.

Package pathio is a package that allows writing to and reading from different types of paths transparently. It supports two types of paths:

1. Local file paths
2. S3 File Paths (s3://bucket/key)

Note that using s3 paths requires setting two environment variables

1. AWS_SECRET_ACCESS_KEY
2. AWS_ACCESS_KEY_ID

Index

• func ListFiles(path string) ([]string, error)
• func Reader(path string) (rc io.ReadCloser, err error)
• func Write(path string, input []byte) error
• func WriteReader(path string, input io.ReadSeeker) error
• type Client
  • func (c *Client) Exists(path string) (bool, error)
  • func (c *Client) ListFiles(path string) ([]string, error)
  • func (c *Client) Reader(path string) (rc io.ReadCloser, err error)
  • func (c *Client) Write(path string, input []byte) error
  • func (c *Client) WriteReader(path string, input io.ReadSeeker) error
• type MockPathio
  • func NewMockPathio(ctrl *gomock.Controller) *MockPathio
  • func (m *MockPathio) EXPECT() *MockPathioMockRecorder
  • func (m *MockPathio) ListFiles(path string) ([]string, error)
  • func (m *MockPathio) Reader(path string) (io.ReadCloser, error)
  • func (m *MockPathio) Write(path string, input []byte) error
  • func (m *MockPathio) WriteReader(path string, input io.ReadSeeker) error
• type MockPathioMockRecorder
  • func (mr *MockPathioMockRecorder) ListFiles(path interface{}) *gomock.Call
  • func (mr *MockPathioMockRecorder) Reader(path interface{}) *gomock.Call
  • func (mr *MockPathioMockRecorder) Write(path, input interface{}) *gomock.Call
  • func (mr *MockPathioMockRecorder) WriteReader(path, input interface{}) *gomock.Call
• type Mocks3Handler
  • func NewMocks3Handler(ctrl *gomock.Controller) *Mocks3Handler
  • func (m *Mocks3Handler) EXPECT() *Mocks3HandlerMockRecorder
  • func (m *Mocks3Handler) GetBucketLocation(input *s3.GetBucketLocationInput) (*s3.GetBucketLocationOutput, error)
  • func (m *Mocks3Handler) GetObject(input *s3.GetObjectInput) (*s3.GetObjectOutput, error)
  • func (m *Mocks3Handler) HeadObject(input *s3.HeadObjectInput) (*s3.HeadObjectOutput, error)
  • func (m *Mocks3Handler) ListObjects(input *s3.ListObjectsInput) (*s3.ListObjectsOutput, error)
  • func (m *Mocks3Handler) PutObject(input *s3.PutObjectInput) (*s3.PutObjectOutput, error)
• type Mocks3HandlerMockRecorder
  • func (mr *Mocks3HandlerMockRecorder) GetBucketLocation(input interface{}) *gomock.Call
  • func (mr *Mocks3HandlerMockRecorder) GetObject(input interface{}) *gomock.Call
  • func (mr *Mocks3HandlerMockRecorder) HeadObject(input interface{}) *gomock.Call
  • func (mr *Mocks3HandlerMockRecorder) ListObjects(input interface{}) *gomock.Call
  • func (mr *Mocks3HandlerMockRecorder) PutObject(input interface{}) *gomock.Call
• type Pathio

Functions

func ListFiles

func ListFiles(path string) ([]string, error)

ListFiles calls DefaultClient's ListFiles method.

func Reader

func Reader(path string) (rc io.ReadCloser, err error)

Reader calls DefaultClient's Reader method.

func Write

func Write(path string, input []byte) error

Write calls DefaultClient's Write method.

func WriteReader

func WriteReader(path string, input io.ReadSeeker) error

WriteReader calls DefaultClient's WriteReader method.

Types

type Client

type Client struct {
	Region string
	// contains filtered or unexported fields
}

Client is the pathio client used to access the local file system and S3. To configure options on the client, create a new Client and call its methods directly.

`&Client{
		disableS3Encryption: true, // disables encryption
		Region: "us-east-1", // hardcodes the s3 region, instead of looking it up
}.Write(...)`

func (*Client) Exists

func (c *Client) Exists(path string) (bool, error)

Exists determines if a path does or does not exist. NOTE: S3 is eventually consistent so keep in mind that there is a delay.

func (*Client) ListFiles

func (c *Client) ListFiles(path string) ([]string, error)

ListFiles lists all the files/directories in the directory. It does not recurse

func (*Client) Reader

func (c *Client) Reader(path string) (rc io.ReadCloser, err error)

Reader returns an io.Reader for the specified path. The path can either be a local file path or an S3 path. It is the caller's responsibility to close rc.

func (*Client) Write

func (c *Client) Write(path string, input []byte) error

Write writes a byte array to the specified path. The path can be either a local file path or an S3 path.

func (*Client) WriteReader

func (c *Client) WriteReader(path string, input io.ReadSeeker) error

WriteReader writes all the data read from the specified io.Reader to the output path. The path can either a local file path or an S3 path.

type MockPathio

type MockPathio struct {
	// contains filtered or unexported fields
}

MockPathio is a mock of Pathio interface

func NewMockPathio

func NewMockPathio(ctrl *gomock.Controller) *MockPathio

NewMockPathio creates a new mock instance

func (*MockPathio) EXPECT

func (m *MockPathio) EXPECT() *MockPathioMockRecorder

EXPECT returns an object that allows the caller to indicate expected use

func (*MockPathio) ListFiles

func (m *MockPathio) ListFiles(path string) ([]string, error)

ListFiles mocks base method

func (*MockPathio) Reader

func (m *MockPathio) Reader(path string) (io.ReadCloser, error)

Reader mocks base method

func (*MockPathio) Write

func (m *MockPathio) Write(path string, input []byte) error

Write mocks base method

func (*MockPathio) WriteReader

func (m *MockPathio) WriteReader(path string, input io.ReadSeeker) error

WriteReader mocks base method

type MockPathioMockRecorder

type MockPathioMockRecorder struct {
	// contains filtered or unexported fields
}

MockPathioMockRecorder is the mock recorder for MockPathio

func (*MockPathioMockRecorder) ListFiles

func (mr *MockPathioMockRecorder) ListFiles(path interface{}) *gomock.Call

ListFiles indicates an expected call of ListFiles

func (*MockPathioMockRecorder) Reader

func (mr *MockPathioMockRecorder) Reader(path interface{}) *gomock.Call

Reader indicates an expected call of Reader

func (*MockPathioMockRecorder) Write

func (mr *MockPathioMockRecorder) Write(path, input interface{}) *gomock.Call

Write indicates an expected call of Write

func (*MockPathioMockRecorder) WriteReader

func (mr *MockPathioMockRecorder) WriteReader(path, input interface{}) *gomock.Call

WriteReader indicates an expected call of WriteReader

type Mocks3Handler

type Mocks3Handler struct {
	// contains filtered or unexported fields
}

Mocks3Handler is a mock of s3Handler interface

func NewMocks3Handler

func NewMocks3Handler(ctrl *gomock.Controller) *Mocks3Handler

NewMocks3Handler creates a new mock instance

func (*Mocks3Handler) EXPECT

func (m *Mocks3Handler) EXPECT() *Mocks3HandlerMockRecorder

EXPECT returns an object that allows the caller to indicate expected use

func (*Mocks3Handler) GetBucketLocation

func (m *Mocks3Handler) GetBucketLocation(input *s3.GetBucketLocationInput) (*s3.GetBucketLocationOutput, error)

GetBucketLocation mocks base method

func (*Mocks3Handler) GetObject

func (m *Mocks3Handler) GetObject(input *s3.GetObjectInput) (*s3.GetObjectOutput, error)

GetObject mocks base method

func (*Mocks3Handler) HeadObject

func (m *Mocks3Handler) HeadObject(input *s3.HeadObjectInput) (*s3.HeadObjectOutput, error)

HeadObject mocks base method

func (*Mocks3Handler) ListObjects

func (m *Mocks3Handler) ListObjects(input *s3.ListObjectsInput) (*s3.ListObjectsOutput, error)

ListObjects mocks base method

func (*Mocks3Handler) PutObject

func (m *Mocks3Handler) PutObject(input *s3.PutObjectInput) (*s3.PutObjectOutput, error)

PutObject mocks base method

type Mocks3HandlerMockRecorder

type Mocks3HandlerMockRecorder struct {
	// contains filtered or unexported fields
}

Mocks3HandlerMockRecorder is the mock recorder for Mocks3Handler

func (*Mocks3HandlerMockRecorder) GetBucketLocation

func (mr *Mocks3HandlerMockRecorder) GetBucketLocation(input interface{}) *gomock.Call

GetBucketLocation indicates an expected call of GetBucketLocation

func (*Mocks3HandlerMockRecorder) GetObject

func (mr *Mocks3HandlerMockRecorder) GetObject(input interface{}) *gomock.Call

GetObject indicates an expected call of GetObject

func (*Mocks3HandlerMockRecorder) HeadObject

func (mr *Mocks3HandlerMockRecorder) HeadObject(input interface{}) *gomock.Call

HeadObject indicates an expected call of HeadObject

func (*Mocks3HandlerMockRecorder) ListObjects

func (mr *Mocks3HandlerMockRecorder) ListObjects(input interface{}) *gomock.Call

ListObjects indicates an expected call of ListObjects

func (*Mocks3HandlerMockRecorder) PutObject

func (mr *Mocks3HandlerMockRecorder) PutObject(input interface{}) *gomock.Call

PutObject indicates an expected call of PutObject

type Pathio

type Pathio interface {
	Reader(path string) (rc io.ReadCloser, err error)
	Write(path string, input []byte) error
	WriteReader(path string, input io.ReadSeeker) error
	ListFiles(path string) ([]string, error)
}

Pathio is a defined interface for accessing both S3 and local files.

var DefaultClient Pathio = &Client{}

DefaultClient is the default pathio client called by the Reader, Writer, and WriteReader methods. It has S3 encryption enabled.