pathio

README

pathio

-- import "gopkg.in/Clever/pathio.v3"

Package pathio is a package that allows writing to and reading from different types of paths transparently. It supports two types of paths:

1. Local file paths
2. S3 File Paths (s3://bucket/key)

Note that using s3 paths requires setting two environment variables

1. AWS_SECRET_ACCESS_KEY
2. AWS_ACCESS_KEY_ID

Usage

Pathio has a very easy to use interface, with 4 main functions:

import "gopkg.in/Clever/pathio.v3"
var err error

ListFiles

// func ListFiles(path string) ([]string, error)
files, err = pathio.ListFiles("s3://bucket/my/key") // s3
files, err = pathio.ListFiles("/home/me")           // local

Write / WriteReader

// func Write(path string, input []byte) error
toWrite := []byte("hello world\n")
err = pathio.Write("s3://bucket/my/key", toWrite)   // s3
err = pathio.Write("/home/me/hello_world", toWrite) // local

// func WriteReader(path string, input io.ReadSeeker) error
toWriteReader, err := os.Open("test.txt") // this implements Read and Seek
err = pathio.WriteReader("s3://bucket/my/key", toWriteReader)   // s3
err = pathio.WriteReader("/home/me/hello_world", toWriteReader) // local

Read

// func Reader(path string) (rc io.ReadCloser, err error)
reader, err = pathio.Reader("s3://bucket/key/to/read") // s3
reader, err = pathio.Reader("/home/me/file/to/read")   // local

Documentation

Overview

Package pathio is a generated GoMock package.

Package pathio is a package that allows writing to and reading from different types of paths transparently. It supports two types of paths:

1. Local file paths
2. S3 File Paths (s3://bucket/key)

Note that using s3 paths requires setting two environment variables

1. AWS_SECRET_ACCESS_KEY
2. AWS_ACCESS_KEY_ID

Index

• Variables
• func ListFiles(path string) ([]string, error)
• func Reader(path string) (rc io.ReadCloser, err error)
• func Write(path string, input []byte) error
• func WriteReader(path string, input io.ReadSeeker) error
• type Client
  • func (c *Client) Exists(path string) (bool, error)
  • func (c *Client) ListFiles(path string) ([]string, error)
  • func (c *Client) Reader(path string) (rc io.ReadCloser, err error)
  • func (c *Client) Write(path string, input []byte) error
  • func (c *Client) WriteReader(path string, input io.ReadSeeker) error
• type MockS3Handler
  • func NewMockS3Handler(ctrl *gomock.Controller) *MockS3Handler
  • func (m *MockS3Handler) EXPECT() *MockS3HandlerMockRecorder
  • func (m *MockS3Handler) GetBucketLocation(input *s3.GetBucketLocationInput) (*s3.GetBucketLocationOutput, error)
  • func (m *MockS3Handler) GetObject(input *s3.GetObjectInput) (*s3.GetObjectOutput, error)
  • func (m *MockS3Handler) HeadObject(input *s3.HeadObjectInput) (*s3.HeadObjectOutput, error)
  • func (m *MockS3Handler) ListObjects(input *s3.ListObjectsInput) (*s3.ListObjectsOutput, error)
  • func (m *MockS3Handler) PutObject(input *s3.PutObjectInput) (*s3.PutObjectOutput, error)
• type MockS3HandlerMockRecorder
  • func (mr *MockS3HandlerMockRecorder) GetBucketLocation(input interface{}) *gomock.Call
  • func (mr *MockS3HandlerMockRecorder) GetObject(input interface{}) *gomock.Call
  • func (mr *MockS3HandlerMockRecorder) HeadObject(input interface{}) *gomock.Call
  • func (mr *MockS3HandlerMockRecorder) ListObjects(input interface{}) *gomock.Call
  • func (mr *MockS3HandlerMockRecorder) PutObject(input interface{}) *gomock.Call
• type S3Handler

Variables

var DefaultClient = &Client{}

DefaultClient is the default pathio client called by the Reader, Writer, and WriteReader methods. It has S3 encryption enabled.

Functions

func ListFiles

func ListFiles(path string) ([]string, error)

ListFiles calls DefaultClient's ListFiles method.

func Reader

func Reader(path string) (rc io.ReadCloser, err error)

Reader calls DefaultClient's Reader method.

func Write

func Write(path string, input []byte) error

Write calls DefaultClient's Write method.

func WriteReader

func WriteReader(path string, input io.ReadSeeker) error

WriteReader calls DefaultClient's WriteReader method.

Types

type Client

type Client struct {
	Region string
	// contains filtered or unexported fields
}

Client is the pathio client used to access the local file system and S3. To configure options on the client, create a new Client and call its methods directly.

`&Client{
		disableS3Encryption: true, // disables encryption
		Region: "us-east-1", // hardcodes the s3 region, instead of looking it up
}.Write(...)`

func (*Client) Exists

func (c *Client) Exists(path string) (bool, error)

Exists determines if a path does or does not exist. NOTE: S3 is eventually consistent so keep in mind that there is a delay.

func (*Client) ListFiles

func (c *Client) ListFiles(path string) ([]string, error)

ListFiles lists all the files/directories in the directory. It does not recurse

func (*Client) Reader

func (c *Client) Reader(path string) (rc io.ReadCloser, err error)

Reader returns an io.Reader for the specified path. The path can either be a local file path or an S3 path. It is the caller's responsibility to close rc.

func (*Client) Write

func (c *Client) Write(path string, input []byte) error

Write writes a byte array to the specified path. The path can be either a local file path or an S3 path.

func (*Client) WriteReader

func (c *Client) WriteReader(path string, input io.ReadSeeker) error

WriteReader writes all the data read from the specified io.Reader to the output path. The path can either a local file path or an S3 path.

type MockS3Handler

type MockS3Handler struct {
	// contains filtered or unexported fields
}

MockS3Handler is a mock of S3Handler interface

func NewMockS3Handler

func NewMockS3Handler(ctrl *gomock.Controller) *MockS3Handler

NewMockS3Handler creates a new mock instance

func (*MockS3Handler) EXPECT

func (m *MockS3Handler) EXPECT() *MockS3HandlerMockRecorder

EXPECT returns an object that allows the caller to indicate expected use

func (*MockS3Handler) GetBucketLocation

func (m *MockS3Handler) GetBucketLocation(input *s3.GetBucketLocationInput) (*s3.GetBucketLocationOutput, error)

GetBucketLocation mocks base method

func (*MockS3Handler) GetObject

func (m *MockS3Handler) GetObject(input *s3.GetObjectInput) (*s3.GetObjectOutput, error)

GetObject mocks base method

func (*MockS3Handler) HeadObject

func (m *MockS3Handler) HeadObject(input *s3.HeadObjectInput) (*s3.HeadObjectOutput, error)

HeadObject mocks base method

func (*MockS3Handler) ListObjects

func (m *MockS3Handler) ListObjects(input *s3.ListObjectsInput) (*s3.ListObjectsOutput, error)

ListObjects mocks base method

func (*MockS3Handler) PutObject

func (m *MockS3Handler) PutObject(input *s3.PutObjectInput) (*s3.PutObjectOutput, error)

PutObject mocks base method

type MockS3HandlerMockRecorder

type MockS3HandlerMockRecorder struct {
	// contains filtered or unexported fields
}

MockS3HandlerMockRecorder is the mock recorder for MockS3Handler

func (*MockS3HandlerMockRecorder) GetBucketLocation

func (mr *MockS3HandlerMockRecorder) GetBucketLocation(input interface{}) *gomock.Call

GetBucketLocation indicates an expected call of GetBucketLocation

func (*MockS3HandlerMockRecorder) GetObject

func (mr *MockS3HandlerMockRecorder) GetObject(input interface{}) *gomock.Call

GetObject indicates an expected call of GetObject

func (*MockS3HandlerMockRecorder) HeadObject

func (mr *MockS3HandlerMockRecorder) HeadObject(input interface{}) *gomock.Call

HeadObject indicates an expected call of HeadObject

func (*MockS3HandlerMockRecorder) ListObjects

func (mr *MockS3HandlerMockRecorder) ListObjects(input interface{}) *gomock.Call

ListObjects indicates an expected call of ListObjects

func (*MockS3HandlerMockRecorder) PutObject

func (mr *MockS3HandlerMockRecorder) PutObject(input interface{}) *gomock.Call

PutObject indicates an expected call of PutObject

type S3Handler

type S3Handler interface {
	GetBucketLocation(input *s3.GetBucketLocationInput) (*s3.GetBucketLocationOutput, error)
	GetObject(input *s3.GetObjectInput) (*s3.GetObjectOutput, error)
	PutObject(input *s3.PutObjectInput) (*s3.PutObjectOutput, error)
	ListObjects(input *s3.ListObjectsInput) (*s3.ListObjectsOutput, error)
	HeadObject(input *s3.HeadObjectInput) (*s3.HeadObjectOutput, error)
}

S3Handler defines the interface that pathio exposes.