pathio

package module
v3.7.1+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 9, 2019 License: Apache-2.0 Imports: 12 Imported by: 5

README

pathio

GoDoc

-- import "gopkg.in/Clever/pathio.v3"

Package pathio is a package that allows writing to and reading from different types of paths transparently. It supports two types of paths:

1. Local file paths
2. S3 File Paths (s3://bucket/key)

Note that using s3 paths requires setting two environment variables

1. AWS_SECRET_ACCESS_KEY
2. AWS_ACCESS_KEY_ID

Usage

Pathio has a very easy to use interface, with 5 main functions:

import "gopkg.in/Clever/pathio.v3"
var err error
ListFiles
// func ListFiles(path string) ([]string, error)
files, err = pathio.ListFiles("s3://bucket/my/key") // s3
files, err = pathio.ListFiles("/home/me")           // local
Write / WriteReader
// func Write(path string, input []byte) error
toWrite := []byte("hello world\n")
err = pathio.Write("s3://bucket/my/key", toWrite)   // s3
err = pathio.Write("/home/me/hello_world", toWrite) // local

// func WriteReader(path string, input io.ReadSeeker) error
toWriteReader, err := os.Open("test.txt") // this implements Read and Seek
err = pathio.WriteReader("s3://bucket/my/key", toWriteReader)   // s3
err = pathio.WriteReader("/home/me/hello_world", toWriteReader) // local
Read
// func Reader(path string) (rc io.ReadCloser, err error)
reader, err = pathio.Reader("s3://bucket/key/to/read") // s3
reader, err = pathio.Reader("/home/me/file/to/read")   // local
Delete
// func Delete(path string) error
err = pathio.Delete("s3://bucket/key/to/read") // s3
err = pathio.Delete("/home/me/file/to/read")   // local

Documentation

Overview

Package pathio is a generated GoMock package.

Package pathio is a package that allows writing to and reading from different types of paths transparently. It supports two types of paths:

  1. Local file paths
  2. S3 File Paths (s3://bucket/key)

Note that using s3 paths requires setting two environment variables

  1. AWS_SECRET_ACCESS_KEY
  2. AWS_ACCESS_KEY_ID

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Delete 

func Delete(path string) error

Delete calls DefaultClient's Delete method.

func Exists 

func Exists(path string) (bool, error)

Exists calls DefaultClient's Exists method.

func ListFiles 

func ListFiles(path string) ([]string, error)

ListFiles calls DefaultClient's ListFiles method.

func Reader 

func Reader(path string) (rc io.ReadCloser, err error)

Reader calls DefaultClient's Reader method.

func Write 

func Write(path string, input []byte) error

Write calls DefaultClient's Write method.

func WriteReader 

func WriteReader(path string, input io.ReadSeeker) error

WriteReader calls DefaultClient's WriteReader method.

Types

type Client 

type Client struct {
	Region string
	// contains filtered or unexported fields
}

Client is the pathio client used to access the local file system and S3. To configure options on the client, create a new Client and call its methods directly. 

&Client{
	disableS3Encryption: true, // disables encryption
	Region: "us-east-1", // hardcodes the s3 region, instead of looking it up
}.Write(...)

func NewClient 

func NewClient(cfg *aws.Config) *Client

NewClient creates a new client that utilizes the provided AWS config. This can be leveraged to enforce more limited permissions.

func (*Client) Delete 

func (c *Client) Delete(path string) error

Delete deletes the object at the specified path. The path can be either a local file path or an S3 path.

func (*Client) Exists 

func (c *Client) Exists(path string) (bool, error)

Exists determines if a path does or does not exist. NOTE: S3 is eventually consistent so keep in mind that there is a delay.

func (*Client) ListFiles 

func (c *Client) ListFiles(path string) ([]string, error)

ListFiles lists all the files/directories in the directory. It does not recurse

func (*Client) Reader 

func (c *Client) Reader(path string) (rc io.ReadCloser, err error)

Reader returns an io.Reader for the specified path. The path can either be a local file path or an S3 path. It is the caller's responsibility to close rc.

func (*Client) Write 

func (c *Client) Write(path string, input []byte) error

Write writes a byte array to the specified path. The path can be either a local file path or an S3 path.

func (*Client) WriteReader 

func (c *Client) WriteReader(path string, input io.ReadSeeker) error

WriteReader writes all the data read from the specified io.Reader to the output path. The path can either a local file path or an S3 path.

type MockPathio 

type MockPathio struct {
	// contains filtered or unexported fields
}

MockPathio is a mock of Pathio interface

func NewMockPathio 

func NewMockPathio(ctrl *gomock.Controller) *MockPathio

NewMockPathio creates a new mock instance

func (*MockPathio) Delete 

func (m *MockPathio) Delete(path string) error

Delete mocks base method

func (*MockPathio) EXPECT 

func (m *MockPathio) EXPECT() *MockPathioMockRecorder

EXPECT returns an object that allows the caller to indicate expected use

func (*MockPathio) Exists 

func (m *MockPathio) Exists(path string) (bool, error)

Exists mocks base method

func (*MockPathio) ListFiles 

func (m *MockPathio) ListFiles(path string) ([]string, error)

ListFiles mocks base method

func (*MockPathio) Reader 

func (m *MockPathio) Reader(path string) (io.ReadCloser, error)

Reader mocks base method

func (*MockPathio) Write 

func (m *MockPathio) Write(path string, input []byte) error

Write mocks base method

func (*MockPathio) WriteReader 

func (m *MockPathio) WriteReader(path string, input io.ReadSeeker) error

WriteReader mocks base method

type MockPathioMockRecorder 

type MockPathioMockRecorder struct {
	// contains filtered or unexported fields
}

MockPathioMockRecorder is the mock recorder for MockPathio

func (*MockPathioMockRecorder) Delete 

func (mr *MockPathioMockRecorder) Delete(path interface{}) *gomock.Call

Delete indicates an expected call of Delete

func (*MockPathioMockRecorder) Exists 

func (mr *MockPathioMockRecorder) Exists(path interface{}) *gomock.Call

Exists indicates an expected call of Exists

func (*MockPathioMockRecorder) ListFiles 

func (mr *MockPathioMockRecorder) ListFiles(path interface{}) *gomock.Call

ListFiles indicates an expected call of ListFiles

func (*MockPathioMockRecorder) Reader 

func (mr *MockPathioMockRecorder) Reader(path interface{}) *gomock.Call

Reader indicates an expected call of Reader

func (*MockPathioMockRecorder) Write 

func (mr *MockPathioMockRecorder) Write(path, input interface{}) *gomock.Call

Write indicates an expected call of Write

func (*MockPathioMockRecorder) WriteReader 

func (mr *MockPathioMockRecorder) WriteReader(path, input interface{}) *gomock.Call

WriteReader indicates an expected call of WriteReader

type Mocks3Handler 

type Mocks3Handler struct {
	// contains filtered or unexported fields
}

Mocks3Handler is a mock of s3Handler interface

func NewMocks3Handler 

func NewMocks3Handler(ctrl *gomock.Controller) *Mocks3Handler

NewMocks3Handler creates a new mock instance

func (*Mocks3Handler) DeleteObject 

func (m *Mocks3Handler) DeleteObject(input *s3.DeleteObjectInput) (*s3.DeleteObjectOutput, error)

DeleteObject mocks base method

func (*Mocks3Handler) EXPECT 

func (m *Mocks3Handler) EXPECT() *Mocks3HandlerMockRecorder

EXPECT returns an object that allows the caller to indicate expected use

func (*Mocks3Handler) GetBucketLocation 

func (m *Mocks3Handler) GetBucketLocation(input *s3.GetBucketLocationInput) (*s3.GetBucketLocationOutput, error)

GetBucketLocation mocks base method

func (*Mocks3Handler) GetObject 

func (m *Mocks3Handler) GetObject(input *s3.GetObjectInput) (*s3.GetObjectOutput, error)

GetObject mocks base method

func (*Mocks3Handler) HeadObject 

func (m *Mocks3Handler) HeadObject(input *s3.HeadObjectInput) (*s3.HeadObjectOutput, error)

HeadObject mocks base method

func (*Mocks3Handler) ListObjects 

func (m *Mocks3Handler) ListObjects(input *s3.ListObjectsInput) (*s3.ListObjectsOutput, error)

ListObjects mocks base method

func (*Mocks3Handler) PutObject 

func (m *Mocks3Handler) PutObject(input *s3.PutObjectInput) (*s3.PutObjectOutput, error)

PutObject mocks base method

type Mocks3HandlerMockRecorder 

type Mocks3HandlerMockRecorder struct {
	// contains filtered or unexported fields
}

Mocks3HandlerMockRecorder is the mock recorder for Mocks3Handler

func (*Mocks3HandlerMockRecorder) DeleteObject 

func (mr *Mocks3HandlerMockRecorder) DeleteObject(input interface{}) *gomock.Call

DeleteObject indicates an expected call of DeleteObject

func (*Mocks3HandlerMockRecorder) GetBucketLocation 

func (mr *Mocks3HandlerMockRecorder) GetBucketLocation(input interface{}) *gomock.Call

GetBucketLocation indicates an expected call of GetBucketLocation

func (*Mocks3HandlerMockRecorder) GetObject 

func (mr *Mocks3HandlerMockRecorder) GetObject(input interface{}) *gomock.Call

GetObject indicates an expected call of GetObject

func (*Mocks3HandlerMockRecorder) HeadObject 

func (mr *Mocks3HandlerMockRecorder) HeadObject(input interface{}) *gomock.Call

HeadObject indicates an expected call of HeadObject

func (*Mocks3HandlerMockRecorder) ListObjects 

func (mr *Mocks3HandlerMockRecorder) ListObjects(input interface{}) *gomock.Call

ListObjects indicates an expected call of ListObjects

func (*Mocks3HandlerMockRecorder) PutObject 

func (mr *Mocks3HandlerMockRecorder) PutObject(input interface{}) *gomock.Call

PutObject indicates an expected call of PutObject

type Pathio 

type Pathio interface {
	Reader(path string) (rc io.ReadCloser, err error)
	Write(path string, input []byte) error
	WriteReader(path string, input io.ReadSeeker) error
	Delete(path string) error
	ListFiles(path string) ([]string, error)
	Exists(path string) (bool, error)
}

Pathio is a defined interface for accessing both S3 and local files. 

var DefaultClient Pathio = &Client{}

DefaultClient is the default pathio client called by the Reader, Writer, and WriteReader methods. It has S3 encryption enabled.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.