pathio

package module
v3.7.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 9, 2019 License: Apache-2.0 Imports: 12 Imported by: 0

README

pathio

GoDoc

-- import "gopkg.in/Clever/pathio.v3"

Package pathio is a package that allows writing to and reading from different types of paths transparently. It supports two types of paths:

1. Local file paths
2. S3 File Paths (s3://bucket/key)

Note that using s3 paths requires setting two environment variables

1. AWS_SECRET_ACCESS_KEY
2. AWS_ACCESS_KEY_ID

Usage

Pathio has a very easy to use interface, with 5 main functions:

import "gopkg.in/Clever/pathio.v3"
var err error
ListFiles
// func ListFiles(path string) ([]string, error)
files, err = pathio.ListFiles("s3://bucket/my/key") // s3
files, err = pathio.ListFiles("/home/me")           // local
Write / WriteReader
// func Write(path string, input []byte) error
toWrite := []byte("hello world\n")
err = pathio.Write("s3://bucket/my/key", toWrite)   // s3
err = pathio.Write("/home/me/hello_world", toWrite) // local

// func WriteReader(path string, input io.ReadSeeker) error
toWriteReader, err := os.Open("test.txt") // this implements Read and Seek
err = pathio.WriteReader("s3://bucket/my/key", toWriteReader)   // s3
err = pathio.WriteReader("/home/me/hello_world", toWriteReader) // local
Read
// func Reader(path string) (rc io.ReadCloser, err error)
reader, err = pathio.Reader("s3://bucket/key/to/read") // s3
reader, err = pathio.Reader("/home/me/file/to/read")   // local
Delete
// func Delete(path string) error
err = pathio.Delete("s3://bucket/key/to/read") // s3
err = pathio.Delete("/home/me/file/to/read")   // local

Documentation

Overview

Package pathio is a generated GoMock package.

Package pathio is a package that allows writing to and reading from different types of paths transparently. It supports two types of paths:

  1. Local file paths
  2. S3 File Paths (s3://bucket/key)

Note that using s3 paths requires setting two environment variables

  1. AWS_SECRET_ACCESS_KEY
  2. AWS_ACCESS_KEY_ID

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Delete added in v3.7.0 

func Delete(path string) error

Delete calls DefaultClient's Delete method.

func Exists added in v3.7.0 

func Exists(path string) (bool, error)

Exists calls DefaultClient's Exists method.

func ListFiles added in v3.2.0 

func ListFiles(path string) ([]string, error)

ListFiles calls DefaultClient's ListFiles method.

func Reader 

func Reader(path string) (rc io.ReadCloser, err error)

Reader calls DefaultClient's Reader method.

func Write 

func Write(path string, input []byte) error

Write calls DefaultClient's Write method.

func WriteReader 

func WriteReader(path string, input io.ReadSeeker) error

WriteReader calls DefaultClient's WriteReader method.

Types

type Client 

type Client struct {
	Region string
	// contains filtered or unexported fields
}

Client is the pathio client used to access the local file system and S3. To configure options on the client, create a new Client and call its methods directly. 

&Client{
	disableS3Encryption: true, // disables encryption
	Region: "us-east-1", // hardcodes the s3 region, instead of looking it up
}.Write(...)

func NewClient added in v3.6.0 

func NewClient(cfg *aws.Config) *Client

NewClient creates a new client that utilizes the provided AWS config. This can be leveraged to enforce more limited permissions.

func (*Client) Delete added in v3.7.0 

func (c *Client) Delete(path string) error

Delete deletes the object at the specified path. The path can be either a local file path or an S3 path.

func (*Client) Exists added in v3.3.1 

func (c *Client) Exists(path string) (bool, error)

Exists determines if a path does or does not exist. NOTE: S3 is eventually consistent so keep in mind that there is a delay.

func (*Client) ListFiles added in v3.2.0 

func (c *Client) ListFiles(path string) ([]string, error)

ListFiles lists all the files/directories in the directory. It does not recurse

func (*Client) Reader 

func (c *Client) Reader(path string) (rc io.ReadCloser, err error)

Reader returns an io.Reader for the specified path. The path can either be a local file path or an S3 path. It is the caller's responsibility to close rc.

func (*Client) Write 

func (c *Client) Write(path string, input []byte) error

Write writes a byte array to the specified path. The path can be either a local file path or an S3 path.

func (*Client) WriteReader 

func (c *Client) WriteReader(path string, input io.ReadSeeker) error

WriteReader writes all the data read from the specified io.Reader to the output path. The path can either a local file path or an S3 path.

type MockPathio added in v3.3.4 

type MockPathio struct {
	// contains filtered or unexported fields
}

MockPathio is a mock of Pathio interface

func NewMockPathio added in v3.3.4 

func NewMockPathio(ctrl *gomock.Controller) *MockPathio

NewMockPathio creates a new mock instance

func (*MockPathio) Delete added in v3.7.0 

func (m *MockPathio) Delete(path string) error

Delete mocks base method

func (*MockPathio) EXPECT added in v3.3.4 

func (m *MockPathio) EXPECT() *MockPathioMockRecorder

EXPECT returns an object that allows the caller to indicate expected use

func (*MockPathio) Exists added in v3.4.0 

func (m *MockPathio) Exists(path string) (bool, error)

Exists mocks base method

func (*MockPathio) ListFiles added in v3.3.4 

func (m *MockPathio) ListFiles(path string) ([]string, error)

ListFiles mocks base method

func (*MockPathio) Reader added in v3.3.4 

func (m *MockPathio) Reader(path string) (io.ReadCloser, error)

Reader mocks base method

func (*MockPathio) Write added in v3.3.4 

func (m *MockPathio) Write(path string, input []byte) error

Write mocks base method

func (*MockPathio) WriteReader added in v3.3.4 

func (m *MockPathio) WriteReader(path string, input io.ReadSeeker) error

WriteReader mocks base method

type MockPathioMockRecorder added in v3.3.4 

type MockPathioMockRecorder struct {
	// contains filtered or unexported fields
}

MockPathioMockRecorder is the mock recorder for MockPathio

func (*MockPathioMockRecorder) Delete added in v3.7.0 

func (mr *MockPathioMockRecorder) Delete(path interface{}) *gomock.Call

Delete indicates an expected call of Delete

func (*MockPathioMockRecorder) Exists added in v3.4.0 

func (mr *MockPathioMockRecorder) Exists(path interface{}) *gomock.Call

Exists indicates an expected call of Exists

func (*MockPathioMockRecorder) ListFiles added in v3.3.4 

func (mr *MockPathioMockRecorder) ListFiles(path interface{}) *gomock.Call

ListFiles indicates an expected call of ListFiles

func (*MockPathioMockRecorder) Reader added in v3.3.4 

func (mr *MockPathioMockRecorder) Reader(path interface{}) *gomock.Call

Reader indicates an expected call of Reader

func (*MockPathioMockRecorder) Write added in v3.3.4 

func (mr *MockPathioMockRecorder) Write(path, input interface{}) *gomock.Call

Write indicates an expected call of Write

func (*MockPathioMockRecorder) WriteReader added in v3.3.4 

func (mr *MockPathioMockRecorder) WriteReader(path, input interface{}) *gomock.Call

WriteReader indicates an expected call of WriteReader

type Mocks3Handler added in v3.3.1 

type Mocks3Handler struct {
	// contains filtered or unexported fields
}

Mocks3Handler is a mock of s3Handler interface

func NewMocks3Handler added in v3.3.1 

func NewMocks3Handler(ctrl *gomock.Controller) *Mocks3Handler

NewMocks3Handler creates a new mock instance

func (*Mocks3Handler) DeleteObject added in v3.7.0 

func (m *Mocks3Handler) DeleteObject(input *s3.DeleteObjectInput) (*s3.DeleteObjectOutput, error)

DeleteObject mocks base method

func (*Mocks3Handler) EXPECT added in v3.3.1 

func (m *Mocks3Handler) EXPECT() *Mocks3HandlerMockRecorder

EXPECT returns an object that allows the caller to indicate expected use

func (*Mocks3Handler) GetBucketLocation added in v3.3.1 

func (m *Mocks3Handler) GetBucketLocation(input *s3.GetBucketLocationInput) (*s3.GetBucketLocationOutput, error)

GetBucketLocation mocks base method

func (*Mocks3Handler) GetObject added in v3.3.1 

func (m *Mocks3Handler) GetObject(input *s3.GetObjectInput) (*s3.GetObjectOutput, error)

GetObject mocks base method

func (*Mocks3Handler) HeadObject added in v3.3.1 

func (m *Mocks3Handler) HeadObject(input *s3.HeadObjectInput) (*s3.HeadObjectOutput, error)

HeadObject mocks base method

func (*Mocks3Handler) ListObjects added in v3.3.1 

func (m *Mocks3Handler) ListObjects(input *s3.ListObjectsInput) (*s3.ListObjectsOutput, error)

ListObjects mocks base method

func (*Mocks3Handler) PutObject added in v3.3.1 

func (m *Mocks3Handler) PutObject(input *s3.PutObjectInput) (*s3.PutObjectOutput, error)

PutObject mocks base method

type Mocks3HandlerMockRecorder added in v3.3.4 

type Mocks3HandlerMockRecorder struct {
	// contains filtered or unexported fields
}

Mocks3HandlerMockRecorder is the mock recorder for Mocks3Handler

func (*Mocks3HandlerMockRecorder) DeleteObject added in v3.7.0 

func (mr *Mocks3HandlerMockRecorder) DeleteObject(input interface{}) *gomock.Call

DeleteObject indicates an expected call of DeleteObject

func (*Mocks3HandlerMockRecorder) GetBucketLocation added in v3.3.4 

func (mr *Mocks3HandlerMockRecorder) GetBucketLocation(input interface{}) *gomock.Call

GetBucketLocation indicates an expected call of GetBucketLocation

func (*Mocks3HandlerMockRecorder) GetObject added in v3.3.4 

func (mr *Mocks3HandlerMockRecorder) GetObject(input interface{}) *gomock.Call

GetObject indicates an expected call of GetObject

func (*Mocks3HandlerMockRecorder) HeadObject added in v3.3.4 

func (mr *Mocks3HandlerMockRecorder) HeadObject(input interface{}) *gomock.Call

HeadObject indicates an expected call of HeadObject

func (*Mocks3HandlerMockRecorder) ListObjects added in v3.3.4 

func (mr *Mocks3HandlerMockRecorder) ListObjects(input interface{}) *gomock.Call

ListObjects indicates an expected call of ListObjects

func (*Mocks3HandlerMockRecorder) PutObject added in v3.3.4 

func (mr *Mocks3HandlerMockRecorder) PutObject(input interface{}) *gomock.Call

PutObject indicates an expected call of PutObject

type Pathio added in v3.3.4 

type Pathio interface {
	Reader(path string) (rc io.ReadCloser, err error)
	Write(path string, input []byte) error
	WriteReader(path string, input io.ReadSeeker) error
	Delete(path string) error
	ListFiles(path string) ([]string, error)
	Exists(path string) (bool, error)
}

Pathio is a defined interface for accessing both S3 and local files. 

var DefaultClient Pathio = &Client{}

DefaultClient is the default pathio client called by the Reader, Writer, and WriteReader methods. It has S3 encryption enabled.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.