pathio

README

pathio

-- import "gopkg.in/Clever/pathio.v3"

Package pathio is a package that allows writing to and reading from different types of paths transparently. It supports two types of paths:

1. Local file paths
2. S3 File Paths (s3://bucket/key)

Note that using s3 paths requires setting two environment variables

1. AWS_SECRET_ACCESS_KEY
2. AWS_ACCESS_KEY_ID

Usage

Pathio has a very easy to use interface, with 4 main functions:

import "gopkg.in/Clever/pathio.v3"
var err error

ListFiles

// func ListFiles(path string) ([]string, error)
files, err = pathio.ListFiles("s3://bucket/my/key") // s3
files, err = pathio.ListFiles("/home/me")           // local

Write / WriteReader

// func Write(path string, input []byte) error
toWrite := []byte("hello world\n")
err = pathio.Write("s3://bucket/my/key", toWrite)   // s3
err = pathio.Write("/home/me/hello_world", toWrite) // local

// func WriteReader(path string, input io.ReadSeeker) error
toWriteReader, err := os.Open("test.txt") // this implements Write and Seek
err = pathio.WriteReader("s3://bucket/my/key", toWriteReader)   // s3
err = pathio.WriteReader("/home/me/hello_world", toWriteReader) // local

Read

// func Reader(path string) (rc io.ReadCloser, err error)
reader, err = pathio.Reader("s3://bucket/key/to/read") // s3
reader, err = pathio.Reader("/home/me/file/to/read")   // local

Documentation

Overview

Package pathio is a package that allows writing to and reading from different types of paths transparently. It supports two types of paths:

1. Local file paths
2. S3 File Paths (s3://bucket/key)

Note that using s3 paths requires setting two environment variables

1. AWS_SECRET_ACCESS_KEY
2. AWS_ACCESS_KEY_ID

Index

• Variables
• func ListFiles(path string) ([]string, error)
• func Reader(path string) (rc io.ReadCloser, err error)
• func Write(path string, input []byte) error
• func WriteReader(path string, input io.ReadSeeker) error
• type Client
  • func (c *Client) ListFiles(path string) ([]string, error)
  • func (c *Client) Reader(path string) (rc io.ReadCloser, err error)
  • func (c *Client) Write(path string, input []byte) error
  • func (c *Client) WriteReader(path string, input io.ReadSeeker) error

Variables

var DefaultClient = &Client{}

DefaultClient is the default pathio client called by the Reader, Writer, and WriteReader methods. It has S3 encryption enabled.

Functions

func ListFiles

func ListFiles(path string) ([]string, error)

ListFiles calls DefaultClient's ListFiles method.

func Reader

func Reader(path string) (rc io.ReadCloser, err error)

Reader calls DefaultClient's Reader method.

func Write

func Write(path string, input []byte) error

Write calls DefaultClient's Write method.

func WriteReader

func WriteReader(path string, input io.ReadSeeker) error

WriteReader calls DefaultClient's WriteReader method.

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is the pathio client used to access the local file system and S3. It includes an option to disable S3 encryption. To disable S3 encryption, create a new Client and call it directly: `&Client{disableS3Encryption: true}.Write(...)`

func (*Client) ListFiles

func (c *Client) ListFiles(path string) ([]string, error)

ListFiles lists all the files/directories in the directory. It does not recurse

func (*Client) Reader

func (c *Client) Reader(path string) (rc io.ReadCloser, err error)

Reader returns an io.Reader for the specified path. The path can either be a local file path or an S3 path. It is the caller's responsibility to close rc.

func (*Client) Write

func (c *Client) Write(path string, input []byte) error

Write writes a byte array to the specified path. The path can be either a local file path or an S3 path.

func (*Client) WriteReader

func (c *Client) WriteReader(path string, input io.ReadSeeker) error

WriteReader writes all the data read from the specified io.Reader to the output path. The path can either a local file path or an S3 path.