Documentation ¶
Overview ¶
Package grw provides the interfaces, embedded structs, and implementing code for normalizing the reading/writing of a stream of bytes from archive/compressed files. This package supports the bzip2, gzip, snappy, and zip archive/compression algorithms. No compression can be identified as "none" or a blank string. This package is used by the go-stream package.
Usage ¶
You can import reader as a package into your own Go project or use the command line interface.
import ( "github.com/spatialcurrent/go-reader-writer/grw" ) r, err := grw.ReadFromFilePath("data-for-2018.sz", "snappy", false, 4096) if err != nil { panic(err) } for { b, err := input_reader.ReadBytes([]byte("\n")[0]) if err != nil { if err != io.EOF { fmt.Println(fmt.Errorf("error reading bytes from file: %w", err)) os.Exit(1) } } if len(b) > 0 { fmt.Println(string(b)) } if err != nil && err == io.EOF { break } }
See the github.com/go-reader/cmd/go-reader package for a command line tool for testing DFL expressions.
Projects go-reader-writer is used by the railgun project and go-stream
Index ¶
- Variables
- func CloseWriters(input *CloseWritersInput) error
- func ExpandOpenAndStat(path string) (*os.File, os.FileInfo, error)
- func UploadS3Object(input *UploadS3ObjectInput) error
- func WrapReader(r io.ReadCloser, alg string, dict []byte, bufferSize int) (io.ReadCloser, error)
- func WrapWriter(w io.WriteCloser, alg string, dict []byte, bufferSize int) (io.WriteCloser, error)
- func WriteAllAndClose(input *WriteAllAndCloseInput) error
- func WriteBuffers(input *WriteBuffersInput) error
- func WriteFile(input *WriteFileInput) (io.WriteCloser, error)
- func WriteToFileSystem(input *WriteToFileSystemInput) (io.WriteCloser, error)
- type CloseWritersInput
- type Closer
- type ErrFunctionNotImplemented
- type ErrReaderNotImplemented
- type ErrWriterNotImplemented
- type FunctionWriteCloser
- type Metadata
- type ReadFromResourceInput
- type ReadFromResourceOutput
- type Reader
- func (r *Reader) Close() error
- func (r *Reader) Read(p []byte) (n int, err error)
- func (r *Reader) ReadAll() ([]byte, error)
- func (r *Reader) ReadAllAndClose() ([]byte, error)
- func (r *Reader) ReadAt(p []byte, off int64) (n int, err error)
- func (r *Reader) ReadByte() (byte, error)
- func (r *Reader) ReadBytes(delim byte) ([]byte, error)
- func (r *Reader) ReadFirst() (byte, error)
- func (r *Reader) ReadRange(start int, end int) ([]byte, error)
- func (r *Reader) ReadString(delim byte) (string, error)
- type UploadS3ObjectInput
- type WriteAllAndCloseInput
- type WriteBuffersInput
- type WriteFileInput
- type WriteToFileSystemInput
- type WriteToResourceInput
- type WriteToResourceOutput
- type Writer
- func (w *Writer) Close() error
- func (w *Writer) CloseSafe() error
- func (w *Writer) Flush() error
- func (w *Writer) FlushSafe() error
- func (w *Writer) Write(p []byte) (n int, err error)
- func (w *Writer) WriteByte(b byte) error
- func (w *Writer) WriteError(e error) (n int, err error)
- func (w *Writer) WriteErrorSafe(e error) (n int, err error)
- func (w *Writer) WriteLine(s string) (n int, err error)
- func (w *Writer) WriteLineSafe(s string) (n int, err error)
- func (w *Writer) WriteString(s string) (n int, err error)
Constants ¶
This section is empty.
Variables ¶
var ( NoDict = []byte{} // no dictionary NoBuffer = 0 )
var ( Algorithms = []string{ alg.AlgorithmBzip2, alg.AlgorithmFlate, alg.AlgorithmGzip, alg.AlgorithmNone, alg.AlgorithmSnappy, alg.AlgorithmZip, alg.AlgorithmZlib, } )
var (
DefaultBufferSize = 4096
)
var (
ErrPathMissing = errors.New("path is missing")
)
Functions ¶
func CloseWriters ¶ added in v0.0.3
func CloseWriters(input *CloseWritersInput) error
CloseWriters closes a map of writers. If flush is true, will flush each writer before closing it.
func ExpandOpenAndStat ¶
ExpandOpenAndStat expands a path, returns *os.File and os.FileInfo for a given file path and an error, if any.
func UploadS3Object ¶
func UploadS3Object(input *UploadS3ObjectInput) error
UploadS3Object uploads an object to S3. alg may be "bzip2", "gzip", "snappy", "zip", or "".
func WrapReader ¶
func WrapReader(r io.ReadCloser, alg string, dict []byte, bufferSize int) (io.ReadCloser, error)
func WrapWriter ¶ added in v0.0.3
func WrapWriter(w io.WriteCloser, alg string, dict []byte, bufferSize int) (io.WriteCloser, error)
WrapWriter wraps the given writer with a buffer and the given compression. alg is the algorithm. dict is the initial dictionary (if the algorithm uses one).
func WriteAllAndClose ¶ added in v0.0.3
func WriteAllAndClose(input *WriteAllAndCloseInput) error
WriteAllAndClose writes the bytes to the resource indicated by the uri given, flushes, and closes the resource.
func WriteBuffers ¶
func WriteBuffers(input *WriteBuffersInput) error
WriteBuffers writes a map of buffers to the resources defined by the keys. alg is the compression algorithm. If the buffer already includes compressed data, then use "" or "none" as alg. If append is true, then append to existing files. If mkdirs is true, then parent directories are created on-demand.
func WriteFile ¶ added in v0.0.3
func WriteFile(input *WriteFileInput) (io.WriteCloser, error)
WriteFile returns a Writer for writing to a local file
func WriteToFileSystem ¶
func WriteToFileSystem(input *WriteToFileSystemInput) (io.WriteCloser, error)
WriteToFileSystem returns a ByteWriteCloser for a file with a given compression. alg may be "snappy", "gzip", or "none."
Types ¶
type CloseWritersInput ¶ added in v0.0.3
type Closer ¶
Closer is a helper struct for closing a sequential list of closers associated with a resource. This is used for flushing the footer for a given file compression alogrithm before closing the underlying file.
type ErrFunctionNotImplemented ¶
func (*ErrFunctionNotImplemented) Error ¶
func (e *ErrFunctionNotImplemented) Error() string
type ErrReaderNotImplemented ¶
type ErrReaderNotImplemented struct {
Algorithm string
}
func (*ErrReaderNotImplemented) Error ¶
func (e *ErrReaderNotImplemented) Error() string
type ErrWriterNotImplemented ¶
type ErrWriterNotImplemented struct {
Algorithm string
}
func (*ErrWriterNotImplemented) Error ¶
func (e *ErrWriterNotImplemented) Error() string
type FunctionWriteCloser ¶ added in v0.0.3
func (*FunctionWriteCloser) Close ¶ added in v0.0.3
func (w *FunctionWriteCloser) Close() error
type Metadata ¶
type Metadata struct { ContentType string LastModified *time.Time ContentLength int64 Header map[string][]string }
func NewMetadataFromHeader ¶
func NewMetadataFromS3 ¶
func NewMetadataFromS3(output *s3.GetObjectOutput) *Metadata
type ReadFromResourceInput ¶ added in v0.0.3
type ReadFromResourceInput struct { URI string // uri to read from Alg string // compression algorithm Dict []byte // compression dictionary BufferSize int // input reader buffer size S3Client *s3.S3 // AWS S3 Client SSHClient *ssh.Client // SSH Client SFTPClient *sftp.Client // SFTP Client Password string // password PrivateKey []byte // private key }
type ReadFromResourceOutput ¶ added in v0.0.3
type ReadFromResourceOutput struct { Reader io.ReadCloser Metadata *Metadata }
func ReadFromResource ¶
func ReadFromResource(input *ReadFromResourceInput) (*ReadFromResourceOutput, error)
type Reader ¶
type Reader struct {
Reader io.ReadCloser // the instance of ByteReader used for reading bytes
}
Reader is a struct for normalizing reading of bytes from files with arbitrary compression and for closing underlying resources. Reader implements the ByteReader interface by wrapping around a subordinate ByteReader.
func (*Reader) Read ¶
Read reads a maximum len(p) bytes from the reader and returns an error, if any.
func (*Reader) ReadAllAndClose ¶
ReadAllAndClose reads all the bytes from the underlying reader and attempts to close the reader, even if there was an error reading.
func (*Reader) ReadBytes ¶
ReadBytes returns all bytes up to an including the first occurrence of the delimiter "delim" and an error, if any.
type UploadS3ObjectInput ¶ added in v0.0.3
type WriteAllAndCloseInput ¶ added in v0.0.3
type WriteAllAndCloseInput struct { ACL string // ACL for objects written to AWS S3 BufferSize int // buffer size Bytes []byte // the content to write URI string // uri to write to Alg string // compression algorithm Dict []byte // compression dictionary Append bool // append to output resource Parents bool // automatically create parent directories as necessary S3Client *s3.S3 // AWS S3 Client }
WriteAllAndCloseInput contains the input parameters for WriteAllAndClose.
type WriteBuffersInput ¶
type WriteFileInput ¶ added in v0.0.3
type WriteFileInput struct { Path string // required field BufferSize int // if zero, then no buffer is used. Mode uint32 // defaults to 0600 }
WriteFileInput holds the input for the WriteFile method
type WriteToFileSystemInput ¶ added in v0.0.3
type WriteToFileSystemInput struct { Alg string // compression algorithm BufferSize int // buffer size Dict []byte // compression dictionary Flag int // flag for file descriptor Mode uint32 // mode of the output file Path string // path to write to Parents bool // automatically create parent directories as necessary }
WriteToFileSystemInput contains the input parameters for WriteToFileSystem.
type WriteToResourceInput ¶ added in v0.0.3
type WriteToResourceInput struct { ACL string // ACL for objects written to AWS s3 Alg string // compression algorithm Append bool // append to output resource BufferSize int // buffer size Dict []byte // compression dictionary Mode uint32 // mode of the output file Parents bool // automatically create parent directories as necessary Password string // password PrivateKey []byte // private key S3Client *s3.S3 // AWS S3 Client SSHClient *ssh.Client // SSH Client SFTPClient *sftp.Client // SFTP Client URI string // uri to write to }
type WriteToResourceOutput ¶ added in v0.0.3
type WriteToResourceOutput struct {
Writer io.WriteCloser
}
func WriteToResource ¶
func WriteToResource(input *WriteToResourceInput) (*WriteToResourceOutput, error)
WriteToResource returns a ByteWriteCloser and error, if any.
type Writer ¶
type Writer struct { *sync.Mutex // inherits Lock and Unlock Functions Writer io.ByteWriter // the instance of ByteWriter used for reading bytes }
Writer is a struct for normalizing reading of bytes from files with arbitrary compression and for closing underlying resources. Writer implements the ByteWriter interface by wrapping around a subordinate ByteWriter.
func NewWriter ¶
func NewWriter(w io.ByteWriter) *Writer
func (*Writer) CloseSafe ¶
CloseSafe closes the Closer and the underlying *os.File if not nil. CloseSafe also locks the writer for the duration of flushing using a sync.Mutex.
func (*Writer) FlushSafe ¶
FlushSafe flushes any intermediate writer. FlushSafe also locks the writer for the duration of flushing using a sync.Mutex.
func (*Writer) Write ¶
Write writes a slice of bytes to the underlying writer and returns an error, if any.
func (*Writer) WriteError ¶
WriteError writes a an error as a string with a trailing newline to the underlying writer and returns an error, if any.
func (*Writer) WriteErrorSafe ¶
WriteErrorSafe writes a an error as a string with a trailing newline to the underlying writer and returns an error, if any. WriteErrorSafe also locks the writer for the duration of writing using a sync.Mutex.
func (*Writer) WriteLine ¶
WriteLine writes a string with a trailing newline to the underlying writer and returns an error, if any.
func (*Writer) WriteLineSafe ¶
WriteLineSafe writes a string with a trailing newline to the underlying writer and returns an error, if any. WriteLineSafe also locks the writer for the duration of writing using a sync.Mutex.
func (*Writer) WriteString ¶
WriteString writes a string to the underlying writer and returns an error, if any.
Source Files ¶
- CloseWriters.go
- Closer.go
- ErrFunctionNotImplemented.go
- ErrReaderNotImplemented.go
- ErrWriterNotImplemented.go
- ExpandOpenAndStat.go
- FunctionWriteCloser.go
- Metadata.go
- ReadAllAndClose.go
- ReadFromFile.go
- ReadFromFilePath.go
- ReadFromResource.go
- Reader.go
- UploadS3Object.go
- WrapReader.go
- WrapWriter.go
- WriteAllAndClose.go
- WriteBuffers.go
- WriteFile.go
- WriteToFileSystem.go
- WriteToResource.go
- Writer.go
- grw.go