blobio

Documentation

Overview

Package blobio abstracts the scalable reading and writing of file blobs against conformant services.

Uses gocloud.dev/blob for abstracting the service providers.

Index

• func MatchAll(s *beam.Scope, col beam.PCol[beam.KV[string, string]], opts ...MatchOptionFn) beam.PCol[BlobMetadata]
• func MatchFiles(s *beam.Scope, bucket, glob string, opts ...MatchOptionFn) beam.PCol[BlobMetadata]
• func ReadMatches(s *beam.Scope, col beam.PCol[BlobMetadata], opts ...ReadOptionFn) beam.PCol[ReadableBlob]
• type BlobMetadata
• type MatchOptionFn
  • func MatchEmptyAllow() MatchOptionFn
  • func MatchEmptyAllowIfWildcard() MatchOptionFn
  • func MatchEmptyDisallow() MatchOptionFn
• type ReadOptionFn
  • func ReadAutoCompression() ReadOptionFn
  • func ReadDirectoryDisallow() ReadOptionFn
  • func ReadDirectorySkip() ReadOptionFn
  • func ReadGzip() ReadOptionFn
  • func ReadUncompressed() ReadOptionFn
• type ReadableBlob
  • func (f ReadableBlob) Open(ctx context.Context) (io.ReadSeekCloser, error)
  • func (f ReadableBlob) Read(ctx context.Context) (data []byte, err error)
  • func (f ReadableBlob) ReadString(ctx context.Context) (string, error)

Functions

func MatchAll

func MatchAll(s *beam.Scope, col beam.PCol[beam.KV[string, string]], opts ...MatchOptionFn) beam.PCol[BlobMetadata]

MatchAll finds all files matching the glob patterns given by the incoming PCollection<string> and returns a beam.Output[BlobMetadata] of the matching files. MatchAll accepts a variadic number of MatchOptionFn that can be used to configure the treatment of empty matches. By default, empty matches are allowed if the pattern contains a wildcard.

func MatchFiles

func MatchFiles(s *beam.Scope, bucket, glob string, opts ...MatchOptionFn) beam.PCol[BlobMetadata]

MatchFiles finds all files matching the glob pattern and returns a PCollection<FileMetadata> of the matching files. MatchFiles accepts a variadic number of MatchOptionFn that can be used to configure the treatment of empty matches. By default, empty matches are allowed if the pattern contains a wildcard.

func ReadMatches

func ReadMatches(s *beam.Scope, col beam.PCol[BlobMetadata], opts ...ReadOptionFn) beam.PCol[ReadableBlob]

ReadMatches accepts the result of MatchFiles, MatchAll or MatchContinuously as a beam.Output[BlobMetadata] and converts it to a beam.Output[ReadableBlob]. The ReadableBlob can be used to retrieve blob metadata, open the blob for reading or read the entire blob into memory. ReadMatches accepts a variadic number of ReadOptionFn that can be used to configure the compression type of the blobs and treatment of directories. By default, the compression type is determined by the blob extension and directories are skipped.

Types

type BlobMetadata

type BlobMetadata struct {
	Bucket string
	Key    string

	Size         int64
	LastModified time.Time
}

BlobMetadata contains metadata about a file, namely its path, size in bytes and last modified time.

type MatchOptionFn

type MatchOptionFn func(*matchOption)

MatchOptionFn is a function that can be passed to MatchFiles or MatchAll to configure options for matching files.

func MatchEmptyAllow

func MatchEmptyAllow() MatchOptionFn

MatchEmptyAllow specifies that empty matches are allowed.

func MatchEmptyAllowIfWildcard

func MatchEmptyAllowIfWildcard() MatchOptionFn

MatchEmptyAllowIfWildcard specifies that empty matches are allowed if the pattern contains a wildcard.

func MatchEmptyDisallow

func MatchEmptyDisallow() MatchOptionFn

MatchEmptyDisallow specifies that empty matches are not allowed.

type ReadOptionFn

type ReadOptionFn func(*readOption)

ReadOptionFn is a function that can be passed to ReadMatches to configure options for reading files.

func ReadAutoCompression

func ReadAutoCompression() ReadOptionFn

ReadAutoCompression specifies that the compression type of files should be auto-detected.

func ReadDirectoryDisallow

func ReadDirectoryDisallow() ReadOptionFn

ReadDirectoryDisallow specifies that directories are not allowed.

func ReadDirectorySkip

func ReadDirectorySkip() ReadOptionFn

ReadDirectorySkip specifies that directories are skipped.

func ReadGzip

func ReadGzip() ReadOptionFn

ReadGzip specifies that files have been compressed using gzip.

func ReadUncompressed

func ReadUncompressed() ReadOptionFn

ReadUncompressed specifies that files have not been compressed.

type ReadableBlob

type ReadableBlob struct {
	Metadata    BlobMetadata
	Compression compressionType
}

ReadableBlob is a wrapper around a BlobMetadata and compressionType that can be used to obtain a file descriptor or read the file's contents.

func (ReadableBlob) Open

func (f ReadableBlob) Open(ctx context.Context) (io.ReadSeekCloser, error)

Open opens the file for reading. The compression type is determined by the Compression field of the ReadableFile. If Compression is compressionAuto, the compression type is auto-detected from the file extension. It is the caller's responsibility to close the returned reader.

func (ReadableBlob) Read

func (f ReadableBlob) Read(ctx context.Context) (data []byte, err error)

Read reads the entire file into memory and returns the contents.

func (ReadableBlob) ReadString

func (f ReadableBlob) ReadString(ctx context.Context) (string, error)

ReadString reads the entire file into memory and returns the contents as a string.