gs

package
v0.0.0-...-d3da680 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 4, 2024 License: Apache-2.0 Imports: 15 Imported by: 28

Documentation

Overview

Package gs implements a versatile Google Storage client on top of the standard Google Storage Go API. It adds:

  • The ability to read from specific byte offsets.
  • Exponential backoff retries on transient errors.
  • Logging
  • The ability to easily stub a Google Storage interface.

Index

Constants

This section is empty.

Variables

View Source
var (
	// ReadWriteScopes is the set of scopes needed for read/write Google Storage
	// access.
	ReadWriteScopes = []string{gs.ScopeReadWrite}

	// ReadOnlyScopes is the set of scopes needed for read/write Google Storage
	// read-only access.
	ReadOnlyScopes = []string{gs.ScopeReadOnly}
)

Functions

This section is empty.

Types

type Client

type Client interface {
	// Close closes the connection to Google Storage.
	Close() error

	// Attrs retrieves Object attributes for a given path.
	Attrs(p Path) (*gs.ObjectAttrs, error)

	// Objects retrieves all object attributes for a given path.
	Objects(p Path) ([]*gs.ObjectAttrs, error)

	// SignedURL generates a google storage signed url for a given path.
	SignedURL(p Path, opts *gs.SignedURLOptions) (string, error)

	// NewReader instantiates a new Reader instance for the named bucket/path.
	//
	// The supplied offset must be >= 0, or else this function will panic.
	//
	// If the supplied length is <0, no upper byte bound will be set.
	//
	// The reader inherits the context of the client.
	NewReader(p Path, offset, length int64) (io.ReadCloser, error)

	// NewWriter instantiates a new Writer instance for the named bucket/path.
	//
	// The writer inherits the context of the client.
	NewWriter(p Path) (Writer, error)

	// Delete deletes the object at the specified path.
	//
	// If the object does not exist, it is considered a success.
	Delete(p Path) error

	// Rename renames an object from one path to another.
	//
	// NOTE: The object should be removed from its original path, but current
	// implementation uses two operations (Copy + Delete), so it may
	// occasionally fail.
	Rename(src, dst Path) error
}

Client abstracts functionality to connect with and use Google Storage from the actual Google Storage client.

Non-production implementations are used primarily for testing.

func NewProdClient

func NewProdClient(ctx context.Context, rt http.RoundTripper) (Client, error)

NewProdClient creates a new Client instance that uses production Cloud Storage.

The supplied RoundTripper will be used to make connections. If nil, the default HTTP client will be used.

The given context is used for all logging and for overall deadline and cancellation. The client is no longer usable when this context is canceled.

type LimitedClient

type LimitedClient struct {
	// Base is the base Client instance.
	Client

	// MaxReadBytes, if >0, is the maximum number of bytes that can be read at a
	// time. If more bytes are required, additional read calls will be made.
	MaxReadBytes int64
}

LimitedClient wraps a base Client, allowing additional limits to be applied to its calls.

func (*LimitedClient) NewReader

func (lc *LimitedClient) NewReader(p Path, offset, length int64) (io.ReadCloser, error)

NewReader implements Client.

type ObjectStream

type ObjectStream struct {
	// contains filtered or unexported fields
}

ObjectStream lets you stream a GCS object as lines. This is useful when you are trying to stream something like a logs file that is being updated as you are reading it. It assumes the object always has complete lines.

ObjectStreamOptions has some guidance on how to use it.

TODO(b/332706067): Deprecate this.

func NewObjectStream

func NewObjectStream(ctx context.Context, options *ObjectStreamOptions) *ObjectStream

NewObjectStream creates an ObjectStream and starts a background routine that starts streaming the file.

func (*ObjectStream) Close

func (os *ObjectStream) Close() error

Close stops the background routine.

type ObjectStreamOptions

type ObjectStreamOptions struct {
	Client        Client
	Path          Path
	PollFrequency time.Duration
	LinesC        chan<- string
	ErrorsC       chan<- error
}

ObjectStreamOptions configure the behavior of an ObjectStream.

PollFrequency controls how frequently GCS object at Path is checked for updates once you reach EOF.

LinesC and ErrorsC are used to return lines of object at Path and any errors encountered. If nothing reads these channels sending will block.

type Path

type Path string

Path is a Google Storage path. A full path consists of a Google storage bucket and a series of path components.

An example of a Path is:

gs://test-bucket/path/to/thing.txt

func MakePath

func MakePath(bucket string, parts ...string) Path

MakePath constructs a Google Storage path from optional bucket and filename components.

Trailing forward slashes will be removed from the bucket name, if present.

func (Path) Bucket

func (p Path) Bucket() string

Bucket returns the Google Storage bucket component of the Path. If there is no bucket, an empty string will be returned.

func (Path) Concat

func (p Path) Concat(v string, parts ...string) Path

Concat concatenates a filename component to the end of Path.

Multiple components may be specified. In this case, each will be added as a "/"-delimited component, and will have any present trailing slashes stripped.

func (Path) Filename

func (p Path) Filename() string

Filename returns the filename component of the Path. If there is no filename component, an empty string will be returned.

Leading and trailing slashes will be truncated.

func (Path) IsFullPath

func (p Path) IsFullPath() bool

IsFullPath returns true if the Path contains both a bucket and file name.

func (Path) Split

func (p Path) Split() (bucket string, filename string)

Split returns the bucket and filename components of the Path.

If a bucket is not defined (doesn't begin with "gs://"), the remainder will be considered to be the filename component. If a filename is not defined, an empty string will be returned.

type Writer

type Writer interface {
	io.WriteCloser

	// Count returns the number of bytes written by the object.
	Count() int64
}

Writer is an augmented io.WriteCloser instance.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL