Documentation
¶
Overview ¶
Package gs implements a versatile Google Storage client on top of the standard Google Storage Go API. It adds:
- The ability to read from specific byte offsets.
- Exponential backoff retries on transient errors.
- Logging
- The ability to easily stub a Google Storage interface.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ReadWriteScopes is the set of scopes needed for read/write Google Storage // access. ReadWriteScopes = []string{gs.ScopeReadWrite} // ReadOnlyScopes is the set of scopes needed for read/write Google Storage // read-only access. ReadOnlyScopes = []string{gs.ScopeReadOnly} )
Functions ¶
This section is empty.
Types ¶
type Client ¶
type Client interface {
// Close closes the connection to Google Storage.
Close() error
// Attrs retrieves Object attributes for a given path.
Attrs(p Path) (*gs.ObjectAttrs, error)
// Objects retrieves all object attributes for a given path.
Objects(p Path) ([]*gs.ObjectAttrs, error)
// SignedURL generates a google storage signed url for a given path.
SignedURL(p Path, opts *gs.SignedURLOptions) (string, error)
// NewReader instantiates a new Reader instance for the named bucket/path.
//
// The supplied offset must be >= 0, or else this function will panic.
//
// If the supplied length is <0, no upper byte bound will be set.
//
// The reader inherits the context of the client.
NewReader(p Path, offset, length int64) (io.ReadCloser, error)
// NewWriter instantiates a new Writer instance for the named bucket/path.
//
// The writer inherits the context of the client.
NewWriter(p Path) (Writer, error)
// Delete deletes the object at the specified path.
//
// If the object does not exist, it is considered a success.
Delete(p Path) error
// Rename renames an object from one path to another.
//
// NOTE: The object should be removed from its original path, but current
// implementation uses two operations (Copy + Delete), so it may
// occasionally fail.
Rename(src, dst Path) error
}
Client abstracts functionality to connect with and use Google Storage from the actual Google Storage client.
Non-production implementations are used primarily for testing.
func NewProdClient ¶
NewProdClient creates a new Client instance that uses production Cloud Storage.
The supplied RoundTripper will be used to make connections. If nil, the default HTTP client will be used.
The given context is used for all logging and for overall deadline and cancellation. The client is no longer usable when this context is canceled.
type LimitedClient ¶
type LimitedClient struct {
// Base is the base Client instance.
Client
// MaxReadBytes, if >0, is the maximum number of bytes that can be read at a
// time. If more bytes are required, additional read calls will be made.
MaxReadBytes int64
}
LimitedClient wraps a base Client, allowing additional limits to be applied to its calls.
func (*LimitedClient) NewReader ¶
func (lc *LimitedClient) NewReader(p Path, offset, length int64) (io.ReadCloser, error)
NewReader implements Client.
type ObjectStream ¶
type ObjectStream struct {
// contains filtered or unexported fields
}
ObjectStream lets you stream a GCS object as lines. This is useful when you are trying to stream something like a logs file that is being updated as you are reading it. It assumes the object always has complete lines.
ObjectStreamOptions has some guidance on how to use it.
TODO(b/332706067): Deprecate this.
func NewObjectStream ¶
func NewObjectStream(ctx context.Context, options *ObjectStreamOptions) *ObjectStream
NewObjectStream creates an ObjectStream and starts a background routine that starts streaming the file.
func (*ObjectStream) Close ¶
func (os *ObjectStream) Close() error
Close stops the background routine.
type ObjectStreamOptions ¶
type ObjectStreamOptions struct {
Client Client
Path Path
PollFrequency time.Duration
LinesC chan<- string
ErrorsC chan<- error
}
ObjectStreamOptions configure the behavior of an ObjectStream.
PollFrequency controls how frequently GCS object at Path is checked for updates once you reach EOF.
LinesC and ErrorsC are used to return lines of object at Path and any errors encountered. If nothing reads these channels sending will block.
type Path ¶
type Path string
Path is a Google Storage path. A full path consists of a Google storage bucket and a series of path components.
An example of a Path is:
gs://test-bucket/path/to/thing.txt
func MakePath ¶
MakePath constructs a Google Storage path from optional bucket and filename components.
Trailing forward slashes will be removed from the bucket name, if present.
func (Path) Bucket ¶
Bucket returns the Google Storage bucket component of the Path. If there is no bucket, an empty string will be returned.
func (Path) Concat ¶
Concat concatenates a filename component to the end of Path.
Multiple components may be specified. In this case, each will be added as a "/"-delimited component, and will have any present trailing slashes stripped.
func (Path) Filename ¶
Filename returns the filename component of the Path. If there is no filename component, an empty string will be returned.
Leading and trailing slashes will be truncated.
func (Path) IsFullPath ¶
IsFullPath returns true if the Path contains both a bucket and file name.
type Writer ¶
type Writer interface {
io.WriteCloser
// Count returns the number of bytes written by the object.
Count() int64
}
Writer is an augmented io.WriteCloser instance.
Source Files