gs

package
v6.25.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 9, 2025 License: MIT Imports: 19 Imported by: 2

Documentation

Overview

Package gs - Google Cloud Storage VFS implementation.

Usage

Rely on github.com/c2fo/vfs/backend 

import(
    "github.com/c2fo/vfs/v6/backend"
    "github.com/c2fo/vfs/v6/backend/gs"
)

func UseFs() error {
    fs := backend.Backend(gs.Scheme)
    ...
}

Or call directly: 

import "github.com/c2fo/vfs/v6/backend/gs"

func DoSomething() {
    fs := gs.NewFileSystem()
    ...
}

gs can be augmented with the following implementation-specific methods. Backend returns vfs.FileSystem interface so it would have to be cast as gs.FileSystem to use the following: 

func DoSomething() {
    ...

    // cast if fs was created using backend.Backend().  Not necessary if created directly from gs.NewFileSystem().
    fs = fs.(gs.FileSystem)

    // to use your own "context"
    ctx := context.Background()
    fs = fs.WithContext(ctx)

    // to pass in client options
    fs = fs.WithOptions(
        gs.Options{
            CredentialFile: "/root/.gcloud/account.json",
            Scopes:         []string{"ScopeReadOnly"},
            //default scope is "ScopeFullControl"
        },
    )

    // to pass specific client, for instance no-auth client
    ctx := context.Background()
    client, _ := storage.NewClient(ctx, option.WithoutAuthentication())
    fs = fs.WithClient(client)
}

Authentication

Authentication, by default, occurs automatically when Client() is called. It looks for credentials in the following places, preferring the first location found:

  1. A JSON file whose path is specified by the GOOGLE_APPLICATION_CREDENTIALS environment variable
  2. A JSON file in a location known to the gcloud command-line tool. On Windows, this is %APPDATA%/gcloud/application_default_credentials.json. On other systems, $HOME/.config/gcloud/application_default_credentials.json.
  3. On Google App Engine it uses the appengine.AccessToken function.
  4. On Google Compute Engine and Google App Engine Managed VMs, it fetches credentials from the metadata server.

See https://cloud.google.com/docs/authentication/production for more auth info

See Also

See: https://github.com/googleapis/google-cloud-go/tree/master/storage

Index

Constants

View Source 
const Scheme = "gs"

Scheme defines the file system type.

Variables

This section is empty.

Functions

This section is empty.

Types

type BucketHandle 

type BucketHandle interface {
	Attrs(ctx context.Context) (*storage.BucketAttrs, error)
}

BucketHandle is an interface which contains a subset of the functions provided by storage.BucketHandler. Any function normally called directly by storage.BucketHandler should be added to this interface to allow for proper retry wrapping of the functions which call the GCS API.

type BucketHandleWrapper 

type BucketHandleWrapper interface {
	BucketHandle
	WrappedObjects(ctx context.Context, q *storage.Query) ObjectIteratorWrapper
}

BucketHandleWrapper is a unique, wrapped type which should mimic the behavior of BucketHandler, but with modified return types. Each function that returns a sub type that also should be wrapped should be added to this interface with the 'Wrapped' prefix.

type ClientCreator added in v6.13.0 

type ClientCreator interface {
	NewClient(ctx context.Context, opts ...option.ClientOption) (*storage.Client, error)
}

ClientCreator defines an interface for creating a new Google Cloud Storage client.

type Copier 

type Copier struct {
	Retry vfs.Retry
	// contains filtered or unexported fields
}

Copier implements the CopierWrapper interface.

func (*Copier) ContentType 

func (c *Copier) ContentType(val string)

ContentType is the MIME type of the object's content.

func (*Copier) Run 

func (c *Copier) Run(ctx context.Context) (*storage.ObjectAttrs, error)

Run performs the copy, wrapped in a retry

type CopierWrapper 

type CopierWrapper interface {
	Run(ctx context.Context) (*storage.ObjectAttrs, error)
	ContentType(string)
}

CopierWrapper is an interface which contains a subset of the functions provided by storage.Copier.

type File 

type File struct {
	// contains filtered or unexported fields
}

File implements vfs.File interface for GS fs.

func (*File) Close 

func (f *File) Close() error

Close commits any writes, either from the GCS writer stream or from a tempfile (in the case where Seek or Read are called after Write). It then cleans up any open resources and resets the file's state.

func (*File) CopyToFile 

func (f *File) CopyToFile(file vfs.File) (err error)

CopyToFile puts the contents of File into the target vfs.File passed in. Uses the GCS CopierFrom method if the target file is also on GCS, otherwise uses io.CopyBuffer. This method should be called on a closed file or a file with 0 cursor position to avoid errors.

func (*File) CopyToLocation 

func (f *File) CopyToLocation(location vfs.Location) (vfs.File, error)

CopyToLocation creates a copy of *File, using the file's current name as the new file's name at the given location. If the given location is also GCS, the GCS API for copying files will be utilized, otherwise, standard io.Copy will be done to the new file.

func (*File) Delete 

func (f *File) Delete(opts ...options.DeleteOption) error

Delete clears any local temp file, or write buffer from read/writes to the file, then makes a DeleteObject call to GCS for the file. If delete.AllVersions option is provided, DeleteObject call is made to GCS for each version of the file. Returns any error returned by the API.

func (*File) Exists 

func (f *File) Exists() (bool, error)

Exists returns a boolean of whether or not the object exists in GCS.

func (*File) LastModified 

func (f *File) LastModified() (*time.Time, error)

LastModified returns the 'Updated' property from the GCS attributes.

func (*File) Location 

func (f *File) Location() vfs.Location

Location returns a Location instance for the file's current location.

func (*File) MoveToFile 

func (f *File) MoveToFile(file vfs.File) error

MoveToFile puts the contents of File into the target vfs.File passed in using File.CopyToFile. If the copy succeeds, the source file is deleted. Any errors from the copy or delete are returned.

func (*File) MoveToLocation 

func (f *File) MoveToLocation(location vfs.Location) (vfs.File, error)

MoveToLocation works by first calling File.CopyToLocation(vfs.Location) then, if that succeeds, it deletes the original file, returning the new file. If the copy process fails the error is returned, and the Delete isn't called. If the call to Delete fails, the error and the file generated by the copy are both returned.

func (*File) Name 

func (f *File) Name() string

Name returns the file name.

func (*File) Path 

func (f *File) Path() string

Path returns full path with leading slash of the GCS file key.

func (*File) Read 

func (f *File) Read(p []byte) (n int, err error)

Read implements the standard for io.Reader.

func (*File) Seek 

func (f *File) Seek(offset int64, whence int) (int64, error)

Seek implements the standard for io.Seeker.

func (*File) Size 

func (f *File) Size() (uint64, error)

Size returns the 'Size' property from the GCS attributes.

func (*File) String 

func (f *File) String() string

String returns the file URI string.

func (*File) Touch 

func (f *File) Touch() error

Touch creates a zero-length file on the vfs.File if no File exists. Update File's last modified timestamp. Returns error if unable to touch File.

func (*File) URI 

func (f *File) URI() string

URI returns a full GCS URI string of the file.

func (*File) Write 

func (f *File) Write(data []byte) (int, error)

Write implements the standard for io.Writer. Note that writes are not committed to GCS until CLose() is called.

type FileSystem 

type FileSystem struct {
	// contains filtered or unexported fields
}

FileSystem implements vfs.FileSystem for the GCS file system.

func NewFileSystem 

func NewFileSystem() *FileSystem

NewFileSystem initializer for FileSystem struct accepts google cloud storage client and returns FileSystem or error.

func (*FileSystem) Client 

func (fs *FileSystem) Client() (*storage.Client, error)

Client returns the underlying google storage client, creating it, if necessary See Overview for authentication resolution

func (*FileSystem) Name 

func (fs *FileSystem) Name() string

Name returns "Google Cloud Storage"

func (*FileSystem) NewFile 

func (fs *FileSystem) NewFile(volume, name string, opts ...options.NewFileOption) (vfs.File, error)

NewFile function returns the gcs implementation of vfs.File.

func (*FileSystem) NewLocation 

func (fs *FileSystem) NewLocation(volume, name string) (loc vfs.Location, err error)

NewLocation function returns the GCS implementation of vfs.Location.

func (*FileSystem) Retry 

func (fs *FileSystem) Retry() vfs.Retry

Retry will return a retrier provided via options, or a no-op if none is provided.

func (*FileSystem) Scheme 

func (fs *FileSystem) Scheme() string

Scheme return "gs" as the initial part of a file URI ie: gs://

func (*FileSystem) WithClient 

func (fs *FileSystem) WithClient(client *storage.Client) *FileSystem

WithClient passes in a google storage client and returns the file system (chainable)

func (*FileSystem) WithContext 

func (fs *FileSystem) WithContext(ctx context.Context) *FileSystem

WithContext passes in user context and returns the file system (chainable)

func (*FileSystem) WithOptions 

func (fs *FileSystem) WithOptions(opts vfs.Options) *FileSystem

WithOptions sets options for client and returns the file system (chainable)

type Location 

type Location struct {
	// contains filtered or unexported fields
}

Location implements vfs.Location for gs fs.

func (*Location) ChangeDir 

func (l *Location) ChangeDir(relativePath string) error

ChangeDir changes the current location's path to the new, relative path.

func (*Location) DeleteFile 

func (l *Location) DeleteFile(fileName string, opts ...options.DeleteOption) error

DeleteFile deletes the file at the given path, relative to the current location.

func (*Location) Exists 

func (l *Location) Exists() (bool, error)

Exists returns whether the location exists or not. In the case of an error, false is returned.

func (*Location) FileSystem 

func (l *Location) FileSystem() vfs.FileSystem

FileSystem returns the GCS file system instance.

func (*Location) List 

func (l *Location) List() ([]string, error)

List returns a list of file name strings for the current location.

func (*Location) ListByPrefix 

func (l *Location) ListByPrefix(filenamePrefix string) ([]string, error)

ListByPrefix returns a slice of file base names and any error, if any List functions return only file basenames

func (*Location) ListByRegex 

func (l *Location) ListByRegex(regex *regexp.Regexp) ([]string, error)

ListByRegex returns a list of file names at the location which match the provided regular expression.

func (*Location) NewFile 

func (l *Location) NewFile(filePath string, opts ...options.NewFileOption) (vfs.File, error)

NewFile returns a new file instance at the given path, relative to the current location.

func (*Location) NewLocation 

func (l *Location) NewLocation(relativePath string) (vfs.Location, error)

NewLocation creates a new location instance relative to the current location's path.

func (*Location) Path 

func (l *Location) Path() string

Path returns the path of the file at the current location, starting with a leading '/'

func (*Location) String 

func (l *Location) String() string

String returns the full URI of the location.

func (*Location) URI 

func (l *Location) URI() string

URI returns a URI string for the GCS location.

func (*Location) Volume 

func (l *Location) Volume() string

Volume returns the GCS bucket name.

type ObjectHandleCopier 

type ObjectHandleCopier interface {
	ObjectHandleWrapper
	WrappedCopierFrom(src *storage.ObjectHandle) CopierWrapper
	ObjectHandle() *storage.ObjectHandle
}

ObjectHandleCopier is a unique, wrapped type which should mimic the behavior of ObjectHandler, but with modified return types. Each function that returns a sub type that also should be wrapped should be added to this interface with the 'Wrapped' prefix.

type ObjectHandleWrapper 

type ObjectHandleWrapper interface {
	NewWriter(ctx context.Context) *storage.Writer
	NewReader(ctx context.Context) (*storage.Reader, error)
	NewRangeReader(ctx context.Context, offset, length int64) (*storage.Reader, error)
	Attrs(ctx context.Context) (*storage.ObjectAttrs, error)
	Delete(ctx context.Context) error
	Update(ctx context.Context, attrs storage.ObjectAttrsToUpdate) (*storage.ObjectAttrs, error)
}

ObjectHandleWrapper is an interface which contains a subset of the functions provided by storage.ObjectHandler. Any function normally called directly by storage.ObjectHandler should be added to this interface to allow for proper retry wrapping of the functions which call the GCS API.

type ObjectIteratorWrapper 

type ObjectIteratorWrapper interface {
	Next() (*storage.ObjectAttrs, error)
}

ObjectIteratorWrapper is an interface which contains a subset of the functions provided by storage.ObjectIterator.

type Options 

type Options struct {
	APIKey         string   `json:"apiKey,omitempty"`
	CredentialFile string   `json:"credentialFilePath,omitempty"`
	Endpoint       string   `json:"endpoint,omitempty"`
	Scopes         []string `json:"WithoutAuthentication,omitempty"`
	Retry          vfs.Retry
	FileBufferSize int // Buffer Size In Bytes Used with utils.TouchCopyBuffered
}

Options holds Google Cloud Storage -specific options. Currently only client options are used.

type RetryBucketHandler 

type RetryBucketHandler struct {
	Retry vfs.Retry
	// contains filtered or unexported fields
}

RetryBucketHandler implements the BucketHandle interface

func (*RetryBucketHandler) Attrs 

func (r *RetryBucketHandler) Attrs(ctx context.Context) (*storage.BucketAttrs, error)

Attrs accepts a context and returns bucket attrs wrapped in a retry

func (*RetryBucketHandler) WrappedObjects 

func (r *RetryBucketHandler) WrappedObjects(ctx context.Context, q *storage.Query) ObjectIteratorWrapper

WrappedObjects returns an iterator over the objects in the bucket that match the Query q, all wrapped in a retry. If q is nil, no filtering is done.

type RetryObjectHandler 

type RetryObjectHandler struct {
	Retry vfs.Retry
	// contains filtered or unexported fields
}

RetryObjectHandler implements the ObjectHandleCopier interface (which also is composed with ObjectHandleWrapper)

func (*RetryObjectHandler) Attrs 

func (r *RetryObjectHandler) Attrs(ctx context.Context) (*storage.ObjectAttrs, error)

Attrs represents the metadata for a Google Cloud Storage (GCS) object, wrapped in a retry.

func (*RetryObjectHandler) Delete 

func (r *RetryObjectHandler) Delete(ctx context.Context) error

Delete deletes the single specified object, wrapped in a retry.

func (*RetryObjectHandler) NewRangeReader added in v6.13.0 

func (r *RetryObjectHandler) NewRangeReader(ctx context.Context, offset, length int64) (*storage.Reader, error)

NewRangeReader creates a new Reader to read the contents of the object starting at an offset, wrapped in a retry. ErrObjectNotExist will be returned if the object is not found.

func (*RetryObjectHandler) NewReader 

func (r *RetryObjectHandler) NewReader(ctx context.Context) (*storage.Reader, error)

NewReader creates a new Reader to read the contents of the object, wrapped in a retry. ErrObjectNotExist will be returned if the object is not found.

func (*RetryObjectHandler) NewWriter 

func (r *RetryObjectHandler) NewWriter(ctx context.Context) *storage.Writer

NewWriter returns a storage Writer that writes to the GCS object associated with this ObjectHandle, wrapped in a retry.

func (*RetryObjectHandler) ObjectHandle 

func (r *RetryObjectHandler) ObjectHandle() *storage.ObjectHandle

ObjectHandle returns the underlying GCS object handle.

func (*RetryObjectHandler) Update 

func (r *RetryObjectHandler) Update(ctx context.Context, attrs storage.ObjectAttrsToUpdate) (*storage.ObjectAttrs, error)

Update will update the metadata for a Google Cloud Storage (GCS) object, wrapped in a retry. Here we can re-use the objectAttributeRetry() func because it has the same signature.

func (*RetryObjectHandler) WrappedCopierFrom 

func (r *RetryObjectHandler) WrappedCopierFrom(src *storage.ObjectHandle) CopierWrapper

WrappedCopierFrom creates a Copier that can copy src to dst, wrapped in a retry. You can immediately call Run on the returned Copier, or you can configure it first.

type RetryObjectIterator 

type RetryObjectIterator struct {
	Retry vfs.Retry
	// contains filtered or unexported fields
}

RetryObjectIterator implements the ObjectIteratorWrapper interface

func (*RetryObjectIterator) Next 

func (r *RetryObjectIterator) Next() (*storage.ObjectAttrs, error)

Next returns the next result, wrapped in retry. Its second return value is iterator.Done if there are no more results. Once Next returns iterator.Done, all subsequent calls will return iterator.Done.

If Query.Delimiter is non-empty, some of the ObjectAttrs returned by Next will have a non-empty Prefix field, and a zero value for all other fields. These represent prefixes.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.