gcsstore

package
v2.0.0-...-a21236f Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 14, 2023 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package gcsstore provides a Google cloud storage based backend.

GCSStore is a storage backend that uses the GCSAPI interface in order to store uploads on GCS. Uploads will be represented by two files in GCS; the data file will be stored as an extensionless object uid and the JSON info file will stored as uid.info. In order to store uploads on GCS, make sure to specify the appropriate Google service account file path in the GCS_SERVICE_ACCOUNT_FILE environment variable. Also make sure that this service account file has the "https://www.googleapis.com/auth/devstorage.read_write" scope enabled so you can read and write data to the storage buckets associated with the service account file.

Index

Constants

View Source 
const COMPOSE_RETRIES = 3
View Source 
const CONCURRENT_SIZE_REQUESTS = 32
View Source 
const MAX_OBJECT_COMPOSITION = 32

MAX_OBJECT_COMPOSITION specifies the maximum number of objects that can combined in a compose operation. GCloud storage's limit is 32.

Variables

This section is empty.

Functions

This section is empty.

Types

type GCSAPI 

type GCSAPI interface {
	ReadObject(ctx context.Context, params GCSObjectParams) (GCSReader, error)
	GetObjectSize(ctx context.Context, params GCSObjectParams) (int64, error)
	SetObjectMetadata(ctx context.Context, params GCSObjectParams, metadata map[string]string) error
	DeleteObject(ctx context.Context, params GCSObjectParams) error
	DeleteObjectsWithFilter(ctx context.Context, params GCSFilterParams) error
	WriteObject(ctx context.Context, params GCSObjectParams, r io.Reader) (int64, error)
	ComposeObjects(ctx context.Context, params GCSComposeParams) error
	FilterObjects(ctx context.Context, params GCSFilterParams) ([]string, error)
}

GCSAPI is an interface composed of all the necessary GCS operations that are required to enable the tus protocol to work with Google's cloud storage.

type GCSComposeParams 

type GCSComposeParams struct {
	// Bucket specifies the GCS bucket which the composed objects will be stored in.
	Bucket string

	// Sources is a list of the object IDs that are going to be composed.
	Sources []string

	// Destination specifies the desired ID of the composed object.
	Destination string
}

type GCSFilterParams 

type GCSFilterParams struct {
	// Bucket specifies the GCS bucket of which the objects you want to filter reside in.
	Bucket string

	// Prefix specifies the prefix of which you want to filter object names with.
	Prefix string
}

type GCSObjectParams 

type GCSObjectParams struct {
	// Bucket specifies the GCS bucket that the object resides in.
	Bucket string

	// ID specifies the ID of the GCS object.
	ID string
}

type GCSReader 

type GCSReader interface {
	Close() error
	ContentType() string
	Read(p []byte) (int, error)
	Remain() int64
	Size() int64
}

GCSReader implements cloud.google.com/go/storage.Reader. It is used to read Google Cloud storage objects.

type GCSService 

type GCSService struct {
	Client *storage.Client
}

GCSService holds the cloud.google.com/go/storage client as well as its associated context. Closures are used as minimal wrappers around the Google Cloud Storage API, since the Storage API cannot be mocked. The usage of these closures allow them to be redefined in the testing package, allowing test to be run against this file.

func NewGCSService 

func NewGCSService(filename string) (*GCSService, error)

NewGCSService returns a GCSService object given a GCloud service account file path.

func (*GCSService) ComposeFrom 

func (service *GCSService) ComposeFrom(ctx context.Context, objSrcs []*storage.ObjectHandle, dstParams GCSObjectParams, contentType string) (uint32, error)

ComposeFrom composes multiple object types together,

func (*GCSService) ComposeObjects 

func (service *GCSService) ComposeObjects(ctx context.Context, params GCSComposeParams) error

ComposeObjects composes multiple GCS objects in to a single object. Since GCS limits composition to a max of 32 objects, additional logic has been added to chunk objects in to groups of 32 and then recursively compose those objects together.

func (*GCSService) DeleteObject 

func (service *GCSService) DeleteObject(ctx context.Context, params GCSObjectParams) error

DeleteObject deletes the object defined by GCSObjectParams

func (*GCSService) DeleteObjectsWithFilter 

func (service *GCSService) DeleteObjectsWithFilter(ctx context.Context, params GCSFilterParams) error

DeleteObjectWithPrefix will delete objects who match the provided filter parameters.

func (*GCSService) FilterObjects 

func (service *GCSService) FilterObjects(ctx context.Context, params GCSFilterParams) ([]string, error)

FilterObjects returns a list of GCS object IDs that match the passed GCSFilterParams. It expects GCS objects to be of the format uid_[chunk_idx] where chunk_idx is zero based. The format uid_tmp_[recursion_lvl]_[chunk_idx] can also be used to specify objects that have been composed in a recursive fashion. These different formats are used to ensure that objects are composed in the correct order.

func (*GCSService) GetObjectAttrs 

func (service *GCSService) GetObjectAttrs(ctx context.Context, params GCSObjectParams) (*storage.ObjectAttrs, error)

GetObjectAttrs returns the associated attributes of a GCS object. See: https://pkg.go.dev/cloud.google.com/go/storage#ObjectAttrs

func (*GCSService) GetObjectSize 

func (service *GCSService) GetObjectSize(ctx context.Context, params GCSObjectParams) (int64, error)

GetObjectSize returns the byte length of the specified GCS object.

func (*GCSService) ReadObject 

func (service *GCSService) ReadObject(ctx context.Context, params GCSObjectParams) (GCSReader, error)

ReadObject reads a GCSObjectParams, returning a GCSReader object if successful, and an error otherwise

func (*GCSService) SetObjectMetadata 

func (service *GCSService) SetObjectMetadata(ctx context.Context, params GCSObjectParams, metadata map[string]string) error

SetObjectMetadata reads a GCSObjectParams and a map of metadata, returning a nil on success and an error otherwise

func (*GCSService) WriteObject 

func (service *GCSService) WriteObject(ctx context.Context, params GCSObjectParams, r io.Reader) (int64, error)

Write object writes the file set out by the GCSObjectParams

type GCSStore 

type GCSStore struct {
	// Specifies the GCS bucket that uploads will be stored in
	Bucket string

	// ObjectPrefix is prepended to the name of each GCS object that is created.
	// It can be used to create a pseudo-directory structure in the bucket,
	// e.g. "path/to/my/uploads".
	ObjectPrefix string

	// Service specifies an interface used to communicate with the Google
	// cloud storage backend. Implementation can be seen in gcsservice file.
	Service GCSAPI
}

See the handler.DataStore interface for documentation about the different methods.

func New 

func New(bucket string, service GCSAPI) GCSStore

New constructs a new GCS storage backend using the supplied GCS bucket name and service object.

func (GCSStore) AsTerminatableUpload 

func (store GCSStore) AsTerminatableUpload(upload handler.Upload) handler.TerminatableUpload

func (GCSStore) GetUpload 

func (store GCSStore) GetUpload(ctx context.Context, id string) (handler.Upload, error)

func (GCSStore) NewUpload 

func (store GCSStore) NewUpload(ctx context.Context, info handler.FileInfo) (handler.Upload, error)

func (GCSStore) UseIn 

func (store GCSStore) UseIn(composer *handler.StoreComposer)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.