Documentation
¶
Index ¶
- Variables
- type ChunkUpload
- type Client
- func (c *Client) Close() error
- func (c *Client) Delete(filepath string) error
- func (c *Client) Download(filepath string, w io.WriterAt, p *data.DownloadParams) error
- func (c *Client) DownloadRange(filepath string, w io.WriterAt, start, finish int) error
- func (c *Client) Exists(filepath string) (bool, error)
- func (c *Client) Get(filepath string) (io.ReadCloser, error)
- func (c *Client) GetChunkSize() int
- func (c *Client) GetIncompleteUpload(filepath string) data.Upload
- func (c *Client) Head(filepath string) (*data.HeadInfo, error)
- func (c *Client) MD5(filepath string) ([]byte, error)
- func (c *Client) Objects() (data.ObjectIterator, func())
- func (c *Client) Put(filepath string, r io.Reader) error
- func (c *Client) PutSignedUrl(filepath string, tm time.Duration, cfg *data.SignedUrlConfig) (string, error)
- func (c *Client) PutWithStatus(filepath string, filesize int, r io.Reader) chan data.UploadStatus
- func (c *Client) ResumePutWithStatus(filepath string, r io.Reader) (chan data.UploadStatus, error)
- func (c *Client) SetMimeType(filepath, mimeType string) error
- func (c *Client) SetTimeout(tm time.Duration)
- func (c *Client) SignedUrl(filepath string, tm time.Duration, cfg *data.SignedUrlConfig) (string, error)
- func (c *Client) Stream(filepath string, w io.Writer) error
- func (c *Client) Upload(filepath string, filesize int, r io.Reader) error
Constants ¶
This section is empty.
Variables ¶
var (
ErrTimeout = errors.New("timeout")
)
Functions ¶
This section is empty.
Types ¶
type ChunkUpload ¶
type ChunkUpload struct {
// Number of total upload bytes. This is to provide a progress message giving them number of uploaded bytes.
UploadedBytes int
// Notifier to pass information back to the user.
Notifier chan data.UploadStatus
// Total file size
FileSize int
// contains filtered or unexported fields
}
ChunkUpload satisfies the data.Upload interface for GCS chunked uploads.
func (*ChunkUpload) Copy ¶
func (u *ChunkUpload) Copy(r io.Reader) (bool, error)
Copy writes the contents fo the reader into the chunk upload. It also handles all error and success notifications.
func (*ChunkUpload) GetFilepath ¶
func (u *ChunkUpload) GetFilepath() string
GetFilepath returns the filepath. Required to implement Upload interface.
func (*ChunkUpload) GetUploaded ¶
func (u *ChunkUpload) GetUploaded() int
GetUploaded returns the number of bytes already uplaoded. Required to implement Upload interface.
func (*ChunkUpload) IsComplete ¶
func (u *ChunkUpload) IsComplete() bool
func (*ChunkUpload) UpdateUploadedBytes ¶
func (u *ChunkUpload) UpdateUploadedBytes(newPartBytes int)
type Client ¶
type Client struct {
// The actual storage client for GCS
*storage.Client
// A quick way to access the bucket directly.
Bucket *storage.BucketHandle
// Timeout for each item.
Timeout time.Duration
// contains filtered or unexported fields
}
Client wraps google's *storage.Client to implement: - data.Service - data.HeadService - data.ChunkService - data.SignedUrlService - data.MimeTypeService
Note that it doesn't use data.ChunkManager because the default GCS client is already chunked. Additional chunking would only cause an extra useless goroutine to be run.
Ensure that proper authentication is provided. Note that, if SignedUrl is to be used, that `google auth application-default login` is *not* used as the credential source as it is not adequate.
Instead, use the GOOGLE_APPLICATION_CREDENTIALS environment variable or attach credentials (e.g., via passing in options.WithCredentialsJSON into the New function).
func New ¶
func New(bucket string, opts ...option.ClientOption) (*Client, error)
New generates a new client. option.ClientOption is not necessary unless special credentials are required in order to authenticate against GCS. For example, if the target machine is running on GCP or has the gcloud CLI setup with default authentication credentials, opts is not required.
If nil is provided for context, context.Background is used by default.
func (*Client) Download ¶
Download is supposed to download a file in parallel (by splitting it into different parts to download). However, GCS doesn't seem to have this functionality built in. Thus, we ignore it.
func (*Client) DownloadRange ¶
DownloadRange allows for a part of the file to be downloaded. Dictate the [start, finish) of the download, and the result will be written into io.WriterAt. For example, if start=0 and end=5, bytes 0...4 should be writen.
func (*Client) Get ¶
func (c *Client) Get(filepath string) (io.ReadCloser, error)
Get returns a reader that can be used to retrieve a file.
func (*Client) GetChunkSize ¶
GetChunkSize returns the ChunkSize so that operators know how much data to provide.
func (*Client) GetIncompleteUpload ¶
GetIncompleteUpload returns an incomplete upload based on the filepath.
func (*Client) Objects ¶
func (c *Client) Objects() (data.ObjectIterator, func())
Objects should return an iterator for all objects in a bucket.
func (*Client) PutSignedUrl ¶
func (c *Client) PutSignedUrl(filepath string, tm time.Duration, cfg *data.SignedUrlConfig) (string, error)
PutSignedUrl creates a SignedURL for uploading files directly to the configured GCS bucket. Unfortunately, content-type cannot be inferred from the data.SignedURLConfig object since it requires adding a header into the Signed URL request that needs to be replicated by the one calling the request.
This is burdensome for any application requiring use of this signed URL. Instead, the application can simply include it in its ContentType parameter and signing should still be successful.
func (*Client) PutWithStatus ¶
PutWithStatus tells GCS to start the upload. It adds the upload to the upload registry and proceeds to send data from the provided io.Reader.
func (*Client) ResumePutWithStatus ¶
ResumePutWithStatus tells CSV to add more data to the upload. If the upload cannot be found on the upload registry, it returns data.ErrUploadNotFound.
func (*Client) SetMimeType ¶
SetMimeType sets the MimeType/ContentType with the filepath that has been uploaded.
func (*Client) SetTimeout ¶
func (*Client) SignedUrl ¶
func (c *Client) SignedUrl(filepath string, tm time.Duration, cfg *data.SignedUrlConfig) (string, error)
SignedUrl is for the Client to implement the SignedUrlService. SignedUrlService allows a service to return a signed URL for a filepath.
For the implementation below, the GoogleAccessID is required. However, there is a chance that the environment does *not* provide this GoogleAccessID. For example, a system using `google auth application-default login` while looking similar, is *not* a valid credential to use as it does not contain the required `client_email` parameter.
Ensure (in testing and on the server) that proper credentials are provided when creating the client. https://cloud.google.com/docs/authentication/application-default-credentials
Source Files