Documentation
¶
Overview ¶
Package gs implement Google Storage API wrapper used by CIPD backend.
We don't use "cloud.google.com/go/storage" because it doesn't expose stuff we need (like resumable uploads and ReaderAt implementation), but instead adds a ton of stuff we don't need.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func SplitPath ¶
SplitPath given "/a/b/c" returns ("a", "b/c") or panics.
Use ValidatePath for prior validation if you are concerned.
func StatusCode ¶
StatusCode returns HTTP status code embedded inside the annotated error.
Returns http.StatusOK if err is nil and 0 if the error doesn't have a status code.
func StatusCodeTag ¶
StatusCodeTag can be used to attach HTTP status code to the error.
This code will be available via StatusCode(err) function.
func ValidatePath ¶
ValidatePath returns an error if p doesn't look like "/bucket/path".
Additionally it verifies p is using only printable ASCII characters, since all Google Storage paths used by CIPD are ASCII (assembled from constants fetched from validated configs and hex digests, there are no user-supplied path elements). We want to assert no fancy unicode characters sneak in.
Types ¶
type GoogleStorage ¶
type GoogleStorage interface {
// Exists checks whether given Google Storage file exists.
Exists(ctx context.Context, path string) (exists bool, err error)
// Size returns the size in bytes of the given Google Storage file.
Size(ctx context.Context, path string) (size uint64, exists bool, err error)
// Copy copies a file at 'src' to 'dst'.
//
// Applies ifSourceGenerationMatch and ifGenerationMatch preconditions if
// srcGen or dstGen are non-negative. See Google Storage docs:
// https://cloud.google.com/storage/docs/json_api/v1/objects/copy
Copy(ctx context.Context, dst string, dstGen int64, src string, srcGen int64) error
// Delete removes a file.
//
// Missing file is not an error.
Delete(ctx context.Context, path string) error
// Publish implements conditional copy operation with some caveats, making it
// useful for moving uploaded files from a temporary storage area after they
// have been verified, ensuring they are not modified during this process.
//
// 'src' will be copied to 'dst' only if source generation number matches
// 'srcGen'. If 'srcGen' is negative, the generation check is not performed.
// Also assumes 'dst' is ether missing, or already contains data matching
// 'src' (so 'dst' should usually be a content-addressed path). This allows
// the conditional move operation to be safely retried even if it failed
// midway before.
//
// Note that it keeps 'src' intact. Use Delete to get rid of it when
// necessary. Google Storage doesn't have atomic "move" operation.
Publish(ctx context.Context, dst, src string, srcGen int64) error
// StartUpload opens a new resumable upload session to a given path.
//
// Returns an URL to use in Resumable Upload protocol. It contains uploadId
// that acts as an authentication token, treat the URL as a secret.
//
// The upload protocol is finished by the CIPD client, and so it's not
// implemented here.
StartUpload(ctx context.Context, path string) (uploadURL string, err error)
// CancelUpload cancels a resumable upload session.
CancelUpload(ctx context.Context, uploadURL string) error
// Reader returns an io.ReaderAt implementation to read contents of a file at
// a specific generation (if 'gen' is positive) or at the current live
// generation (if 'gen' is zero or negative).
Reader(ctx context.Context, path string, gen int64) (Reader, error)
}
GoogleStorage is a wrapper over raw Google Cloud Storage JSON API.
Use Get() to grab an implementation.
Uses service's own service account for authentication.
All paths are expected to be in format "/<bucket>/<object>", methods would panic otherwise. Use ValidatePath prior to calling GoogleStorage methods if necessary.
Errors returned by GoogleStorage are annotated with transient tag (when appropriate) and with HTTP status codes of corresponding Google Storage API replies (if available). Use StatusCode(err) to extract them.
Retries on transient errors internally a bunch of times. Logs all calls to the info log.
func Get ¶
func Get(ctx context.Context) GoogleStorage
Get returns Google Storage JSON API wrapper.
Its guts are lazily initializes on first use, to simplify error handling.
The returned object is associated with the given context and it should not outlive it. Each individual method still accepts a context though, which can be a derivative of the root context (for example to provide custom per-method deadline or logging fields).
type Reader ¶
type Reader interface {
io.ReaderAt
// Size is the total file size.
Size() int64
// Generation is generation number of the content we are reading.
Generation() int64
}
Reader can read chunks of a Google Storage file.
Use GoogleStorage.Reader to get the reader.
type RestartUploadError ¶
type RestartUploadError struct {
Offset int64
}
RestartUploadError is returned by Uploader when it resumes an interrupted upload, and Google Storage asks to upload from an offset the Uploader has no data for.
Callers of Uploader should handle this case themselves by restarting the upload from the requested offset.
See https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload#resume-upload
func (*RestartUploadError) Error ¶
func (e *RestartUploadError) Error() string
Error is part of error interface.
type Uploader ¶
type Uploader struct {
Context context.Context // the context for canceling retries and for logging
Client *http.Client // the client to use for sending anonymous requests
UploadURL string // upload URL returned by GoogleStorage.StartUpload
Offset int64 // offset in the file to upload to, mutated by Write
FileSize int64 // total size of the file being uploaded, required
// contains filtered or unexported fields
}
Uploader implements io.Writer for Google Storage Resumable Upload sessions.
Does no buffering inside, thus efficiency of uploads directly depends on granularity of Write(...) calls. Additionally, Google Storage expects the length of each uploaded chunk to be a multiple of 256 Kb, so callers of Write(...) should supply the appropriately-sized chunks.
Retries transient errors internally, but it can potentially end up in a situation where it needs data not available in the current Write(...) operation. In this case Write returns *RestartUploadError error, which indicates an offset the upload should be restarted from.
Source Files