fetcher

Documentation

Overview

Copyright 2018 Google, Inc. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

• type Fetcher
  • func (gf *Fetcher) Fetch(ctx context.Context) error
• type GCS
• type OS

Types

type Fetcher

type Fetcher struct {
	GCS GCS
	OS  OS

	DestDir    string
	KeepSource bool
	StagingDir string

	CreatedDirs map[string]bool

	SourceType     string
	Bucket, Object string
	Generation     int64

	TimeoutGCS  bool
	WorkerCount int
	Retries     int
	Backoff     time.Duration
	Verbose     bool
	Stdout      io.Writer
	Stderr      io.Writer
	// contains filtered or unexported fields
}

Fetcher is the main workhorse of this package and does all the heavy lifting.

func (*Fetcher) Fetch

func (gf *Fetcher) Fetch(ctx context.Context) error

Fetch is the main entry point into Fetcher. Based on configuration, it pulls source from GCS into the destination directory.

type GCS

type GCS interface {
	NewReader(ctx context.Context, bucket, object string) (io.ReadCloser, error)
}

GCS allows us to inject dependencies to facilitate testing.

type OS

type OS interface {
	Rename(oldpath, newpath string) error
	Chmod(name string, mode os.FileMode) error
	Create(name string) (*os.File, error)
	MkdirAll(path string, perm os.FileMode) error
	Open(name string) (*os.File, error)
	RemoveAll(path string) error
}

OS allows us to inject dependencies to facilitate testing.