s3assets

Documentation

Index

• Constants
• Variables
• func DownloadWithRetry(assets []string, saveDir, s3Endpoint string, retries int) error
• func List(assetsDir string) (assets []string, err error)
• func Load(assetsFile string) ([]string, error)
• func UploadWithRetry(assets []string, assetDir, bucket, s3Endpoint string, retries int) error
• func Write(s3Assets []string, output io.Writer) error
• type Downloader
  • func NewManyDownloader(assets []string, saveDir, s3Endpoint string, retries int) (Downloader, error)
  • func NewSingleDownloader(asset, saveDir string, retries int) (Downloader, error)
• type Extractor
  • func (e Extractor) Extract(sourceDir string, extraRegexps ...string) ([]string, error)
• type Uploader
  • func NewManyUploader(assets []string, assetDir, bucket, s3Endpoint string, retries int) (Uploader, error)
  • func NewSingleUploader(asset, saveDir, bucket string, retries int) (Uploader, error)

Constants

const DefaultS3Regexp = `(s3:\/\/[a-zA-Z0-9\/\-\_\.]+)`

DefaultS3Regexp matches s3://.* until a " or \n

Variables

var DiscardCloser io.WriteCloser = discardCloser{}

DiscardCloser implements a noop Write and Close.

Functions

func DownloadWithRetry

func DownloadWithRetry(assets []string, saveDir, s3Endpoint string, retries int) error

DownloadWithRetry will try to download all s3 assets to a directory. Will accept errors `retries` times. The path in `saveDir` that assets are saved to is simply saveDir/bucket/assetKey. E.g. if there's an s3 asset in `assets` with the uri "s3://my-bucket/nested/key/file.zip" and `saveDir` is "download_dir" then the asset will be saved at the path: `download_dir/my-bucket/nested/key/file.zip` with any intermediate directories being created as needed.

func List

func List(assetsDir string) (assets []string, err error)

List reads a directory and returns a list of paths relative to `assetsDir`.

func Load

func Load(assetsFile string) ([]string, error)

Load reads s3 assets from an S3AssetsList yaml file.

func UploadWithRetry

func UploadWithRetry(assets []string, assetDir, bucket, s3Endpoint string, retries int) error

UploadWithRetry will try to upload all assets in a directory to a particular bucket. Will accept errors `retries` times. Asset paths in `assets` are expected to be relative to the `assetDir` directory. The key created in the bucket for each asset will be the path that same relative path in `assetDir`. E.g. if there's an asset path in `assets` "my-bucket/nested/key/file.zip", and an `assetDir` "download_dir", a file at the path "./download_dir/my-bucket/nested/key/file.zip" is expected to exist. And if the target `bucket` is "target-bucket" then the example file will be uploaded to the s3 uri "s3://target-bucket/my-bucket/nested/key/file.zip".

func Write

func Write(s3Assets []string, output io.Writer) error

Write takes a list of s3 assets and writes them as a yaml S3Assets doc to output

Types

type Downloader

type Downloader interface {
	Download() error
}

Downloader implement Download.

func NewManyDownloader

func NewManyDownloader(assets []string, saveDir, s3Endpoint string, retries int) (Downloader, error)

NewManyDownloader returns a Downloader that will download many s3 assets to a save directory.

func NewSingleDownloader

func NewSingleDownloader(asset, saveDir string, retries int) (Downloader, error)

NewSingleDownloader returns a Downloader that will download a single s3 asset to a save directory.

type Extractor

type Extractor struct {
	Log logr.Logger
}

Extractor extracts all the S3 urls in a directory of manifests

func (Extractor) Extract

func (e Extractor) Extract(sourceDir string, extraRegexps ...string) ([]string, error)

Extract takes a source directory of manifests and slice of strings used to extract strings and returns a list of extracted strings

type Uploader

type Uploader interface {
	Upload() error
}

Uploader implement Upload.

func NewManyUploader

func NewManyUploader(assets []string, assetDir, bucket, s3Endpoint string, retries int) (Uploader, error)

NewManyUploader returns an Uploader that will upload many assets in a directory to an s3 bucket.

func NewSingleUploader

func NewSingleUploader(asset, saveDir, bucket string, retries int) (Uploader, error)

NewSingleUploader returns a Uploader that will upload a single asset in a save directory to an s3 bucket. `asset` must be a path relative to `saveDir`.