s3

README

S3 Local Testing

Prerequisites:

• LocalStack CLI installed

  • python3 -m pip install localstack

• If you need to make changes

  • localstack config validate to verify docker-compose setup

• To validate in Docker-compose docker-compose up -d

  • localhost:4566/health will show status of all mimicked services

Documentation

Overview

Package s3 provides an implementation of Vervet Underground storage backed by Amazon S3.

Index

• func New(ctx context.Context, awsCfg *Config, options ...Option) (storage.Storage, error)
• type Config
• type Option
  • func NewCollator(newCollator func() (*storage.Collator, error)) Option
• type StaticKeyCredentials
• type Storage
  • func (s *Storage) CollateVersions(ctx context.Context, serviceFilter map[string]bool) error
  • func (s *Storage) CreateBucket(ctx context.Context) error
  • func (s *Storage) DeleteObject(ctx context.Context, key string) error
  • func (s *Storage) GetCollatedVersionSpec(ctx context.Context, version string) ([]byte, error)
  • func (s *Storage) GetCollatedVersionSpecs(ctx context.Context) (map[string][]byte, error)
  • func (s *Storage) GetObject(ctx context.Context, key string) ([]byte, error)
  • func (s *Storage) GetObjectWithMetadata(ctx context.Context, key string) (*s3.GetObjectOutput, error)
  • func (s *Storage) HasVersion(ctx context.Context, name string, version string, digest string) (bool, error)
  • func (s *Storage) ListCollatedVersions(ctx context.Context) ([]string, error)
  • func (s *Storage) ListCommonPrefixes(ctx context.Context, key string) ([]types.CommonPrefix, error)
  • func (s *Storage) ListObjects(ctx context.Context, key string, delimeter string) (*s3.ListObjectsV2Output, error)
  • func (s *Storage) NotifyVersion(ctx context.Context, name string, version string, contents []byte, ...) error
  • func (s *Storage) NotifyVersions(ctx context.Context, name string, versions []string, scrapeTime time.Time) error
  • func (s *Storage) PutObject(ctx context.Context, key string, reader io.Reader) (*s3.PutObjectOutput, error)
  • func (s *Storage) Version(ctx context.Context, version string) ([]byte, error)
  • func (s *Storage) VersionIndex(ctx context.Context) (vervet.VersionIndex, error)

Functions

func New

func New(ctx context.Context, awsCfg *Config, options ...Option) (storage.Storage, error)

Types

type Config

type Config struct {
	AwsRegion      string
	AwsEndpoint    string
	BucketName     string
	Credentials    StaticKeyCredentials
	IamRoleEnabled bool
}

Config defines S3 client target used in config.LoadDefaultConfig.

type Option

type Option func(*Storage)

Option defines a Storage constructor option.

func NewCollator

func NewCollator(newCollator func() (*storage.Collator, error)) Option

NewCollator configures the Storage instance to use the given constructor function for creating collator instances.

type StaticKeyCredentials

type StaticKeyCredentials struct {
	AccessKey  string
	SecretKey  string
	SessionKey string
}

StaticKeyCredentials defines credential structure used in config.LoadDefaultConfig.

type Storage

type Storage struct {
	// contains filtered or unexported fields
}

func (*Storage) CollateVersions

func (s *Storage) CollateVersions(ctx context.Context, serviceFilter map[string]bool) error

CollateVersions aggregates versions and revisions from all the services, and produces unified versions and merged specs for all APIs.

func (*Storage) CreateBucket

func (s *Storage) CreateBucket(ctx context.Context) error

CreateBucket idempotently creates an S3 bucket for VU.

func (*Storage) DeleteObject

func (s *Storage) DeleteObject(ctx context.Context, key string) error

DeleteObject performs an S3 DeleteObject request.

func (*Storage) GetCollatedVersionSpec

func (s *Storage) GetCollatedVersionSpec(ctx context.Context, version string) ([]byte, error)

GetCollatedVersionSpec retrieves a single collated vervet.Version and returns the JSON blob.

func (*Storage) GetCollatedVersionSpecs

func (s *Storage) GetCollatedVersionSpecs(ctx context.Context) (map[string][]byte, error)

GetCollatedVersionSpecs retrieves a map of vervet.Version strings and their corresponding JSON blobs and returns the result.

func (*Storage) GetObject

func (s *Storage) GetObject(ctx context.Context, key string) ([]byte, error)

GetObject performs an s3 GetObject request.

func (*Storage) GetObjectWithMetadata

func (s *Storage) GetObjectWithMetadata(ctx context.Context, key string) (*s3.GetObjectOutput, error)

GetObjectWithMetadata performs an S3 GetObject request, returning object metadata.

func (*Storage) HasVersion

func (s *Storage) HasVersion(ctx context.Context, name string, version string, digest string) (bool, error)

HasVersion implements scraper.Storage.

func (*Storage) ListCollatedVersions

func (s *Storage) ListCollatedVersions(ctx context.Context) ([]string, error)

ListCollatedVersions performs an S3 ListCommonPrefixes request on the collated versions folder, returning a slice of available version strings.

The returned result is currently truncated at 1000 results. TODO: Paginate all available results?

func (*Storage) ListCommonPrefixes

func (s *Storage) ListCommonPrefixes(ctx context.Context, key string) ([]types.CommonPrefix, error)

ListCommonPrefixes performs an S3 ListCommonPrefixes request. For an example key `collated-versions/`, this function may return a result such as `[]types.CommonPrefix{"collated-versions/2022-02-02~wip/", "collated-versions/2022-12-02~beta/"}`.

The returned result is currently truncated at 1000 results. TODO: Paginate all available results?

func (*Storage) ListObjects

func (s *Storage) ListObjects(ctx context.Context, key string, delimeter string) (*s3.ListObjectsV2Output, error)

ListObjects performs an S3 ListObjects request.

The returned result is currently truncated at 1000 results. TODO: Paginate all available results?

func (*Storage) NotifyVersion

func (s *Storage) NotifyVersion(
	ctx context.Context,
	name string,
	version string,
	contents []byte,
	scrapeTime time.Time,
) error

NotifyVersion implements scraper.Storage.

func (*Storage) NotifyVersions

func (s *Storage) NotifyVersions(ctx context.Context, name string, versions []string, scrapeTime time.Time) error

NotifyVersions implements scraper.Storage.

func (*Storage) PutObject

func (s *Storage) PutObject(ctx context.Context, key string, reader io.Reader) (*s3.PutObjectOutput, error)

PutObject performs an S3 PutObject request.

func (*Storage) Version

func (s *Storage) Version(ctx context.Context, version string) ([]byte, error)

Version implements scraper.Storage.

func (*Storage) VersionIndex

func (s *Storage) VersionIndex(ctx context.Context) (vervet.VersionIndex, error)

Versions implements scraper.Storage.