s3

README

S3 Sink

The S3 sink archives every flush to S3 as a separate S3 object.

This sink is still in an experimental state.

Config Options to connect to S3

Mandatory parameters below.

• aws_s3_bucket: util.StringSecret
• aws_region: string

Optional parameters below.

• aws_access_key_id string
• aws_secret_access_key util.StringSecret

The Go AWS SDK will load up Credentials in the following order. https://docs.aws.amazon.com/sdk-for-go/api/aws/session/

1. Environment Variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, AWS_PROFILE, AWS_REGION
2. Shared Credentials file ~/.aws/credentials
3. Shared Configuration file (if SharedConfig is enabled) export AWS_SDK_LOAD_CONFIG=1
4. EC2 Instance Metadata (credentials only).

Documentation

Index

• Variables
• func Create(server *veneur.Server, name string, logger *logrus.Entry, config veneur.Config, ...) (sinks.MetricSink, error)
• func MigrateConfig(config *veneur.Config)
• func ParseConfig(name string, config interface{}) (veneur.MetricSinkConfig, error)
• func S3Path(hostname string, ft filetype) *string
• type S3Sink
  • func (p *S3Sink) Flush(ctx context.Context, metrics []samplers.InterMetric) error
  • func (p *S3Sink) FlushOtherSamples(ctx context.Context, samples []ssf.SSFSample)
  • func (p *S3Sink) Name() string
  • func (p *S3Sink) S3Post(hostname string, data io.ReadSeeker, ft filetype) error
  • func (p *S3Sink) Start(traceClient *trace.Client) error
• type S3SinkConfig

Variables

var S3ClientUninitializedError = errors.New("s3 client has not been initialized")

Functions

func Create

func Create(
	server *veneur.Server, name string, logger *logrus.Entry,
	config veneur.Config, sinkConfig veneur.MetricSinkConfig,
) (sinks.MetricSink, error)

Create creates a new S3 sink for metrics. This function should match the signature of a value in veneur.MetricSinkTypes, and is intended to be passed into veneur.NewFromConfig to be called based on the provided configuration.

func MigrateConfig

func MigrateConfig(config *veneur.Config)

TODO(arnavdugar): Remove this once the old configuration format has been removed.

func ParseConfig

func ParseConfig(
	name string, config interface{},
) (veneur.MetricSinkConfig, error)

ParseConfig decodes the map config for an S3 sink into an S3SinkConfig struct.

func S3Path

func S3Path(hostname string, ft filetype) *string

Types

type S3Sink

type S3Sink struct {
	Hostname string
	Interval int
	Logger   *logrus.Entry

	S3Bucket string
	Svc      s3iface.S3API
	// contains filtered or unexported fields
}

func (*S3Sink) Flush

func (p *S3Sink) Flush(
	ctx context.Context, metrics []samplers.InterMetric,
) error

func (*S3Sink) FlushOtherSamples

func (p *S3Sink) FlushOtherSamples(ctx context.Context, samples []ssf.SSFSample)

func (*S3Sink) Name

func (p *S3Sink) Name() string

func (*S3Sink) S3Post

func (p *S3Sink) S3Post(hostname string, data io.ReadSeeker, ft filetype) error

func (*S3Sink) Start

func (p *S3Sink) Start(traceClient *trace.Client) error

type S3SinkConfig

type S3SinkConfig struct {
	AccessKeyID     util.StringSecret `yaml:"access_key_id"`
	S3Bucket        string            `yaml:"s3_bucket"`
	Region          string            `yaml:"region"`
	SecretAccessKey util.StringSecret `yaml:"secret_access_key"`
}