s3

README

S3 Plugin

The S3 plugin archives every flush to S3 as a separate S3 object.

This plugin is still in an experimental state.

Documentation

Index

• Constants
• Variables
• func EncodeDDMetricCSV(d samplers.DDMetric, w *csv.Writer, partitionDate *time.Time, hostName string) error
• func EncodeDDMetricsCSV(metrics []samplers.DDMetric, delimiter rune, includeHeaders bool, ...) (io.ReadSeeker, error)
• func S3Path(hostname string, ft filetype) *string
• type S3Plugin
  • func (p *S3Plugin) Flush(metrics []samplers.DDMetric, hostname string) error
  • func (p *S3Plugin) Name() string
  • func (p *S3Plugin) S3Post(hostname string, data io.ReadSeeker, ft filetype) error

Constants

const (
	// the order in which these appear determines the
	// order of the fields in the resultant TSV
	TsvName tsvField = iota
	TsvTags
	TsvMetricType

	// The hostName attached to the metric
	TsvHostname

	// The hostName of the server flushing the data
	TsvVeneurHostname

	TsvDeviceName
	TsvInterval

	TsvTimestamp
	TsvValue

	// This is the _partition field
	// required by the Redshift IncrementalLoader.
	// For our purposes, the current date is a good partition.
	TsvPartition
)

const PartitionDateFormat = "20060102"

const RedshiftDateFormat = "2006-01-02 03:04:05"

Variables

var S3Bucket = "stripe-veneur"

var S3ClientUninitializedError = errors.New("s3 client has not been initialized")

Functions

func EncodeDDMetricCSV

func EncodeDDMetricCSV(d samplers.DDMetric, w *csv.Writer, partitionDate *time.Time, hostName string) error

EncodeDDMetricCSV generates a newline-terminated CSV row that describes the data represented by the DDMetric. The caller is responsible for setting w.Comma as the appropriate delimiter. For performance, encodeCSV does not flush after every call; the caller is expected to flush at the end of the operation cycle

func EncodeDDMetricsCSV

func EncodeDDMetricsCSV(metrics []samplers.DDMetric, delimiter rune, includeHeaders bool, hostname string) (io.ReadSeeker, error)

EncodeDDMetricsCSV returns a reader containing the gzipped CSV representation of the DDMetrics data, one row per DDMetric. the AWS sdk requires seekable input, so we return a ReadSeeker here

func S3Path

func S3Path(hostname string, ft filetype) *string

Types

type S3Plugin

type S3Plugin struct {
	Logger   *logrus.Logger
	Svc      s3iface.S3API
	S3Bucket string
	Hostname string
}

func (*S3Plugin) Flush

func (p *S3Plugin) Flush(metrics []samplers.DDMetric, hostname string) error

func (*S3Plugin) Name

func (p *S3Plugin) Name() string

func (*S3Plugin) S3Post

func (p *S3Plugin) S3Post(hostname string, data io.ReadSeeker, ft filetype) error