deprecated

README

Deprecated producers

Here you find a collection of deprecated producers which reached the EOL (End of Life) or replaced by a new one. If you want to use these deprecated producers pls active them explicitly by your contrib_loader.go:

import (
	_ "github.com/trivago/gollum/contrib/deprecated/producer"
)

S3 Producer

Example config

Console to S3:

StdIn:
    Type: consumer.Console
    Streams: console

S3Out:
    Type: deprecated.producer.S3
    Region: eu-west-1

    Credential:
        Type: shared
        File: /Users/MYUSER/.aws/credentials
        Profile: default

    Streams: "*"
    StreamMapping:
        "console": "my-s3-bucket/subfolder/"

    SendTimeframeMs: 1000

    LocalPath: /tmp/aws
    UploadOnShutdown: true

Documentation

Index

• type S3
  • func (prod *S3) Configure(conf core.PluginConfigReader)
  • func (prod *S3) Produce(workers *sync.WaitGroup)

Types

type S3

type S3 struct {
	core.BufferedProducer `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

S3 producer plugin This producer sends data to an AWS S3 Bucket. Configuration example

S3OutDeprecated:

Type: deprecated.producer.S3
Region: "eu-west-1"
Endpoint: "s3-eu-west-1.amazonaws.com"
StorageClass: "STANDARD"
Credential:
  Type: "none"
  Id: ""
  Token: ""
  Secret: ""
  File: ""
  Profile: ""
BatchMaxMessages: 5000
ObjectMaxMessages: 5000
ObjectMessageDelimiter: "\n"
SendTimeframeMs: 10000
BatchTimeoutSec: 30
TimestampWrite: "2006-01-02T15:04:05"
PathFormatter: ""
Compress: false
LocalPath: ""
UploadOnShutdown: false
FileMaxAgeSec: 3600
FileMaxMB: 1000
StreamMapping:
  "*" : "bucket/path"

Region defines the amazon region of your s3 bucket. By default this is set to "eu-west-1".

Endpoint defines the amazon endpoint for your s3 bucket. By default this is set to "s3-eu-west-1.amazonaws.com"

StorageClass defines the amazon s3 storage class for objects created, from http://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html By default this is set to "STANDARD".

CredentialType defines the credentials that are to be used when connecting to s3. This can be one of the following: environment, static, shared, none. Static enables the parameters CredentialId, CredentialToken and CredentialSecret shared enables the parameters CredentialFile and CredentialProfile. None will not use any credentials and environment will pull the credentials from environmental settings. By default this is set to none.

BatchMaxMessages defines the maximum number of messages to upload per batch. By default this is set to 5000.

ObjectMaxMessages defines the maximum number of messages to join into an s3 object. By default this is set to 5000.

ObjectMessageDelimiter defines the string to delimit messages within an s3 object. By default this is set to "\n".

SendTimeframeMs defines the timeframe in milliseconds in which a second batch send can be triggered. By default this is set to 10000, i.e. ten upload operations per second per s3 path.

BatchTimeoutSec defines the number of seconds after which a batch is flushed automatically. By default this is set to 30.

TimestampWrite defines the go timestamp format that will be used in naming objects. Objects are named <s3_path><timestamp><sha1>. By default timestamp is set to "2006-01-02T15:04:05".

PathFormatter can define a formatter that extracts the path suffix for an s3 object from the object data. By default this is uses the sha1 of the object. A good formatter for this can be format.Identifier.

Compress defines whether to gzip compress the object before uploading. This adds a ".gz" extension to objects. By default this is set to false.

LocalPath defines the local output directory for temporary object files. Files will be stored as "<path>/<number>". Compressed files will have a .gz extension. State will be stored in "<path>/state". By default this is not set, and objects will be built in memory.

UploadOnShutdown defines whether to upload all temporary object files on shutdown. This has no effect if LocalPath is not set. By default this is false.

FileMaxAgeSec defines the maximum age of a local file before it is uploaded. This defaults to 3600 (1 hour).

FileMaxMB defines the maximum size of a local file before it is uploaded. This limit is imposed before compression occurs. This defaults to 1000 (1 GB).

StreamMapping defines a translation from gollum stream to s3 bucket/path. If no mapping is given the gollum stream name is used as s3 bucket. Values are of the form bucket/path or bucket, s3:// prefix is not allowed. The full path of the object will be s3://<StreamMapping><Timestamp><PathFormat> where Timestamp is time the object is written formatted with TimestampWrite, and PathFormat is the output of PathFormatter when passed the object data.

func (*S3) Configure

func (prod *S3) Configure(conf core.PluginConfigReader)

Configure initializes this producer with values from a plugin config.

func (*S3) Produce

func (prod *S3) Produce(workers *sync.WaitGroup)

Produce writes to a buffer that is sent to amazon s3.