README
¶
s3 output
Sends events to s3 output of one or multiple buckets.
bucket is default bucket for events. Addition buckets can be described in multi_buckets section, example down here.
Field "bucket_field_event" is filed name, that will be searched in event.
If appears we try to send event to this bucket instead of described here.
⚠ Currently bucket names for bucket and multi_buckets can't intersect.
⚠ If dynamic bucket moved to config it can leave some not send data behind. To send this data to s3 move bucket dir from /var/log/dynamic_buckets/bucketName to /var/log/static_buckets/bucketName (/var/log is default path) and restart file.d
Example Standard example:
pipelines:
mkk:
settings:
capacity: 128
# input plugin is not important in this case, let's emulate http input.
input:
type: http
emulate_mode: "no"
address: ":9200"
actions:
- type: json_decode
field: message
output:
type: s3
file_plugin:
retention_interval: 10s
# endpoint, access_key, secret_key, bucket are required.
endpoint: "s3.fake_host.org:80"
access_key: "access_key1"
secret_key: "secret_key2"
bucket: "bucket-logs"
bucket_field_event: "bucket_name"
Example with fan-out buckets:
pipelines:
mkk:
settings:
capacity: 128
# input plugin is not important in this case, let's emulate http input.
input:
type: http
emulate_mode: "no"
address: ":9200"
actions:
- type: json_decode
field: message
output:
type: s3
file_plugin:
retention_interval: 10s
# endpoint, access_key, secret_key, bucket are required.
endpoint: "s3.fake_host.org:80"
access_key: "access_key1"
secret_key: "secret_key2"
bucket: "bucket-logs"
# bucket_field_event - event with such field will be sent to bucket with its value
# if such exists: {"bucket_name": "secret", "message": 123} to bucket "secret".
bucket_field_event: "bucket_name"
# multi_buckets is optional, contains array of buckets.
multi_buckets:
- endpoint: "otherS3.fake_host.org:80"
access_key: "access_key2"
secret_key: "secret_key2"
bucket: "bucket-logs-2"
- endpoint: "yet_anotherS3.fake_host.ru:80"
access_key: "access_key3"
secret_key: "secret_key3"
bucket: "bucket-logs-3"
Config params
file_config file.Config
Under the hood this plugin uses /plugin/output/file/ to collect logs
compression_type string default=zip options=zip
Compression type
endpoint string required
Endpoint address of default bucket.
access_key string required
s3 access key.
secret_key string required
s3 secret key.
bucket string required
s3 default bucket.
multi_buckets *``json:"multi_buckets"* MultiBuckets is additional buckets, which can also receive event. Event must contain bucket_namefield which value is s3 bucket name. Events withoutbucket_name` sends to DefaultBucket.
secure bool default=false
s3 connection secure option.
bucket_field_event string
BucketEventField field change destination bucket of event to fields value.
Fallback to DefaultBucket if BucketEventField bucket doesn't exist.
dynamic_buckets_limit int default=32
DynamicBucketsLimit regulates how many buckets can be created dynamically.
Prevents problems when some random strings in BucketEventField where
Generated using insane-doc
Documentation
¶
Index ¶
Constants ¶
const ( StaticBucketDir = "static_buckets" DynamicBucketDir = "dynamic_buckets" )
Variables ¶
var ( ErrCreateOutputPluginCantCheckBucket = errors.New("could not check bucket") ErrCreateOutputPluginNoSuchBucket = errors.New("bucket doesn't exist") )
Functions ¶
Types ¶
type Config ¶
type Config struct {
//> @3@4@5@6
//> Under the hood this plugin uses /plugin/output/file/ to collect logs
FileConfig file.Config `json:"file_config" child:"true"` //*
//> @3@4@5@6
//> Compression type
CompressionType string `json:"compression_type" default:"zip" options:"zip"` //*
//> @3@4@5@6
//> Endpoint address of default bucket.
Endpoint string `json:"endpoint" required:"true"` //*
//> @3@4@5@6
//> s3 access key.
AccessKey string `json:"access_key" required:"true"` //*
//> @3@4@5@6
//> s3 secret key.
SecretKey string `json:"secret_key" required:"true"` //*
//> @3@4@5@6
//> s3 default bucket.
DefaultBucket string `json:"bucket" required:"true"` //*
//> @3@4@5@6
//> MultiBuckets is additional buckets, which can also receive event.
//> Event must contain `bucket_name` field which value is s3 bucket name.
//> Events without `bucket_name` sends to DefaultBucket.
MultiBuckets `json:"multi_buckets" required:"false"` //*
//> @3@4@5@6
//> s3 connection secure option.
Secure bool `json:"secure" default:"false"` //*
//> @3@4@5@6
//> BucketEventField field change destination bucket of event to fields value.
//> Fallback to DefaultBucket if BucketEventField bucket doesn't exist.
BucketEventField string `json:"bucket_field_event" default:""` //*
//> @3@4@5@6
//> DynamicBucketsLimit regulates how many buckets can be created dynamically.
//> Prevents problems when some random strings in BucketEventField where
DynamicBucketsLimit int `json:"dynamic_buckets_limit" default:"32"` //*
}
! config-params ^ config-params
func (*Config) IsMultiBucketExists ¶
type MultiBuckets ¶
type MultiBuckets []singleBucketConfig
type ObjectStoreClient ¶
type ObjectStoreClientLimiter ¶
type ObjectStoreClientLimiter struct {
// contains filtered or unexported fields
}
func NewObjectStoreClientLimiter ¶
func NewObjectStoreClientLimiter(limit int) *ObjectStoreClientLimiter
func (*ObjectStoreClientLimiter) CanCreate ¶
func (limiter *ObjectStoreClientLimiter) CanCreate() bool
func (*ObjectStoreClientLimiter) Increment ¶
func (limiter *ObjectStoreClientLimiter) Increment()
type Plugin ¶
type Plugin struct {
// contains filtered or unexported fields
}
func (*Plugin) Start ¶
func (p *Plugin) Start(config pipeline.AnyConfig, params *pipeline.OutputPluginParams)
func (*Plugin) StartWithMinio ¶
func (p *Plugin) StartWithMinio(config pipeline.AnyConfig, params *pipeline.OutputPluginParams, factory objStoreFactory)
Source Files
Directories