pumps

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 27, 2021 License: Apache-2.0, BSD-2-Clause, BSD-3-Clause, + 2 more Imports: 38 Imported by: 0

Documentation

Overview

Package pumps defines multiple back-end service used to persistent store iam analytics data.

Index

Constants

View Source 
const (
	KiB = 1 << (10 * iota)
	MiB
	GiB
	TiB
)

Define unit constant.

Variables

This section is empty.

Functions

This section is empty.

Types

type APIKeyTransport 

type APIKeyTransport struct {
	APIKey   string
	APIKeyID string
}

APIKeyTransport defiens elasticsearch api key.

func (*APIKeyTransport) RoundTrip 

func (t *APIKeyTransport) RoundTrip(r *http.Request) (*http.Response, error)

RoundTrip for APIKeyTransport auth.

type BaseMongoConf 

type BaseMongoConf struct {
	MongoURL                      string    `json:"mongo_url"                         mapstructure:"mongo_url"`
	MongoUseSSL                   bool      `json:"mongo_use_ssl"                     mapstructure:"mongo_use_ssl"`
	MongoSSLInsecureSkipVerify    bool      `json:"mongo_ssl_insecure_skip_verify"    mapstructure:"mongo_ssl_insecure_skip_verify"`
	MongoSSLAllowInvalidHostnames bool      `json:"mongo_ssl_allow_invalid_hostnames" mapstructure:"mongo_ssl_allow_invalid_hostnames"`
	MongoSSLCAFile                string    `json:"mongo_ssl_ca_file"                 mapstructure:"mongo_ssl_ca_file"`
	MongoSSLPEMKeyfile            string    `json:"mongo_ssl_pem_keyfile"             mapstructure:"mongo_ssl_pem_keyfile"`
	MongoDBType                   MongoType `json:"mongo_db_type"                     mapstructure:"mongo_db_type"`
}

BaseMongoConf defines options needed when connnect to mongo db.

type CSVConf 

type CSVConf struct {
	// Specify the directory used to store automatically generated csv file which contains analyzed data.
	CSVDir string `mapstructure:"csv_dir"`
}

CSVConf defines csv specific options.

type CSVPump 

type CSVPump struct {
	CommonPumpConfig
	// contains filtered or unexported fields
}

CSVPump defines a csv pump with csv specific options and common options.

func (*CSVPump) GetName 

func (c *CSVPump) GetName() string

GetName returns the csv pump name.

func (*CSVPump) Init 

func (c *CSVPump) Init(conf interface{}) error

Init initialize the csv pump instance.

func (*CSVPump) New 

func (c *CSVPump) New() Pump

New create a csv pump instance.

func (*CSVPump) WriteData 

func (c *CSVPump) WriteData(ctx context.Context, data []interface{}) error

WriteData write analyzed data to csv persistent back-end storage.

type CommonPumpConfig 

type CommonPumpConfig struct {
	OmitDetailedRecording bool
	// contains filtered or unexported fields
}

CommonPumpConfig defines common options used by all persistent store, like elasticsearch, kafka, mongo and etc.

func (*CommonPumpConfig) GetFilters 

func (p *CommonPumpConfig) GetFilters() analytics.AnalyticsFilters

GetFilters get attributes `filters` for CommonPumpConfig.

func (*CommonPumpConfig) GetOmitDetailedRecording 

func (p *CommonPumpConfig) GetOmitDetailedRecording() bool

GetOmitDetailedRecording get attributes `OmitDetailedRecording` for CommonPumpConfig.

func (*CommonPumpConfig) GetTimeout 

func (p *CommonPumpConfig) GetTimeout() int

GetTimeout get attributes `timeout` for CommonPumpConfig.

func (*CommonPumpConfig) SetFilters 

func (p *CommonPumpConfig) SetFilters(filters analytics.AnalyticsFilters)

SetFilters set attributes `filters` for CommonPumpConfig.

func (*CommonPumpConfig) SetOmitDetailedRecording 

func (p *CommonPumpConfig) SetOmitDetailedRecording(omitDetailedRecording bool)

SetOmitDetailedRecording set attributes `OmitDetailedRecording` for CommonPumpConfig.

func (*CommonPumpConfig) SetTimeout 

func (p *CommonPumpConfig) SetTimeout(timeout int)

SetTimeout set attributes `timeout` for CommonPumpConfig.

type DummyPump 

type DummyPump struct {
	CommonPumpConfig
}

DummyPump defines a dummy pump with dummy specific options and common options.

func (*DummyPump) GetName 

func (p *DummyPump) GetName() string

GetName returns the dummy pump name.

func (*DummyPump) Init 

func (p *DummyPump) Init(conf interface{}) error

Init initialize the dummy pump instance.

func (*DummyPump) New 

func (p *DummyPump) New() Pump

New create a dummy pump instance.

func (*DummyPump) WriteData 

func (p *DummyPump) WriteData(ctx context.Context, data []interface{}) error

WriteData write analyzed data to dummy persistent back-end storage.

type Elasticsearch7Operator 

type Elasticsearch7Operator struct {
	// contains filtered or unexported fields
}

Elasticsearch7Operator defines elasticsearch6 operator.

type ElasticsearchBulkConfig 

type ElasticsearchBulkConfig struct {
	Workers       int `mapstructure:"workers"`
	FlushInterval int `mapstructure:"flush_interval"`
	BulkActions   int `mapstructure:"bulk_actions"`
	BulkSize      int `mapstructure:"bulk_size"`
}

ElasticsearchBulkConfig defines elasticsearch bulk config.

type ElasticsearchConf 

type ElasticsearchConf struct {
	BulkConfig       ElasticsearchBulkConfig `mapstructure:"bulk_config"`
	IndexName        string                  `mapstructure:"index_name"`
	ElasticsearchURL string                  `mapstructure:"elasticsearch_url"`
	DocumentType     string                  `mapstructure:"document_type"`
	AuthAPIKeyID     string                  `mapstructure:"auth_api_key_id"`
	AuthAPIKey       string                  `mapstructure:"auth_api_key"`
	Username         string                  `mapstructure:"auth_basic_username"`
	Password         string                  `mapstructure:"auth_basic_password"`
	EnableSniffing   bool                    `mapstructure:"use_sniffing"`
	RollingIndex     bool                    `mapstructure:"rolling_index"`
	DisableBulk      bool                    `mapstructure:"disable_bulk"`
}

ElasticsearchConf defines elasticsearch specific options.

type ElasticsearchOperator 

type ElasticsearchOperator interface {
	// contains filtered or unexported methods
}

ElasticsearchOperator defines interface for all elasticsearch operator.

type ElasticsearchPump 

type ElasticsearchPump struct {
	CommonPumpConfig
	// contains filtered or unexported fields
}

ElasticsearchPump defines a elasticsearch pump with elasticsearch specific options and common options.

func (*ElasticsearchPump) GetName 

func (e *ElasticsearchPump) GetName() string

GetName returns the elasticsearch pump name.

func (*ElasticsearchPump) Init 

func (e *ElasticsearchPump) Init(config interface{}) error

Init initialize the elasticsearch pump instance.

func (*ElasticsearchPump) New 

func (e *ElasticsearchPump) New() Pump

New create a elasticsearch pump instance.

func (*ElasticsearchPump) WriteData 

func (e *ElasticsearchPump) WriteData(ctx context.Context, data []interface{}) error

WriteData write analyzed data to elasticsearch persistent back-end storage.

type InfluxConf 

type InfluxConf struct {
	DatabaseName string   `mapstructure:"database_name"`
	Addr         string   `mapstructure:"address"`
	Username     string   `mapstructure:"username"`
	Password     string   `mapstructure:"password"`
	Fields       []string `mapstructure:"fields"`
	Tags         []string `mapstructure:"tags"`
}

InfluxConf defines influx specific options.

type InfluxPump 

type InfluxPump struct {
	CommonPumpConfig
	// contains filtered or unexported fields
}

InfluxPump defines a influx pump whth influx specific options and common options.

func (*InfluxPump) GetName 

func (i *InfluxPump) GetName() string

GetName returns the influx pump name.

func (*InfluxPump) Init 

func (i *InfluxPump) Init(config interface{}) error

Init initialize the influx pump instance.

func (*InfluxPump) New 

func (i *InfluxPump) New() Pump

New create a influx pump instance.

func (*InfluxPump) WriteData 

func (i *InfluxPump) WriteData(ctx context.Context, data []interface{}) error

WriteData write analyzed data to influx persistent back-end storage.

type KafkaConf 

type KafkaConf struct {
	Broker                []string          `mapstructure:"broker"`
	ClientID              string            `mapstructure:"client_id"`
	Topic                 string            `mapstructure:"topic"`
	SSLCertFile           string            `mapstructure:"ssl_cert_file"`
	SSLKeyFile            string            `mapstructure:"ssl_key_file"`
	SASLMechanism         string            `mapstructure:"sasl_mechanism"`
	Username              string            `mapstructure:"sasl_username"`
	Password              string            `mapstructure:"sasl_password"`
	Algorithm             string            `mapstructure:"sasl_algorithm"`
	Timeout               time.Duration     `mapstructure:"timeout"`
	MetaData              map[string]string `mapstructure:"meta_data"`
	Compressed            bool              `mapstructure:"compressed"`
	UseSSL                bool              `mapstructure:"use_ssl"`
	SSLInsecureSkipVerify bool              `mapstructure:"ssl_insecure_skip_verify"`
}

KafkaConf defines kafka specific options.

type KafkaPump 

type KafkaPump struct {
	CommonPumpConfig
	// contains filtered or unexported fields
}

KafkaPump defines a kafka pump with kafka specific options and common options.

func (*KafkaPump) GetName 

func (k *KafkaPump) GetName() string

GetName returns the kafka pump name.

func (*KafkaPump) Init 

func (k *KafkaPump) Init(config interface{}) error

Init initialize the kafka pump instance.

func (*KafkaPump) New 

func (k *KafkaPump) New() Pump

New create a kafka pump instance.

func (*KafkaPump) WriteData 

func (k *KafkaPump) WriteData(ctx context.Context, data []interface{}) error

WriteData write analyzed data to kafka persistent back-end storage.

type Message 

type Message map[string]interface{}

Message contains the messages need to push to pump.

type MongoConf 

type MongoConf struct {
	BaseMongoConf

	CollectionName            string `json:"collection_name"               mapstructure:"collection_name"`
	MaxInsertBatchSizeBytes   int    `json:"max_insert_batch_size_bytes"   mapstructure:"max_insert_batch_size_bytes"`
	MaxDocumentSizeBytes      int    `json:"max_document_size_bytes"       mapstructure:"max_document_size_bytes"`
	CollectionCapMaxSizeBytes int    `json:"collection_cap_max_size_bytes" mapstructure:"collection_cap_max_size_bytes"`
	CollectionCapEnable       bool   `json:"collection_cap_enable"         mapstructure:"collection_cap_enable"`
}

MongoConf defines mongo specific options.

type MongoPump 

type MongoPump struct {
	CommonPumpConfig
	// contains filtered or unexported fields
}

MongoPump defines a mongo pump with mongo specific options and common options.

func (*MongoPump) AccumulateSet 

func (m *MongoPump) AccumulateSet(data []interface{}) [][]interface{}

AccumulateSet accumulate data.

func (*MongoPump) GetName 

func (m *MongoPump) GetName() string

GetName returns the mongo pump name.

func (*MongoPump) Init 

func (m *MongoPump) Init(config interface{}) error

Init initialize the mongo pump instance.

func (*MongoPump) New 

func (m *MongoPump) New() Pump

New create a mongo pump instance.

func (*MongoPump) WriteData 

func (m *MongoPump) WriteData(ctx context.Context, data []interface{}) error

WriteData write analyzed data to mongo persistent back-end storage.

type MongoType 

type MongoType int

MongoType define a new mongo type. 

const (
	StandardMongo MongoType = iota
	AWSDocumentDB
)

Defines mongo type.

type PrometheusConf 

type PrometheusConf struct {
	Addr string `mapstructure:"listen_address"`
	Path string `mapstructure:"path"`
}

PrometheusConf defines prometheus specific options.

type PrometheusPump 

type PrometheusPump struct {

	// Per service
	TotalStatusMetrics *prometheus.CounterVec

	CommonPumpConfig
	// contains filtered or unexported fields
}

PrometheusPump defines a prometheus pump with prometheus specific options and common options.

func (*PrometheusPump) GetName 

func (p *PrometheusPump) GetName() string

GetName returns the prometheus pump name.

func (*PrometheusPump) Init 

func (p *PrometheusPump) Init(conf interface{}) error

Init initialize the prometheus pump instance.

func (*PrometheusPump) New 

func (p *PrometheusPump) New() Pump

New create a prometheus pump instance.

func (*PrometheusPump) WriteData 

func (p *PrometheusPump) WriteData(ctx context.Context, data []interface{}) error

WriteData write analyzed data to prometheus persistent back-end storage.

type Pump 

type Pump interface {
	GetName() string
	New() Pump
	Init(interface{}) error
	WriteData(context.Context, []interface{}) error
	SetFilters(analytics.AnalyticsFilters)
	GetFilters() analytics.AnalyticsFilters
	SetTimeout(timeout int)
	GetTimeout() int
	SetOmitDetailedRecording(bool)
	GetOmitDetailedRecording() bool
}

Pump defines the interface for all analytics back-end.

func GetPumpByName 

func GetPumpByName(name string) (Pump, error)

GetPumpByName returns the pump instance by given name.

type SyslogConf 

type SyslogConf struct {
	Transport   string `mapstructure:"transport"`
	NetworkAddr string `mapstructure:"network_addr"`
	LogLevel    int    `mapstructure:"log_level"`
	Tag         string `mapstructure:"tag"`
}

SyslogConf defines syslog specific options.

type SyslogPump 

type SyslogPump struct {
	CommonPumpConfig
	// contains filtered or unexported fields
}

SyslogPump defines a syslog pump with syslog specific options and common options.

func (*SyslogPump) GetFilters 

func (s *SyslogPump) GetFilters() analytics.AnalyticsFilters

GetFilters get attributes `filters` for SyslogPump.

func (*SyslogPump) GetName 

func (s *SyslogPump) GetName() string

GetName returns the syslog pump name.

func (*SyslogPump) GetTimeout 

func (s *SyslogPump) GetTimeout() int

GetTimeout get attributes `timeout` for SyslogPump.

func (*SyslogPump) Init 

func (s *SyslogPump) Init(config interface{}) error

Init initialize the syslog pump instance.

func (*SyslogPump) New 

func (s *SyslogPump) New() Pump

New create a syslog pump instance.

func (*SyslogPump) SetFilters 

func (s *SyslogPump) SetFilters(filters analytics.AnalyticsFilters)

SetFilters set attributes `filters` for SyslogPump.

func (*SyslogPump) SetTimeout 

func (s *SyslogPump) SetTimeout(timeout int)

SetTimeout set attributes `timeout` for SyslogPump.

func (*SyslogPump) WriteData 

func (s *SyslogPump) WriteData(ctx context.Context, data []interface{}) error

WriteData write analyzed data to syslog persistent back-end storage.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.