downloadfile

README

downloadfile

This filter downloads a file to the local filesystem. The source is a HTTP or HTTPS GET call. Authentication is supported by having means for adding headers to the request. (Including Authorization headers.)

{
  "filter": [
    {
      "type": "downloadfile",
      "url": "message",
      "response": "response_headers",
      "authenticators": [
        {
          "name": "",
          "restrict_to": [
            "www.helge.net"
          ],
          "headers": {
            "Authorization": "Basic YWRtaW46YWRtaW4="
          }
        }
      ]
    }
  ]
}

Including headers

Headers are loaded in this order:

1. From a field in the gogstash event. (The name of the field is specified by the parameter "headers".)
2. From the gogstash configuration file, as shown above.
3. From a file specified by the environment variable FILTERDOWNLOAD_AUTHENTICATOR. (Must be in JSON format.)
4. From the file authenticator.json in the working directory. (Must be in JSON format.)

Each authenticator has an optional name. An authenticator with a blank name will be loaded automatically if the site matches. In the event you can specify what authenticator to use - the configuration parameter auth specifies what field to look into.

Placement of the file

The downloaded file is placed in a directory of your choice by specifying "download_dir". If left blank the system default temp dir is used.

The filename is a unique name generated by the system. The name of the file is placed in the field specified by the configuration parameter "file_name". The default field is "field_name".

Documentation

Index

• Constants
• func InitHandler(ctx context.Context, raw config.ConfigRaw, control config.Control) (config.TypeFilterConfig, error)
• func IsIntIn(value int, set []int) bool
• func IsStringIn(value string, set []string) bool
• type Authenticator
• type FilterConfig
  • func DefaultFilterConfig() FilterConfig
  • func (f *FilterConfig) DownloadFile(event *logevent.LogEvent) error
  • func (f *FilterConfig) Event(ctx context.Context, event logevent.LogEvent) (logevent.LogEvent, bool)
  • func (f *FilterConfig) GetAuthenticatorHeaders(name string, host string) map[string]string
  • func (f *FilterConfig) ValidateEvent(event *logevent.LogEvent) error

Constants

const ErrorTag = "gogstash_filter_downloadfile_error"

ErrorTag tag added to event when process module failed

const ModuleName = "downloadfile"

ModuleName is the name used in config file

Functions

func InitHandler

func InitHandler(ctx context.Context, raw config.ConfigRaw, control config.Control) (config.TypeFilterConfig, error)

InitHandler initialize the filter plugin

func IsIntIn

func IsIntIn(value int, set []int) bool

IsIntIn checks if value is in set, returns true if true, false otherwise

func IsStringIn

func IsStringIn(value string, set []string) bool

IsStringIn checks is a value is in the set

Types

type Authenticator

type Authenticator struct {
	Name       string            `json:"name" yaml:"name"`               // name of this authenticator
	Headers    map[string]string `json:"headers" yaml:"headers"`         // extra headers to add to request
	RestrictTo []string          `json:"restrict_to" yaml:"restrict_to"` // hostnames to limit this authenticator against
}

Authenticator describes a way to authenticate to a web service by adding headers to the request

type FilterConfig

type FilterConfig struct {
	config.FilterConfig

	URL               string          `json:"URL" yaml:"URL"`                               // field name of file to download
	Headers           string          `json:"headers" yaml:"headers"`                       // field name of extra headers to send in request, must be map[string]string
	Auth              string          `json:"auth" yaml:"auth"`                             // field name of specific authentication to use in client
	Size              string          `json:"size" yaml:"size"`                             // field to store the file size in
	FileName          string          `json:"file_name" yaml:"file_name"`                   // field to store name of output file
	Response          string          `json:"response" yaml:"response"`                     // field to save response headers to
	DownloadDir       string          `json:"download_dir" yaml:"download_dir"`             // folder to download file to
	Authenticators    []Authenticator `json:"authenticators" yaml:"authenticators"`         // a list of preconfigured authenticators
	AuthenticatorFile string          `json:"authenticator_file" yaml:"authenticator_file"` // a file with authenticators to read
	SuccessCodes      []int           `json:"success_codes" yaml:"success_codes"`           // result codes that indicate success
	RetryCodes        []int           `json:"retry_codes" yaml:"retry_codes"`               // codes that indicate that this error can be retried
}

FilterConfig holds the configuration json fields and internal objects

func DefaultFilterConfig

func DefaultFilterConfig() FilterConfig

DefaultFilterConfig returns an FilterConfig struct with default values

func (*FilterConfig) DownloadFile

func (f *FilterConfig) DownloadFile(event *logevent.LogEvent) error

DownloadFile downloads the file for the given event

func (*FilterConfig) Event

func (f *FilterConfig) Event(ctx context.Context, event logevent.LogEvent) (logevent.LogEvent, bool)

Event the main filter event

func (*FilterConfig) GetAuthenticatorHeaders

func (f *FilterConfig) GetAuthenticatorHeaders(name string, host string) map[string]string

GetAuthenticatorHeaders returns an empty map or the content of this authenticator. host is compared against allowed hostnames and both name and host must match if something is to be returned.

func (*FilterConfig) ValidateEvent

func (f *FilterConfig) ValidateEvent(event *logevent.LogEvent) error

ValidateEvent is a pre-flight check that checks our input parameters to see if they are sane. An error is returned if they are not sane.