funnel

README

funnel

The 12 factor rule for logging says that an app "should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout." The execution environment should take care of capturing the logs and perform further processing with it.

Funnel is meant to be a replacement for your app's "logger + logrotate" pipeline. No more sending SIGHUP signals, or reload commands to open a new file. No more setting up conditional flags to switch between writing to console and file when changing from dev to production. All you need to do is just print to stdout and pipe it to funnel. And let it take care of the rest.

Features quick tour

• Basic feature set of a log rotator:

• Rolling over to a new file
• Deleting old files
• Gzipping files
• File rename policies

• Prepend each log line with a custom string
• Live reloading of config file on save. No more messing around with SIGHUP or SIGUSR1.

Quickstart

Grab the binary for your platform and the config file from here.

To run, just pipe the output of your app to the funnel binary. Note that, funnel only consumes from stdout, so you might need to redirect stderr to stdout.

$/etc/myapp/bin 2>&1 | funnel

P.S. You also need to drop the funnel binary to your $PATH.

The funnel app itself does not give any output except errors. For eg- invalid config file, or insufficient permissions for the log directory. If you are running your app as a service, you can just pipe its output to logger.

$/etc/myapp/bin 2>&1 | funnel 2>&1 | logger

Or just leave it at standard output, if you are running from the terminal.

Configuration

The config can be specified in a .toml file. The file is part of the repo, which you can see here. All the settings are documented and are populated with the default values. The same defaults are embedded in the app itself, so the app can even run without a config file.

To read the config, the app looks for a file named config.toml in these locations one by one -

• /etc/funnel/config.toml
• $HOME/.funnel/config.toml
• ./config.toml (i.e. in the current directory of your target app)

You can place a global file in /etc/funnel/ and have separate files in each app directory to have config values overriding the global ones.

Environment variables are also supported and takes the highest precedence. To get the env variable name, just capitalize the config variable. For eg-

• logging.directory becomes LOGGING_DIRECTORY
• rollup.file_rename_policy becomes ROLLUP_FILE_RENAME_POLICY

TODO:

• Add benchmarks
• Add new output targets like ElasticSearch, InfluxDB, AmazonS3
• Add stats endpoint to expose metrics.

Footnote - This project was heavily inspired from the logsend project.

Documentation

Index

• Constants
• Variables
• type ByModTime
  • func (a ByModTime) Len() int
  • func (a ByModTime) Less(i, j int) bool
  • func (a ByModTime) Swap(i, j int)
• type Config
  • func GetConfig(v *viper.Viper, logger *syslog.Writer) (*Config, chan *Config, error)
• type ConfigValueError
  • func (e *ConfigValueError) Error() string
• type Consumer
  • func (c *Consumer) Start(inputStream io.Reader)
• type LineProcessor
  • func GetLineProcessor(cfg *Config) LineProcessor
• type NoProcessor
  • func (*NoProcessor) Write(w io.Writer, line string) error
• type SimpleLineProcessor
  • func (lp *SimpleLineProcessor) Write(w io.Writer, line string) error
• type TemplateLineProcessor
  • func (lp *TemplateLineProcessor) Write(w io.Writer, line string) error

Constants

const (
	// config keys
	LoggingDirectory         = "logging.directory"
	LoggingActiveFileName    = "logging.active_file_name"
	RotationMaxLines         = "rotation.max_lines"
	RotationMaxFileSizeBytes = "rotation.max_file_size_bytes"
	FlushingTimeIntervalSecs = "flushing.time_interval_secs"
	PrependValue             = "misc.prepend_value"
	FileRenamePolicy         = "rollup.file_rename_policy"
	MaxAge                   = "rollup.max_age"
	MaxCount                 = "rollup.max_count"
	Gzip                     = "rollup.gzip"
)

XXX: Move it to constants.go if needed

Variables

var (
	// ErrInvalidFileRenamePolicy is raised for invalid values to file rename policy
	ErrInvalidFileRenamePolicy = errors.New(FileRenamePolicy + " can only be timestamp or serial")
	// ErrInvalidMaxAge is raised for invalid value in max age - life bad suffixes or no integer value at all
	ErrInvalidMaxAge = errors.New(MaxAge + " must end with either d or h and start with a number")
)

Types

type ByModTime

type ByModTime []os.FileInfo

ByModTime implements sorting for files by mod time from recent to old

func (ByModTime) Len

func (a ByModTime) Len() int

func (ByModTime) Less

func (a ByModTime) Less(i, j int) bool

func (ByModTime) Swap

func (a ByModTime) Swap(i, j int)

type Config

type Config struct {
	DirName        string
	ActiveFileName string

	RotationMaxLines int
	RotationMaxBytes uint64

	FlushingTimeIntervalSecs int

	PrependValue string

	FileRenamePolicy string
	MaxAge           int64
	MaxCount         int
	Gzip             bool
}

Config holds all the config settings

func GetConfig

func GetConfig(v *viper.Viper, logger *syslog.Writer) (*Config, chan *Config, error)

GetConfig returns the config struct which is then passed to the consumer

type ConfigValueError

type ConfigValueError struct {
	Key string
}

ConfigValueError holds the error value if a config key contains an invalid value

func (*ConfigValueError) Error

func (e *ConfigValueError) Error() string

type Consumer

type Consumer struct {
	Config        *Config
	LineProcessor LineProcessor
	Logger        *syslog.Writer

	ReloadChan chan *Config
	// contains filtered or unexported fields
}

Consumer is the main struct which holds all the stuff necessary to run the code

func (*Consumer) Start

func (c *Consumer) Start(inputStream io.Reader)

Start takes the input stream and begins reading line by line buffering the output to a file and flushing at set intervals

type LineProcessor

type LineProcessor interface {
	Write(io.Writer, string) error
}

LineProcessor interface is passed down to the consumer which just calls the write function

func GetLineProcessor

func GetLineProcessor(cfg *Config) LineProcessor

GetLineProcessor function returns the particular processor depending on the config.

type NoProcessor

type NoProcessor struct {
}

NoProcessor is used when there is no prepend value. It just prints the line without any other action

func (*NoProcessor) Write

func (*NoProcessor) Write(w io.Writer, line string) error

type SimpleLineProcessor

type SimpleLineProcessor struct {
	// contains filtered or unexported fields
}

SimpleLineProcessor is used when the prependValue is only a simple string It just concatenates the string with the line and prints it

func (*SimpleLineProcessor) Write

func (lp *SimpleLineProcessor) Write(w io.Writer, line string) error

type TemplateLineProcessor

type TemplateLineProcessor struct {
	// contains filtered or unexported fields
}

TemplateLineProcessor is used when there is a template action in the prependValue It parses the prependValue and store the template. Then for every write call, it executes the template and writes it

func (*TemplateLineProcessor) Write

func (lp *TemplateLineProcessor) Write(w io.Writer, line string) error