gonx

README

gonx

gonx is Nginx access log reader library for Go. In fact you can use it for any format.

Usage

The library provides Reader type and two constructors for it.

Common constructor NewReader gets opened file (io.Reader) and log format (string) as argumets. format is in form os nginx log_format string.

reader := gonx.NewReader(file, format)

NewNginxReader provides mo magic. It gets nginx config file (io.Reader) as second argument and log_format name (string) a third.

reader := gonx.NewNginxReader(file, nginxConfig, format_name)

Reader implements io.Reader. Here is example usage

for {
	rec, err := reader.Read()
	if err == io.EOF {
		break
	}
	// Process the record... e.g.
}

See more examples in example/*.go sources.

Format

As I said above this library is primary for nginx access log parsing, but it can be configured to parse any other format. NewReader accepts format argument, it will be transformed to regular expression and used for log line by line parsing. Format is nginx-like, here is example

`$remote_addr [$time_local] "$request"`

It should contain variables inn form $name. The regular expression will be created using this string format representation

`^(?P<remote_addr>[^ ]+) \[(?P<time_local>[^]]+)\] "(?P<request>[^"]+)"$`

If log line does not match this format, the Reader.Read returns an error. Otherwise you will get the record of type Entry (which is customized map[string][string]) with remote_addr, time_local and request keys filled with parsed values.

Stability

This library API and internal representation can be changed at any moment, but I guarantee that backward capability will be supported for the following public interfaces.

• func NewReader(logFile io.Reader, format string) *Reader
• func NewNginxReader(logFile io.Reader, nginxConf io.Reader, formatName string) (reader *Reader, err error)
• func (r *Reader) Read() (record Entry, err error)

Changelog

All major changes will be noticed in release notes.

Contributing

Fork the repo, create a feature branch then send me pull request. Feel free to create new issues or contact me using email.

Documentation

Index

• func MapReduce(file io.Reader, parser *Parser, reducer Reducer) interface{}
• type Entry
  • func (entry *Entry) Get(name string) (value string, err error)
• type Parser
  • func NewNginxParser(conf io.Reader, name string) (parser *Parser, err error)
  • func NewParser(format string) *Parser
  • func (parser *Parser) ParseString(line string) (entry Entry, err error)
• type ReadAll
  • func (r *ReadAll) Reduce(input chan Entry, output chan interface{})
• type Reader
  • func NewNginxReader(logFile io.Reader, nginxConf io.Reader, formatName string) (reader *Reader, err error)
  • func NewReader(logFile io.Reader, format string) *Reader
  • func (r *Reader) Read() (entry Entry, err error)
• type Reducer

Functions

func MapReduce

func MapReduce(file io.Reader, parser *Parser, reducer Reducer) interface{}

Iterate over given file and map each it's line into Entry record using parser and apply reducer to the Entries channel. Execution terminates when result will be readed from reducer's output channel, but the mapper works and fills input Entries channel until all lines will be read from the fiven file.

Types

type Entry

type Entry map[string]string

Parsed log record. Use Get method to retrieve a value by name instead of threating this as a map, because inner representation is in design.

func (*Entry) Get

func (entry *Entry) Get(name string) (value string, err error)

Return entry field value by name or empty string and error if it does not exist.

type Parser

type Parser struct {
	// contains filtered or unexported fields
}

Log record parser. Use specific constructors to initialize it.

func NewNginxParser

func NewNginxParser(conf io.Reader, name string) (parser *Parser, err error)

NewNginxParser parse nginx conf file to find log_format with given name and returns parser for this format. It returns an error if cannot find the needle.

func NewParser

func NewParser(format string) *Parser

Returns a new Parser, use given log format to create its internal strings parsing regexp.

func (*Parser) ParseString

func (parser *Parser) ParseString(line string) (entry Entry, err error)

Parse log file line using internal format regexp. If line do not match given format an error will be returned.

type ReadAll

type ReadAll struct {
}

Implements Reducer interface for simple input entries redirection to the output channel.

func (*ReadAll) Reduce

func (r *ReadAll) Reduce(input chan Entry, output chan interface{})

Redirect input Entries channel directly to the output without any modifications. It is useful when you want jast to read file fast using asynchronous with mapper routines.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Log file reader. Use specific constructors to create it.

func NewNginxReader

func NewNginxReader(logFile io.Reader, nginxConf io.Reader, formatName string) (reader *Reader, err error)

Creates reader for nginx log format. Nginx config parser will be used to get particular format from the conf file.

func NewReader

func NewReader(logFile io.Reader, format string) *Reader

Creates reader for custom log format.

func (*Reader) Read

func (r *Reader) Read() (entry Entry, err error)

Get next parsed Entry from the log file. Return EOF if there is no Entries to read.

type Reducer

type Reducer interface {
	Reduce(input chan Entry, output chan interface{})
}

Reducer interface for Entries channel redure.

Each Reduce method should accept input channel of Entries, do it's job and the result should be written to the output channel.

It does not return values because usually it runs in a separate goroutine and it is handy to use channel for reduced data retrieval.