tokenizer

Documentation

Index

• type SimpleTokenizer
  • func NewSimpleTokenizer() *SimpleTokenizer
  • func (x *SimpleTokenizer) DisableRegex()
  • func (x *SimpleTokenizer) EnableRegex()
  • func (x *SimpleTokenizer) SetDelim(d string)
  • func (x *SimpleTokenizer) Split(msg string) []*Token
• type Token
  • func (x *Token) IsSpace() bool
• type Tokenizer

Types

type SimpleTokenizer

type SimpleTokenizer struct {
	// contains filtered or unexported fields
}

SimpleTokenizer is one of implementation for Tokenizer

func NewSimpleTokenizer

func NewSimpleTokenizer() *SimpleTokenizer

NewSimpleTokenizer is a constructor of SimpleTokenizer

func (*SimpleTokenizer) DisableRegex

func (x *SimpleTokenizer) DisableRegex()

DisableRegex is disabler of heuristics patterns

func (*SimpleTokenizer) EnableRegex

func (x *SimpleTokenizer) EnableRegex()

EnableRegex is disabler of heuristics patterns

func (*SimpleTokenizer) SetDelim

func (x *SimpleTokenizer) SetDelim(d string)

SetDelim is a function set characters as delimiter

func (*SimpleTokenizer) Split

func (x *SimpleTokenizer) Split(msg string) []*Token

Split is a function to split log message.

type Token

type Token struct {
	Data    string
	IsDelim bool
	// contains filtered or unexported fields
}

Token is a part of log message

func (*Token) IsSpace

func (x *Token) IsSpace() bool

IsSpace checks all runes in string

type Tokenizer

type Tokenizer interface {
	Split(msg string) []*Token
}

Tokenizer splits log message string