tokenizer

Documentation

Index

• type Policy
  • func NewPolicy(policyType PolicyType, values ...string) *Policy
  • func (p *Policy) Allow(value string) bool
• type PolicyType
• type Spec
  • func NewSpec(expression string, tokenType Type) *Spec
• type Token
  • func NewToken(tokenType Type, value string) *Token
• type Tokenizer
  • func NewTokenizer(query string, skipTokenType, policyCheckType Type, spec []*Spec, ...) *Tokenizer
  • func (t *Tokenizer) GetCursorPosition() int
  • func (t *Tokenizer) GetNextToken() (*Token, error)
  • func (t *Tokenizer) HasMoreTokens() bool
• type Type
  • func (t Type) String() string

Types

type Policy

type Policy struct {
	// contains filtered or unexported fields
}

Policy handles policy checks based on configuration.

func NewPolicy

func NewPolicy(policyType PolicyType, values ...string) *Policy

NewPolicy create a new policy instance with given arguments.

func (*Policy) Allow

func (p *Policy) Allow(value string) bool

Allow check if a value is allowed.

type PolicyType

type PolicyType byte

PolicyType represent policy type values.

const (
	// WhitelistPolicy declare a policy
	// to use whitelist logic.
	WhitelistPolicy PolicyType = 0
	// BlacklistPolicy declare a policy
	// to use blacklist logic.
	BlacklistPolicy PolicyType = 1
)

type Spec

type Spec struct {
	// contains filtered or unexported fields
}

spec for the tokenizer.

func NewSpec

func NewSpec(expression string, tokenType Type) *Spec

NewSpec creates a new spec with given arguments.

type Token

type Token struct {
	Type  Type
	Value string
}

Token represents a single token.

func NewToken

func NewToken(tokenType Type, value string) *Token

NewToken creates a new token with given arguments.

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

tokenizer that lazily pulls a token from a stream.

func NewTokenizer

func NewTokenizer(query string, skipTokenType, policyCheckType Type, spec []*Spec, policy *Policy) *Tokenizer

NewTokenizer create a new tokenizer instance with given parameters.

func (*Tokenizer) GetCursorPosition

func (t *Tokenizer) GetCursorPosition() int

GetCursorPosition return the position of the cursor.

func (*Tokenizer) GetNextToken

func (t *Tokenizer) GetNextToken() (*Token, error)

GetNextToken obtains next token.

func (*Tokenizer) HasMoreTokens

func (t *Tokenizer) HasMoreTokens() bool

HasMoreTokens checks aether we still have more tokens.

type Type

type Type string

TokenType is the type of the token.

func (Type) String

func (t Type) String() string

String return as string.