tokenizer

package
v0.0.0-...-710d95f Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 16, 2023 License: MIT Imports: 5 Imported by: 0

Documentation

Overview

Package tokenizer implements a rudimentary tokens parser of buffered io.Reader while respecting quotes and parenthesis boundaries.

Example

tk := tokenizer.NewFromString("a, b, (c, d)")
result, _ := tk.ScanAll() // ["a", "b", "(c, d)"]

Index

Constants

This section is empty.

Variables

View Source
var DefaultSeparators = []rune{','}

DefaultSeparators is a list with the default token separator characters.

Functions

This section is empty.

Types

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer defines a struct that parses a reader into tokens while respecting quotes and parenthesis boundaries.

func New

func New(r io.Reader) *Tokenizer

New creates new Tokenizer from the provided reader with DefaultSeparators.

func NewFromBytes

func NewFromBytes(b []byte) *Tokenizer

NewFromBytes creates new Tokenizer from the provided bytes slice.

func NewFromString

func NewFromString(str string) *Tokenizer

NewFromString creates new Tokenizer from the provided string.

func (*Tokenizer) KeepSeparator

func (s *Tokenizer) KeepSeparator(state bool)

KeepSeparator defines whether to keep the separator rune as part of the token (default to false).

func (*Tokenizer) Scan

func (s *Tokenizer) Scan() (string, error)

Scan reads and returns the next available token from the Tokenizer's buffer (trimmed).

Returns io.EOF error when there are no more tokens to scan.

func (*Tokenizer) ScanAll

func (s *Tokenizer) ScanAll() ([]string, error)

ScanAll reads the entire Tokenizer's buffer and return all found tokens.

func (*Tokenizer) Separators

func (s *Tokenizer) Separators(separators ...rune)

Separators defines the provided separatos of the current Tokenizer.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL