util

Documentation

Overview

Package util provides some helpful functions.

Index

• type MultiToken
  • func NewMultiToken(s string) MultiToken
  • func (m MultiToken) ScanToken(rd *bufio.Reader) (rns []rune, err error)

Examples

• MultiToken.ScanToken

Types

type MultiToken

type MultiToken []rune

MultiToken represents a run of runes that can be used to tokenize a stream of runes.

func NewMultiToken

func NewMultiToken(s string) MultiToken

func (MultiToken) ScanToken

func (m MultiToken) ScanToken(rd *bufio.Reader) (rns []rune, err error)

ScanToken returns the run of runes terminated by m, not including the terminator.

Example

const input = "hi\r\nhow\r\nare\r\nyou"
var tok = MultiToken([]rune{'\r', '\n'})

r := bufio.NewReader(strings.NewReader(input))

for {
	rs, err := tok.ScanToken(r)

	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Printf("Got input: %s\n", string(rs))
}

Output:

Got input: hi
Got input: how
Got input: are
EOF