scanner

Documentation

Index

• type Scanner
  • func New(input string) *Scanner
  • func (s *Scanner) NextToken(ctx context.Context) token.Token

Types

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner converts a sequence of characters into a sequence of tokens.

func New

func New(input string) *Scanner

New returns new Scanner.

func (*Scanner) NextToken

func (s *Scanner) NextToken(ctx context.Context) token.Token

NextToken scans the next token and returns the token position, the token, and its literal string if applicable. The source end is indicated by token.EOF.

If the returned token is a literal (token.Identifier, token.Value, token.RawValue) or token.Comment, the literal string has the corresponding value.

If the returned token is token.Illegal, the literal string is the offending character.