tokens

Documentation

Overview

Package tokens contains the token definitions for the tokenizer.

Index

• type Token

Types

type Token

type Token byte

Token is the type that represents a single token.

const (
	Space Token = iota

	// Special Characters
	Colon        // :
	Semicolon    // ;
	Dash         // -
	Underscore   // _
	Fslash       // /
	Bslash       // \
	Period       // .
	Comma        // ,
	Singlequote  // '
	Doublequote  // "
	Backtick     // `
	Tilda        // ~
	Star         // *
	Plus         // +
	Equal        // =
	Parenopen    // (
	Parenclose   // )
	Braceopen    // {
	Braceclose   // }
	Bracketopen  // [
	Bracketclose // ]
	Ampersand    // &
	Exclamation  // !
	At           // @
	Pound        // #
	Dollar       // $
	Percent      // %
	Uparrow      // ^

	// Digit runs
	D1
	D2
	D3
	D4
	D5
	D6
	D7
	D8
	D9
	D10

	// Char runs
	C1
	C2
	C3
	C4
	C5
	C6
	C7
	C8
	C9
	C10

	// Special tokens
	Month
	Day
	Apm  // am or pm
	Zone // Represents a timezone
	T    // t (often `T`) denotes a time separator in many timestamp formats

	End // Not a valid token. Used to mark the end of the token list or as a terminator.
)

Disable linter since the token list is self explanatory, or documented where needed.