parser

README

Zed parser

This directory contains the Zed parser implemented in PEG.

There is a single PEG input file that works with pigeon to generate the Go parser.

Build

To build the parser, just run make:

make

This will ensure the required libraries are installed and then produce the Go parser (parser.go).

Testing

The zed dev compile command can be used for easily testing the output of the Zed parser.

Development

During development, the easiest way to run the parser is with this make command at the root of this repository:

make peg

This will ensure the PEG-generated Go parser is up to date with parser.peg

To update the parser and launch the zc -repl, your can run make peg-run.

Documentation

Index

• func Parse(filename string, b []byte, opts ...Option) (any, error)
• func ParseFile(filename string, opts ...Option) (i any, err error)
• func ParseReader(filename string, r io.Reader, opts ...Option) (any, error)
• type Cloner
• type Error
  • func (e *Error) Error() string
• type ErrorList
  • func (e *ErrorList) Append(msg string, pos, end int)
  • func (e ErrorList) Error() string
  • func (e ErrorList) SetSourceSet(sset *SourceSet)
• type Option
  • func AllowInvalidUTF8(b bool) Option
  • func Debug(b bool) Option
  • func Entrypoint(ruleName string) Option
  • func GlobalStore(key string, value any) Option
  • func InitState(key string, value any) Option
  • func MaxExpressions(maxExprCnt uint64) Option
  • func Memoize(b bool) Option
  • func Recover(b bool) Option
  • func Statistics(stats *Stats, choiceNoMatch string) Option
• type Position
  • func (p Position) IsValid() bool
• type SourceInfo
  • func (s *SourceInfo) LineOfPos(src string, pos int) string
  • func (s *SourceInfo) Position(pos int) Position
• type SourceSet
  • func ConcatSource(filenames []string, src string) (*SourceSet, error)
  • func ParseZed(filenames []string, src string) (ast.Seq, *SourceSet, error)
  • func (s *SourceSet) SourceOf(pos int) *SourceInfo
• type Stats

Functions

func Parse

func Parse(filename string, b []byte, opts ...Option) (any, error)

Parse parses the data from b using filename as information in the error messages.

func ParseFile

func ParseFile(filename string, opts ...Option) (i any, err error)

ParseFile parses the file identified by filename.

func ParseReader

func ParseReader(filename string, r io.Reader, opts ...Option) (any, error)

ParseReader parses the data from r using filename as information in the error messages.

Types

type Cloner

type Cloner interface {
	Clone() any
}

Cloner is implemented by any value that has a Clone method, which returns a copy of the value. This is mainly used for types which are not passed by value (e.g map, slice, chan) or structs that contain such types.

This is used in conjunction with the global state feature to create proper copies of the state to allow the parser to properly restore the state in the case of backtracking.

type Error

type Error struct {
	Msg string
	Pos int
	End int
	// contains filtered or unexported fields
}

func (*Error) Error

func (e *Error) Error() string

type ErrorList

type ErrorList []*Error

ErrList is a list of Errors.

func (*ErrorList) Append

func (e *ErrorList) Append(msg string, pos, end int)

Append appends an Error to e.

func (ErrorList) Error

func (e ErrorList) Error() string

Error concatenates the errors in e with a newline between each.

func (ErrorList) SetSourceSet

func (e ErrorList) SetSourceSet(sset *SourceSet)

SetSourceSet sets the SourceSet for every Error in e.

type Option

type Option func(*parser) Option

Option is a function that can set an option on the parser. It returns the previous setting as an Option.

func AllowInvalidUTF8

func AllowInvalidUTF8(b bool) Option

AllowInvalidUTF8 creates an Option to allow invalid UTF-8 bytes. Every invalid UTF-8 byte is treated as a utf8.RuneError (U+FFFD) by character class matchers and is matched by the any matcher. The returned matched value, c.text and c.offset are NOT affected.

The default is false.

func Debug

func Debug(b bool) Option

Debug creates an Option to set the debug flag to b. When set to true, debugging information is printed to stdout while parsing.

The default is false.

func Entrypoint

func Entrypoint(ruleName string) Option

Entrypoint creates an Option to set the rule name to use as entrypoint. The rule name must have been specified in the -alternate-entrypoints if generating the parser with the -optimize-grammar flag, otherwise it may have been optimized out. Passing an empty string sets the entrypoint to the first rule in the grammar.

The default is to start parsing at the first rule in the grammar.

func GlobalStore

func GlobalStore(key string, value any) Option

GlobalStore creates an Option to set a key to a certain value in the globalStore.

func InitState

func InitState(key string, value any) Option

InitState creates an Option to set a key to a certain value in the global "state" store.

func MaxExpressions

func MaxExpressions(maxExprCnt uint64) Option

MaxExpressions creates an Option to stop parsing after the provided number of expressions have been parsed, if the value is 0 then the parser will parse for as many steps as needed (possibly an infinite number).

The default for maxExprCnt is 0.

func Memoize

func Memoize(b bool) Option

Memoize creates an Option to set the memoize flag to b. When set to true, the parser will cache all results so each expression is evaluated only once. This guarantees linear parsing time even for pathological cases, at the expense of more memory and slower times for typical cases.

The default is false.

func Recover

func Recover(b bool) Option

Recover creates an Option to set the recover flag to b. When set to true, this causes the parser to recover from panics and convert it to an error. Setting it to false can be useful while debugging to access the full stack trace.

The default is true.

func Statistics

func Statistics(stats *Stats, choiceNoMatch string) Option

Statistics adds a user provided Stats struct to the parser to allow the user to process the results after the parsing has finished. Also the key for the "no match" counter is set.

Example usage:

input := "input"
stats := Stats{}
_, err := Parse("input-file", []byte(input), Statistics(&stats, "no match"))
if err != nil {
    log.Panicln(err)
}
b, err := json.MarshalIndent(stats.ChoiceAltCnt, "", "  ")
if err != nil {
    log.Panicln(err)
}
fmt.Println(string(b))

type Position

type Position struct {
	Pos    int `json:"pos"`    // Offset relative to SourceSet.
	Offset int `json:"offset"` // Offset relative to file start.
	Line   int `json:"line"`   // 1-based line number.
	Column int `json:"column"` // 1-based column number.
}

func (Position) IsValid

func (p Position) IsValid() bool

type SourceInfo

type SourceInfo struct {
	Filename string
	// contains filtered or unexported fields
}

SourceInfo holds source file offsets.

func (*SourceInfo) LineOfPos

func (s *SourceInfo) LineOfPos(src string, pos int) string

func (*SourceInfo) Position

func (s *SourceInfo) Position(pos int) Position

type SourceSet

type SourceSet struct {
	Text    string
	Sources []*SourceInfo
}

func ConcatSource

func ConcatSource(filenames []string, src string) (*SourceSet, error)

ConcatSource concatenates the source files in filenames followed by src, returning a SourceSet.

func ParseZed

func ParseZed(filenames []string, src string) (ast.Seq, *SourceSet, error)

ParseZed calls ConcatSource followed by Parse. If Parse returns an error, ConcatSource tries to convert it to an ErrorList.

func (*SourceSet) SourceOf

func (s *SourceSet) SourceOf(pos int) *SourceInfo

type Stats

type Stats struct {
	// ExprCnt counts the number of expressions processed during parsing
	// This value is compared to the maximum number of expressions allowed
	// (set by the MaxExpressions option).
	ExprCnt uint64

	// ChoiceAltCnt is used to count for each ordered choice expression,
	// which alternative is used how may times.
	// These numbers allow to optimize the order of the ordered choice expression
	// to increase the performance of the parser
	//
	// The outer key of ChoiceAltCnt is composed of the name of the rule as well
	// as the line and the column of the ordered choice.
	// The inner key of ChoiceAltCnt is the number (one-based) of the matching alternative.
	// For each alternative the number of matches are counted. If an ordered choice does not
	// match, a special counter is incremented. The name of this counter is set with
	// the parser option Statistics.
	// For an alternative to be included in ChoiceAltCnt, it has to match at least once.
	ChoiceAltCnt map[string]map[string]int
}

Stats stores some statistics, gathered during parsing