parse

package
v1.0.28 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 28, 2023 License: BSD-3-Clause Imports: 15 Imported by: 3

Documentation

Overview

Package parse does the parsing stage after lexing

Package parse does the parsing stage after lexing, using a top-down recursive-descent (TDRD) strategy, with a special reverse mode to deal with left-associative binary expressions which otherwise end up being right-associative for TDRD parsing. Higher-level rules provide scope to lower-level ones, with a special EOS end-of-statement scope recognized for

Index

Constants

This section is empty.

Variables

View Source 
var AstProps = ki.Props{
	"EnumType:Flag": ki.KiT_Flags,
	"StructViewFields": ki.Props{
		"Flag":  `view:"-"`,
		"Props": `view:"-"`,
	},
}
View Source 
var DepthLimit = 10000

DepthLimit is the infinite recursion prevention cutoff

View Source 
var GuiActive = false

Set GuiActive to true if the gui (piview) is active -- ensures that the Ast tree is updated when nodes are swapped in reverse mode, and maybe other things

View Source 
var KiT_Actions = kit.Enums.AddEnum(ActionsN, kit.NotBitFlag, nil)
View Source 
var KiT_Ast = kit.Types.AddType(&Ast{}, AstProps)
View Source 
var KiT_AstActs = kit.Enums.AddEnum(AstActsN, kit.NotBitFlag, nil)
View Source 
var KiT_Rule = kit.Types.AddType(&Rule{}, RuleProps)
View Source 
var KiT_Steps = kit.Enums.AddEnum(StepsN, kit.NotBitFlag, nil)
View Source 
var RuleMap map[string]*Rule

RuleMap is a map of all the rule names, for quick lookup

View Source 
var RuleProps = ki.Props{
	"EnumType:Flag": ki.KiT_Flags,
}

Functions

This section is empty.

Types

type Act added in v0.5.5 

type Act struct {

	// at what point during sequence of sub-rules / tokens should this action be run?  -1 = at end, 0 = before first rule, 1 = before second rule, etc -- must be at point when relevant Ast nodes have been added, but for scope setting, must be early enough so that scope is present
	RunIdx int `` /* 280-byte string literal not displayed */

	// what action to perform
	Act Actions `desc:"what action to perform"`

	// Ast path, relative to current node: empty = current node; [idx] specifies a child node by index, and a name specifies it by name -- include name/name for sub-nodes etc -- multiple path options can be specified by | or & and will be tried in order until one succeeds (for |) or all that succeed will be used for &. ... means use all nodes with given name (only for change token) -- for PushStack, this is what to push on the stack
	Path string `` /* 447-byte string literal not displayed */

	// for ChgToken, the new token type to assign to token at given path
	Tok token.Tokens `desc:"for ChgToken, the new token type to assign to token at given path"`

	// for ChgToken, only change if token is this to start with (only if != None))
	FmTok token.Tokens `desc:"for ChgToken, only change if token is this to start with (only if != None))"`
}

Act is one action to perform, operating on the Ast output

func (*Act) ChgTok added in v0.5.5 

func (ac *Act) ChgTok(lx *lex.Lex)

ChgTok changes the token type, using FmTok logic

func (Act) String added in v0.5.5 

func (ac Act) String() string

String satisfies fmt.Stringer interface

type Actions 

type Actions int

Actions are parsing actions to perform 

const (
	// ChgToken changes the token to the Tok specified in the Act action
	ChgToken Actions = iota

	// AddSymbol means add name as a symbol, using current scoping and token type
	// or the token specified in the Act action if != None
	AddSymbol

	// PushScope means look for an existing symbol of given name
	// to push onto current scope -- adding a new one if not found --
	// does not add new item to overall symbol list.  This is useful
	// for e.g., definitions of methods on a type, where this is not
	// the definition of the type itself.
	PushScope

	// PushNewScope means add a new symbol to the list and also push
	// onto scope stack, using given token type or the token specified
	// in the Act action if != None
	PushNewScope

	// PopScope means remove the most recently-added scope item
	PopScope

	// PopScopeReg means remove the most recently-added scope item, and also
	// updates the source region for that item based on final SrcReg from
	// corresponding Ast node -- for "definitional" scope
	PopScopeReg

	// AddDetail adds src at given path as detail info for the last-added symbol
	// if there is already something there, a space is added for this new addition
	AddDetail

	// AddType Adds a type with the given name -- sets the Ast node for this rule
	// and actual type is resolved later in a second language-specific pass
	AddType

	// PushStack adds name to stack -- provides context-sensitivity option for
	// optimizing and ambiguity resolution
	PushStack

	// PopStack pops the stack
	PopStack

	ActionsN
)

The parsing acts

func (*Actions) FromString 

func (i *Actions) FromString(s string) error

func (Actions) MarshalJSON 

func (ev Actions) MarshalJSON() ([]byte, error)

func (Actions) String 

func (i Actions) String() string

func (*Actions) UnmarshalJSON 

func (ev *Actions) UnmarshalJSON(b []byte) error

type Acts added in v0.5.5 

type Acts []Act

Acts are multiple actions

func (Acts) String added in v0.5.5 

func (ac Acts) String() string

String satisfies fmt.Stringer interface

type Ast 

type Ast struct {
	ki.Node

	// region in source lexical tokens corresponding to this Ast node -- Ch = index in lex lines
	TokReg lex.Reg `desc:"region in source lexical tokens corresponding to this Ast node -- Ch = index in lex lines"`

	// region in source file corresponding to this Ast node
	SrcReg lex.Reg `desc:"region in source file corresponding to this Ast node"`

	// source code corresponding to this Ast node
	Src string `desc:"source code corresponding to this Ast node"`

	// stack of symbols created for this node
	Syms syms.SymStack `desc:"stack of symbols created for this node"`
}

Ast is a node in the abstract syntax tree generated by the parsing step the name of the node (from ki.Node) is the type of the element (e.g., expr, stmt, etc) These nodes are generated by the parse.Rule's by matching tokens

func (*Ast) ChildAst added in v0.5.5 

func (ast *Ast) ChildAst(idx int) *Ast

ChildAst returns the Child at given index as an Ast. Will panic if index is invalid -- use Try if unsure.

func (*Ast) ChildAstTry added in v0.5.7 

func (ast *Ast) ChildAstTry(idx int) (*Ast, error)

ChildAstTry returns the child at given index as an Ast -- error if not valid

func (*Ast) NextAst added in v0.9.11 

func (ast *Ast) NextAst() *Ast

NextAst returns the next node in the Ast tree, or nil if none

func (*Ast) NextSiblingAst added in v0.9.11 

func (ast *Ast) NextSiblingAst() *Ast

NextSiblingAst returns the next sibling node in the Ast tree, or nil if none

func (*Ast) ParAst added in v0.9.11 

func (ast *Ast) ParAst() *Ast

ParAst returns the Parent as an Ast.

func (*Ast) PrevAst added in v0.9.11 

func (ast *Ast) PrevAst() *Ast

PrevAst returns the previous node in the Ast tree, or nil if none

func (*Ast) SetTokReg 

func (ast *Ast) SetTokReg(reg lex.Reg, src *lex.File)

SetTokReg sets the token region for this rule to given region

func (*Ast) SetTokRegEnd 

func (ast *Ast) SetTokRegEnd(pos lex.Pos, src *lex.File)

SetTokRegEnd updates the ending token region to given position -- token regions are typically over-extended and get narrowed as tokens actually match

func (*Ast) WriteTree added in v0.5.5 

func (ast *Ast) WriteTree(out io.Writer, depth int)

WriteTree writes the AST tree data to the writer -- not attempting to re-render source code -- just for debugging etc

type AstActs 

type AstActs int

AstActs are actions to perform on the Ast nodes 

const (
	// NoAst means don't create an Ast node for this rule
	NoAst AstActs = iota

	// AddAst means create an Ast node for this rule, adding it to the current anchor Ast.
	// Any sub-rules within this rule are *not* added as children of this node -- see
	// SubAst and AnchorAst.  This is good for token-only terminal nodes and list elements
	// that should be added to a list.
	AddAst

	// SubAst means create an Ast node and add all the elements of *this rule* as
	// children of this new node (including sub-rules), *except* for the very last rule
	// which is assumed to be a recursive rule -- that one goes back up to the parent node.
	// This is good for adding more complex elements with sub-rules to a recursive list,
	// without creating a new hierarchical depth level for every such element.
	SubAst

	// AnchorAst means create an Ast node and set it as the anchor that subsequent
	// sub-nodes are added into.  This is for a new hierarchical depth level
	// where everything under this rule gets organized.
	AnchorAst

	// AnchorFirstAst means create an Ast node and set it as the anchor that subsequent
	// sub-nodes are added into, *only* if this is the first time that this rule has
	// matched within the current sequence (i.e., if the parent of this rule is the same
	// rule then don't add a new Ast node).  This is good for starting a new list
	// of recursively-defined elements, without creating increasing depth levels.
	AnchorFirstAst

	AstActsN
)

The Ast actions

func (*AstActs) FromString 

func (i *AstActs) FromString(s string) error

func (AstActs) MarshalJSON 

func (ev AstActs) MarshalJSON() ([]byte, error)

func (AstActs) String 

func (i AstActs) String() string

func (*AstActs) UnmarshalJSON 

func (ev *AstActs) UnmarshalJSON(b []byte) error

type MatchStack 

type MatchStack []MatchState

MatchStack is the stack of rules that matched or ran for each token point

func (*MatchStack) Add 

func (rs *MatchStack) Add(pr *Rule, scope lex.Reg, regs Matches)

Add given rule to stack

func (*MatchStack) Find 

func (rs *MatchStack) Find(pr *Rule, scope lex.Reg) (*MatchState, bool)

Find looks for given rule and scope on the stack

type MatchState 

type MatchState struct {

	// rule that either matched or ran here
	Rule *Rule `desc:"rule that either matched or ran here"`

	// scope for match
	Scope lex.Reg `desc:"scope for match"`

	// regions of match for each sub-region
	Regs Matches `desc:"regions of match for each sub-region"`
}

MatchState holds state info for rules that matched, recorded at starting position of match

func (MatchState) String 

func (rs MatchState) String() string

String is fmt.Stringer

type Matches 

type Matches []lex.Reg

Matches encodes the regions of each match, Err for no match

func (Matches) StartEnd 

func (mm Matches) StartEnd() lex.Reg

StartEnd returns the first and last non-zero positions in the Matches list as a region

func (Matches) StartEndExcl added in v0.5.5 

func (mm Matches) StartEndExcl(ps *State) lex.Reg

StartEndExcl returns the first and last non-zero positions in the Matches list as a region moves the end to next toke to make it the usual exclusive end pos

type Parser 

type Parser interface {
	ki.Ki

	// Compile compiles string rules into their runnable elements
	Compile(ps *State) bool

	// Validate checks for any errors in the rules and issues warnings,
	// returns true if valid (no err) and false if invalid (errs)
	Validate(ps *State) bool

	// Parse tries to apply rule to given input state, returns rule that matched or nil
	// par is the parent rule that we're being called from
	// ast is the current ast node that we add to
	Parse(ps *State, par *Rule, ast *Ast, scope lex.Reg, optMap lex.TokenMap, depth int) *Rule

	// AsParseRule returns object as a parse.Rule
	AsParseRule() *Rule
}

Parser is the interface type for parsers -- likely not necessary except is essential for defining the BaseIface for gui in making new nodes

type Rule 

type Rule struct {
	ki.Node

	// disable this rule -- useful for testing and exploration
	Off bool `desc:"disable this rule -- useful for testing and exploration"`

	// description / comments about this rule
	Desc string `desc:"description / comments about this rule"`

	// the rule as a space-separated list of rule names and token(s) -- use single quotes around 'tokens' (using token.Tokens names or symbols). For keywords use 'key:keyword'.  All tokens are matched at the same nesting depth as the start of the scope of this rule, unless they have a +D relative depth value differential before the token.  Use @ prefix for a sub-rule to require that rule to match -- by default explicit tokens are used if available, and then only the first sub-rule failing that.  Use ! by itself to define start of an exclusionary rule -- doesn't match when those rule elements DO match.  Use : prefix for a special group node that matches a single token at start of scope, and then defers to the child rules to perform full match -- this is used for FirstTokMap when there are multiple versions of a given keyword rule.  Use - prefix for tokens anchored by the end (next token) instead of the previous one -- typically just for token prior to 'EOS' but also a block of tokens that need to go backward in the middle of a sequence to avoid ambiguity can be marked with -
	Rule string `` /* 1090-byte string literal not displayed */

	// if present, this rule only fires if stack has this on it
	StackMatch string `desc:"if present, this rule only fires if stack has this on it"`

	// what action should be take for this node when it matches
	Ast AstActs `desc:"what action should be take for this node when it matches"`

	// actions to perform based on parsed Ast tree data, when this rule is done executing
	Acts Acts `desc:"actions to perform based on parsed Ast tree data, when this rule is done executing"`

	// for group-level rules having lots of children and lots of recursiveness, and also of high-frequency, when we first encounter such a rule, make a map of all the tokens in the entire scope, and use that for a first-pass rejection on matching tokens
	OptTokMap bool `` /* 253-byte string literal not displayed */

	// for group-level rules with a number of rules that match based on first tokens / keywords, build map to directly go to that rule -- must also organize all of these rules sequentially from the start -- if no match, goes directly to first non-lookup case
	FirstTokMap bool `` /* 258-byte string literal not displayed */

	// rule elements compiled from Rule string
	Rules RuleList `json:"-" xml:"-" desc:"rule elements compiled from Rule string"`

	// strategic matching order for matching the rules
	Order []int `inactive:"+" json:"-" xml:"-" desc:"strategic matching order for matching the rules"`

	// map from first tokens / keywords to rules for FirstTokMap case
	FiTokMap map[string]*Rule `inactive:"+" json:"-" xml:"-" desc:"map from first tokens / keywords to rules for FirstTokMap case"`

	// for FirstTokMap, the start of the else cases not covered by the map
	FiTokElseIdx int `inactive:"+" json:"-" xml:"-" desc:"for FirstTokMap, the start of the else cases not covered by the map"`

	// exclusionary key index -- this is the token in Rules that we need to exclude matches for using ExclFwd and ExclRev rules
	ExclKeyIdx int `` /* 157-byte string literal not displayed */

	// exclusionary forward-search rule elements compiled from Rule string
	ExclFwd RuleList `inactive:"+" json:"-" xml:"-" desc:"exclusionary forward-search rule elements compiled from Rule string"`

	// exclusionary reverse-search rule elements compiled from Rule string
	ExclRev RuleList `inactive:"+" json:"-" xml:"-" desc:"exclusionary reverse-search rule elements compiled from Rule string"`
}

The first step is matching which searches in order for matches within the children of parent nodes, and for explicit rule nodes, it looks first through all the explicit tokens in the rule. If there are no explicit tokens then matching defers to ONLY the first node listed by default -- you can add a @ prefix to indicate a rule that is also essential to match.

After a rule matches, it then proceeds through the rules narrowing the scope and calling the sub-nodes..

func (*Rule) AsParseRule 

func (pr *Rule) AsParseRule() *Rule

func (*Rule) BaseIface 

func (pr *Rule) BaseIface() reflect.Type

func (*Rule) Compile 

func (pr *Rule) Compile(ps *State) bool

Compile compiles string rules into their runnable elements. Returns true if everything is ok, false if there were compile errors.

func (*Rule) CompileAll 

func (pr *Rule) CompileAll(ps *State) bool

CompileAll is called on the top-level Rule to compile all nodes it calls SetRuleMap first. Returns true if everything is ok, false if there were compile errors

func (*Rule) CompileExcl 

func (pr *Rule) CompileExcl(ps *State, rs []string, rist int) bool

CompileExcl compiles exclusionary rules starting at given point currently only working for single-token matching rule

func (*Rule) CompileTokMap added in v0.5.5 

func (pr *Rule) CompileTokMap(ps *State) bool

CompileTokMap compiles first token map

func (*Rule) DoAct added in v0.5.5 

func (pr *Rule) DoAct(ps *State, act *Act, par *Rule, ourAst, parAst *Ast) bool

DoAct performs one action after a rule executes

func (*Rule) DoActs added in v0.5.5 

func (pr *Rule) DoActs(ps *State, ri int, par *Rule, ourAst, parAst *Ast) bool

DoActs performs actions at given point in rule execution (ri = rule index, is -1 at end)

func (*Rule) DoRules 

func (pr *Rule) DoRules(ps *State, par *Rule, parAst *Ast, scope lex.Reg, mpos Matches, optMap lex.TokenMap, depth int) bool

DoRules after we have matched, goes through rest of the rules -- returns false if there were any issues encountered

func (*Rule) DoRulesRevBinExp 

func (pr *Rule) DoRulesRevBinExp(ps *State, par *Rule, parAst *Ast, scope lex.Reg, mpos Matches, ourAst *Ast, optMap lex.TokenMap, depth int) bool

DoRulesRevBinExp reverse version of do rules for binary expression rule with one key token in the middle -- we just pay attention to scoping rest of sub-rules relative to that, and don't otherwise adjust scope or position. In particular all the position updating taking place in sup-rules is then just ignored and we set the position to the end position matched by the "last" rule (which was the first processed)

func (*Rule) Find 

func (pr *Rule) Find(find string) []*Rule

Find looks for rules in the tree that contain given string in Rule or Name fields

func (*Rule) IsGroup 

func (pr *Rule) IsGroup() bool

IsGroup returns true if this node is a group, else it should have rules

func (*Rule) Match 

func (pr *Rule) Match(ps *State, parAst *Ast, scope lex.Reg, depth int, optMap lex.TokenMap) (bool, lex.Reg, Matches)

Match attempts to match the rule, returns true if it matches, and the match positions, along with any update to the scope

func (*Rule) MatchExclude 

func (pr *Rule) MatchExclude(ps *State, scope lex.Reg, ktpos lex.Reg, depth int, optMap lex.TokenMap) bool

MatchExclude looks for matches of exclusion tokens -- if found, they exclude this rule return is true if exclude matches and rule should be excluded

func (*Rule) MatchGroup added in v0.5.5 

func (pr *Rule) MatchGroup(ps *State, parAst *Ast, scope lex.Reg, depth int, optMap lex.TokenMap) (bool, lex.Reg, Matches)

MatchGroup does matching for Group rules

func (*Rule) MatchMixed added in v0.5.5 

func (pr *Rule) MatchMixed(ps *State, parAst *Ast, scope lex.Reg, depth int, optMap lex.TokenMap) (bool, Matches)

MatchMixed matches mixed tokens and non-tokens

func (*Rule) MatchNoToks added in v0.5.5 

func (pr *Rule) MatchNoToks(ps *State, parAst *Ast, scope lex.Reg, depth int, optMap lex.TokenMap) (bool, Matches)

MatchNoToks matches NoToks case -- just does single sub-rule match

func (*Rule) MatchOnlyToks added in v0.5.5 

func (pr *Rule) MatchOnlyToks(ps *State, parAst *Ast, scope lex.Reg, depth int, optMap lex.TokenMap) (bool, Matches)

MatchOnlyToks matches rules having only tokens

func (*Rule) MatchToken added in v0.5.5 

func (pr *Rule) MatchToken(ps *State, rr *RuleEl, ri int, kt token.KeyToken, creg *lex.Reg, mpos Matches, parAst *Ast, scope lex.Reg, depth int, optMap lex.TokenMap) (bool, lex.Pos)

MatchToken matches one token sub-rule -- returns true for match and false if no match -- and the position where it was / should have been

func (*Rule) OptimizeOrder added in v0.5.5 

func (pr *Rule) OptimizeOrder(ps *State)

OptimizeOrder optimizes the order of processing rule elements, including: * A block of reversed elements that match from next

func (*Rule) Parse 

func (pr *Rule) Parse(ps *State, par *Rule, parAst *Ast, scope lex.Reg, optMap lex.TokenMap, depth int) *Rule

Parse tries to apply rule to given input state, returns rule that matched or nil par is the parent rule that we're being called from. parAst is the current ast node that we add to. scope is the region to search within, defined by parent or EOS if we have a terminal one

func (*Rule) ParseRules 

func (pr *Rule) ParseRules(ps *State, par *Rule, parAst *Ast, scope lex.Reg, optMap lex.TokenMap, depth int) *Rule

ParseRules parses rules and returns this rule if it matches, nil if not

func (*Rule) Scope 

func (pr *Rule) Scope(ps *State, parAst *Ast, scope lex.Reg) (lex.Reg, bool)

Scope finds the potential scope region for looking for tokens -- either from EOS position or State ScopeStack pushed from parents. Returns new scope and false if no valid scope found.

func (*Rule) SetRuleMap 

func (pr *Rule) SetRuleMap(ps *State)

SetRuleMap is called on the top-level Rule and initializes the RuleMap

func (*Rule) StartParse 

func (pr *Rule) StartParse(ps *State) *Rule

StartParse is called on the root of the parse rule tree to start the parsing process

func (*Rule) Validate 

func (pr *Rule) Validate(ps *State) bool

Validate checks for any errors in the rules and issues warnings, returns true if valid (no err) and false if invalid (errs)

func (*Rule) WriteGrammar 

func (pr *Rule) WriteGrammar(writer io.Writer, depth int)

WriteGrammar outputs the parser rules as a formatted grammar in a BNF-like format it is called recursively

type RuleEl 

type RuleEl struct {

	// sub-rule for this position -- nil if token
	Rule *Rule `desc:"sub-rule for this position -- nil if token"`

	// token, None if rule
	Tok token.KeyToken `desc:"token, None if rule"`

	// start increment for matching -- this is the number of non-optional, non-match items between (start | last match) and this item -- increments start region for matching
	StInc int `` /* 173-byte string literal not displayed */

	// if true, this rule must match for rule to fire -- by default only tokens and, failing that, the first sub-rule is used for matching -- use @ to require a match
	Match bool `` /* 166-byte string literal not displayed */

	// this rule is optional -- will absorb tokens if they exist -- indicated with ? prefix
	Opt bool `desc:"this rule is optional -- will absorb tokens if they exist -- indicated with ? prefix"`

	// match this rule working backward from the next token -- triggered by - (minus) prefix and optimizes cases where there can be a lot of tokens going forward but few going from end -- must be anchored by a terminal EOS or other FmNext elements and is ignored if at the very end
	FmNext bool `` /* 281-byte string literal not displayed */
}

RuleEl is an element of a parsing rule -- either a pointer to another rule or a token

func (RuleEl) IsRule 

func (re RuleEl) IsRule() bool

func (RuleEl) IsToken 

func (re RuleEl) IsToken() bool

type RuleFlags added in v0.5.5 

type RuleFlags int

RuleFlags define bitflags for rule options compiled from rule syntax 

const (
	// SetsScope means that this rule sets its own scope, because it ends with EOS
	SetsScope RuleFlags = RuleFlags(ki.FlagsN) + iota

	// Reverse means that this rule runs in reverse (starts with - sign) -- for arithmetic
	// binary expressions only: this is needed to produce proper associativity result for
	// mathematical expressions in the recursive descent parser.
	// Only for rules of form: Expr '+' Expr -- two sub-rules with a token operator
	// in the middle.
	Reverse

	// NoToks means that this rule doesn't have any explicit tokens -- only refers to
	// other rules
	NoToks

	// OnlyToks means that this rule only has explicit tokens for matching -- can be
	// optimized
	OnlyToks

	// MatchEOS means that the rule ends with a *matched* EOS with StInc = 1.
	// SetsScope applies for optional and matching EOS rules alike.
	MatchEOS

	// MultiEOS means that the rule has multiple EOS tokens within it --
	// changes some of the logic
	MultiEOS

	// TokMatchGroup is a group node that also has a single token match, so it can
	// be used in a FirstTokMap to optimize lookup of rules
	TokMatchGroup
)

type RuleList 

type RuleList []RuleEl

RuleList is a list (slice) of rule elements

func (RuleList) Last 

func (rl RuleList) Last() *RuleEl

Last returns the last rule -- only used in cases where there are rules

type ScopeRule added in v0.5.5 

type ScopeRule struct {
	Scope lex.Reg
	Rule  *Rule
}

ScopeRule is a scope and a rule, for storing matches / nonmatch

type ScopeRuleSet 

type ScopeRuleSet map[ScopeRule]struct{}

ScopeRuleSet is a map by scope of RuleSets, for non-matching rules

func (ScopeRuleSet) Add 

func (rs ScopeRuleSet) Add(scope lex.Reg, pr *Rule)

Add a rule to scope set, with auto-alloc

func (ScopeRuleSet) Has 

func (rs ScopeRuleSet) Has(scope lex.Reg, pr *Rule) bool

Has checks if scope rule set has given scope, rule

type State 

type State struct {

	// [view: no-inline] source and lexed version of source we're parsing
	Src *lex.File `view:"no-inline" desc:"source and lexed version of source we're parsing"`

	// tracing for this parser
	Trace TraceOpts `desc:"tracing for this parser"`

	// root of the Ast abstract syntax tree we're updating
	Ast *Ast `desc:"root of the Ast abstract syntax tree we're updating"`

	// symbol map that everything gets added to from current file of parsing -- typically best for subsequent management to just have a single outer-most scoping symbol here (e.g., in Go it is the package), and then everything is a child under that
	Syms syms.SymMap `` /* 248-byte string literal not displayed */

	// stack of scope(s) added to FileSyms e.g., package, library, module-level elements of which this file is a part -- these are reset at the start and must be added by parsing actions within the file itself
	Scopes syms.SymStack `` /* 209-byte string literal not displayed */

	// the current lex token position
	Pos lex.Pos `desc:"the current lex token position"`

	// [view: no-inline] any error messages accumulated during parsing specifically
	Errs lex.ErrorList `view:"no-inline" desc:"any error messages accumulated during parsing specifically"`

	// [view: no-inline] rules that matched and ran at each point, in 1-to-1 correspondence with the Src.Lex tokens for the lines and char pos dims
	Matches [][]MatchStack `` /* 146-byte string literal not displayed */

	// [view: no-inline] rules that did NOT match -- represented as a map by scope of a RuleSet
	NonMatches ScopeRuleSet `view:"no-inline" desc:"rules that did NOT match -- represented as a map by scope of a RuleSet"`

	// [view: no-inline] stack for context-sensitive rules
	Stack lex.Stack `view:"no-inline" desc:"stack for context-sensitive rules"`
}

parse.State is the state maintained for parsing

func (*State) AddAst 

func (ps *State) AddAst(parAst *Ast, rule string, reg lex.Reg) *Ast

AddAst adds a child Ast node to given parent Ast node

func (*State) AddMatch 

func (ps *State) AddMatch(pr *Rule, scope lex.Reg, regs Matches)

AddMatch adds given rule to rule stack at given scope

func (*State) AddNonMatch 

func (ps *State) AddNonMatch(scope lex.Reg, pr *Rule)

AddNonMatch adds given rule to non-matching rule set for this scope

func (*State) AllocRules 

func (ps *State) AllocRules()

AllocRules allocate the match, nonmatch rule state in correspondence with the src state

func (*State) AtEof 

func (ps *State) AtEof() bool

AtEof returns true if current position is at end of file -- this includes common situation where it is just at the very last token

func (*State) AtEofNext added in v0.5.5 

func (ps *State) AtEofNext() bool

AtEofNext returns true if current OR NEXT position is at end of file -- this includes common situation where it is just at the very last token

func (*State) Error 

func (ps *State) Error(pos lex.Pos, msg string, rule *Rule)

Error adds a parsing error at given lex token position

func (*State) FindNameScoped added in v0.5.5 

func (ps *State) FindNameScoped(nm string) (*syms.Symbol, bool)

FindNameScoped searches top-down in the stack for something with the given name in symbols that are of subcategory token.NameScope (i.e., namespace, module, package, library) also looks in ps.Syms if not found in Scope stack.

func (*State) FindNameTopScope added in v0.9.15 

func (ps *State) FindNameTopScope(nm string) (*syms.Symbol, bool)

FindNameTopScope searches only in top of current scope for something 

with the given name in symbols

also looks in ps.Syms if not found in Scope stack.

func (*State) FindToken 

func (ps *State) FindToken(tkey token.KeyToken, reg lex.Reg) (lex.Pos, bool)

FindToken looks for token in given region, returns position where found, false if not. Only matches when depth is same as at reg.St start at the start of the search. All positions in token indexes.

func (*State) FindTokenReverse 

func (ps *State) FindTokenReverse(tkey token.KeyToken, reg lex.Reg) (lex.Pos, bool)

FindTokenReverse looks *backwards* for token in given region, with same depth as reg.Ed-1 end where the search starts. Returns position where found, false if not. Automatically deals with possible confusion with unary operators -- if there are two ambiguous operators in a row, automatically gets the first one. This is mainly / only used for binary operator expressions (mathematical binary operators). All positions are in token indexes.

func (*State) GotoEof added in v0.5.5 

func (ps *State) GotoEof()

GotoEof sets current position at EOF

func (*State) Init 

func (ps *State) Init(src *lex.File, ast *Ast)

Init initializes the state at start of parsing

func (*State) IsMatch 

func (ps *State) IsMatch(pr *Rule, scope lex.Reg) (*MatchState, bool)

IsMatch looks for rule at given scope in list of matches, if found returns match state info

func (*State) IsNonMatch 

func (ps *State) IsNonMatch(scope lex.Reg, pr *Rule) bool

IsNonMatch looks for rule in nonmatch list at given scope

func (*State) MatchLex 

func (ps *State) MatchLex(lx *lex.Lex, tkey token.KeyToken, isCat, isSubCat bool, cp lex.Pos) bool

MatchLex is our optimized matcher method, matching tkey depth as well

func (*State) MatchToken 

func (ps *State) MatchToken(tkey token.KeyToken, pos lex.Pos) bool

MatchToken returns true if token matches at given position -- must be a valid position!

func (*State) NextSrcLine 

func (ps *State) NextSrcLine() string

NextSrcLine returns the next line of text

func (*State) ResetNonMatches added in v0.5.5 

func (ps *State) ResetNonMatches()

ResetNonMatches resets the non-match map -- do after every EOS

func (*State) RuleString 

func (ps *State) RuleString(full bool) string

RuleString returns the rule info for entire source -- if full then it includes the full stack at each point -- otherwise just the top of stack

type Steps 

type Steps int

Steps are the different steps of the parsing processing 

const (
	// Match happens when a rule matches
	Match Steps = iota

	// SubMatch is when a sub-rule within a rule matches
	SubMatch

	// NoMatch is when the rule fails to match (recorded at first non-match, which terminates
	// matching process
	NoMatch

	// Run is when the rule is running and iterating through its sub-rules
	Run

	// RunAct is when the rule is running and performing actions
	RunAct

	StepsN
)

The parsing steps

func (*Steps) FromString 

func (i *Steps) FromString(s string) error

func (Steps) MarshalJSON 

func (ev Steps) MarshalJSON() ([]byte, error)

func (Steps) String 

func (i Steps) String() string

func (*Steps) UnmarshalJSON 

func (ev *Steps) UnmarshalJSON(b []byte) error

type TraceOpts 

type TraceOpts struct {

	// perform tracing
	On bool `desc:"perform tracing"`

	// trace specific named rules here (space separated) -- if blank, then all rules are traced
	Rules string `width:"50" desc:"trace specific named rules here (space separated) -- if blank, then all rules are traced"`

	// trace full rule matches -- when a rule fully matches
	Match bool `desc:"trace full rule matches -- when a rule fully matches"`

	// trace sub-rule matches -- when the parts of each rule match
	SubMatch bool `desc:"trace sub-rule matches -- when the parts of each rule match"`

	// trace sub-rule non-matches -- why a rule doesn't match -- which terminates the matching process at first non-match (can be a lot of info)
	NoMatch bool `` /* 144-byte string literal not displayed */

	// trace progress running through each of the sub-rules when a rule has matched and is 'running'
	Run bool `desc:"trace progress running through each of the sub-rules when a rule has matched and is 'running'"`

	// trace actions performed by running rules
	RunAct bool `desc:"trace actions performed by running rules"`

	// if true, shows the full scope source for every trace statement
	ScopeSrc bool `desc:"if true, shows the full scope source for every trace statement"`

	// for the ParseOut display, whether to display the full stack of rules at each position, or just the deepest one
	FullStackOut bool `desc:"for the ParseOut display, whether to display the full stack of rules at each position, or just the deepest one"`

	// [view: -] list of rules
	RulesList []string `view:"-" json:"-" xml:"-" desc:"list of rules"`

	// [view: -] trace output is written here, connected via os.Pipe to OutRead
	OutWrite *os.File `view:"-" json:"-" xml:"-" desc:"trace output is written here, connected via os.Pipe to OutRead"`

	// [view: -] trace output is read here -- can connect this to a TextBuf via giv.OutBuf to monitor tracing output
	OutRead *os.File `` /* 132-byte string literal not displayed */
}

TraceOpts provides options for debugging / monitoring the rule matching and execution process

func (*TraceOpts) CheckRule 

func (pt *TraceOpts) CheckRule(rule string) bool

CheckRule checks if given rule should be traced

func (*TraceOpts) CopyOpts added in v0.5.5 

func (pt *TraceOpts) CopyOpts(ot *TraceOpts)

CopyOpts copies just the options

func (*TraceOpts) FullOn added in v0.5.5 

func (pt *TraceOpts) FullOn()

FullOn sets all options on

func (*TraceOpts) Init 

func (pt *TraceOpts) Init()

Init intializes tracer after any changes -- opens pipe if not already open

func (*TraceOpts) Out 

func (pt *TraceOpts) Out(ps *State, pr *Rule, step Steps, pos lex.Pos, scope lex.Reg, ast *Ast, msg string) bool

Out outputs a trace message -- returns true if actually output

func (*TraceOpts) PipeOut added in v0.5.5 

func (pt *TraceOpts) PipeOut()

PipeOut sets output to a pipe for monitoring (OutWrite -> OutRead)

func (*TraceOpts) StdOut added in v0.5.5 

func (pt *TraceOpts) StdOut()

StdOut sets OutWrite to os.Stdout

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.