asp

package
v11.4.0+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 19, 2018 License: Apache-2.0 Imports: 21 Imported by: 0

README

ASP

This package implements a custom parser for BUILD files.

The so-far-unnamed BUILD language is (almost) a subset of Python, with many of its more advanced or dynamic features stripped out. Obviously it is not easy to implement even a subset of it and so many aspects remain unimplemented. Some of the notable differences are:

  • The import, try, except, finally, class, global, nonlocal, while and async keywords are not available. It is therefore possible but discouraged to use these as identifiers.
  • The raise and assert statements are supported, but since it is not possible to catch exceptions they only serve to signal catastrophic errors.
  • List and dict comprehensions are supported, but not Python's more general generator expressions. Up to two 'for' clauses are permitted.
  • Most builtin functions are not available.
  • Dictionaries are supported, but can only be keyed by strings.
  • The only builtin types are bool, int, str, list, dict and functions. There are no float, complex, set, frozenset or bytes types.
  • Operators +, <, >, %, and, or, in, not in, ==, >=, <= and != are supported in most appropriate cases. Other operators are not available.
  • Limited string interpolation is available via %. format() is also available but its implementation is incomplete and use is discouraged.
  • The += augmented assignment operator is available in addition to = for normal assignment. Other augmented assignment operations aren't available.
  • Operator precedence is not always the same as Python's at present. This may well change in later versions, although users are encouraged to avoid relying on precedence details where possible.
  • The accepted syntax likely differs in minor ways from Python's due to implementation details on our part or limitations in how our parser treats files. In general we try to match Python's semantics but only as a best-effort; many of the differences mentioned here are deliberate.
  • append and extend on list objects are troublesome - our lists are a little more immutable (80% of the time they're immutable every time) than Python lists and hence those functions sort of work but don't have exactly the same semantics as the original (e.g. they may not always modify the original if it originated from a surrounding scope). Users are encouraged to use the += augmented assignment operator instead.
  • subinclude() has slightly different semantics now. Subincluded files must include their dependencies (it is not sufficient simply to do them in order). Subincludes should not attempt to mutate global objects.

TODOs (in no particular order of importance):

  • Better guarantees around ordering. Consider banishing sorted().
  • Better operator precedence.

The name is reminiscent of AST, and a play on an asp being a smaller snake than a python.

Documentation

Overview

Package asp implements an experimental BUILD-language parser. Parsing is doing using Participle (github.com/alecthomas/participle) in native Go, with a custom and also native partial Python interpreter.

Index

Constants

View Source 
const (
	EOF = -(iota + 1)
	Ident
	Int
	String
	EOL
	Unindent
	Colon       // Would prefer not to have this but literal colons seem to deeply upset the parser.
	LexOperator // Similarly this, it doesn't seem to be able to handle == otherwise.
)

Token types.

Variables

View Source 
var (
	True         pyBool = 1
	False        pyBool
	None         pyBool = -1
	FileNotFound pyBool = -2
)

True, False and None are the singletons representing those values in Python. None isn't really a bool of course, but it's easier to use an instance of bool than another custom type.

Functions

func AddReader 

func AddReader(err error, r io.ReadSeeker) error

AddReader adds an io.Reader to an errStack, which will allow it to recover more information from that file.

func AddStackFrame 

func AddStackFrame(pos lexer.Position, err interface{}) error

AddStackFrame adds a new stack frame to the given errorStack, or wraps an existing error if not.

func GetExtents 

func GetExtents(statements []*Statement, statement *Statement, max int) (int, int)

GetExtents returns the "extents" of a statement, i.e. the lines that it covers in source. The caller must pass a value for the maximum extent of the file; we can't detect it here because the AST only contains positions for the beginning of the statements.

func NewLexer 

func NewLexer() lexer.Definition

NewLexer is a slightly nicer interface to creating a new lexer definition.

func RequiresSubinclude 

func RequiresSubinclude(err error) (bool, core.BuildLabel)

RequiresSubinclude returns true if the error requires another target to be built, along with the target in question.

Types

type Argument 

type Argument struct {
	Name  string      `@Ident`
	Type  []string    `[ ":" @( { ( "bool" | "str" | "int" | "list" | "dict" | "function" ) [ "|" ] } ) ]`
	Value *Expression `[ "=" @@ ]`
}

An Argument represents an argument to a function definition.

type Call 

type Call struct {
	Arguments []CallArgument `[ @@ ] { "," [ @@ ] }`
}

A Call represents a call site of a function.

type CallArgument 

type CallArgument struct {
	Expr  *Expression `@@`
	Value *Expression `[ "=" @@ ]`
	// contains filtered or unexported fields
}

A CallArgument represents a single argument at a call site of a function.

type Comprehension 

type Comprehension struct {
	Names  []string    `"for" @Ident [ { "," @Ident } ] "in"`
	Expr   *Expression `@@`
	Second *struct {
		Names []string    `"for" @Ident [ { "," @Ident } ] "in"`
		Expr  *Expression `@@`
	} `[ @@ ]`
	If *Expression `[ "if" @@ ]`
}

A Comprehension represents a list or dict comprehension clause.

type Dict 

type Dict struct {
	Items         []*DictItem    `[ @@ ] { "," [ @@ ] }`
	Comprehension *Comprehension `[ @@ ]`
}

A Dict represents a dict literal, either with or without a comprehension clause.

type DictItem 

type DictItem struct {
	Key   string     `@( Ident | String ) ":"`
	Value Expression `@@`
}

A DictItem represents a single key-value pair in a dict literal.

type Environment 

type Environment struct {
	Functions map[string]Function `json:"functions"`
}

An Environment describes the global environment of the parser. TODO(peterebden): We may refactor this out in favour of exposing the AST instead.

type Expression 

type Expression struct {
	Pos     lexer.Position
	UnaryOp *UnaryOp         `( @@`
	Val     *ValueExpression `| @@ )`
	Op      *struct {
		Op   Operator    `@("+" | "-" | "%" | "<" | ">" | "and" | "or" | "is" | "in" | "not" "in" | "==" | "!=" | ">=" | "<=")`
		Expr *Expression `@@`
	} `[ @@ ]`
	If *InlineIf `[ @@ ]`
	// contains filtered or unexported fields
}

An Expression is a generalised Python expression, i.e. anything that can appear where an expression is allowed (including the extra parts like inline if-then-else, operators, etc).

type FileInput 

type FileInput struct {
	Statements []*Statement `{ @@ } EOF`
}

A FileInput is the top-level structure of a BUILD file.

type ForStatement 

type ForStatement struct {
	Names      []string     `"for" @Ident [ { "," @Ident } ] "in"`
	Expr       Expression   `@@ Colon EOL`
	Statements []*Statement `{ @@ } Unindent`
}

A ForStatement implements the 'for' statement. Note that it does not support Python's "for-else" construction.

type FuncDef 

type FuncDef struct {
	Name       string       `"def" @Ident`
	Arguments  []*Argument  `"(" [ @@ { "," @@ } ] ")" Colon EOL`
	Docstring  string       `[ @String EOL ]`
	Statements []*Statement `{ @@ } Unindent`
}

A FuncDef implements definition of a new function.

type Function 

type Function struct {
	Args      []FunctionArg `json:"args"`
	Comment   string        `json:"comment,omitempty"`
	Docstring string        `json:"docstring,omitempty"`
	Language  string        `json:"language,omitempty"`
}

A Function describes a function within the global environment

type FunctionArg 

type FunctionArg struct {
	Comment    string   `json:"comment,omitempty"`
	Deprecated bool     `json:"deprecated,omitempty"`
	Name       string   `json:"name"`
	Required   bool     `json:"required,omitempty"`
	Types      []string `json:"types"`
}

A FunctionArg represents a single argument to a function.

type IdentExpr 

type IdentExpr struct {
	Name   string `@Ident`
	Action []struct {
		Property *IdentExpr `  "." @@`
		Call     *Call      `| "(" @@ ")"`
	} `{ @@ }`
}

An IdentExpr implements parts of an expression that begin with an identifier (i.e. anything that might be a variable name).

type IdentStatement 

type IdentStatement struct {
	Name   string `@Ident`
	Unpack *struct {
		Names []string    `@Ident { "," @Ident }`
		Expr  *Expression `"=" @@`
	} `( "," @@ `
	Index *struct {
		Expr      *Expression `@@ "]"`
		Assign    *Expression `( "=" @@`
		AugAssign *Expression `| "+=" @@ )`
	} `| "[" @@`
	Action *IdentStatementAction `| @@ )`
}

An IdentStatement implements a statement that begins with an identifier (i.e. anything that starts off with a variable name). It is a little fiddly due to parser limitations.

type IdentStatementAction 

type IdentStatementAction struct {
	Property  *IdentExpr  `  "." @@`
	Call      *Call       `| "(" @@ ")"`
	Assign    *Expression `| "=" @@`
	AugAssign *Expression `| "+=" @@`
}

An IdentStatementAction implements actions on an IdentStatement.

type IfStatement 

type IfStatement struct {
	Condition  Expression   `"if" @@ Colon EOL`
	Statements []*Statement `{ @@ } Unindent`
	Elif       []struct {
		Condition  *Expression  `"elif" @@ Colon EOL`
		Statements []*Statement `{ @@ } Unindent`
	} `{ @@ }`
	ElseStatements []*Statement `[ "else" Colon EOL { @@ } Unindent ]`
}

An IfStatement implements the if-elif-else statement.

type InlineIf 

type InlineIf struct {
	Condition *Expression `"if" @@`
	Else      *Expression `[ "else" @@ ]`
}

An InlineIf implements the single-line if-then-else construction

type Lambda 

type Lambda struct {
	Arguments []LambdaArgument `[ @@ { "," @@ } ] Colon`
	Expr      Expression       `@@`
}

A Lambda is the inline lambda function.

type LambdaArgument 

type LambdaArgument struct {
	Name  string      `@Ident`
	Value *Expression `[ "=" @@ ]`
}

A LambdaArgument represents an argument to a lambda function. Vexingly these can't be normal function arguments any more, because the : for type annotations gets preferentially consumed to the one that ends the lambda itself :(

type List 

type List struct {
	Values        []*Expression  `[ @@ ] { "," [ @@ ] }`
	Comprehension *Comprehension `[ @@ ]`
}

A List represents a list literal, either with or without a comprehension clause.

type Operator 

type Operator int

An Operator wraps up a Python binary operator to be faster to switch on and to add some useful extra methods. 

const (
	// ComparisonOperator is used to mark comparison operators
	ComparisonOperator Operator = 0x100
	// LogicalOperator is used to mark logical operators.
	LogicalOperator Operator = 0x200
	// Add etc are arithmetic operators - these are implemented on a per-type basis
	Add Operator = iota
	// Subtract implements binary - (only works on integers)
	Subtract
	// Modulo implements % (including string interpolation)
	Modulo
	// LessThan implements <
	LessThan
	// GreaterThan implements >
	GreaterThan
	// LessThanOrEqual implements <=
	LessThanOrEqual
	// GreaterThanOrEqual implements >=
	GreaterThanOrEqual
	// Equal etc are comparison operators - also on a per-type basis but have slightly different rules.
	Equal = iota | ComparisonOperator
	// NotEqual implements !=
	NotEqual
	// In implements the in operator
	In
	// NotIn implements "not in" as a single operator.
	NotIn
	// And etc are logical operators - these are implemented type-independently
	And Operator = iota | LogicalOperator
	// Or implements the or operator
	Or
	// Is implements type identity.
	Is
	// Index is used in the parser, but not when parsing code.
	Index = -iota
)

func (*Operator) Capture 

func (o *Operator) Capture(values []string) error

Capture implements capturing for the parser.

func (Operator) IsComparison 

func (o Operator) IsComparison() bool

IsComparison returns true if this operator is a comparison operator.

func (Operator) IsLogical 

func (o Operator) IsLogical() bool

IsLogical returns true if this operator is a logical operator.

func (Operator) String 

func (o Operator) String() string

String implements the fmt.Stringer interface. It is not especially efficient and is normally only used for errors & debugging.

type Parser 

type Parser struct {
	// contains filtered or unexported fields
}

A Parser implements parsing of BUILD files.

func NewParser 

func NewParser(state *core.BuildState) *Parser

NewParser creates a new parser instance. One is normally sufficient for a process lifetime.

func (*Parser) Environment 

func (p *Parser) Environment() *Environment

Environment returns the current global environment of the parser.

func (*Parser) LoadBuiltins 

func (p *Parser) LoadBuiltins(filename string, contents, encoded []byte) error

LoadBuiltins instructs the parser to load rules from this file as built-ins. Optionally the file contents can be supplied directly. Also optionally a previously parsed form (acquired from ParseToFile) can be supplied.

func (*Parser) MustLoadBuiltins 

func (p *Parser) MustLoadBuiltins(filename string, contents, encoded []byte)

MustLoadBuiltins calls LoadBuiltins, and dies on any errors.

func (*Parser) ParseFile 

func (p *Parser) ParseFile(state *core.BuildState, pkg *core.Package, filename string) error

ParseFile parses the contents of a single file in the BUILD language.

func (*Parser) ParseFileOnly 

func (p *Parser) ParseFileOnly(filename string) ([]*Statement, error)

ParseFileOnly parses the given file but does not interpret it.

func (*Parser) ParseToFile 

func (p *Parser) ParseToFile(input, output string) error

ParseToFile parses the given file and writes a binary form of the result to the output file.

type ReturnStatement 

type ReturnStatement struct {
	Values []*Expression `[ @@ { "," @@ } ]`
}

A ReturnStatement implements the Python 'return' statement.

type Slice 

type Slice struct {
	Start *Expression `"[" [ @@ ]`
	Colon string      `[ @":" ]`
	End   *Expression `[ @@ ] "]"`
}

A Slice represents a slice or index expression (e.g. [1], [1:2], [2:], [:], etc).

type Statement 

type Statement struct {
	Pos      lexer.Position
	Pass     string           `( @"pass" EOL`
	Continue string           `| @"continue" EOL`
	FuncDef  *FuncDef         `| @@`
	For      *ForStatement    `| @@`
	If       *IfStatement     `| @@`
	Return   *ReturnStatement `| "return" @@ EOL`
	Raise    *Expression      `| "raise" @@ EOL`
	Assert   *struct {
		Expr    *Expression `@@`
		Message string      `["," @String]`
	} `| "assert" @@ EOL`
	Ident   *IdentStatement `| @@ EOL`
	Literal *Expression     `| @@ EOL)`
}

A Statement is the type we work with externally the most; it's a single Python statement. Note that some mildly excessive fiddling is needed since the parser we're using doesn't support backoff (i.e. if an earlier entry matches to its completion but can't consume following tokens, it doesn't then make another choice :( )

func FindTarget 

func FindTarget(statements []*Statement, name string) *Statement

FindTarget returns the top-level call in a BUILD file that corresponds to a target of the given name (or nil if one does not exist).

func NextStatement 

func NextStatement(statements []*Statement, statement *Statement) *Statement

NextStatement finds the statement that follows the given one. This is often useful to find the extent of a statement in source code. It will return nil if there is not one following it.

type UnaryOp 

type UnaryOp struct {
	Op   string          `@( "-" | "not" )`
	Expr ValueExpression `@@`
}

A UnaryOp represents a unary operation - in our case the only ones we support are negation and not.

type ValueExpression 

type ValueExpression struct {
	String string `( @String`
	Int    *struct {
		Int int `@Int`
	} `| @@` // Should just be *int, but https://github.com/golang/go/issues/23498 :(
	Bool     string     `| @( "True" | "False" | "None" )`
	List     *List      `| "[" @@ "]"`
	Dict     *Dict      `| "{" @@ "}"`
	Tuple    *List      `| "(" @@ ")"`
	Lambda   *Lambda    `| "lambda" @@`
	Ident    *IdentExpr `| @@ )`
	Slice    *Slice     `[ @@ ]`
	Property *IdentExpr `[ ( "." @@`
	Call     *Call      `| "(" @@ ")" ) ]`
}

A ValueExpression is the value part of an expression, i.e. without surrounding operators.

Source Files

View all Source files

Directories

Path Synopsis
Package main implements a compiler for the builtin build rules, which is used at bootstrap time.
 Package main implements a compiler for the builtin build rules, which is used at bootstrap time.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.