asp

package
v13.4.0+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 25, 2019 License: Apache-2.0 Imports: 21 Imported by: 36

README

ASP

This package implements a custom parser for BUILD files.

The so-far-unnamed BUILD language is (almost) a subset of Python, with many of its more advanced or dynamic features stripped out. Obviously it is not easy to implement even a subset of it and so many aspects remain unimplemented. Some of the notable differences are:

  • The import, try, except, finally, class, global, nonlocal, while and async keywords are not available. It is therefore possible but discouraged to use these as identifiers.
  • The raise and assert statements are supported, but since it is not possible to catch exceptions they only serve to signal catastrophic errors.
  • List and dict comprehensions are supported, but not Python's more general generator expressions. Up to two 'for' clauses are permitted.
  • Most builtin functions are not available.
  • Dictionaries are supported, but can only be keyed by strings.
  • The only builtin types are bool, int, str, list, dict and functions. There are no float, complex, set, frozenset or bytes types.
  • Operators +, <, >, %, and, or, in, not in, ==, >=, <= and != are supported in most appropriate cases. Other operators are not available.
  • Limited string interpolation is available via %. format() is also available but its implementation is incomplete and use is discouraged.
  • The += augmented assignment operator is available in addition to = for normal assignment. Other augmented assignment operations aren't available.
  • Operator precedence is not always the same as Python's at present. This may well change in later versions, although users are encouraged to avoid relying on precedence details where possible.
  • The accepted syntax likely differs in minor ways from Python's due to implementation details on our part or limitations in how our parser treats files. In general we try to match Python's semantics but only as a best-effort; many of the differences mentioned here are deliberate.
  • append and extend on list objects are troublesome - our lists are a little more immutable (80% of the time they're immutable every time) than Python lists and hence those functions sort of work but don't have exactly the same semantics as the original (e.g. they may not always modify the original if it originated from a surrounding scope). Users are encouraged to use the += augmented assignment operator instead.
  • subinclude() has slightly different semantics now. Subincluded files must include their dependencies (it is not sufficient simply to do them in order). Subincludes should not attempt to mutate global objects.

TODOs (in no particular order of importance):

  • Better guarantees around ordering. Consider banishing sorted().
  • Better operator precedence.

The name is reminiscent of AST, and a play on an asp being a smaller snake than a python.

Documentation

Overview

Package asp implements an experimental BUILD-language parser. Parsing is doing using Participle (github.com/alecthomas/participle) in native Go, with a custom and also native partial Python interpreter.

Index

Constants

View Source 
const (
	// Add etc are arithmetic operators - these are implemented on a per-type basis
	Add Operator = '+'
	// Subtract implements binary - (only works on integers)
	Subtract = '-'
	// Modulo implements % (including string interpolation)
	Modulo = '%'
	// LessThan implements <
	LessThan = '<'
	// GreaterThan implements >
	GreaterThan = '>'
	// LessThanOrEqual implements <=
	LessThanOrEqual = '≤'
	// GreaterThanOrEqual implements >=
	GreaterThanOrEqual = '≥'
	// Equal etc are comparison operators - also on a per-type basis but have slightly different rules.
	Equal = '＝'
	// NotEqual implements !=
	NotEqual = '≠'
	// In implements the in operator
	In = '∈'
	// NotIn implements "not in" as a single operator.
	NotIn = '∉'
	// And etc are logical operators - these are implemented type-independently
	And Operator = '&'
	// Or implements the or operator
	Or = '|'
	// Is implements type identity.
	Is = '≡'
	// Index is used in the parser, but not when parsing code.
	Index = '['
)
View Source 
const (
	EOF = -(iota + 1)
	Ident
	Int
	String
	LexOperator
	EOL
	Unindent
)

Token types.

Variables

View Source 
var (
	True         pyBool = 1
	False        pyBool
	None         pyBool = -1
	FileNotFound pyBool = -2
)

True, False and None are the singletons representing those values in Python. None isn't really a bool of course, but it's easier to use an instance of bool than another custom type.

Functions

func AddReader 

func AddReader(err error, r io.ReadSeeker) error

AddReader adds an io.Reader to an errStack, which will allow it to recover more information from that file.

func AddStackFrame 

func AddStackFrame(pos Position, err interface{}) error

AddStackFrame adds a new stack frame to the given errorStack, or wraps an existing error if not.

func GetExtents 

func GetExtents(statements []*Statement, statement *Statement, max int) (int, int)

GetExtents returns the "extents" of a statement, i.e. the lines that it covers in source. The caller must pass a value for the maximum extent of the file; we can't detect it here because the AST only contains positions for the beginning of the statements.

func NameOfReader 

func NameOfReader(r io.Reader) string

NameOfReader returns a name for the given reader, if one can be determined.

func StatementOrExpressionFromAst 

func StatementOrExpressionFromAst(stmts []*Statement, position Position) (statement *Statement, expression *Expression)

StatementOrExpressionFromAst recursively finds asp.IdentStatement and asp.Expression in the ast and returns a valid statement pointer if within range

func WalkAST 

func WalkAST(astStruct interface{}, callback func(astStruct interface{}) interface{}) interface{}

WalkAST is a generic function that walks through the ast recursively, astStruct can be anything inside of the AST, such as asp.Statement, asp.Expression it accepts a callback for any operations

Types

type Argument 

type Argument struct {
	Name string   `@Ident`
	Type []string `[ ":" @( { ( "bool" | "str" | "int" | "list" | "dict" | "function" ) [ "|" ] } ) ]`
	// Aliases are an experimental non-Python concept where function arguments can be aliased to different names.
	// We use this to support compatibility with Bazel & Buck etc in some cases.
	Aliases []string    `[ "&" ( { @Ident [ "&" ] } ) ]`
	Value   *Expression `[ "=" @@ ]`

	IsPrivate bool
}

An Argument represents an argument to a function definition.

type Call 

type Call struct {
	Arguments []CallArgument `[ @@ ] { "," [ @@ ] }`
}

A Call represents a call site of a function.

type CallArgument 

type CallArgument struct {
	Pos   Position
	Name  string     `[ @@ "=" ]`
	Value Expression `@@`
}

A CallArgument represents a single argument at a call site of a function.

func FindArgument 

func FindArgument(statement *Statement, args ...string) *CallArgument

FindArgument finds an argument of any one of the given names, or nil if there isn't one. The statement must be a function call (e.g. as returned by FindTarget).

type Comprehension 

type Comprehension struct {
	Names  []string    `"for" @Ident [ { "," @Ident } ] "in"`
	Expr   *Expression `@@`
	Second *struct {
		Names []string    `"for" @Ident [ { "," @Ident } ] "in"`
		Expr  *Expression `@@`
	} `[ @@ ]`
	If *Expression `[ "if" @@ ]`
}

A Comprehension represents a list or dict comprehension clause.

type Dict 

type Dict struct {
	Items         []*DictItem    `[ @@ ] { "," [ @@ ] }`
	Comprehension *Comprehension `[ @@ ]`
}

A Dict represents a dict literal, either with or without a comprehension clause.

type DictItem 

type DictItem struct {
	Key   Expression `@( Ident | String ) ":"`
	Value Expression `@@`
}

A DictItem represents a single key-value pair in a dict literal.

type Expression 

type Expression struct {
	Pos     Position
	EndPos  Position
	UnaryOp *UnaryOp         `( @@`
	Val     *ValueExpression `| @@ )`
	Op      []OpExpression   `{ @@ }`
	If      *InlineIf        `[ @@ ]`
	// For internal optimisation - do not use outside this package.
	Optimised *OptimisedExpression
}

An Expression is a generalised Python expression, i.e. anything that can appear where an expression is allowed (including the extra parts like inline if-then-else, operators, etc).

type FString 

type FString struct {
	Vars []struct {
		Prefix string // Preceding string bit
		Var    string // Variable name to interpolate
		Config string // Config variable to look up
	}
	Suffix string // Following string bit
}

A FString represents a minimal version of a Python literal format string. Note that we only support a very small subset of what Python allows there; essentially only variable substitution, which gives a much simpler AST structure here.

type FileInput 

type FileInput struct {
	Statements []*Statement `{ @@ } EOF`
}

A FileInput is the top-level structure of a BUILD file.

type ForStatement 

type ForStatement struct {
	Names      []string     `"for" @Ident [ { "," @Ident } ] "in"`
	Expr       Expression   `@@ Colon EOL`
	Statements []*Statement `{ @@ } Unindent`
}

A ForStatement implements the 'for' statement. Note that it does not support Python's "for-else" construction.

type FuncDef 

type FuncDef struct {
	Name       string       `"def" @Ident`
	Arguments  []Argument   `"(" [ @@ { "," @@ } ] ")" Colon EOL`
	Docstring  string       `[ @String EOL ]`
	Statements []*Statement `{ @@ } Unindent`
	EoDef      Position
	// allowed return type of the FuncDef
	Return string
	// Not part of the grammar. Used to indicate internal targets that can only
	// be called using keyword arguments.
	KeywordsOnly bool
	IsPrivate    bool
}

A FuncDef implements definition of a new function.

type IdentExpr 

type IdentExpr struct {
	Pos    Position
	EndPos Position
	Name   string `@Ident`
	Action []struct {
		Property *IdentExpr `  "." @@`
		Call     *Call      `| "(" @@ ")"`
	} `{ @@ }`
}

An IdentExpr implements parts of an expression that begin with an identifier (i.e. anything that might be a variable name).

type IdentStatement 

type IdentStatement struct {
	Name   string `@Ident`
	Unpack *struct {
		Names []string    `@Ident { "," @Ident }`
		Expr  *Expression `"=" @@`
	} `( "," @@ `
	Index *struct {
		Expr      *Expression `@@ "]"`
		Assign    *Expression `( "=" @@`
		AugAssign *Expression `| "+=" @@ )`
	} `| "[" @@`
	Action *IdentStatementAction `| @@ )`
}

An IdentStatement implements a statement that begins with an identifier (i.e. anything that starts off with a variable name). It is a little fiddly due to parser limitations.

type IdentStatementAction 

type IdentStatementAction struct {
	Property  *IdentExpr  `  "." @@`
	Call      *Call       `| "(" @@ ")"`
	Assign    *Expression `| "=" @@`
	AugAssign *Expression `| "+=" @@`
}

An IdentStatementAction implements actions on an IdentStatement.

type IfStatement 

type IfStatement struct {
	Condition  Expression   `"if" @@ Colon EOL`
	Statements []*Statement `{ @@ } Unindent`
	Elif       []struct {
		Condition  Expression   `"elif" @@ Colon EOL`
		Statements []*Statement `{ @@ } Unindent`
	} `{ @@ }`
	ElseStatements []*Statement `[ "else" Colon EOL { @@ } Unindent ]`
}

An IfStatement implements the if-elif-else statement.

type InlineIf 

type InlineIf struct {
	Condition *Expression `"if" @@`
	Else      *Expression `[ "else" @@ ]`
}

An InlineIf implements the single-line if-then-else construction

type Lambda 

type Lambda struct {
	Arguments []Argument `[ @@ { "," @@ } ] Colon`
	Expr      Expression `@@`
}

A Lambda is the inline lambda function.

type List 

type List struct {
	Values        []*Expression  `[ @@ ] { "," [ @@ ] }`
	Comprehension *Comprehension `[ @@ ]`
}

A List represents a list literal, either with or without a comprehension clause.

type OpExpression 

type OpExpression struct {
	Op   Operator    `@("+" | "-" | "%" | "<" | ">" | "and" | "or" | "is" | "in" | "not" "in" | "==" | "!=" | ">=" | "<=")`
	Expr *Expression `@@`
}

An OpExpression is a operator combined with its following expression.

type Operator 

type Operator rune

An Operator defines a unary or binary operator.

func (Operator) String 

func (o Operator) String() string

String implements the fmt.Stringer interface. It is not especially efficient and is normally only used for errors & debugging.

type OptimisedExpression 

type OptimisedExpression struct {
	// Used to optimise constant expressions.
	Constant pyObject
	// Similarly applied to optimise simple lookups of local variables.
	Local string
	// And similarly applied to optimise lookups into configuration.
	Config string
}

An OptimisedExpression contains information to optimise certain aspects of execution of an expression. It must be public for serialisation but shouldn't be used outside this package.

type Parser 

type Parser struct {
	// contains filtered or unexported fields
}

A Parser implements parsing of BUILD files.

func NewParser 

func NewParser(state *core.BuildState) *Parser

NewParser creates a new parser instance. One is normally sufficient for a process lifetime.

func (*Parser) LoadBuiltins 

func (p *Parser) LoadBuiltins(filename string, contents, encoded []byte) error

LoadBuiltins instructs the parser to load rules from this file as built-ins. Optionally the file contents can be supplied directly. Also optionally a previously parsed form (acquired from ParseToFile) can be supplied.

func (*Parser) MustLoadBuiltins 

func (p *Parser) MustLoadBuiltins(filename string, contents, encoded []byte)

MustLoadBuiltins calls LoadBuiltins, and dies on any errors.

func (*Parser) ParseData 

func (p *Parser) ParseData(data []byte, filename string) ([]*Statement, error)

ParseData reads the given byteslice and parses it into a set of statements. The 'filename' argument is only used in case of errors so doesn't necessarily have to correspond to a real file.

func (*Parser) ParseFile 

func (p *Parser) ParseFile(pkg *core.Package, filename string) error

ParseFile parses the contents of a single file in the BUILD language. It returns true if the call was deferred at some point awaiting target to build, along with any error encountered.

func (*Parser) ParseFileOnly 

func (p *Parser) ParseFileOnly(filename string) ([]*Statement, error)

ParseFileOnly parses the given file but does not interpret it.

func (*Parser) ParseReader 

func (p *Parser) ParseReader(pkg *core.Package, r io.ReadSeeker) (bool, error)

ParseReader parses the contents of the given ReadSeeker as a BUILD file. The first return value is true if parsing succeeds - if the error is still non-nil that indicates that interpretation failed.

func (*Parser) ParseToFile 

func (p *Parser) ParseToFile(input, output string) error

ParseToFile parses the given file and writes a binary form of the result to the output file.

type Position 

type Position struct {
	Filename string
	Offset   int
	Line     int
	Column   int
}

A Position describes a position in a source file. All properties in Position are one(1) indexed

type ReturnStatement 

type ReturnStatement struct {
	Values []*Expression `[ @@ { "," @@ } ]`
}

A ReturnStatement implements the Python 'return' statement.

type Slice 

type Slice struct {
	Start *Expression `"[" [ @@ ]`
	Colon string      `[ @":" ]`
	End   *Expression `[ @@ ] "]"`
}

A Slice represents a slice or index expression (e.g. [1], [1:2], [2:], [:], etc).

type Statement 

type Statement struct {
	Pos     Position
	EndPos  Position
	FuncDef *FuncDef         `| @@`
	For     *ForStatement    `| @@`
	If      *IfStatement     `| @@`
	Return  *ReturnStatement `| "return" @@ EOL`
	Raise   *Expression      `| "raise" @@ EOL`
	Assert  *struct {
		Expr    *Expression `@@`
		Message string      `["," @String]`
	} `| "assert" @@ EOL`
	Ident    *IdentStatement `| @@ EOL`
	Literal  *Expression     `| @@ EOL)`
	Pass     bool            `( @"pass" EOL`
	Continue bool            `| @"continue" EOL`
}

A Statement is the type we work with externally the most; it's a single Python statement. Note that some mildly excessive fiddling is needed since the parser we're using doesn't support backoff (i.e. if an earlier entry matches to its completion but can't consume following tokens, it doesn't then make another choice :( )

func FindTarget 

func FindTarget(statements []*Statement, name string) *Statement

FindTarget returns the top-level call in a BUILD file that corresponds to a target of the given name (or nil if one does not exist).

func NextStatement 

func NextStatement(statements []*Statement, statement *Statement) *Statement

NextStatement finds the statement that follows the given one. This is often useful to find the extent of a statement in source code. It will return nil if there is not one following it.

type Token 

type Token struct {
	// Type of token. If > 0 this is the literal character value; if < 0 it is one of the types above.
	Type rune
	// The literal text of the token. Strings are lightly normalised to always be surrounded by quotes (but only one).
	Value string
	// The position in the input that the token occurred at.
	Pos Position
}

A Token describes each individual lexical element emitted by the lexer.

func (Token) EndPos 

func (tok Token) EndPos() Position

EndPos returns the end position of a token

func (Token) String 

func (tok Token) String() string

String implements the fmt.Stringer interface

type UnaryOp 

type UnaryOp struct {
	Op   string          `@( "-" | "not" )`
	Expr ValueExpression `@@`
}

A UnaryOp represents a unary operation - in our case the only ones we support are negation and not.

type ValueExpression 

type ValueExpression struct {
	String  string   `( @String`
	FString *FString `| @FString`
	Int     *struct {
		Int int `@Int`
	} `| @@` // Should just be *int, but https://github.com/golang/go/issues/23498 :(
	Bool     string     `| @( "True" | "False" | "None" )`
	List     *List      `| "[" @@ "]"`
	Dict     *Dict      `| "{" @@ "}"`
	Tuple    *List      `| "(" @@ ")"`
	Lambda   *Lambda    `| "lambda" @@`
	Ident    *IdentExpr `| @@ )`
	Slice    *Slice     `[ @@ ]`
	Property *IdentExpr `[ ( "." @@`
	Call     *Call      `| "(" @@ ")" ) ]`
}

A ValueExpression is the value part of an expression, i.e. without surrounding operators.

Source Files

View all Source files

Directories

Path Synopsis
Package fuzz implements a fuzzing entry point for asp using go-fuzz.
 Package fuzz implements a fuzzing entry point for asp using go-fuzz.
Package main implements a compiler for the builtin build rules, which is used at bootstrap time.
 Package main implements a compiler for the builtin build rules, which is used at bootstrap time.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.