Documentation ¶
Overview ¶
Package ictiobus provides a compiler-compiler that generates compiler frontends in Go from language specifications. Ictiobus is used and was created as part of personal research into compiling techniques. It is the tunascript compiler pulled out after it turned from "small knowledge gaining side-side project" into a full-blown compilers and translators learning and research project. The name ictiobus is based off of the name for the buffalo fish due to the buffalo's relation with bison, important due to the popularity of the parser generator bison and for being related to fish.
A compiler Frontend is created with ictiobus (usually using the ictcc command to generate one based on a FISHI spec for a langauge), and these frontends are used to parse input code into some intermediate representation (IR). A Frontend holds all information needed to run analysis on input; its Analyze provides a simple interface to accept code in its language and parse it for the IR it was configured for.
For instructions on using the ictcc command to generate a Frontend, see the README.md file included with ictiobus, or the command documentation for github.com/dekarrin/ictiobus/cmd/ictcc.
While the ictiobus package itself contains a few convenience functions for creating the components of a Frontend, most of the functionality of the compiler is delegated to packages in sub-directories of this project:
The grammar and automaton packages contain fundamental types for language analysis that the rest of the compiler frontend uses.
The lex package handles the lexing stage of input analysis. Tokens lexed during this stage are provided to the parsing stage via a lex.TokenStream produced by calling lex.Lexer.Lex.
The parse package handles the parsing stage of input analysis. This phase parses the tokens lexed from the stream into a parse.Tree by calling parse.Parser.Parse on a lex.TokenStream.
The trans package handles the translation stage of input analysis. This phase translates a parse tree into an intermediate representation (IR) that is ultimately returned as the final result of frontend analysis. The translation is applied by calling trans.SDTS.Evaluate on a parse.Tree and specifying the name of the final attribute(s) requested.
The syntaxerr package provides a unifying interface for errors produced from the lexing, parsing, or translation phases of input code analysis.
The fishi package reads in language specifications written in the FISHI language and produces new ictiobus parsers. It is the primary module used by the ictcc command.
All of the above functionality is unified in the Frontend.Analyze(io.Reader) function, which will run all three stages of input analysis and produce the final intermediate representation.
Index ¶
- func NewCLRParser(g grammar.CFG, allowAmbiguous bool) (parser parse.Parser, ambigWarns []string, err error)
- func NewLALRParser(g grammar.CFG, allowAmbiguous bool) (parser parse.Parser, ambigWarns []string, err error)
- func NewLLParser(g grammar.CFG) (parser parse.Parser, err error)
- func NewLazyLexer() lex.Lexer
- func NewLexer() lex.Lexer
- func NewParser(g grammar.CFG, allowAmbiguous bool) (parser parse.Parser, ambigWarns []string, err error)
- func NewSDTS() trans.SDTS
- func NewSLRParser(g grammar.CFG, allowAmbiguous bool) (parser parse.Parser, ambigWarns []string, err error)
- type Frontend
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewCLRParser ¶
func NewCLRParser(g grammar.CFG, allowAmbiguous bool) (parser parse.Parser, ambigWarns []string, err error)
NewCLRParser returns a canonical LR(k) parser for the given grammar. The value of k will be the highest possible to provide with ictiobus. Returns an error if the grammar is not CLR(k).
At the time of this writing, the greatest k = 1.
func NewLALRParser ¶ added in v0.8.0
func NewLALRParser(g grammar.CFG, allowAmbiguous bool) (parser parse.Parser, ambigWarns []string, err error)
NewLALRParser returns an LALR(k) parser for the given grammar. The value of k will be the highest possible to provide with ictiobus. Returns an error if the grammar is not LALR(k).
At the time of this writing, the greatest k = 1.
func NewLLParser ¶ added in v0.8.0
NewLLParser returns an LL(k) parser for the given grammar. The value of k will be the highest possible to provide with ictiobus. Returns an error if the grammar is not LL(k).
At the time of this writing, the greatest k = 1.
func NewLazyLexer ¶
NewLazyLexer returns a Lexer whose Lex method will return a TokenStream that is lazily executed; that is to say, calling Next() on the token stream will perform only enough lexical analysis to produce the next token. Additionally, that TokenStream may produce an error token.
func NewLexer ¶
NewLexer returns a lexer whose Lex method will immediately lex the entire input source, finding errors and reporting them and stopping as soon as the first lexing error is encountered or the input has been completely lexed.
The TokenStream returned by the Lex function is guaranteed to not have any error tokens.
func NewParser ¶
func NewParser(g grammar.CFG, allowAmbiguous bool) (parser parse.Parser, ambigWarns []string, err error)
NewParser returns what is the most flexible and efficient parser in this package that can parse the given grammar. The following parsers will be attempted to be built, in order, with each subsequent one attempted after the prior one fails: LALR(1), CLR(1), SLR(1), LL(1).
Returns an error if no parser can be generated for the given grammar.
allowAmbiguous allows the use of ambiguous grammars in LR parsers. It has no effect on LL(1) parser generation; LL(1) grammars must be unambiguous.
func NewSDTS ¶ added in v0.3.0
NewSDTS returns a new Syntax-Directed Translation Scheme. The SDTS will be empty and ready to accept bindings, which must be manually added by callers.
func NewSLRParser ¶
func NewSLRParser(g grammar.CFG, allowAmbiguous bool) (parser parse.Parser, ambigWarns []string, err error)
NewSLRParser returns an SLR(k) parser for the given grammar. The value of k will be the highest possible to provide with ictiobus. Returns an error if the grammar is not SLR(k).
At the time of this writing, the greatest k = 1.
Types ¶
type Frontend ¶
type Frontend[E any] struct { Lexer lex.Lexer Parser parse.Parser SDTS trans.SDTS IRAttribute string Language string Version string }
Frontend is a complete input-to-intermediate representation compiler frontend, including all information necessary to process input written in the language it was created for. When Analyze is called, it reads input from the provided io.Reader and produces the IR value or a syntax error if the input could not be parsed.
Creation of a Frontend is complicated and requires setting up all three phases of frontend analysis; manually doing so is not recommended. Instead, Frontends can be generated by running ictcc on a FISHI langage spec to produce Go source code that provides a pre-configured Frontend for the language. Information on doing this can be found in the README.md in the root of the ictiobus repository.
func (Frontend[E]) Analyze ¶
Analyze reads input text in the Frontend's language from r and parses it to produce an analyzed value. First, it lexes the input read from r using fe.Lexer to produce a stream of tokens. This stream is consumed by fe.Parser to produce a parse tree. Finally the actions of the syntax-directed translation scheme in fe.SDTS are applied to the parse tree to annotate it, and the final value for the IR is taken from the attribute named fe.IRAttribute in the root node of the annotated tree.
If there is a problem with the input, it will be returned in a syntaxerr.Error containing information about the location where it occured in the source text read from r. The returned parse tree may be valid even if there is an error, in which case pt will be non-nil.
Directories ¶
Path | Synopsis |
---|---|
Package automaton provides automata, theoretical structures that change state based on reading some input.
|
Package automaton provides automata, theoretical structures that change state based on reading some input. |
regex
Package regex converts regular expressions into finite automata that accept them but is incomplete at this time.
|
Package regex converts regular expressions into finite automata that accept them but is incomplete at this time. |
cmd
|
|
ictcc
Ictcc produces compiler frontends written in Go from frontend specifications written in FISHI.
|
Ictcc produces compiler frontends written in Go from frontend specifications written in FISHI. |
Package fishi provides parsing and reading of FISHI frontend specifications for ictiobus.
|
Package fishi provides parsing and reading of FISHI frontend specifications for ictiobus. |
fe
Package fe contains the frontend for analyzing FISHI code.
|
Package fe contains the frontend for analyzing FISHI code. |
fe/fetoken
Package fetoken contains the token classes used by the frontend of FISHI.
|
Package fetoken contains the token classes used by the frontend of FISHI. |
format
Package format contains functions for producing a [CodeReader] from a stream that contains markdown files with fishi codeblocks.
|
Package format contains functions for producing a [CodeReader] from a stream that contains markdown files with fishi codeblocks. |
syntax
Package syntax provides functions for building up an abstract syntax tree from a FISHI markdown file.
|
Package syntax provides functions for building up an abstract syntax tree from a FISHI markdown file. |
Package grammar implements context-free grammars and associated constructs.
|
Package grammar implements context-free grammars and associated constructs. |
internal
|
|
box
Package box contains generic types and interfaces for various container data types.
|
Package box contains generic types and interfaces for various container data types. |
rezi
Package rezi, the Rarefied Encoding (Compressible) Interpreter, contains functions for marshaling data as binary bytes using a simple encoding scheme.
|
Package rezi, the Rarefied Encoding (Compressible) Interpreter, contains functions for marshaling data as binary bytes using a simple encoding scheme. |
shellout
Package shellout contains functions for executing shell commands.
|
Package shellout contains functions for executing shell commands. |
slices
Package slices contains functions useful for operating on slices.
|
Package slices contains functions useful for operating on slices. |
tmatch
Package tmatch provides some quick and dirty matching functions with detailed mismatch error messages for use with unit tests.
|
Package tmatch provides some quick and dirty matching functions with detailed mismatch error messages for use with unit tests. |
Package lex provides lexing functionality for the ictiobus parser generator.
|
Package lex provides lexing functionality for the ictiobus parser generator. |
Package parse provides language parsers for the ictiobus parser generator.
|
Package parse provides language parsers for the ictiobus parser generator. |
Package syntaxerr provides a common syntax error type for all of ictiobus.
|
Package syntaxerr provides a common syntax error type for all of ictiobus. |
Package trans provides syntax-directed translations of parse trees for the ictiobus parser generator.
|
Package trans provides syntax-directed translations of parse trees for the ictiobus parser generator. |