scanner

Documentation

Index

• type ErrorHandler
• type Mode
• type Scanner
  • func (s *Scanner) Init(file *token.File, src []byte, err ErrorHandler, mode Mode)
  • func (s *Scanner) Scan() (pos token.Pos, tok token.Token, tokValue string)

Examples

• Scanner (Hello)
• Scanner.Scan

Types

type ErrorHandler

type ErrorHandler func(pos token.Position, msg string)

An ErrorHandler may be provided to Scanner.Init. If a syntax error is encountered and a handler was installed, the handler is called with a position and an error message. The position points to the beginning of the offending token.

type Mode

type Mode uint

A mode value is a set of flags (or 0). They control scanner behavior.

const (
	ScanComments Mode = 1 << iota // return comments as COMMENT tokens

)

type Scanner

type Scanner struct {

	// public state - ok to modify
	Source     []byte // source
	ErrorCount int    // number of errors encountered
	// contains filtered or unexported fields
}

A Scanner holds the scanner's internal state while processing a given text. It can be allocated as part of another data structure but must be initialized via Init before use.

Example (Hello)

package main

import (
	"fmt"

	"wa-lang.org/wa/internal/wat/scanner"
	"wa-lang.org/wa/internal/wat/token"
)

func main() {
	var src = []byte(tHello)
	var file = token.NewFile("", len(src))

	var s scanner.Scanner
	s.Init(file, src, nil, scanner.ScanComments)

	for {
		pos, tok, lit := s.Scan()
		if tok == token.EOF {
			break
		}
		if tok == token.ILLEGAL {
			fmt.Printf("failed: %v: %s %q\n", file.Position(pos), tok, lit)
			return
		}
	}
	fmt.Println("ok")

}

const tHello = `(module $hello_wasi
    ;; type iov struct { iov_base, iov_len int32 }
    ;; func fd_write(fd int32, id *iov, iovs_len int32, nwritten *int32) (errno int32)
    (import "wasi_snapshot_preview1" "fd_write" (func $fd_write (param i32 i32 i32 i32) (result i32)))

    (memory 1)(export "memory" (memory 0))

    ;; 前 8 个字节保留给 iov 数组, 字符串从地址 8 开始
    (data (i32.const 8) "hello world\n")

    ;; _start 类似 main 函数, 自动执行
    (func $main (export "_start")
        (i32.store (i32.const 0) (i32.const 8))  ;; iov.iov_base - 字符串地址为 8
        (i32.store (i32.const 4) (i32.const 12)) ;; iov.iov_len  - 字符串长度

        (call $fd_write
            (i32.const 1)  ;; 1 对应 stdout
            (i32.const 0)  ;; *iovs - 前 8 个字节保留给 iov 数组
            (i32.const 1)  ;; len(iovs) - 只有1个字符串
            (i32.const 20) ;; nwritten - 指针, 里面是要写到数据长度
        )
        drop ;; 忽略返回值
    )
)`

Output:

ok

func (*Scanner) Init

func (s *Scanner) Init(file *token.File, src []byte, err ErrorHandler, mode Mode)

Init prepares the scanner s to tokenize the text src by setting the scanner at the beginning of src. The scanner uses the file set file for position information and it adds line information for each line. It is ok to re-use the same file when re-scanning the same file as line information which is already present is ignored. Init causes a panic if the file size does not match the src size.

Calls to Scan will invoke the error handler err if they encounter a syntax error and err is not nil. Also, for each error encountered, the Scanner field ErrorCount is incremented by one. The mode parameter determines how comments are handled.

Note that Init may call err if there is an error in the first character of the file.

func (*Scanner) Scan

func (s *Scanner) Scan() (pos token.Pos, tok token.Token, tokValue string)

Scan scans the next token and returns the token position, the token, and its string encoded value if applicable. The source end is indicated by token.EOF.

If the returned token is a literal (token.IDENT, token.INT, token.FLOAT, token.IMAG, token.CHAR, token.STRING) or token.COMMENT, the literal string has the corresponding value.

If the returned token is a keyword, the literal string is the keyword.

If the returned token is token.ILLEGAL, the literal string is the offending character.

In all other cases, Scan returns an empty literal string.

For more tolerant parsing, Scan will return a valid token if possible even if a syntax error was encountered. Thus, even if the resulting token sequence contains no illegal tokens, a client may not assume that no error occurred. Instead it must check the scanner's ErrorCount or the number of calls of the error handler, if there was one installed.

Scan adds line information to the file with Init. Token positions are relative to that file.

Example

package main

import (
	"fmt"

	"wa-lang.org/wa/internal/wat/scanner"
	"wa-lang.org/wa/internal/wat/token"
)

func main() {
	var src = []byte("(module $__walang__)")
	var file = token.NewFile("", len(src))

	var s scanner.Scanner
	s.Init(file, src, nil, scanner.ScanComments)

	for {
		_, tok, lit := s.Scan()
		if tok == token.EOF {
			break
		}
		fmt.Printf("%s %q\n", tok, lit)
	}

}

Output:

( ""
module "module"
IDENT "__walang__"
) ""