parser

README

Parser

TiDB SQL Parser

How to update parser for TiDB

Assuming that you want to file a PR (pull request) to TiDB, and your PR includes a change in the parser, follow these steps to update the parser in TiDB.

Step 1: Make changes in your parser repository

Fork this repository to your own account and commit the changes to your repository.

Note:

• Don't forget to run make test before you commit!
• Make sure parser.go is updated.

Suppose the forked repository is https://github.com/your-repo/parser.

Step 2: Make your parser changes take effect in TiDB and run CI

1. In your TiDB repository, modify the go.mod file, remove github.com/pingcap/parser from the require instruction, and add a new line at the end of the file like this:

   replace github.com/pingcap/parser => github.com/your-repo/parser v0.0.0-20181102150703-4acd198f5092
   

   This change tells TiDB to use the modified parser from your repository.

2. You can get correct version information by running this command in your TiDB directory:

   GO111MODULE=on go get -u github.com/your-repo/parser@master
   

   If some error is reported, you can ignore it and still edit the go.mod file manually.

3. File a PR to TiDB.

Step 3: Merge the PR about the parser to this repository

File a PR to this repository. Link the related PR in TiDB in your PR description or comment.

This PR will be reviewed, and if everything goes well, it will be merged.

Step 4: Update TiDB to use the latest parser

In your TiDB pull request, modify the go.mod file manually or use this command:

GO111MODULE=on go get -u github.com/pingcap/parser@master

Make sure the replace instruction is changed back to the require instruction and the version is the latest.

Documentation

Overview

Example (ParseSQL)

This example show how to parse a text sql into ast.

package main

import (
	"fmt"
	"github.com/pingcap/parser"

	_ "github.com/pingcap/tidb/types/parser_driver"
)

func main() {

	// 0. make sure import parser_driver implemented by TiDB(user also can implement own driver by self).
	// and add `import _ "github.com/pingcap/tidb/types/parser_driver"` in the head of file.

	// 1. Create a parser. The parser is NOT goroutine safe and should
	// not be shared among multiple goroutines. However, parser is also
	// heavy, so each goroutine should reuse its own local instance if
	// possible.
	p := parser.New()

	// 2. Parse a text SQL into AST([]ast.StmtNode).
	stmtNodes, _, err := p.Parse("select * from tbl where id = 1", "", "")

	// 3. Use AST to do cool things.
	fmt.Println(stmtNodes[0], err)
}

Index

• Variables
• func DigestHash(sql string) (result string)
• func Normalize(sql string) (result string)
• func NormalizeDigest(sql string) (normalized, digest string)
• func ParseErrorWith(errstr string, lineno int) error
• func TrimComment(txt string) string
• type Parser
  • func New() *Parser
  • func (parser *Parser) EnableWindowFunc()
  • func (parser *Parser) Parse(sql, charset, collation string) (stmt []ast.StmtNode, warns []error, err error)
  • func (parser *Parser) ParseOneStmt(sql, charset, collation string) (ast.StmtNode, error)
  • func (parser *Parser) SetSQLMode(mode mysql.SQLMode)
• type Pos
• type Scanner
  • func NewScanner(s string) *Scanner
  • func (s *Scanner) AppendError(err error)
  • func (s *Scanner) EnableWindowFunc()
  • func (s *Scanner) Errorf(format string, a ...interface{}) (err error)
  • func (s *Scanner) Errors() (warns []error, errs []error)
  • func (s *Scanner) GetSQLMode() mysql.SQLMode
  • func (s *Scanner) Lex(v *yySymType) int
  • func (s *Scanner) SetSQLMode(mode mysql.SQLMode)

Examples

• Package (ParseSQL)

Variables

var (
	// ErrSyntax returns for sql syntax error.
	ErrSyntax = terror.ClassParser.New(codeErrSyntax, mysql.MySQLErrName[mysql.ErrSyntax])
	// ErrParse returns for sql parse error.
	ErrParse = terror.ClassParser.New(codeErrParse, mysql.MySQLErrName[mysql.ErrParse])
	// ErrUnknownCharacterSet returns for no character set found error.
	ErrUnknownCharacterSet = terror.ClassParser.New(codeErrUnknownCharacterSet, mysql.MySQLErrName[mysql.ErrUnknownCharacterSet])
	// SpecFieldPattern special result field pattern
	SpecFieldPattern = regexp.MustCompile(`(\/\*!(M?[0-9]{5,6})?|\*\/)`)
)

Functions

func DigestHash

func DigestHash(sql string) (result string)

DigestHash generates the digest of statements. it will generate a hash on normalized form of statement text which removes general property of a statement but keeps specific property.

for example: both DigestHash('select 1') and DigestHash('select 2') => e1c71d1661ae46e09b7aaec1c390957f0d6260410df4e4bc71b9c8d681021471

func Normalize

func Normalize(sql string) (result string)

Normalize generates the normalized statements. it will get normalized form of statement text which removes general property of a statement but keeps specific property.

for example: Normalize('select 1 from b where a = 1') => 'select ? from b where a = ?'

func NormalizeDigest

func NormalizeDigest(sql string) (normalized, digest string)

NormalizeDigest combines Normalize and DigestHash into one method.

func ParseErrorWith

func ParseErrorWith(errstr string, lineno int) error

ParseErrorWith returns "You have a syntax error near..." error message compatible with mysql.

func TrimComment

func TrimComment(txt string) string

TrimComment trim comment for special comment code of MySQL.

Types

type Parser

type Parser struct {
	// contains filtered or unexported fields
}

Parser represents a parser instance. Some temporary objects are stored in it to reduce object allocation during Parse function.

func New

func New() *Parser

New returns a Parser object.

func (*Parser) EnableWindowFunc

func (parser *Parser) EnableWindowFunc()

EnableWindowFunc enables the parser to parse syntax related with window function.

func (*Parser) Parse

func (parser *Parser) Parse(sql, charset, collation string) (stmt []ast.StmtNode, warns []error, err error)

Parse parses a query string to raw ast.StmtNode. If charset or collation is "", default charset and collation will be used.

func (*Parser) ParseOneStmt

func (parser *Parser) ParseOneStmt(sql, charset, collation string) (ast.StmtNode, error)

ParseOneStmt parses a query and returns an ast.StmtNode. The query must have one statement, otherwise ErrSyntax is returned.

func (*Parser) SetSQLMode

func (parser *Parser) SetSQLMode(mode mysql.SQLMode)

SetSQLMode sets the SQL mode for parser.

type Pos

type Pos struct {
	Line   int
	Col    int
	Offset int
}

Pos represents the position of a token.

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner implements the yyLexer interface.

func NewScanner

func NewScanner(s string) *Scanner

NewScanner returns a new scanner object.

func (*Scanner) AppendError

func (s *Scanner) AppendError(err error)

AppendError sets error into scanner. Scanner satisfies yyLexer interface which need this function.

func (*Scanner) EnableWindowFunc

func (s *Scanner) EnableWindowFunc()

EnableWindowFunc enables the scanner to recognize the keywords of window function.

func (*Scanner) Errorf

func (s *Scanner) Errorf(format string, a ...interface{}) (err error)

Errorf tells scanner something is wrong. Scanner satisfies yyLexer interface which need this function.

func (*Scanner) Errors

func (s *Scanner) Errors() (warns []error, errs []error)

Errors returns the errors and warns during a scan.

func (*Scanner) GetSQLMode

func (s *Scanner) GetSQLMode() mysql.SQLMode

GetSQLMode return the SQL mode of scanner.

func (*Scanner) Lex

func (s *Scanner) Lex(v *yySymType) int

Lex returns a token and store the token value in v. Scanner satisfies yyLexer interface. 0 and invalid are special token id this function would return: return 0 tells parser that scanner meets EOF, return invalid tells parser that scanner meets illegal character.

func (*Scanner) SetSQLMode

func (s *Scanner) SetSQLMode(mode mysql.SQLMode)

SetSQLMode sets the SQL mode for scanner.