parser

README

Parser - A MySQL Compatible SQL Parser

The goal of this project is to build a Golang parser that is fully compatible with MySQL syntax, easy to extend, and high performance. Currently, features supported by parser are as follows:

• Highly compatible with MySQL: it supports almost all features of MySQL. For the complete details, see parser.y and hintparser.y.
• Extensible: adding a new syntax requires only a few lines of Yacc and Golang code changes. As an example, see PR-680.
• Good performance: the parser is generated by goyacc in a bottom-up approach. It is efficient to build an AST tree with a state machine.

How to use it

Please read the quickstart.

Future

• Support more MySQL syntax
• Optimize the code structure, make it easier to extend
• Improve performance and benchmark
• Improve the quality of code and comments

Getting Help

• GitHub Issue
• Stack Overflow
• User Group (Chinese)

If you have any questions, feel free to discuss in sig-ddl. Here are the steps to join:

1. Join TiDB Slack community, and then
2. Join sig-ddl Slack channel.

If you want to join as a special interest group member, see DDL Special Interest Group.

Users

These projects use this parser. Please feel free to extend this list if you found you are one of the users but not listed here:

• pingcap/tidb
• XiaoMi/soar
• XiaoMi/Gaea
• sql-machine-learning/sqlflow
• nooncall/shazam
• bytebase/bytebase

Contributing

Contributions are welcomed and greatly appreciated. See Contribution Guide for details on submitting patches and the contribution workflow.

Acknowledgments

Thanks cznic for providing some great open-source tools.

License

Parser is under the Apache 2.0 license. See the LICENSE file for details.

More resources

• TiDB documentation

  • English
  • 简体中文

• TiDB blog

  • English
  • 简体中文

Documentation

Index

• Variables
• func Normalize(sql string) (result string)
• func ParseErrorWith(errstr string, lineno int) error
• func ParseHint(input string, sqlMode mysql.SQLMode, initPos Pos) ([]*ast.TableOptimizerHint, []error)
• func TrimComment(txt string) string
• type CharsetClient
  • func (c CharsetClient) ApplyOn(p *Parser) error
• type CharsetConnection
  • func (c CharsetConnection) ApplyOn(p *Parser) error
• type CollationConnection
  • func (c CollationConnection) ApplyOn(p *Parser) error
• type Digest
  • func DigestHash(sql string) (digest *Digest)
  • func DigestNormalized(normalized string) (digest *Digest)
  • func NewDigest(b []byte) *Digest
  • func NormalizeDigest(sql string) (normalized string, digest *Digest)
  • func (d *Digest) Bytes() []byte
  • func (d *Digest) String() string
• type ParseParam
• type Parser
  • func New() *Parser
  • func (parser *Parser) EnableWindowFunc(val bool)
  • func (parser *Parser) Parse(sql, charset, collation string) (stmt []ast.StmtNode, warns []error, err error)
  • func (parser *Parser) ParseOneStmt(sql, charset, collation string) (ast.StmtNode, error)
  • func (parser *Parser) ParseSQL(sql string, params ...ParseParam) (stmt []ast.StmtNode, warns []error, err error)
  • func (parser *Parser) SetParserConfig(config ParserConfig)
  • func (parser *Parser) SetSQLMode(mode mysql.SQLMode)
  • func (parser *Parser) SetStrictDoubleTypeCheck(val bool)
• type ParserConfig
• type Pos
• type Scanner
  • func NewScanner(s string) *Scanner
  • func (s *Scanner) AppendError(err error)
  • func (s *Scanner) AppendWarn(err error)
  • func (s *Scanner) EnableWindowFunc(val bool)
  • func (s *Scanner) Errorf(format string, a ...interface{}) (err error)
  • func (s *Scanner) Errors() (warns []error, errs []error)
  • func (s *Scanner) GetSQLMode() mysql.SQLMode
  • func (s *Scanner) InheritScanner(sql string) *Scanner
  • func (s *Scanner) Lex(v *yySymType) int
  • func (s *Scanner) LexLiteral() interface{}
  • func (s *Scanner) SetSQLMode(mode mysql.SQLMode)

Variables

var (
	ErrWarnOptimizerHintUnsupportedHint = terror.ClassParser.NewStd(mysql.ErrWarnOptimizerHintUnsupportedHint)
	ErrWarnOptimizerHintInvalidToken    = terror.ClassParser.NewStd(mysql.ErrWarnOptimizerHintInvalidToken)
	ErrWarnMemoryQuotaOverflow          = terror.ClassParser.NewStd(mysql.ErrWarnMemoryQuotaOverflow)
	ErrWarnOptimizerHintParseError      = terror.ClassParser.NewStd(mysql.ErrWarnOptimizerHintParseError)
	ErrWarnOptimizerHintInvalidInteger  = terror.ClassParser.NewStd(mysql.ErrWarnOptimizerHintInvalidInteger)
)

var (
	// ErrSyntax returns for sql syntax error.
	ErrSyntax = terror.ClassParser.NewStd(mysql.ErrSyntax)
	// ErrParse returns for sql parse error.
	ErrParse = terror.ClassParser.NewStd(mysql.ErrParse)
	// ErrUnknownCharacterSet returns for no character set found error.
	ErrUnknownCharacterSet = terror.ClassParser.NewStd(mysql.ErrUnknownCharacterSet)
	// ErrInvalidYearColumnLength returns for illegal column length for year type.
	ErrInvalidYearColumnLength = terror.ClassParser.NewStd(mysql.ErrInvalidYearColumnLength)
	// ErrWrongArguments returns for illegal argument.
	ErrWrongArguments = terror.ClassParser.NewStd(mysql.ErrWrongArguments)
	// ErrWrongFieldTerminators returns for illegal field terminators.
	ErrWrongFieldTerminators = terror.ClassParser.NewStd(mysql.ErrWrongFieldTerminators)
	// ErrTooBigDisplayWidth returns for data display width exceed limit .
	ErrTooBigDisplayWidth = terror.ClassParser.NewStd(mysql.ErrTooBigDisplaywidth)
	// ErrTooBigPrecision returns for data precision exceed limit.
	ErrTooBigPrecision = terror.ClassParser.NewStd(mysql.ErrTooBigPrecision)
	// ErrUnknownAlterLock returns for no alter lock type found error.
	ErrUnknownAlterLock = terror.ClassParser.NewStd(mysql.ErrUnknownAlterLock)
	// ErrUnknownAlterAlgorithm returns for no alter algorithm found error.
	ErrUnknownAlterAlgorithm = terror.ClassParser.NewStd(mysql.ErrUnknownAlterAlgorithm)
	// ErrWrongValue returns for wrong value
	ErrWrongValue = terror.ClassParser.NewStd(mysql.ErrWrongValue)
	// ErrWarnDeprecatedSyntaxNoReplacement return when the syntax was deprecated and there is no replacement.
	ErrWarnDeprecatedSyntaxNoReplacement = terror.ClassParser.NewStd(mysql.ErrWarnDeprecatedSyntaxNoReplacement)
	// ErrWarnDeprecatedIntegerDisplayWidth share the same code 1681, and it will be returned when length is specified in integer.
	ErrWarnDeprecatedIntegerDisplayWidth = terror.ClassParser.NewStdErr(mysql.ErrWarnDeprecatedSyntaxNoReplacement, mysql.Message("Integer display width is deprecated and will be removed in a future release.", nil))
	// ErrWrongUsage returns for incorrect usages.
	ErrWrongUsage = terror.ClassParser.NewStd(mysql.ErrWrongUsage)
	// SpecFieldPattern special result field pattern
	SpecFieldPattern = regexp.MustCompile(`(\/\*!(M?[0-9]{5,6})?|\*\/)`)
)

Functions

func Normalize

func Normalize(sql string) (result string)

Normalize generates the normalized statements. it will get normalized form of statement text which removes general property of a statement but keeps specific property.

for example: Normalize('select 1 from b where a = 1') => 'select ? from b where a = ?'

func ParseErrorWith

func ParseErrorWith(errstr string, lineno int) error

ParseErrorWith returns "You have a syntax error near..." error message compatible with mysql.

func ParseHint

func ParseHint(input string, sqlMode mysql.SQLMode, initPos Pos) ([]*ast.TableOptimizerHint, []error)

ParseHint parses an optimizer hint (the interior of `/*+ ... */`).

func TrimComment

func TrimComment(txt string) string

TrimComment trim comment for special comment code of MySQL.

Types

type CharsetClient

type CharsetClient string

CharsetClient specifies the charset of a SQL. This is used to decode the SQL into a utf-8 string.

func (CharsetClient) ApplyOn

func (c CharsetClient) ApplyOn(p *Parser) error

ApplyOn implements ParseParam interface.

type CharsetConnection

type CharsetConnection string

CharsetConnection is used for literals specified without a character set.

func (CharsetConnection) ApplyOn

func (c CharsetConnection) ApplyOn(p *Parser) error

ApplyOn implements ParseParam interface.

type CollationConnection

type CollationConnection string

CollationConnection is used for literals specified without a collation.

func (CollationConnection) ApplyOn

func (c CollationConnection) ApplyOn(p *Parser) error

ApplyOn implements ParseParam interface.

type Digest

type Digest struct {
	// contains filtered or unexported fields
}

func DigestHash

func DigestHash(sql string) (digest *Digest)

DigestHash generates the digest of statements. it will generate a hash on normalized form of statement text which removes general property of a statement but keeps specific property.

for example: both DigestHash('select 1') and DigestHash('select 2') => e1c71d1661ae46e09b7aaec1c390957f0d6260410df4e4bc71b9c8d681021471

Deprecated: It is logically consistent with NormalizeDigest.

func DigestNormalized

func DigestNormalized(normalized string) (digest *Digest)

DigestNormalized generates the digest of a normalized sql. it will generate a hash on a normalized sql. Normalize + DigestNormalized equals to NormalizeDigest.

for example: DigestNormalized('select ?') DigestNormalized should be called with a normalized SQL string (like 'select ?') generated by function Normalize. do not call with SQL which is not normalized, DigestNormalized('select 1') and DigestNormalized('select 2') is not the same

func NewDigest

func NewDigest(b []byte) *Digest

NewDigest returns a new digest.

func NormalizeDigest

func NormalizeDigest(sql string) (normalized string, digest *Digest)

NormalizeDigest combines Normalize and DigestNormalized into one method.

func (*Digest) Bytes

func (d *Digest) Bytes() []byte

Bytes returns the digest byte slice.

func (*Digest) String

func (d *Digest) String() string

String returns the digest hex string.

type ParseParam

type ParseParam interface {
	ApplyOn(*Parser) error
}

ParseParam represents the parameter of parsing.

type Parser

type Parser struct {
	// contains filtered or unexported fields
}

Parser represents a parser instance. Some temporary objects are stored in it to reduce object allocation during Parse function.

func New

func New() *Parser

New returns a Parser object with default SQL mode.

func (*Parser) EnableWindowFunc

func (parser *Parser) EnableWindowFunc(val bool)

EnableWindowFunc controls whether the parser to parse syntax related with window function.

func (*Parser) Parse

func (parser *Parser) Parse(sql, charset, collation string) (stmt []ast.StmtNode, warns []error, err error)

Parse parses a query string to raw ast.StmtNode. If charset or collation is "", default charset and collation will be used.

func (*Parser) ParseOneStmt

func (parser *Parser) ParseOneStmt(sql, charset, collation string) (ast.StmtNode, error)

ParseOneStmt parses a query and returns an ast.StmtNode. The query must have one statement, otherwise ErrSyntax is returned.

func (*Parser) ParseSQL

func (parser *Parser) ParseSQL(sql string, params ...ParseParam) (stmt []ast.StmtNode, warns []error, err error)

ParseSQL parses a query string to raw ast.StmtNode.

func (*Parser) SetParserConfig

func (parser *Parser) SetParserConfig(config ParserConfig)

func (*Parser) SetSQLMode

func (parser *Parser) SetSQLMode(mode mysql.SQLMode)

SetSQLMode sets the SQL mode for parser.

func (*Parser) SetStrictDoubleTypeCheck

func (parser *Parser) SetStrictDoubleTypeCheck(val bool)

type ParserConfig

type ParserConfig struct {
	EnableWindowFunction        bool
	EnableStrictDoubleTypeCheck bool
	SkipPositionRecording       bool
}

type Pos

type Pos struct {
	Line   int
	Col    int
	Offset int
}

Pos represents the position of a token.

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner implements the yyLexer interface.

func NewScanner

func NewScanner(s string) *Scanner

NewScanner returns a new scanner object.

func (*Scanner) AppendError

func (s *Scanner) AppendError(err error)

AppendError sets error into scanner. Scanner satisfies yyLexer interface which need this function.

func (*Scanner) AppendWarn

func (s *Scanner) AppendWarn(err error)

AppendWarn sets warning into scanner.

func (*Scanner) EnableWindowFunc

func (s *Scanner) EnableWindowFunc(val bool)

EnableWindowFunc controls whether the scanner recognize the keywords of window function.

func (*Scanner) Errorf

func (s *Scanner) Errorf(format string, a ...interface{}) (err error)

Errorf tells scanner something is wrong. Scanner satisfies yyLexer interface which need this function.

func (*Scanner) Errors

func (s *Scanner) Errors() (warns []error, errs []error)

Errors returns the errors and warns during a scan.

func (*Scanner) GetSQLMode

func (s *Scanner) GetSQLMode() mysql.SQLMode

GetSQLMode return the SQL mode of scanner.

func (*Scanner) InheritScanner

func (s *Scanner) InheritScanner(sql string) *Scanner

InheritScanner returns a new scanner object which inherits configurations from the parent scanner.

func (*Scanner) Lex

func (s *Scanner) Lex(v *yySymType) int

Lex returns a token and store the token value in v. Scanner satisfies yyLexer interface. 0 and invalid are special token id this function would return: return 0 tells parser that scanner meets EOF, return invalid tells parser that scanner meets illegal character.

func (*Scanner) LexLiteral

func (s *Scanner) LexLiteral() interface{}

LexLiteral returns the value of the converted literal

func (*Scanner) SetSQLMode

func (s *Scanner) SetSQLMode(mode mysql.SQLMode)

SetSQLMode sets the SQL mode for scanner.