filter

README

Binlog Filter

introduction

Binlog Filter is a libary to provide a simple and unified way to filter binlog events with the following features:

• Do/Ignore binlog events

  Synchronize/Ignore some specified replicated Binlog Events from these specfied databases/tables by given rules.

• Do/Ignore queries

  Synchronize/Ignore some specified replicated queries that is in Binog Query Event from these specfied databases/tables by given rules.

binlog event rule

we define a rule BinlogEventRule to filter specified Binlog Events and queries that is in Binog Query Event

type BinlogEventRule struct {
	SchemaPattern string      `json:"schema-pattern" toml:"schema-pattern" yaml:"schema-pattern"`
	TablePattern  string      `json:"table-pattern" toml:"table-pattern" yaml:"table-pattern"`
	Events        []EventType `json:"events" toml:"events" yaml:"events"`
	SQLPattern    []string    `json:"sql-pattern" toml:"sql-pattern" yaml:"sql-pattern"` // regular expression

	Action ActionType `json:"action" toml:"action" yaml:"action"`
}

now we support following events

// it indicates all dml/ddl events in rule
AllEvent
AllDDL
AllDML
    
// it indicates no any dml/ddl events in rule,
// and equals empty rule.DDLEvent/DMLEvent array
NoneEvent
NoneDDL
NoneDML

// DML events
InsertEvent
UpdateEvent
DeleteEvent

// DDL events
CreateDatabase
DropDatabase
CreateTable
DropTable
TruncateTable
RenameTable
CreateIndex
DropIndex
AlertTable

// unknown event
NullEvent EventType = ""

notice

if you want to use BinlogEventRule to synchronize/ignore some table, you may need to pay attention to setting AllEvent and NoneEvent.

like synchronizing all events from specified table or setting a do table, ignore is opposite.

BinlogEventRule {
	SchemaPattern: test*,
	TablePattern:  test*,
	DMLEvent:     []EventType{AllEvent},               
	DDLEvent:     []EventType{AllEvent},

    Action: Do,
}

Documentation

Index

• type ActionType
• type BinlogEvent
  • func NewBinlogEvent(caseSensitive bool, rules []*BinlogEventRule) (*BinlogEvent, error)
  • func (b *BinlogEvent) AddRule(rule *BinlogEventRule) error
  • func (b *BinlogEvent) Filter(schema, table string, event EventType, rawQuery string) (ActionType, error)
  • func (b *BinlogEvent) RemoveRule(rule *BinlogEventRule) error
  • func (b *BinlogEvent) UpdateRule(rule *BinlogEventRule) error
• type BinlogEventRule
  • func (b *BinlogEventRule) ToLower()
  • func (b *BinlogEventRule) Valid() error
• type EventType
  • func AstToDDLEvent(node ast.StmtNode) EventType
  • func ClassifyEvent(event EventType) (EventType, error)

Types

type ActionType

type ActionType string

ActionType indicates how to handle matched items

const (
	Ignore ActionType = "Ignore"
	Do     ActionType = "Do"
)

show that how to handle rules

type BinlogEvent

type BinlogEvent struct {
	selector.Selector
	// contains filtered or unexported fields
}

BinlogEvent filters binlog events by given rules

func NewBinlogEvent

func NewBinlogEvent(caseSensitive bool, rules []*BinlogEventRule) (*BinlogEvent, error)

NewBinlogEvent returns a binlog event filter

func (*BinlogEvent) AddRule

func (b *BinlogEvent) AddRule(rule *BinlogEventRule) error

AddRule adds a rule into binlog event filter

func (*BinlogEvent) Filter

func (b *BinlogEvent) Filter(schema, table string, event EventType, rawQuery string) (ActionType, error)

Filter filters events or queries by given rules returns action and error

func (*BinlogEvent) RemoveRule

func (b *BinlogEvent) RemoveRule(rule *BinlogEventRule) error

RemoveRule removes a rule from binlog event filter

func (*BinlogEvent) UpdateRule

func (b *BinlogEvent) UpdateRule(rule *BinlogEventRule) error

UpdateRule updates binlog event filter rule

type BinlogEventRule

type BinlogEventRule struct {
	SchemaPattern string      `json:"schema-pattern" toml:"schema-pattern" yaml:"schema-pattern"`
	TablePattern  string      `json:"table-pattern" toml:"table-pattern" yaml:"table-pattern"`
	Events        []EventType `json:"events" toml:"events" yaml:"events"`
	SQLPattern    []string    `json:"sql-pattern" toml:"sql-pattern" yaml:"sql-pattern"` // regular expression

	Action ActionType `json:"action" toml:"action" yaml:"action"`
	// contains filtered or unexported fields
}

BinlogEventRule is a rule to filter binlog events

func (*BinlogEventRule) ToLower

func (b *BinlogEventRule) ToLower()

ToLower covert schema/table pattern to lower case

func (*BinlogEventRule) Valid

func (b *BinlogEventRule) Valid() error

Valid checks validity of rule. TODO: check validity of dml/ddl event.

type EventType

type EventType string

EventType is DML/DDL Event type

const (

	// it indicates all dml/ddl events in rule
	AllEvent EventType = "all"
	AllDDL   EventType = "all ddl"
	AllDML   EventType = "all dml"

	// it indicates no any dml/ddl events in rule,
	// and equals empty rule.DDLEvent/DMLEvent
	NoneEvent EventType = "none"
	NoneDDL   EventType = "none ddl"
	NoneDML   EventType = "none dml"

	InsertEvent EventType = "insert"
	UpdateEvent EventType = "update"
	DeleteEvent EventType = "delete"

	CreateDatabase EventType = "create database"
	DropDatabase   EventType = "drop database"
	CreateTable    EventType = "create table"
	DropTable      EventType = "drop table"
	TruncateTable  EventType = "truncate table"
	RenameTable    EventType = "rename table"
	CreateIndex    EventType = "create index"
	DropIndex      EventType = "drop index"
	AlertTable     EventType = "alter table"

	NullEvent EventType = ""
)

show DML/DDL Events

func AstToDDLEvent

func AstToDDLEvent(node ast.StmtNode) EventType

AstToDDLEvent returns filter.DDLEvent

func ClassifyEvent

func ClassifyEvent(event EventType) (EventType, error)

ClassifyEvent classify event into dml/ddl