bpl

package module
v1.1.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 31, 2022 License: Apache-2.0 Imports: 15 Imported by: 0

README

BPL - Binary Processing Language

Build Status Go Report Card GitHub release Coverage Status GoDoc

快速入门

了解 BPL 最快的方式是学习 qbpl 和 qbplproxy 两个实用程序:

qbpl

qbpl 可用来分析任意的文件格式。使用方法如下:

qbpl [-p <protocol>.bpl -o <output>.log] <file>

多数情况下,你不需要指定 -p <protocol>.bpl 参数,我们根据文件后缀来确定应该使用何种 protocol 来解析这个文件。例如:

qbpl 1.gif

不过为了让 qbpl 能够找到所有的 protocols,我们需要先安装:

make install # 这将将所有的bpl文件拷贝到 ~/.qbpl/formats/
qbplproxy

qbplproxy 可用来分析服务器和客户端之间的网络包。它通过代理要分析的服务,让客户端请求自己来分析请求包和返回包。使用方式如下:

qbplproxy -h <listenIp:port> -b <backendIp:port> [-p <protocol>.bpl -f <filter> -o <output>.log]

其中,<listenIp:port> 是 qbplproxy 自身监听的IP和端口,<backendIp:port> 是原始的服务。-f <filter> 是过滤条件,这个条件通过 BPL_FILTER 全局变量传递到 bpl 中。

多数情况下,你不需要指定 -p <protocol>.bpl 参数,qbplproxy 程序可以根据你监听的端口来猜测网络协议。例如:

mongod --port 37017
qbplproxy -h localhost:27017 -b localhost:37017

我们会依据端口 27017 知道你要分析的是 mongodb 的网络协议。

BPL 文法

请参见 BPL 文法

网络协议研究

RTMP 协议

格式描述:

测试:

  1. 启动一个 rtmp server,让其监听 1936 端口(而不是默认的 1935 端口)。比如我们可以用 node-rtsp-rtmp-server
git clone git@github.com:iizukanao/node-rtsp-rtmp-server.git
cd node-rtsp-rtmp-server
修改 config.coffee,将:
  * rtmpServerPort: 1935 改为 rtmpServerPort: 1936;
  * serverPort: 80 改为 serverPort: 8080(这样就不用 sudo 来运行了)
coffee server.coffee
  1. 启动 qbplproxy:
qbplproxy -h localhost:1935 -b localhost:1936 -p formats/rtmp.bpl | tee rtmp.log
  1. 推流:
ffmpeg -re -i test.m4v -c:v copy -c:a copy -f flv rtmp://localhost/live/123
  1. 播流:

在 Mac 下可以考虑用 VLC Player,打开网址 rtmp://localhost/live/123 进行播放即可。

  1. 选择性查看

有时候我们并不希望看到所有的信息,rtmp.bpl 支持以 flashVer 作为过滤条件。如:

qbplproxy -f 'flashVer=LNX 9,0,124,2' -h localhost:1935 -b localhost:1936 -p formats/rtmp.bpl | tee <output>.log

或者我们直接用 reqMode(用来区分是推流publish还是播流play) 来过滤。如:

qbplproxy -f 'reqMode=play' -h localhost:1935 -b localhost:1936 -p formats/rtmp.bpl | tee <output>.log

这样就可以只捕获 VLC Player 的播流过程了。

当然,其实还有一个不用过滤条件的办法:就是让推流直接推到 rtmp server,但是播流请求发到 qbplproxy。

FLV 协议

格式描述:

测试:

  1. 启动一个 rtmp/flv server,让其监听 1935/8135 端口。

  2. 启动 qbplproxy:

qbplproxy -h localhost:8888 -b localhost:8135 -p formats/flv.bpl | tee flv.log
  1. 推流:
ffmpeg -re -i test.m4v -c:v copy -c:a copy -f flv rtmp://localhost/live/123
  1. 播流:

在 Mac 下可以考虑用 VLC Player,打开网址 http://localhost:8888/live/123.flv 进行播放即可。

WebRTC 协议

格式描述:

MongoDB 协议

格式描述:

测试:

  1. 启动 MongoDB,让其监听 37017 端口(而不是默认的 27017 端口):
./mongod --port 37017 --dbpath ~/data/db
  1. 启动 qbplproxy:
qbplproxy -h localhost:27017 -b localhost:37017 -p formats/mongo.bpl | tee mongo.log
  1. 使用 MongoDB,比如通过 mongo shell 操作:
./mongo

文件格式研究

MongoDB binlog 格式

TODO

MySQL binlog 格式

TODO

HLS TS 格式

格式描述:

测试:TODO

FLV 格式

格式描述:

测试:TODO

MP4 格式

格式描述:

测试:

qbpl -p formats/mp4.bpl <example>.mp4
GIF 格式

格式描述:

测试:

qbpl -p formats/gif.bpl formats/1.gif

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	// Int8 is the matching unit for int8
	Int8 = BaseType(reflect.Int8)

	// Int16 is the matching unit for int16
	Int16 = BaseType(reflect.Int16)

	// Int32 is the matching unit for int32
	Int32 = BaseType(reflect.Int32)

	// Int64 is the matching unit for int64
	Int64 = BaseType(reflect.Int64)

	// Uint8 is the matching unit for uint8
	Uint8 = BaseType(reflect.Uint8)

	// Uint16 is the matching unit for uint16
	Uint16 = BaseType(reflect.Uint16)

	// Uint32 is the matching unit for uint32
	Uint32 = BaseType(reflect.Uint32)

	// Uint64 is the matching unit for uint64
	Uint64 = BaseType(reflect.Uint64)

	// Float32 is the matching unit for float32
	Float32 = BaseType(reflect.Float32)

	// Float64 is the matching unit for float64
	Float64 = BaseType(reflect.Float64)
)
View Source
var (
	// ErrVarNotAssigned is returned when TypeVar.Elem is not assigned.
	ErrVarNotAssigned = errors.New("variable is not assigned")

	// ErrVarAssigned is returned when TypeVar.Elem is already assigned.
	ErrVarAssigned = errors.New("variable is already assigned")

	// ErrNotEOF is returned when current position is not at EOF.
	ErrNotEOF = errors.New("current position is not at EOF")
)
View Source
var (
	TyInterface = reflect.TypeOf((*interface{})(nil)).Elem()
)
View Source
var Uint24 = Uintle(3)

Uint24 is a matching unit that matches a uintle(3) type.

Functions

func MatchStream

func MatchStream(r Ruler, in *bufio.Reader, ctx *Context) (v interface{}, err error)

MatchStream matches a stream.

Types

type BaseType

type BaseType uint

A BaseType represents a matching unit of a builtin fixed size type.

func (BaseType) Match

func (p BaseType) Match(in *bufio.Reader, ctx *Context) (v interface{}, err error)

Match is required by a matching unit. see Ruler interface.

func (BaseType) RetType

func (p BaseType) RetType() reflect.Type

RetType returns matching result type.

func (BaseType) SizeOf

func (p BaseType) SizeOf() int

SizeOf is required by a matching unit. see Ruler interface.

type Context

type Context struct {
	Stack   *exec.Stack
	Parent  *Context
	Globals Globals
	// contains filtered or unexported fields
}

A Context represents the matching context of bpl.

func NewContext

func NewContext() *Context

NewContext returns a new matching Context.

func (*Context) Dom

func (p *Context) Dom() interface{}

Dom returns matching result.

func (*Context) LetVar

func (p *Context) LetVar(name string, v interface{})

LetVar sets a variable to matching context.

func (*Context) NewSub

func (p *Context) NewSub() *Context

NewSub returns a new sub Context.

func (*Context) SetDom

func (p *Context) SetDom(v interface{})

SetDom set matching result of matching result.

func (*Context) SetVar

func (p *Context) SetVar(name string, v interface{})

SetVar sets a new variable to matching context.

func (*Context) Var

func (p *Context) Var(name string) (v interface{}, ok bool)

Var gets a variable from matching context.

type Globals

type Globals struct {
	Impl map[string]interface{}
}

A Globals represents global variables.

func NewGlobals

func NewGlobals() Globals

NewGlobals returns a `Globals` instance.

func (Globals) GetAndSetVar

func (p Globals) GetAndSetVar(name string, v interface{}) (old interface{}, ok bool)

GetAndSetVar gets old value of a global variable and sets new value to it.

func (Globals) SetVar

func (p Globals) SetVar(name string, v interface{})

SetVar sets a global variable to new value.

func (Globals) Var

func (p Globals) Var(name string) (v interface{}, ok bool)

Var returns value of a global variable.

type Member

type Member struct {
	Name string
	Type Ruler
}

A Member is typeinfo of a `Struct` member.

func (*Member) Match

func (p *Member) Match(in *bufio.Reader, ctx *Context) (v interface{}, err error)

Match is required by a matching unit. see Ruler interface.

func (*Member) RetType

func (p *Member) RetType() reflect.Type

RetType returns matching result type.

func (*Member) SizeOf

func (p *Member) SizeOf() int

SizeOf is required by a matching unit. see Ruler interface.

type Ruler

type Ruler interface {
	// Match matches input stream `in`, and returns matching result.
	Match(in *bufio.Reader, ctx *Context) (v interface{}, err error)

	// RetType returns matching result type.
	RetType() reflect.Type

	// SizeOf returns expected length of result. If length is variadic, it returns -1.
	SizeOf() int
}

A Ruler interface is required to a matching unit.

var ByteArray0 Ruler = byteArray0(0)

ByteArray0 is a matching unit that matches `*byte`.

var ByteArray1 Ruler = byteArray1(0)

ByteArray1 is a matching unit that matches `+byte`.

var CString Ruler = cstring(0)

CString is a matching unit that matches a C style string.

var Char Ruler = charType(0)

Char is a matching unit that matches a character.

var Done Ruler = done(0)

Done is a matching unit that seeks current position to EOF.

var EOF Ruler = eof(0)

EOF is a matching unit that matches EOF.

var Float32be Ruler = float32be(0)

Float32be returns a matching unit that matches a float32be type.

var Float64be Ruler = float64be(0)

Float64be returns a matching unit that matches a float64be type.

var Nil Ruler = nilType(0)

Nil is a matching unit that matches zero bytes.

func And

func And(rs ...Ruler) Ruler

And returns a matching unit that matches R1 R2 ... RN

func Array

func Array(r Ruler, n int) Ruler

Array returns a matching unit that matches R n times.

func Array0

func Array0(R Ruler) Ruler

Array0 returns a matching unit that matches R*

func Array01

func Array01(R Ruler) Ruler

Array01 returns a matching unit that matches R?

func Array1

func Array1(R Ruler) Ruler

Array1 returns a matching unit that matches R+

func Assert

func Assert(expr func(ctx *Context) bool, msg string) Ruler

Assert returns a matching unit that assert expr(ctx).

func BaseArray

func BaseArray(r BaseType, n int) Ruler

BaseArray returns a matching unit that matches R n times.

func BaseDynarray

func BaseDynarray(r BaseType, n func(ctx *Context) int) Ruler

BaseDynarray returns a matching unit that matches R n(ctx) times.

func ByteArray

func ByteArray(n int) Ruler

ByteArray returns a matching unit that matches `[n]byte`.

func ByteDynarray

func ByteDynarray(n func(ctx *Context) int) Ruler

ByteDynarray returns a matching unit that matches `[n(ctx)]byte`.

func CharArray

func CharArray(n int) Ruler

CharArray returns a matching unit that matches `[n]char`.

func CharDynarray

func CharDynarray(n func(ctx *Context) int) Ruler

CharDynarray returns a matching unit that matches `[n(ctx)]char`.

func Do

func Do(fn func(ctx *Context) error) Ruler

Do returns a matching unit that executes action fn(ctx).

func Dynarray

func Dynarray(r Ruler, n func(ctx *Context) int) Ruler

Dynarray returns a matching unit that matches R n(ctx) times.

func Dyntype

func Dyntype(r func(ctx *Context) (Ruler, error)) Ruler

Dyntype returns a dynamic matching unit.

func Eval

func Eval(expr func(ctx *Context) interface{}, r Ruler) Ruler

Eval returns a matching unit that eval expr(ctx) and matches it with R.

func FileLine

func FileLine(file string, line int, R Ruler) Ruler

FileLine is a matching rule that reports error file line when error occurs.

func FixedType

func FixedType(t reflect.Type) Ruler

FixedType returns a matching unit that matches a C style fixed size struct.

func If

func If(cond func(ctx *Context) bool, r Ruler) Ruler

If returns a matching unit that if cond(ctx) then matches it with R.

func Read

func Read(n func(ctx *Context) int, r Ruler) Ruler

Read returns a matching unit that reads n(ctx) bytes and matches R.

func Repeat0

func Repeat0(R Ruler) Ruler

Repeat0 returns a matching unit that matches R*

func Repeat01

func Repeat01(R Ruler) Ruler

Repeat01 returns a matching unit that matches R?

func Repeat1

func Repeat1(R Ruler) Ruler

Repeat1 returns a matching unit that matches R+

func Return

func Return(fnRet func(ctx *Context) (v interface{}, err error)) Ruler

Return returns a matching unit that returns fnRet(ctx).

func Seq

func Seq(rs ...Ruler) Ruler

Seq returns a matching unit that matches R1 R2 ... RN and returns matching result.

func Skip

func Skip(n func(ctx *Context) int) Ruler

Skip returns a matching unit that skips n(ctx) bytes.

func Struct

func Struct(members []Ruler) Ruler

Struct returns a compound matching unit.

func TypeFrom

func TypeFrom(t reflect.Type) (r Ruler, err error)

TypeFrom creates a matching unit from a Go type.

func Uintbe

func Uintbe(n int) Ruler

Uintbe returns a matching unit that matches a uintbe(n) type.

func Uintle

func Uintle(n int) Ruler

Uintle returns a matching unit that matches a uintle(n) type.

type TypeVar

type TypeVar struct {
	Name string
	Elem Ruler
}

A TypeVar is typeinfo of a `Struct` member.

func (*TypeVar) Assign

func (p *TypeVar) Assign(r Ruler) error

Assign assigns TypeVar.Elem.

func (*TypeVar) Match

func (p *TypeVar) Match(in *bufio.Reader, ctx *Context) (v interface{}, err error)

Match is required by a matching unit. see Ruler interface.

func (*TypeVar) RetType

func (p *TypeVar) RetType() reflect.Type

RetType returns matching result type.

func (*TypeVar) SizeOf

func (p *TypeVar) SizeOf() int

SizeOf is required by a matching unit. see Ruler interface.

Directories

Path Synopsis
cmd
go

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL