util

Documentation

Overview

The protoparse util package deals with reading and writing protocol buffer encoded records in, for example, files. functions provided here treat the underlying file as something that will be opened for either Reading or Writing (create or append)

Index

• Constants
• func GetIP(a *pbcom.IPAddressWrapper) []byte
• func IPToRadixkey(b []byte, mask uint8) string
• func MarshalBytes(a *Footer) []byte
• func MaskStrToUint8(m string) (uint8, error)
• func OpenWithBufferSizes(p *RecordFile, readersize, writersize, openmode int) (err error, created bool)
• type FlatRecordFile
  • func NewFlatRecordFile(fname string) *FlatRecordFile
  • func (p *FlatRecordFile) Entries() (uint64, error)
  • func (p *FlatRecordFile) Footer() (*Footer, error)
  • func (p *FlatRecordFile) Version() uint16
• type FootedRecordFile
  • func NewFootedRecordFile(fname string) *FootedRecordFile
  • func (p *FootedRecordFile) Close() error
  • func (p *FootedRecordFile) MakeFooter() *Footer
  • func (p *FootedRecordFile) OpenRead() error
  • func (p *FootedRecordFile) OpenWithFooter(mode int) error
  • func (p *FootedRecordFile) OpenWrite() error
  • func (f *FootedRecordFile) ReadFooter() (*Footer, error)
• type Footer
  • func ParseFooter(a []byte) (*Footer, error)
  • func (f *Footer) String() string
• type Offset
• type PrefixTree
  • func NewPrefixTree() PrefixTree
  • func (pt PrefixTree) Add(IP net.IP, mask uint8)
  • func (pt PrefixTree) ContainsIPMask(IP net.IP, mask uint8) bool
• type RecordFile
  • func NewRecordFile(fname string) *RecordFile
  • func (p *RecordFile) Close() error
  • func (p *RecordFile) Flush() (err error)
  • func (p *RecordFile) Fname() string
  • func (p *RecordFile) IncEntries(n uint64)
  • func (p *RecordFile) OpenRead() error
  • func (p *RecordFile) OpenWrite() error
  • func (p *RecordFile) Read(b []byte) (int, error)
  • func (p *RecordFile) Write(b []byte) (n int, err error)
• type RecordFiler
• type Section

Constants

const (
	RecordFile_Flat = iota
	RecordFile_FlatFooted
	RecordFile_Indexed
)

const (
	OMode_Read = iota
	OMode_Write
)

Functions

func GetIP

func GetIP(a *pbcom.IPAddressWrapper) []byte

func IPToRadixkey

func IPToRadixkey(b []byte, mask uint8) string

IPToRadixkey creates a binary string representation of an IP address. the length is 32 chars for IPv4 and 128 chars for IPv6. The mask is applied and zeroes out the bits it masks out on the resulting string.

func MarshalBytes

func MarshalBytes(a *Footer) []byte

MarshalBytes returns the bytes of [bytes of footer string][len of footer string][magic num] when this is writen to our underlying record file it is prepeneded by the length of the bytes mentioned above so this will appear as a normal record (but it won't parse normally)

func MaskStrToUint8

func MaskStrToUint8(m string) (uint8, error)

MaskStrToUint8 is a helper that just converts a possible mask string to a 10 based uint8.

func OpenWithBufferSizes

func OpenWithBufferSizes(p *RecordFile, readersize, writersize, openmode int) (err error, created bool)

Opens the underlying reader with buffer sizes specified in the arguments. useful for larger tokens than the default 64k returns if it created a new file or if it just opened an existing one

Types

type FlatRecordFile

type FlatRecordFile struct {
	*RecordFile
}

func NewFlatRecordFile

func NewFlatRecordFile(fname string) *FlatRecordFile

func (*FlatRecordFile) Entries

func (p *FlatRecordFile) Entries() (uint64, error)

Entries in a flat record file are not guaranteed to be correct. they are dependant to the position of the writer.

func (*FlatRecordFile) Footer

func (p *FlatRecordFile) Footer() (*Footer, error)

Flat record files don't have headers.

func (*FlatRecordFile) Version

func (p *FlatRecordFile) Version() uint16

type FootedRecordFile

type FootedRecordFile struct {
	*RecordFile
	*Footer
	// contains filtered or unexported fields
}

func NewFootedRecordFile

func NewFootedRecordFile(fname string) *FootedRecordFile

func (*FootedRecordFile) Close

func (p *FootedRecordFile) Close() error

func (*FootedRecordFile) MakeFooter

func (p *FootedRecordFile) MakeFooter() *Footer

func (*FootedRecordFile) OpenRead

func (p *FootedRecordFile) OpenRead() error

OpenRead will try to read the footer

func (*FootedRecordFile) OpenWithFooter

func (p *FootedRecordFile) OpenWithFooter(mode int) error

func (*FootedRecordFile) OpenWrite

func (p *FootedRecordFile) OpenWrite() error

OpenWrite will try to read the footer

func (*FootedRecordFile) ReadFooter

func (f *FootedRecordFile) ReadFooter() (*Footer, error)

Seeks to the end of the file, validates the magic number, reads the bytes of the footer, reads the footer as a string and calls ParseFooter on it. the last argument it returns are the number of bytes of the footer

type Footer

type Footer struct {
	Numentries uint64
	Filever    uint16
	Filedir    string
	Filename   string
	Sections   []Section
}

A Footer is appended at the very end of a RecordFile (all types except the FlatRecordFile type ) The end of a footer should always be the magicbytes uint32 using that an application can easily see if a file is of our type Right before that the previous uint32 is the length of the footer bytes. The footer is a utf-8 encoded JSON string.In a sense it is a reversed entry than the length prefixed records it follows. The length should not include the 4 magic bytes. neither the 4 bytes of the footer length. it should be just the number of string bytes so in the end the size of the file should be the sum of all the entry bytes + footer size + 4. Length in the end of the file should be encoded in BigEndian

func ParseFooter

func ParseFooter(a []byte) (*Footer, error)

func (*Footer) String

func (f *Footer) String() string

type Offset

type Offset struct {
	Recnum uint64
	Off    uint64
}

type PrefixTree

type PrefixTree struct {
	// contains filtered or unexported fields
}

PrefixTree holds a radix tree which clients can insert IPs and masks in , and also lookup for their existence.

func NewPrefixTree

func NewPrefixTree() PrefixTree

NewPrefixTree creates a new PrefixTree with an empty radix tree.

func (PrefixTree) Add

func (pt PrefixTree) Add(IP net.IP, mask uint8)

Add adds an IP and a mask to that PrefixTree.

func (PrefixTree) ContainsIPMask

func (pt PrefixTree) ContainsIPMask(IP net.IP, mask uint8) bool

ContainsIPMask checks for the existance of that IP and mask in the PrefixTree. It performs a longest prefix match and if it is found it retuns true.

type RecordFile

type RecordFile struct {
	Scanner *bufio.Scanner
	// contains filtered or unexported fields
}

A record file knows the number of records it has stored, every record is preceded by a 32bit unsigned value that is the length of that record in Big Endian and after that the bytes of the record

func NewRecordFile

func NewRecordFile(fname string) *RecordFile

func (*RecordFile) Close

func (p *RecordFile) Close() error

implements io.Closer

func (*RecordFile) Flush

func (p *RecordFile) Flush() (err error)

func (*RecordFile) Fname

func (p *RecordFile) Fname() string

func (*RecordFile) IncEntries

func (p *RecordFile) IncEntries(n uint64)

func (*RecordFile) OpenRead

func (p *RecordFile) OpenRead() error

func (*RecordFile) OpenWrite

func (p *RecordFile) OpenWrite() error

func (*RecordFile) Read

func (p *RecordFile) Read(b []byte) (int, error)

func (*RecordFile) Write

func (p *RecordFile) Write(b []byte) (n int, err error)

implements io.Writer but enforces the bufio interfaces underneath bytes written here increase the recorded size of the file.

type RecordFiler

type RecordFiler interface {
	Version() uint16
	Fname() string
	Footer() (*Footer, error)
	Entries() (uint64, error)
}

type Section

type Section struct {
	Compressed bool
	Secnum     uint32
	Start_off  uint64
	End_off    uint64
	Offsets    []Offset
}