msg

package module
v0.0.0-...-bb015b3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 8, 2020 License: MPL-2.0 Imports: 12 Imported by: 0

README

github.com/zutto/msg

Godoc

todo:

  • documentation
  • sanity checks
  • checksum support
  • whatever else i forgot.

Documentation

Overview

msg is a byte serialization format for 'packets' or 'messages' or 'envelopes'.

Goal of msg is to be able to store these 'packets' or 'messages' on storage mediums, be able to send them through ip links, fifo, unix pipes. (disclaimer: checksums are not.. yet.. part of the packets, so these are not to be used on unstable/unreliable mediums.)

Each packet contains ID, Feature and data. Data can be of any length, and it shall automatically be packed into a sequence of packets that can be later recompiled into one.

Index

Constants

View Source 
const (
	//init byte, not very necessary but helps to keep track of the stream.
	INIT_BYTE = 1

	//static values, may change between versions
	PREFIX_LEN          = 8
	HEADER_LEN          = 16 + 20
	ENVELOPE_HEADER_LEN = 8
	MESSAGE_PREFIX_LEN  = 2
	//CHECKSUM_LEN        = 20
	TOTAL_HEADER_LEN = PREFIX_LEN + HEADER_LEN + ENVELOPE_HEADER_LEN + MESSAGE_PREFIX_LEN // + CHECKSUM_LEN

	//default message size - Can be modified to fit use cases, such as but not limited to Tcp jumbo frames, large files, etc.
	//by default, the size is calculated to fit into TCP frame.
	SIZE = 1500 - TOTAL_HEADER_LEN - INIT_BYTE //1445

	//packet types. this is to be changed possibly, or one could make custom types? these are just hints for receiver.
	INIT = iota
	FRAGMENT
	STREAM
	ROUTE
	BROADCAST
	MULTICAST
)

Envelope constants.

View Source 
const (
	NIL = iota
	LZ4
	GZIP
)

Compression types

Variables

This section is empty.

Functions

This section is empty.

Types

type Box 

type Box struct {
	// contains filtered or unexported fields
}

func NewBox 

func NewBox() *Box

func (*Box) InsertEnvelope 

func (b *Box) InsertEnvelope(e *Envelope) error

func (*Box) IsFull 

func (b *Box) IsFull() bool

func (*Box) RecornstructMessage 

func (b *Box) RecornstructMessage() (*Envelope, error)

func (*Box) Sort 

func (b *Box) Sort()

type Envelope 

type Envelope struct {
	Prefix          Prefix          // Prefix for every envelope
	Headers         Headers         // headers for the content
	EnvelopeHeaders EnvelopeHeaders // envelope headers
	EnvelopeLabels  EnvelopeLabels  //envelope labels
	Message         Message         // message

	MessageSizeLimit int  //limit of message size (for tcp frames)
	AutoIncrement    bool //automatically increment message index
	IncludeChecksum  bool
}

Envelope structure contains all parts of the envelope

func NewEnvelope 

func NewEnvelope() *Envelope

NewEnvelope creates new envelope structure.

func (*Envelope) AddLabels 

func (e *Envelope) AddLabels(id, feature []byte) error

AddLabels is a convenience function to add ID and feature byte slices into the Envelope's EnvelopeLabels struct. you can manually insert these by doing something like this: 

e.EnvelopeLabels.MessageId = []byte{"my id"}

func (*Envelope) AddMessage 

func (e *Envelope) AddMessage(data []byte) error

AddMessage is a convenience function to insert data into the envelopes Message. You can also insert data manually by doing something like this: 

myData := []byte{"foo bar"}
e.Message.Data = &myData

func (*Envelope) Checksum 

func (e *Envelope) Checksum(from, to int) ([]byte, error)

Checksum generates checksum from all of the envelope feature checksums

func (*Envelope) Generate 

func (e *Envelope) Generate() (*[]byte, int, error)

Generate generates the envelope from zero. This function retuns data, length and error.

func (*Envelope) GenerateFromByte 

func (e *Envelope) GenerateFromByte(n int) (*[]byte, int, error)

GenerateFromByte generates envelope starting from N byte. This function retuns data, length and error. This is for sending large, or multipart envelopes. 

processedBytes := 0
for processedBytes < e.Envelope.Message.Len() {
	data, n, err := s.envelope.GenerateFromByte(processed)
	if err != nil {
		//..handle error
	}
	//..send/store/whatever with the data.

	processedBytes += n
}

func (*Envelope) GetHeaderSize 

func (e *Envelope) GetHeaderSize() int

GetHeaderSize Calculates header size of the envelope. Does not include the hidden headers included in Message used for compression.

func (*Envelope) SetCompression 

func (e *Envelope) SetCompression(cmp uint8) error

SetCompression sets the compression method for data, set to 'NIL' for no compression.

func (*Envelope) SetVersion 

func (e *Envelope) SetVersion(ver uint16) error

SetVersions sets the envelope version TODO - this is not currently used or validated as this is version 1.

type EnvelopeHeaders 

type EnvelopeHeaders struct {
	MessageIdLength      uint16 //[0:4] length of the message id
	MessageFeatureLength uint16 //[4:8] length of the message feature
}

EnvelopeHeaders contain the headers for the message header, including ID and Feature labels lengths.

func (*EnvelopeHeaders) Checksum 

func (eh *EnvelopeHeaders) Checksum() ([]byte, error)

Checksum generates SHA-1 checksum from the EnvelopeHeaders for envelope.

func (*EnvelopeHeaders) Generate 

func (eh *EnvelopeHeaders) Generate() (*[]byte, error)

Generate generates the EnvelopeHeaders for the envelope.

func (*EnvelopeHeaders) Parse 

func (eh *EnvelopeHeaders) Parse(data *[]byte, pad int) error

Parse parses input data, data can be read from padded starting position.

func (*EnvelopeHeaders) SetLengths 

func (eh *EnvelopeHeaders) SetLengths(el EnvelopeLabels)

SetLengths sets the values for the EnvelopeHeaders from EnvelopeLabels.

type EnvelopeLabels 

type EnvelopeLabels struct {
	MessageId      []byte
	MessageFeature []byte
}

EnvelopeLabels contain optional user defined ID and Feature for the packet. TODO - limits for the sizes to ensure that the labels cannot be larger than the packet max size itself.

func (*EnvelopeLabels) Checksum 

func (el *EnvelopeLabels) Checksum() ([]byte, error)

Checksum returns SHA-1 checksum for the EnvelopeLabels

func (*EnvelopeLabels) Generate 

func (el *EnvelopeLabels) Generate() (*[]byte, error)

Generate generates the EnvelopeLables for the envelope

func (*EnvelopeLabels) GetLengths 

func (el *EnvelopeLabels) GetLengths() (uint16, uint16)

GetLengths returns the lengths of MessageId and Messagefeature.

func (*EnvelopeLabels) Len 

func (el *EnvelopeLabels) Len() int

Len returns combined length of the MessageId and MessageFeature.

func (*EnvelopeLabels) Parse 

func (el *EnvelopeLabels) Parse(eh EnvelopeHeaders, data *[]byte, padding int) error

Parse parses input data, data can be read from padded starting position.

type Headers 

type Headers struct {
	PacketIndex    uint32   //[0:4] number of packets in the message
	TotalPackets   uint32   //[4:8] total number of packets in this message
	TotalLength    uint64   //[8:16] total length of the message
	PacketChecksum [20]byte //[16:36] Checksum (potentially zeroes.)
}

Headers contain the headers of the message, which includes the index of the packet (if multipart packet), total number of packets, and total length of the combined data contained of all the packets.

func (*Headers) CalculatePacketCount 

func (h *Headers) CalculatePacketCount(size int)

func (*Headers) Checksum 

func (h *Headers) Checksum() ([]byte, error)

Checksum returns SHA-1 checksum of the headers.

func (*Headers) Generate 

func (h *Headers) Generate() (*[]byte, error)

Generates generates the headers for envelope.

func (*Headers) IncrementIndex 

func (h *Headers) IncrementIndex()

func (*Headers) Parse 

func (h *Headers) Parse(data *[]byte, pad int) error

Parse parses input data, data can be read from padded starting position.

func (*Headers) SetChecksum 

func (h *Headers) SetChecksum(checksum []byte) error

SetChecksum is a convenience function to set checksum

func (*Headers) ValidateChecksum 

func (h *Headers) ValidateChecksum(checksum [20]byte) bool

type Message 

type Message struct {
	Data *[]byte //Data of any size

	Compressed      bool  //true false if current data is compressed
	CompressionType uint8 //compression type used/to be used.
	// contains filtered or unexported fields
}

Message contains the (optional) user defined data for the envelope.

func (*Message) AppendData 

func (m *Message) AppendData(data []byte) error

func (*Message) Checksum 

func (m *Message) Checksum() ([]byte, error)

Checksum generates SHA-1 checksum from the message.

func (*Message) ChecksumFromBytes 

func (m *Message) ChecksumFromBytes(from, to int) ([]byte, error)

func (*Message) Compress 

func (m *Message) Compress(level int) error

Compress compresses the current data and marks it as compressed.

func (*Message) DeCompress 

func (m *Message) DeCompress() error

DeCompress decompresses the current data and marks it as uncompressed.

func (*Message) GetData 

func (m *Message) GetData(from, to int) (*[]byte, int, error)

GetData retusn data from X position to Y position. Returns data, length, and error.

func (*Message) Len 

func (m *Message) Len() int

func (*Message) Parse 

func (m *Message) Parse(data *[]byte, pad int) error

Parse parses input data, data can be read from padded starting position.

func (*Message) SetData 

func (m *Message) SetData(d []byte) error

SetData is convenience function to set the data.

type Prefix 

type Prefix struct {
	Init         uint8  // [0:1] always 1?
	Version      uint16 // [1:3] 1..to be changed?
	Length       uint16 // [3:5] length of whole packet
	HeaderLength uint16 // [5:7] length of headers
	Type         uint8  // [7:8] packet type
}

Prefix contains the basic data for the current packet Including: Init byte, version of the packet, length of the current packet, length of headers and type.

func (*Prefix) Checksum 

func (p *Prefix) Checksum() ([]byte, error)

Checksum generates SHA-1 checksum for the prefix.

func (*Prefix) Generate 

func (p *Prefix) Generate() (*[]byte, error)

Generate generates the prefix for the envelope.

func (*Prefix) Parse 

func (p *Prefix) Parse(data *[]byte, pad int) error

Parse parses the input data, data can be read from padded startin position.

type StreamReader 

type StreamReader struct {
	// contains filtered or unexported fields
}

func NewStreamReader 

func NewStreamReader(e *Envelope, r io.Reader) (*StreamReader, error)

func (*StreamReader) Close 

func (s *StreamReader) Close() error

func (*StreamReader) GetEnvelope 

func (s *StreamReader) GetEnvelope() *Envelope

func (*StreamReader) GetState 

func (s *StreamReader) GetState() *envelopeReadState

func (*StreamReader) IsFull 

func (s *StreamReader) IsFull() bool

func (*StreamReader) Read 

func (s *StreamReader) Read(data []byte) (int, error)

todo FIX does not implement reader{} .. wtf was I thinking

func (*StreamReader) ReadWhole 

func (s *StreamReader) ReadWhole() (int, error)

func (*StreamReader) Reset 

func (s *StreamReader) Reset(e *Envelope, r io.Reader) error

func (*StreamReader) SetState 

func (s *StreamReader) SetState(es *envelopeReadState)

type StreamWriter 

type StreamWriter struct {
	// contains filtered or unexported fields
}

func NewStreamWriter 

func NewStreamWriter(e *Envelope, writer io.Writer) (*StreamWriter, error)

func (*StreamWriter) Close 

func (s *StreamWriter) Close() error

Close - this does nothing, does it even need to do anything?

func (*StreamWriter) Flush 

func (s *StreamWriter) Flush() error

Flush implements io.Writer (does nothing)

func (*StreamWriter) Reset 

func (s *StreamWriter) Reset(e *Envelope, writer io.Writer) error

Reset resets the StreamWriter

func (*StreamWriter) Write 

func (s *StreamWriter) Write(d []byte) (int, error)

Write implements io.reader

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.