README
¶
Reference:
CBOR Encoding is described in RFC7049
Comparison of JSON vs CBOR
Two main areas of reduction are:
- CPU usage to write a log msg
- Size (in bytes) of log messages.
CPU Usage savings are below:
name JSON time/op CBOR time/op delta
Info-32 15.3ns ± 1% 11.7ns ± 3% -23.78% (p=0.000 n=9+10)
ContextFields-32 16.2ns ± 2% 12.3ns ± 3% -23.97% (p=0.000 n=9+9)
ContextAppend-32 6.70ns ± 0% 6.20ns ± 0% -7.44% (p=0.000 n=9+9)
LogFields-32 66.4ns ± 0% 24.6ns ± 2% -62.89% (p=0.000 n=10+9)
LogArrayObject-32 911ns ±11% 768ns ± 6% -15.64% (p=0.000 n=10+10)
LogFieldType/Floats-32 70.3ns ± 2% 29.5ns ± 1% -57.98% (p=0.000 n=10+10)
LogFieldType/Err-32 14.0ns ± 3% 12.1ns ± 8% -13.20% (p=0.000 n=8+10)
LogFieldType/Dur-32 17.2ns ± 2% 13.1ns ± 1% -24.27% (p=0.000 n=10+9)
LogFieldType/Object-32 54.3ns ±11% 52.3ns ± 7% ~ (p=0.239 n=10+10)
LogFieldType/Ints-32 20.3ns ± 2% 15.1ns ± 2% -25.50% (p=0.000 n=9+10)
LogFieldType/Interfaces-32 642ns ±11% 621ns ± 9% ~ (p=0.118 n=10+10)
LogFieldType/Interface(Objects)-32 635ns ±13% 632ns ± 9% ~ (p=0.592 n=10+10)
LogFieldType/Times-32 294ns ± 0% 27ns ± 1% -90.71% (p=0.000 n=10+9)
LogFieldType/Durs-32 121ns ± 0% 33ns ± 2% -72.44% (p=0.000 n=9+9)
LogFieldType/Interface(Object)-32 56.6ns ± 8% 52.3ns ± 8% -7.54% (p=0.007 n=10+10)
LogFieldType/Errs-32 17.8ns ± 3% 16.1ns ± 2% -9.71% (p=0.000 n=10+9)
LogFieldType/Time-32 40.5ns ± 1% 12.7ns ± 6% -68.66% (p=0.000 n=8+9)
LogFieldType/Bool-32 12.0ns ± 5% 10.2ns ± 2% -15.18% (p=0.000 n=10+8)
LogFieldType/Bools-32 17.2ns ± 2% 12.6ns ± 4% -26.63% (p=0.000 n=10+10)
LogFieldType/Int-32 12.3ns ± 2% 11.2ns ± 4% -9.27% (p=0.000 n=9+10)
LogFieldType/Float-32 16.7ns ± 1% 12.6ns ± 2% -24.42% (p=0.000 n=7+9)
LogFieldType/Str-32 12.7ns ± 7% 11.3ns ± 7% -10.88% (p=0.000 n=10+9)
LogFieldType/Strs-32 20.3ns ± 3% 18.2ns ± 3% -10.25% (p=0.000 n=9+10)
LogFieldType/Interface-32 183ns ±12% 175ns ± 9% ~ (p=0.078 n=10+10)
Log message size savings is greatly dependent on the number and type of fields in the log message. Assuming this log message (with an Integer, timestamp and string, in addition to level).
{"level":"error","Fault":41650,"time":"2018-04-01T15:18:19-07:00","message":"Some Message"}
Two measurements were done for the log file sizes - one without any compression, second using compress/zlib.
Results for 10,000 log messages:
| Log Format | Plain File Size (in KB) | Compressed File Size (in KB) |
|---|---|---|
| JSON | 920 | 28 |
| CBOR | 550 | 28 |
The example used to calculate the above data is available in Examples.
Documentation
¶
Overview ¶
Package cbor provides primitives for storing different data in the CBOR (binary) format. CBOR is defined in RFC7049.
Index ¶
- Variables
- func AppendEmbeddedJSON(dst, s []byte) []byte
- func Cbor2JsonManyObjects(src io.Reader, dst io.Writer) (err error)
- func DecodeIfBinaryToBytes(in []byte) []byte
- func DecodeIfBinaryToString(in []byte) string
- func DecodeObjectToStr(in []byte) string
- type Encoder
- func (Encoder) AppendArrayDelim(dst []byte) []byte
- func (Encoder) AppendArrayEnd(dst []byte) []byte
- func (Encoder) AppendArrayStart(dst []byte) []byte
- func (Encoder) AppendBeginMarker(dst []byte) []byte
- func (Encoder) AppendBool(dst []byte, val bool) []byte
- func (e Encoder) AppendBools(dst []byte, vals []bool) []byte
- func (Encoder) AppendBytes(dst, s []byte) []byte
- func (e Encoder) AppendDuration(dst []byte, d time.Duration, unit time.Duration, useInt bool) []byte
- func (e Encoder) AppendDurations(dst []byte, vals []time.Duration, unit time.Duration, useInt bool) []byte
- func (Encoder) AppendEndMarker(dst []byte) []byte
- func (Encoder) AppendFloat32(dst []byte, val float32) []byte
- func (Encoder) AppendFloat64(dst []byte, val float64) []byte
- func (e Encoder) AppendFloats32(dst []byte, vals []float32) []byte
- func (e Encoder) AppendFloats64(dst []byte, vals []float64) []byte
- func (e Encoder) AppendHex(dst []byte, val []byte) []byte
- func (e Encoder) AppendIPAddr(dst []byte, ip net.IP) []byte
- func (e Encoder) AppendIPPrefix(dst []byte, pfx net.IPNet) []byte
- func (Encoder) AppendInt(dst []byte, val int) []byte
- func (e Encoder) AppendInt16(dst []byte, val int16) []byte
- func (e Encoder) AppendInt32(dst []byte, val int32) []byte
- func (Encoder) AppendInt64(dst []byte, val int64) []byte
- func (e Encoder) AppendInt8(dst []byte, val int8) []byte
- func (e Encoder) AppendInterface(dst []byte, i interface{}) []byte
- func (e Encoder) AppendInts(dst []byte, vals []int) []byte
- func (e Encoder) AppendInts16(dst []byte, vals []int16) []byte
- func (e Encoder) AppendInts32(dst []byte, vals []int32) []byte
- func (e Encoder) AppendInts64(dst []byte, vals []int64) []byte
- func (e Encoder) AppendInts8(dst []byte, vals []int8) []byte
- func (e Encoder) AppendKey(dst []byte, key string) []byte
- func (Encoder) AppendLineBreak(dst []byte) []byte
- func (e Encoder) AppendMACAddr(dst []byte, ha net.HardwareAddr) []byte
- func (Encoder) AppendNil(dst []byte) []byte
- func (Encoder) AppendObjectData(dst []byte, o []byte) []byte
- func (Encoder) AppendString(dst []byte, s string) []byte
- func (e Encoder) AppendStrings(dst []byte, vals []string) []byte
- func (e Encoder) AppendTime(dst []byte, t time.Time, unused string) []byte
- func (e Encoder) AppendTimes(dst []byte, vals []time.Time, unused string) []byte
- func (e Encoder) AppendUint(dst []byte, val uint) []byte
- func (e Encoder) AppendUint16(dst []byte, val uint16) []byte
- func (e Encoder) AppendUint32(dst []byte, val uint32) []byte
- func (Encoder) AppendUint64(dst []byte, val uint64) []byte
- func (e Encoder) AppendUint8(dst []byte, val uint8) []byte
- func (e Encoder) AppendUints(dst []byte, vals []uint) []byte
- func (e Encoder) AppendUints16(dst []byte, vals []uint16) []byte
- func (e Encoder) AppendUints32(dst []byte, vals []uint32) []byte
- func (e Encoder) AppendUints64(dst []byte, vals []uint64) []byte
- func (e Encoder) AppendUints8(dst []byte, vals []uint8) []byte
Constants ¶
This section is empty.
Variables ¶
var IntegerTimeFieldFormat = time.RFC3339
IntegerTimeFieldFormat indicates the format of timestamp decoded from an integer (time in seconds).
var JSONMarshalFunc func(v interface{}) ([]byte, error)
JSONMarshalFunc is used to marshal interface to JSON encoded byte slice. Making it package level instead of embedded in Encoder brings some extra efforts at importing, but avoids value copy when the functions of Encoder being invoked. DO REMEMBER to set this variable at importing, or you might get a nil pointer dereference panic at runtime.
var NanoTimeFieldFormat = time.RFC3339Nano
NanoTimeFieldFormat indicates the format of timestamp decoded from a float value (time in seconds and nano seconds).
Functions ¶
func AppendEmbeddedJSON ¶
AppendEmbeddedJSON adds a tag and embeds input JSON as such.
func Cbor2JsonManyObjects ¶
Cbor2JsonManyObjects decodes all the CBOR Objects read from src reader. It keeps on decoding until reader returns EOF (error when reading). Decoded string is written to the dst. At the end of every CBOR Object newline is written to the output stream.
Returns error (if any) that was encountered during decode. The child functions will generate a panic when error is encountered and this function will recover non-runtime Errors and return the reason as error.
func DecodeIfBinaryToBytes ¶
DecodeIfBinaryToBytes checks if the input is a binary format, if so, it will decode all Objects and return the decoded string as byte array.
func DecodeIfBinaryToString ¶
DecodeIfBinaryToString converts a binary formatted log msg to a JSON formatted String Log message - suitable for printing to Console/Syslog.
func DecodeObjectToStr ¶
DecodeObjectToStr checks if the input is a binary format, if so, it will decode a single Object and return the decoded string.
Types ¶
type Encoder ¶ added in v1.7.0
type Encoder struct{}
func (Encoder) AppendArrayDelim ¶ added in v1.7.0
AppendArrayDelim adds markers to indicate end of a particular array element.
func (Encoder) AppendArrayEnd ¶ added in v1.7.0
AppendArrayEnd adds markers to indicate the end of an array.
func (Encoder) AppendArrayStart ¶ added in v1.7.0
AppendArrayStart adds markers to indicate the start of an array.
func (Encoder) AppendBeginMarker ¶ added in v1.7.0
AppendBeginMarker inserts a map start into the dst byte array.
func (Encoder) AppendBool ¶ added in v1.7.0
AppendBool encodes and inserts a boolean value into the dst byte array.
func (Encoder) AppendBools ¶ added in v1.7.0
AppendBools encodes and inserts an array of boolean values into the dst byte array.
func (Encoder) AppendBytes ¶ added in v1.7.0
AppendBytes encodes and adds an array of bytes to the dst byte array.
func (Encoder) AppendDuration ¶ added in v1.7.0
func (e Encoder) AppendDuration(dst []byte, d time.Duration, unit time.Duration, useInt bool) []byte
AppendDuration encodes and adds a duration to the dst byte array. useInt field indicates whether to store the duration as seconds (integer) or as seconds+nanoseconds (float).
func (Encoder) AppendDurations ¶ added in v1.7.0
func (e Encoder) AppendDurations(dst []byte, vals []time.Duration, unit time.Duration, useInt bool) []byte
AppendDurations encodes and adds an array of durations to the dst byte array. useInt field indicates whether to store the duration as seconds (integer) or as seconds+nanoseconds (float).
func (Encoder) AppendEndMarker ¶ added in v1.7.0
AppendEndMarker inserts a map end into the dst byte array.
func (Encoder) AppendFloat32 ¶ added in v1.7.0
AppendFloat32 encodes and inserts a single precision float value into the dst byte array.
func (Encoder) AppendFloat64 ¶ added in v1.7.0
AppendFloat64 encodes and inserts a double precision float value into the dst byte array.
func (Encoder) AppendFloats32 ¶ added in v1.7.0
AppendFloats32 encodes and inserts an array of single precision float value into the dst byte array.
func (Encoder) AppendFloats64 ¶ added in v1.7.0
AppendFloats64 encodes and inserts an array of double precision float values into the dst byte array.
func (Encoder) AppendHex ¶ added in v1.7.0
AppendHex adds a TAG and inserts a hex bytes as a string.
func (Encoder) AppendIPAddr ¶ added in v1.7.0
AppendIPAddr encodes and inserts an IP Address (IPv4 or IPv6).
func (Encoder) AppendIPPrefix ¶ added in v1.7.0
AppendIPPrefix encodes and inserts an IP Address Prefix (Address + Mask Length).
func (Encoder) AppendInt ¶ added in v1.7.0
AppendInt encodes and inserts an integer value into the dst byte array.
func (Encoder) AppendInt16 ¶ added in v1.7.0
AppendInt16 encodes and inserts a int16 value into the dst byte array.
func (Encoder) AppendInt32 ¶ added in v1.7.0
AppendInt32 encodes and inserts a int32 value into the dst byte array.
func (Encoder) AppendInt64 ¶ added in v1.7.0
AppendInt64 encodes and inserts a int64 value into the dst byte array.
func (Encoder) AppendInt8 ¶ added in v1.7.0
AppendInt8 encodes and inserts an int8 value into the dst byte array.
func (Encoder) AppendInterface ¶ added in v1.7.0
AppendInterface takes an arbitrary object and converts it to JSON and embeds it dst.
func (Encoder) AppendInts ¶ added in v1.7.0
AppendInts encodes and inserts an array of integer values into the dst byte array.
func (Encoder) AppendInts16 ¶ added in v1.7.0
AppendInts16 encodes and inserts an array of int16 values into the dst byte array.
func (Encoder) AppendInts32 ¶ added in v1.7.0
AppendInts32 encodes and inserts an array of int32 values into the dst byte array.
func (Encoder) AppendInts64 ¶ added in v1.7.0
AppendInts64 encodes and inserts an array of int64 values into the dst byte array.
func (Encoder) AppendInts8 ¶ added in v1.7.0
AppendInts8 encodes and inserts an array of integer values into the dst byte array.
func (Encoder) AppendKey ¶ added in v1.7.0
AppendKey adds a key (string) to the binary encoded log message
func (Encoder) AppendLineBreak ¶ added in v1.7.0
AppendLineBreak is a noop that keep API compat with json encoder.
func (Encoder) AppendMACAddr ¶ added in v1.7.0
func (e Encoder) AppendMACAddr(dst []byte, ha net.HardwareAddr) []byte
AppendMACAddr encodes and inserts an Hardware (MAC) address.
func (Encoder) AppendNil ¶ added in v1.7.0
AppendNil inserts a 'Nil' object into the dst byte array.
func (Encoder) AppendObjectData ¶ added in v1.7.0
AppendObjectData takes an object in form of a byte array and appends to dst.
func (Encoder) AppendString ¶ added in v1.7.0
AppendString encodes and adds a string to the dst byte array.
func (Encoder) AppendStrings ¶ added in v1.7.0
AppendStrings encodes and adds an array of strings to the dst byte array.
func (Encoder) AppendTime ¶ added in v1.7.0
AppendTime encodes and adds a timestamp to the dst byte array.
func (Encoder) AppendTimes ¶ added in v1.7.0
AppendTimes encodes and adds an array of timestamps to the dst byte array.
func (Encoder) AppendUint ¶ added in v1.7.0
AppendUint encodes and inserts an unsigned integer value into the dst byte array.
func (Encoder) AppendUint16 ¶ added in v1.7.0
AppendUint16 encodes and inserts a uint16 value into the dst byte array.
func (Encoder) AppendUint32 ¶ added in v1.7.0
AppendUint32 encodes and inserts a uint32 value into the dst byte array.
func (Encoder) AppendUint64 ¶ added in v1.7.0
AppendUint64 encodes and inserts a uint64 value into the dst byte array.
func (Encoder) AppendUint8 ¶ added in v1.7.0
AppendUint8 encodes and inserts a unsigned int8 value into the dst byte array.
func (Encoder) AppendUints ¶ added in v1.7.0
AppendUints encodes and inserts an array of unsigned integer values into the dst byte array.
func (Encoder) AppendUints16 ¶ added in v1.7.0
AppendUints16 encodes and inserts an array of uint16 values into the dst byte array.
func (Encoder) AppendUints32 ¶ added in v1.7.0
AppendUints32 encodes and inserts an array of uint32 values into the dst byte array.
func (Encoder) AppendUints64 ¶ added in v1.7.0
AppendUints64 encodes and inserts an array of uint64 values into the dst byte array.
Source Files
Directories