json

package
v1.36.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 31, 2021 License: MIT, MIT Imports: 21 Imported by: 0

README

encoding/json GoDoc

Go package offering a replacement implementation of the standard library's encoding/json package, with much better performance.

Usage

The exported API of this package mirrors the standard library's encoding/json package, the only change needed to take advantage of the performance improvements is the import path of the json package, from:

import (
    "encoding/json"
)

to

import (
    "github.com/segmentio/encoding/json"
)

One way to gain higher encoding throughput is to disable HTML escaping. It allows the string encoding to use a much more efficient code path which does not require parsing UTF-8 runes most of the time.

Performance Improvements

The internal implementation uses a fair amount of unsafe operations (untyped code, pointer arithmetic, etc...) to avoid using reflection as much as possible, which is often the reason why serialization code has a large CPU and memory footprint.

The package aims for zero unnecessary dynamic memory allocations and hot code paths that are mostly free from calls into the reflect package.

Compatibility with encoding/json

This package aims to be a drop-in replacement, therefore it is tested to behave exactly like the standard library's package. However, there are still a few missing features that have not been ported yet:

  • Streaming decoder, currently the Decoder implementation offered by the package does not support progressively reading values from a JSON array (unlike the standard library). In our experience this is a very rare use-case, if you need it you're better off sticking to the standard library, or spend a bit of time implementing it in here ;)

Note that none of those features should result in performance degradations if they were implemented in the package, and we welcome contributions!

Trade-offs

As one would expect, we had to make a couple of trade-offs to achieve greater performance than the standard library, but there were also features that we did not want to give away.

Other open-source packages offering a reduced CPU and memory footprint usually do so by designing a different API, or require code generation (therefore adding complexity to the build process). These were not acceptable conditions for us, as we were not willing to trade off developer productivity for better runtime performance. To achieve this, we chose to exactly replicate the standard library interfaces and behavior, which meant the package implementation was the only area that we were able to work with. The internals of this package make heavy use of unsafe pointer arithmetics and other performance optimizations, and therefore are not as approachable as typical Go programs. Basically, we put a bigger burden on maintainers to achieve better runtime cost without sacrificing developer productivity.

For these reasons, we also don't believe that this code should be ported upstream to the standard encoding/json package. The standard library has to remain readable and approachable to maximize stability and maintainability, and make projects like this one possible because a high quality reference implementation already exists.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Append 

func Append(b []byte, x interface{}, flags AppendFlags) ([]byte, error)

Append acts like Marshal but appends the json representation to b instead of always reallocating a new slice.

func Compact 

func Compact(dst *bytes.Buffer, src []byte) error

Compact is documented at https://golang.org/pkg/encoding/json/#Compact

func HTMLEscape 

func HTMLEscape(dst *bytes.Buffer, src []byte)

HTMLEscape is documented at https://golang.org/pkg/encoding/json/#HTMLEscape

func Indent 

func Indent(dst *bytes.Buffer, src []byte, prefix, indent string) error

Indent is documented at https://golang.org/pkg/encoding/json/#Indent

func Marshal 

func Marshal(x interface{}) ([]byte, error)

Marshal is documented at https://golang.org/pkg/encoding/json/#Marshal

func MarshalIndent 

func MarshalIndent(x interface{}, prefix, indent string) ([]byte, error)

MarshalIndent is documented at https://golang.org/pkg/encoding/json/#MarshalIndent

func Parse 

func Parse(b []byte, x interface{}, flags ParseFlags) ([]byte, error)

Parse behaves like Unmarshal but the caller can pass a set of flags to configure the parsing behavior.

func Unmarshal 

func Unmarshal(b []byte, x interface{}) error

Unmarshal is documented at https://golang.org/pkg/encoding/json/#Unmarshal

func Valid 

func Valid(data []byte) bool

Valid is documented at https://golang.org/pkg/encoding/json/#Valid

Types

type AppendFlags 

type AppendFlags int

AppendFlags is a type used to represent configuration options that can be applied when formatting json output. 

const (
	// EscapeHTML is a formatting flag used to to escape HTML in json strings.
	EscapeHTML AppendFlags = 1 << iota

	// SortMapKeys is formatting flag used to enable sorting of map keys when
	// encoding JSON (this matches the behavior of the standard encoding/json
	// package).
	SortMapKeys

	// TrustRawMessage is a performance optimization flag to skip value
	// checking of raw messages. It should only be used if the values are
	// known to be valid json (e.g., they were created by json.Unmarshal).
	TrustRawMessage
)

type Decoder 

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder is documented at https://golang.org/pkg/encoding/json/#Decoder

func NewDecoder 

func NewDecoder(r io.Reader) *Decoder

NewDecoder is documented at https://golang.org/pkg/encoding/json/#NewDecoder

func (*Decoder) Buffered 

func (dec *Decoder) Buffered() io.Reader

Buffered is documented at https://golang.org/pkg/encoding/json/#Decoder.Buffered

func (*Decoder) Decode 

func (dec *Decoder) Decode(v interface{}) error

Decode is documented at https://golang.org/pkg/encoding/json/#Decoder.Decode

func (*Decoder) DisallowUnknownFields 

func (dec *Decoder) DisallowUnknownFields()

DisallowUnknownFields is documented at https://golang.org/pkg/encoding/json/#Decoder.DisallowUnknownFields

func (*Decoder) DontCopyNumber 

func (dec *Decoder) DontCopyNumber()

DontCopyNumber is an extension to the standard encoding/json package which instructs the decoder to not copy numbers loaded from the json payloads.

func (*Decoder) DontCopyRawMessage 

func (dec *Decoder) DontCopyRawMessage()

DontCopyRawMessage is an extension to the standard encoding/json package which instructs the decoder to not allocate RawMessage values in separate memory buffers (see the documentation of the DontcopyRawMessage flag for more detais).

func (*Decoder) DontCopyString 

func (dec *Decoder) DontCopyString()

DontCopyString is an extension to the standard encoding/json package which instructs the decoder to not copy strings loaded from the json payloads when possible.

func (*Decoder) DontMatchCaseInsensitiveStructFields 

func (dec *Decoder) DontMatchCaseInsensitiveStructFields()

DontMatchCaseInsensitiveStructFields is an extension to the standard encoding/json package which instructs the decoder to not match object fields against struct fields in a case-insensitive way, the field names have to match exactly to be decoded into the struct field values.

func (*Decoder) InputOffset 

func (dec *Decoder) InputOffset() int64

InputOffset returns the input stream byte offset of the current decoder position. The offset gives the location of the end of the most recently returned token and the beginning of the next token.

func (*Decoder) UseNumber 

func (dec *Decoder) UseNumber()

UseNumber is documented at https://golang.org/pkg/encoding/json/#Decoder.UseNumber

func (*Decoder) ZeroCopy 

func (dec *Decoder) ZeroCopy()

ZeroCopy is an extension to the standard encoding/json package which enables all the copy optimizations of the decoder.

type Delim 

type Delim = json.Delim

Delim is documented at https://golang.org/pkg/encoding/json/#Delim

type Encoder 

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder is documented at https://golang.org/pkg/encoding/json/#Encoder

func NewEncoder 

func NewEncoder(w io.Writer) *Encoder

NewEncoder is documented at https://golang.org/pkg/encoding/json/#NewEncoder

func (*Encoder) Encode 

func (enc *Encoder) Encode(v interface{}) error

Encode is documented at https://golang.org/pkg/encoding/json/#Encoder.Encode

func (*Encoder) SetEscapeHTML 

func (enc *Encoder) SetEscapeHTML(on bool)

SetEscapeHTML is documented at https://golang.org/pkg/encoding/json/#Encoder.SetEscapeHTML

func (*Encoder) SetIndent 

func (enc *Encoder) SetIndent(prefix, indent string)

SetIndent is documented at https://golang.org/pkg/encoding/json/#Encoder.SetIndent

func (*Encoder) SetSortMapKeys 

func (enc *Encoder) SetSortMapKeys(on bool)

SetSortMapKeys is an extension to the standard encoding/json package which allows the program to toggle sorting of map keys on and off.

func (*Encoder) SetTrustRawMessage 

func (enc *Encoder) SetTrustRawMessage(on bool)

SetTrustRawMessage skips value checking when encoding a raw json message. It should only be used if the values are known to be valid json, e.g. because they were originally created by json.Unmarshal.

type InvalidUTF8Error 

type InvalidUTF8Error = json.InvalidUTF8Error

InvalidUTF8Error is documented at https://golang.org/pkg/encoding/json/#InvalidUTF8Error

type InvalidUnmarshalError 

type InvalidUnmarshalError = json.InvalidUnmarshalError

InvalidUnmarshalError is documented at https://golang.org/pkg/encoding/json/#InvalidUnmarshalError

type Marshaler 

type Marshaler = json.Marshaler

Marshaler is documented at https://golang.org/pkg/encoding/json/#Marshaler

type MarshalerError 

type MarshalerError = json.MarshalerError

MarshalerError is documented at https://golang.org/pkg/encoding/json/#MarshalerError

type Number 

type Number = json.Number

Number is documented at https://golang.org/pkg/encoding/json/#Number

type ParseFlags 

type ParseFlags int

ParseFlags is a type used to represent configuration options that can be applied when parsing json input. 

const (
	// DisallowUnknownFields is a parsing flag used to prevent decoding of
	// objects to Go struct values when a field of the input does not match
	// with any of the struct fields.
	DisallowUnknownFields ParseFlags = 1 << iota

	// UseNumber is a parsing flag used to load numeric values as Number
	// instead of float64.
	UseNumber

	// DontCopyString is a parsing flag used to provide zero-copy support when
	// loading string values from a json payload. It is not always possible to
	// avoid dynamic memory allocations, for example when a string is escaped in
	// the json data a new buffer has to be allocated, but when the `wire` value
	// can be used as content of a Go value the decoder will simply point into
	// the input buffer.
	DontCopyString

	// DontCopyNumber is a parsing flag used to provide zero-copy support when
	// loading Number values (see DontCopyString and DontCopyRawMessage).
	DontCopyNumber

	// DontCopyRawMessage is a parsing flag used to provide zero-copy support
	// when loading RawMessage values from a json payload. When used, the
	// RawMessage values will not be allocated into new memory buffers and
	// will instead point directly to the area of the input buffer where the
	// value was found.
	DontCopyRawMessage

	// DontMatchCaseInsensitiveStructFields is a parsing flag used to prevent
	// matching fields in a case-insensitive way. This can prevent degrading
	// performance on case conversions, and can also act as a stricter decoding
	// mode.
	DontMatchCaseInsensitiveStructFields

	// ZeroCopy is a parsing flag that combines all the copy optimizations
	// available in the package.
	//
	// The zero-copy optimizations are better used in request-handler style
	// code where none of the values are retained after the handler returns.
	ZeroCopy = DontCopyString | DontCopyNumber | DontCopyRawMessage
)

type RawMessage 

type RawMessage = json.RawMessage

RawMessage is documented at https://golang.org/pkg/encoding/json/#RawMessage

type RawValue 

type RawValue []byte

RawValue represents a raw json value, it is intended to carry null, true, false, number, and string values only.

func (RawValue) AppendUnquote 

func (v RawValue) AppendUnquote(b []byte) []byte

AppendUnquote writes the unquoted version of the string value in v into b.

func (RawValue) False 

func (v RawValue) False() bool

False returns true if v contains a false value.

func (RawValue) Null 

func (v RawValue) Null() bool

Null returns true if v contains a null value.

func (RawValue) Number 

func (v RawValue) Number() bool

Number returns true if v contains a number value.

func (RawValue) String 

func (v RawValue) String() bool

String returns true if v contains a string value.

func (RawValue) True 

func (v RawValue) True() bool

True returns true if v contains a true value.

func (RawValue) Unquote 

func (v RawValue) Unquote() []byte

Unquote returns the unquoted version of the string value in v.

type SyntaxError 

type SyntaxError = json.SyntaxError

A SyntaxError is a description of a JSON syntax error.

type Token 

type Token = json.Token

Token is documented at https://golang.org/pkg/encoding/json/#Token

type Tokenizer 

type Tokenizer struct {
	// When the tokenizer is positioned on a json delimiter this field is not
	// zero. In this case the possible values are '{', '}', '[', ']', ':', and
	// ','.
	Delim Delim

	// This field contains the raw json token that the tokenizer is pointing at.
	// When Delim is not zero, this field is a single-element byte slice
	// continaing the delimiter value. Otherwise, this field holds values like
	// null, true, false, numbers, or quoted strings.
	Value RawValue

	// When the tokenizer has encountered invalid content this field is not nil.
	Err error

	// When the value is in an array or an object, this field contains the depth
	// at which it was found.
	Depth int

	// When the value is in an array or an object, this field contains the
	// position at which it was found.
	Index int

	// This field is true when the value is the key of an object.
	IsKey bool
	// contains filtered or unexported fields
}

Tokenizer is an iterator-style type which can be used to progressively parse through a json input.

Tokenizing json is useful to build highly efficient parsing operations, for example when doing tranformations on-the-fly where as the program reads the input and produces the transformed json to an output buffer.

Here is a common pattern to use a tokenizer: 

for t := json.NewTokenizer(b); t.Next(); {
	switch t.Delim {
	case '{':
		...
	case '}':
		...
	case '[':
		...
	case ']':
		...
	case ':':
		...
	case ',':
		...
	}

	switch {
	case t.Value.String():
		...
	case t.Value.Null():
		...
	case t.Value.True():
		...
	case t.Value.False():
		...
	case t.Value.Number():
		...
	}
}

func NewTokenizer 

func NewTokenizer(b []byte) *Tokenizer

NewTokenizer constructs a new Tokenizer which reads its json input from b.

func (*Tokenizer) Next 

func (t *Tokenizer) Next() bool

Next returns a new tokenizer pointing at the next token, or the zero-value of Tokenizer if the end of the json input has been reached.

If the tokenizer encounters malformed json while reading the input the method sets t.Err to an error describing the issue, and returns false. Once an error has been encountered, the tokenizer will always fail until its input is cleared by a call to its Reset method.

func (*Tokenizer) Reset 

func (t *Tokenizer) Reset(b []byte)

Reset erases the state of t and re-initializes it with the json input from b.

type UnmarshalFieldError 

type UnmarshalFieldError = json.UnmarshalFieldError

UnmarshalFieldError is documented at https://golang.org/pkg/encoding/json/#UnmarshalFieldError

type UnmarshalTypeError 

type UnmarshalTypeError = json.UnmarshalTypeError

UnmarshalTypeError is documented at https://golang.org/pkg/encoding/json/#UnmarshalTypeError

type Unmarshaler 

type Unmarshaler = json.Unmarshaler

Unmarshaler is documented at https://golang.org/pkg/encoding/json/#Unmarshaler

type UnsupportedTypeError 

type UnsupportedTypeError = json.UnsupportedTypeError

UnsupportedTypeError is documented at https://golang.org/pkg/encoding/json/#UnsupportedTypeError

type UnsupportedValueError 

type UnsupportedValueError = json.UnsupportedValueError

UnsupportedValueError is documented at https://golang.org/pkg/encoding/json/#UnsupportedValueError

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.