json

package
v0.2.8-0...-eb39950 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 24, 2021 License: MIT Imports: 21 Imported by: 0

README

encoding/json GoDoc

Go package offering a replacement implementation of the standard library's encoding/json package, with much better performance.

Usage

The exported API of this package mirrors the standard library's encoding/json package, the only change needed to take advantage of the performance improvements is the import path of the json package, from:

import (
    "encoding/json"
)

to

import (
    "github.com/segmentio/encoding/json"
)

One way to gain higher encoding throughput is to disable HTML escaping. It allows the string encoding to use a much more efficient code path which does not require parsing UTF-8 runes most of the time.

Performance Improvements

The internal implementation uses a fair amount of unsafe operations (untyped code, pointer arithmetic, etc...) to avoid using reflection as much as possible, which is often the reason why serialization code has a large CPU and memory footprint.

The package aims for zero unnecessary dynamic memory allocations and hot code paths that are mostly free from calls into the reflect package.

Compatibility with encoding/json

This package aims to be a drop-in replacement, therefore it is tested to behave exactly like the standard library's package. However, there are still a few missing features that have not been ported yet:

  • Streaming decoder, currently the Decoder implementation offered by the package does not support progressively reading values from a JSON array (unlike the standard library). In our experience this is a very rare use-case, if you need it you're better off sticking to the standard library, or spend a bit of time implementing it in here ;)

Note that none of those features should result in performance degradations if they were implemented in the package, and we welcome contributions!

Trade-offs

As one would expect, we had to make a couple of trade-offs to achieve greater performance than the standard library, but there were also features that we did not want to give away.

Other open-source packages offering a reduced CPU and memory footprint usually do so by designing a different API, or require code generation (therefore adding complexity to the build process). These were not acceptable conditions for us, as we were not willing to trade off developer productivity for better runtime performance. To achieve this, we chose to exactly replicate the standard library interfaces and behavior, which meant the package implementation was the only area that we were able to work with. The internals of this package make heavy use of unsafe pointer arithmetics and other performance optimizations, and therefore are not as approachable as typical Go programs. Basically, we put a bigger burden on maintainers to achieve better runtime cost without sacrificing developer productivity.

For these reasons, we also don't believe that this code should be ported upstream to the standard encoding/json package. The standard library has to remain readable and approachable to maximize stability and maintainability, and make projects like this one possible because a high quality reference implementation already exists.

Documentation

Overview

Example (CustomMarshalJSON) 
package main

import (
	"encoding/json"
	"fmt"
	"log"
	"strings"
)

type Animal int

const (
	Unknown Animal = iota
	Gopher
	Zebra
)

func (a *Animal) UnmarshalJSON(b []byte) error {
	var s string
	if err := json.Unmarshal(b, &s); err != nil {
		return err
	}
	switch strings.ToLower(s) {
	default:
		*a = Unknown
	case "gopher":
		*a = Gopher
	case "zebra":
		*a = Zebra
	}

	return nil
}

func (a Animal) MarshalJSON() ([]byte, error) {
	var s string
	switch a {
	default:
		s = "unknown"
	case Gopher:
		s = "gopher"
	case Zebra:
		s = "zebra"
	}

	return json.Marshal(s)
}

func main() {
	blob := `["gopher","armadillo","zebra","unknown","gopher","bee","gopher","zebra"]`
	var zoo []Animal
	if err := json.Unmarshal([]byte(blob), &zoo); err != nil {
		log.Fatal(err)
	}

	census := make(map[Animal]int)
	for _, animal := range zoo {
		census[animal] += 1
	}

	fmt.Printf("Zoo Census:\n* Gophers: %d\n* Zebras:  %d\n* Unknown: %d\n",
		census[Gopher], census[Zebra], census[Unknown])

}

Output:

Zoo Census:
* Gophers: 3
* Zebras:  2
* Unknown: 3

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

func Append 

func Append(b []byte, x interface{}, flags AppendFlags) ([]byte, error)

Append acts like Marshal but appends the json representation to b instead of always reallocating a new slice.

func AppendEscape 

func AppendEscape(b []byte, s string, flags AppendFlags) []byte

AppendEscape appends s to b with the string escaped as a JSON value. This will include the starting and ending quote characters, and the appropriate characters will be escaped correctly for JSON encoding.

func Compact 

func Compact(dst *bytes.Buffer, src []byte) error

Compact is documented at https://golang.org/pkg/encoding/json/#Compact

func Escape 

func Escape(s string) []byte

Escape is a convenience helper to construct an escaped JSON string from s. The function escales HTML characters, for more control over the escape behavior and to write to a pre-allocated buffer, use AppendEscape.

func HTMLEscape 

func HTMLEscape(dst *bytes.Buffer, src []byte)

HTMLEscape is documented at https://golang.org/pkg/encoding/json/#HTMLEscape

Example 
package main

import (
	"bytes"
	"encoding/json"
	"os"
)

func main() {
	var out bytes.Buffer
	json.HTMLEscape(&out, []byte(`{"Name":"<b>HTML content</b>"}`))
	out.WriteTo(os.Stdout)
}

Output:

{"Name":"\u003cb\u003eHTML content\u003c/b\u003e"}

func Indent 

func Indent(dst *bytes.Buffer, src []byte, prefix, indent string) error

Indent is documented at https://golang.org/pkg/encoding/json/#Indent

Example 
package main

import (
	"bytes"
	"encoding/json"
	"log"
	"os"
)

func main() {
	type Road struct {
		Name   string
		Number int
	}
	roads := []Road{
		{"Diamond Fork", 29},
		{"Sheep Creek", 51},
	}

	b, err := json.Marshal(roads)
	if err != nil {
		log.Fatal(err)
	}

	var out bytes.Buffer
	json.Indent(&out, b, "=", "\t")
	out.WriteTo(os.Stdout)
}

Output:

[
=	{
=		"Name": "Diamond Fork",
=		"Number": 29
=	},
=	{
=		"Name": "Sheep Creek",
=		"Number": 51
=	}
=]

func Marshal 

func Marshal(x interface{}) ([]byte, error)

Marshal is documented at https://golang.org/pkg/encoding/json/#Marshal

Example 
package main

import (
	"encoding/json"
	"fmt"
	"os"
)

func main() {
	type ColorGroup struct {
		ID     int
		Name   string
		Colors []string
	}
	group := ColorGroup{
		ID:     1,
		Name:   "Reds",
		Colors: []string{"Crimson", "Red", "Ruby", "Maroon"},
	}
	b, err := json.Marshal(group)
	if err != nil {
		fmt.Println("error:", err)
	}
	os.Stdout.Write(b)
}

Output:

{"ID":1,"Name":"Reds","Colors":["Crimson","Red","Ruby","Maroon"]}

func MarshalIndent 

func MarshalIndent(x interface{}, prefix, indent string) ([]byte, error)

MarshalIndent is documented at https://golang.org/pkg/encoding/json/#MarshalIndent

Example 
package main

import (
	"encoding/json"
	"fmt"
	"log"
)

func main() {
	data := map[string]int{
		"a": 1,
		"b": 2,
	}

	json, err := json.MarshalIndent(data, "<prefix>", "<indent>")
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(string(json))
}

Output:

{
<prefix><indent>"a": 1,
<prefix><indent>"b": 2
<prefix>}

func Parse 

func Parse(b []byte, x interface{}, flags ParseFlags) ([]byte, error)

Parse behaves like Unmarshal but the caller can pass a set of flags to configure the parsing behavior.

func Unmarshal 

func Unmarshal(b []byte, x interface{}) error

Unmarshal is documented at https://golang.org/pkg/encoding/json/#Unmarshal

Example 
package main

import (
	"encoding/json"
	"fmt"
)

func main() {
	var jsonBlob = []byte(`[
	{"Name": "Platypus", "Order": "Monotremata"},
	{"Name": "Quoll",    "Order": "Dasyuromorphia"}
]`)
	type Animal struct {
		Name  string
		Order string
	}
	var animals []Animal
	err := json.Unmarshal(jsonBlob, &animals)
	if err != nil {
		fmt.Println("error:", err)
	}
	fmt.Printf("%+v", animals)
}

Output:

[{Name:Platypus Order:Monotremata} {Name:Quoll Order:Dasyuromorphia}]

func Valid 

func Valid(data []byte) bool

Valid is documented at https://golang.org/pkg/encoding/json/#Valid

Example 
package main

import (
	"encoding/json"
	"fmt"
)

func main() {
	goodJSON := `{"example": 1}`
	badJSON := `{"example":2:]}}`

	fmt.Println(json.Valid([]byte(goodJSON)), json.Valid([]byte(badJSON)))
}

Output:

true false

Types

type AppendFlags 

type AppendFlags int

AppendFlags is a type used to represent configuration options that can be applied when formatting json output. 

const (
	// EscapeHTML is a formatting flag used to to escape HTML in json strings.
	EscapeHTML AppendFlags = 1 << iota

	// SortMapKeys is formatting flag used to enable sorting of map keys when
	// encoding JSON (this matches the behavior of the standard encoding/json
	// package).
	SortMapKeys

	// TrustRawMessage is a performance optimization flag to skip value
	// checking of raw messages. It should only be used if the values are
	// known to be valid json (e.g., they were created by json.Unmarshal).
	TrustRawMessage
)

type Decoder 

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder is documented at https://golang.org/pkg/encoding/json/#Decoder

Example

This example uses a Decoder to decode a stream of distinct JSON values. 

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"log"
	"strings"
)

func main() {
	const jsonStream = `
	{"Name": "Ed", "Text": "Knock knock."}
	{"Name": "Sam", "Text": "Who's there?"}
	{"Name": "Ed", "Text": "Go fmt."}
	{"Name": "Sam", "Text": "Go fmt who?"}
	{"Name": "Ed", "Text": "Go fmt yourself!"}
`
	type Message struct {
		Name, Text string
	}
	dec := json.NewDecoder(strings.NewReader(jsonStream))
	for {
		var m Message
		if err := dec.Decode(&m); err == io.EOF {
			break
		} else if err != nil {
			log.Fatal(err)
		}
		fmt.Printf("%s: %s\n", m.Name, m.Text)
	}
}

Output:

Ed: Knock knock.
Sam: Who's there?
Ed: Go fmt.
Sam: Go fmt who?
Ed: Go fmt yourself!

func NewDecoder 

func NewDecoder(r io.Reader) *Decoder

NewDecoder is documented at https://golang.org/pkg/encoding/json/#NewDecoder

func (*Decoder) Buffered 

func (dec *Decoder) Buffered() io.Reader

Buffered is documented at https://golang.org/pkg/encoding/json/#Decoder.Buffered

func (*Decoder) Decode 

func (dec *Decoder) Decode(v interface{}) error

Decode is documented at https://golang.org/pkg/encoding/json/#Decoder.Decode

Example (Stream)

This example uses a Decoder to decode a streaming array of JSON objects. 

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"strings"
)

func main() {
	const jsonStream = `
	[
		{"Name": "Ed", "Text": "Knock knock."},
		{"Name": "Sam", "Text": "Who's there?"},
		{"Name": "Ed", "Text": "Go fmt."},
		{"Name": "Sam", "Text": "Go fmt who?"},
		{"Name": "Ed", "Text": "Go fmt yourself!"}
	]
`
	type Message struct {
		Name, Text string
	}
	dec := json.NewDecoder(strings.NewReader(jsonStream))

	// read open bracket
	t, err := dec.Token()
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%T: %v\n", t, t)

	// while the array contains values
	for dec.More() {
		var m Message
		// decode an array value (Message)
		err := dec.Decode(&m)
		if err != nil {
			log.Fatal(err)
		}

		fmt.Printf("%v: %v\n", m.Name, m.Text)
	}

	// read closing bracket
	t, err = dec.Token()
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%T: %v\n", t, t)

}

Output:

json.Delim: [
Ed: Knock knock.
Sam: Who's there?
Ed: Go fmt.
Sam: Go fmt who?
Ed: Go fmt yourself!
json.Delim: ]

func (*Decoder) DisallowUnknownFields 

func (dec *Decoder) DisallowUnknownFields()

DisallowUnknownFields is documented at https://golang.org/pkg/encoding/json/#Decoder.DisallowUnknownFields

func (*Decoder) DontCopyNumber 

func (dec *Decoder) DontCopyNumber()

DontCopyNumber is an extension to the standard encoding/json package which instructs the decoder to not copy numbers loaded from the json payloads.

func (*Decoder) DontCopyRawMessage 

func (dec *Decoder) DontCopyRawMessage()

DontCopyRawMessage is an extension to the standard encoding/json package which instructs the decoder to not allocate RawMessage values in separate memory buffers (see the documentation of the DontcopyRawMessage flag for more detais).

func (*Decoder) DontCopyString 

func (dec *Decoder) DontCopyString()

DontCopyString is an extension to the standard encoding/json package which instructs the decoder to not copy strings loaded from the json payloads when possible.

func (*Decoder) DontMatchCaseInsensitiveStructFields 

func (dec *Decoder) DontMatchCaseInsensitiveStructFields()

DontMatchCaseInsensitiveStructFields is an extension to the standard encoding/json package which instructs the decoder to not match object fields against struct fields in a case-insensitive way, the field names have to match exactly to be decoded into the struct field values.

func (*Decoder) InputOffset 

func (dec *Decoder) InputOffset() int64

InputOffset returns the input stream byte offset of the current decoder position. The offset gives the location of the end of the most recently returned token and the beginning of the next token.

func (*Decoder) UseNumber 

func (dec *Decoder) UseNumber()

UseNumber is documented at https://golang.org/pkg/encoding/json/#Decoder.UseNumber

func (*Decoder) ZeroCopy 

func (dec *Decoder) ZeroCopy()

ZeroCopy is an extension to the standard encoding/json package which enables all the copy optimizations of the decoder.

type Delim 

type Delim = json.Delim

Delim is documented at https://golang.org/pkg/encoding/json/#Delim

type Encoder 

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder is documented at https://golang.org/pkg/encoding/json/#Encoder

func NewEncoder 

func NewEncoder(w io.Writer) *Encoder

NewEncoder is documented at https://golang.org/pkg/encoding/json/#NewEncoder

func (*Encoder) Encode 

func (enc *Encoder) Encode(v interface{}) error

Encode is documented at https://golang.org/pkg/encoding/json/#Encoder.Encode

func (*Encoder) SetEscapeHTML 

func (enc *Encoder) SetEscapeHTML(on bool)

SetEscapeHTML is documented at https://golang.org/pkg/encoding/json/#Encoder.SetEscapeHTML

func (*Encoder) SetIndent 

func (enc *Encoder) SetIndent(prefix, indent string)

SetIndent is documented at https://golang.org/pkg/encoding/json/#Encoder.SetIndent

func (*Encoder) SetSortMapKeys 

func (enc *Encoder) SetSortMapKeys(on bool)

SetSortMapKeys is an extension to the standard encoding/json package which allows the program to toggle sorting of map keys on and off.

func (*Encoder) SetTrustRawMessage 

func (enc *Encoder) SetTrustRawMessage(on bool)

SetTrustRawMessage skips value checking when encoding a raw json message. It should only be used if the values are known to be valid json, e.g. because they were originally created by json.Unmarshal.

type InvalidUTF8Error 

type InvalidUTF8Error = json.InvalidUTF8Error

InvalidUTF8Error is documented at https://golang.org/pkg/encoding/json/#InvalidUTF8Error

type InvalidUnmarshalError 

type InvalidUnmarshalError = json.InvalidUnmarshalError

InvalidUnmarshalError is documented at https://golang.org/pkg/encoding/json/#InvalidUnmarshalError

type Marshaler 

type Marshaler = json.Marshaler

Marshaler is documented at https://golang.org/pkg/encoding/json/#Marshaler

type MarshalerError 

type MarshalerError = json.MarshalerError

MarshalerError is documented at https://golang.org/pkg/encoding/json/#MarshalerError

type Number 

type Number = json.Number

Number is documented at https://golang.org/pkg/encoding/json/#Number

type ParseFlags 

type ParseFlags int

ParseFlags is a type used to represent configuration options that can be applied when parsing json input. 

const (
	// DisallowUnknownFields is a parsing flag used to prevent decoding of
	// objects to Go struct values when a field of the input does not match
	// with any of the struct fields.
	DisallowUnknownFields ParseFlags = 1 << iota

	// UseNumber is a parsing flag used to load numeric values as Number
	// instead of float64.
	UseNumber

	// DontCopyString is a parsing flag used to provide zero-copy support when
	// loading string values from a json payload. It is not always possible to
	// avoid dynamic memory allocations, for example when a string is escaped in
	// the json data a new buffer has to be allocated, but when the `wire` value
	// can be used as content of a Go value the decoder will simply point into
	// the input buffer.
	DontCopyString

	// DontCopyNumber is a parsing flag used to provide zero-copy support when
	// loading Number values (see DontCopyString and DontCopyRawMessage).
	DontCopyNumber

	// DontCopyRawMessage is a parsing flag used to provide zero-copy support
	// when loading RawMessage values from a json payload. When used, the
	// RawMessage values will not be allocated into new memory buffers and
	// will instead point directly to the area of the input buffer where the
	// value was found.
	DontCopyRawMessage

	// DontMatchCaseInsensitiveStructFields is a parsing flag used to prevent
	// matching fields in a case-insensitive way. This can prevent degrading
	// performance on case conversions, and can also act as a stricter decoding
	// mode.
	DontMatchCaseInsensitiveStructFields

	// ZeroCopy is a parsing flag that combines all the copy optimizations
	// available in the package.
	//
	// The zero-copy optimizations are better used in request-handler style
	// code where none of the values are retained after the handler returns.
	ZeroCopy = DontCopyString | DontCopyNumber | DontCopyRawMessage
)

type RawMessage 

type RawMessage = json.RawMessage

RawMessage is documented at https://golang.org/pkg/encoding/json/#RawMessage

Example (Marshal)

This example uses RawMessage to use a precomputed JSON during marshal. 

package main

import (
	"encoding/json"
	"fmt"
	"os"
)

func main() {
	h := json.RawMessage(`{"precomputed": true}`)

	c := struct {
		Header *json.RawMessage `json:"header"`
		Body   string           `json:"body"`
	}{Header: &h, Body: "Hello Gophers!"}

	b, err := json.MarshalIndent(&c, "", "\t")
	if err != nil {
		fmt.Println("error:", err)
	}
	os.Stdout.Write(b)

}

Output:

{
	"header": {
		"precomputed": true
	},
	"body": "Hello Gophers!"
}
Example (Unmarshal)

This example uses RawMessage to delay parsing part of a JSON message. 

package main

import (
	"encoding/json"
	"fmt"
	"log"
)

func main() {
	type Color struct {
		Space string
		Point json.RawMessage // delay parsing until we know the color space
	}
	type RGB struct {
		R uint8
		G uint8
		B uint8
	}
	type YCbCr struct {
		Y  uint8
		Cb int8
		Cr int8
	}

	var j = []byte(`[
	{"Space": "YCbCr", "Point": {"Y": 255, "Cb": 0, "Cr": -10}},
	{"Space": "RGB",   "Point": {"R": 98, "G": 218, "B": 255}}
]`)
	var colors []Color
	err := json.Unmarshal(j, &colors)
	if err != nil {
		log.Fatalln("error:", err)
	}

	for _, c := range colors {
		var dst interface{}
		switch c.Space {
		case "RGB":
			dst = new(RGB)
		case "YCbCr":
			dst = new(YCbCr)
		}
		err := json.Unmarshal(c.Point, dst)
		if err != nil {
			log.Fatalln("error:", err)
		}
		fmt.Println(c.Space, dst)
	}
}

Output:

YCbCr &{255 0 -10}
RGB &{98 218 255}

type RawValue 

type RawValue []byte

RawValue represents a raw json value, it is intended to carry null, true, false, number, and string values only.

func (RawValue) AppendUnquote 

func (v RawValue) AppendUnquote(b []byte) []byte

AppendUnquote writes the unquoted version of the string value in v into b.

func (RawValue) False 

func (v RawValue) False() bool

False returns true if v contains a false value.

func (RawValue) Null 

func (v RawValue) Null() bool

Null returns true if v contains a null value.

func (RawValue) Number 

func (v RawValue) Number() bool

Number returns true if v contains a number value.

func (RawValue) String 

func (v RawValue) String() bool

String returns true if v contains a string value.

func (RawValue) True 

func (v RawValue) True() bool

True returns true if v contains a true value.

func (RawValue) Unquote 

func (v RawValue) Unquote() []byte

Unquote returns the unquoted version of the string value in v.

type SyntaxError 

type SyntaxError = json.SyntaxError

A SyntaxError is a description of a JSON syntax error.

type Token 

type Token = json.Token

Token is documented at https://golang.org/pkg/encoding/json/#Token

type Tokenizer 

type Tokenizer struct {
	// When the tokenizer is positioned on a json delimiter this field is not
	// zero. In this case the possible values are '{', '}', '[', ']', ':', and
	// ','.
	Delim Delim

	// This field contains the raw json token that the tokenizer is pointing at.
	// When Delim is not zero, this field is a single-element byte slice
	// continaing the delimiter value. Otherwise, this field holds values like
	// null, true, false, numbers, or quoted strings.
	Value RawValue

	// When the tokenizer has encountered invalid content this field is not nil.
	Err error

	// When the value is in an array or an object, this field contains the depth
	// at which it was found.
	Depth int

	// When the value is in an array or an object, this field contains the
	// position at which it was found.
	Index int

	// This field is true when the value is the key of an object.
	IsKey bool
	// contains filtered or unexported fields
}

Tokenizer is an iterator-style type which can be used to progressively parse through a json input.

Tokenizing json is useful to build highly efficient parsing operations, for example when doing tranformations on-the-fly where as the program reads the input and produces the transformed json to an output buffer.

Here is a common pattern to use a tokenizer: 

for t := json.NewTokenizer(b); t.Next(); {
	switch t.Delim {
	case '{':
		...
	case '}':
		...
	case '[':
		...
	case ']':
		...
	case ':':
		...
	case ',':
		...
	}

	switch {
	case t.Value.String():
		...
	case t.Value.Null():
		...
	case t.Value.True():
		...
	case t.Value.False():
		...
	case t.Value.Number():
		...
	}
}

func NewTokenizer 

func NewTokenizer(b []byte) *Tokenizer

NewTokenizer constructs a new Tokenizer which reads its json input from b.

func (*Tokenizer) Next 

func (t *Tokenizer) Next() bool

Next returns a new tokenizer pointing at the next token, or the zero-value of Tokenizer if the end of the json input has been reached.

If the tokenizer encounters malformed json while reading the input the method sets t.Err to an error describing the issue, and returns false. Once an error has been encountered, the tokenizer will always fail until its input is cleared by a call to its Reset method.

func (*Tokenizer) Reset 

func (t *Tokenizer) Reset(b []byte)

Reset erases the state of t and re-initializes it with the json input from b.

type UnmarshalFieldError 

type UnmarshalFieldError = json.UnmarshalFieldError

UnmarshalFieldError is documented at https://golang.org/pkg/encoding/json/#UnmarshalFieldError

type UnmarshalTypeError 

type UnmarshalTypeError = json.UnmarshalTypeError

UnmarshalTypeError is documented at https://golang.org/pkg/encoding/json/#UnmarshalTypeError

type Unmarshaler 

type Unmarshaler = json.Unmarshaler

Unmarshaler is documented at https://golang.org/pkg/encoding/json/#Unmarshaler

type UnsupportedTypeError 

type UnsupportedTypeError = json.UnsupportedTypeError

UnsupportedTypeError is documented at https://golang.org/pkg/encoding/json/#UnsupportedTypeError

type UnsupportedValueError 

type UnsupportedValueError = json.UnsupportedValueError

UnsupportedValueError is documented at https://golang.org/pkg/encoding/json/#UnsupportedValueError

Source Files

View all Source files

Directories

Path Synopsis
bugs
issue11
issue18

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.