simdjson

package module
v0.0.0-...-21db5a4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 10, 2020 License: Apache-2.0 Imports: 18 Imported by: 0

README

simdjson-go

This repository has moved to github.com/minio/simdjson-go.

Documentation

Index

Constants

View Source 
const (
	TagString      = Tag('"')
	TagInteger     = Tag('l')
	TagUint        = Tag('u')
	TagFloat       = Tag('d')
	TagNull        = Tag('n')
	TagBoolTrue    = Tag('t')
	TagBoolFalse   = Tag('f')
	TagObjectStart = Tag('{')
	TagObjectEnd   = Tag('}')
	TagArrayStart  = Tag('[')
	TagArrayEnd    = Tag(']')
	TagRoot        = Tag('r')
	TagEnd         = Tag(0)
)
View Source 
const DEFAULTDEPTH = 128
View Source 
const GOLANG_NUMBER_PARSING = true

The current parse_number() code has some precision issues (see PR "Exact float parsing", https://github.com/lemire/simdjson/pull/333)

An example of this (from canada.json): 

"-65.619720000000029" --> -65.61972000000004

instead of 

"-65.619720000000029" --> -65.61972000000003

There is a slower code path that uses Golang's Atoi() and ParseFloat()

For benchmarking GOLANG_NUMBER_PARSING is set to false

View Source 
const INDEX_SIZE = 1536 // Seems to be a good size for the index buffering
View Source 
const INDEX_SIZE_WITH_SAFETY_BUFFER = INDEX_SIZE - 128 // Make sure we never write beyond buffer
View Source 
const INDEX_SLOTS = 16
View Source 
const JSONVALUEMASK = 0xffffffffffffff
View Source 
const RET_ADDRESS_ARRAY_CONST = 3
View Source 
const RET_ADDRESS_OBJECT_CONST = 2
View Source 
const RET_ADDRESS_SHIFT = 2

Constants for "return address" modes

View Source 
const RET_ADDRESS_START_CONST = 1
View Source 
const STRINGBUFBIT = 0x80000000000000
View Source 
const STRINGBUFMASK = 0x7fffffffffffff

Variables

View Source 
var TagToType = [256]Type{
	TagString:      TypeString,
	TagInteger:     TypeInt,
	TagUint:        TypeUint,
	TagFloat:       TypeFloat,
	TagNull:        TypeNull,
	TagBoolTrue:    TypeBool,
	TagBoolFalse:   TypeBool,
	TagObjectStart: TypeObject,
	TagArrayStart:  TypeArray,
	TagRoot:        TypeRoot,
}

TagToType converts a tag to type. For arrays and objects only the start tag will return types. All non-existing tags returns TypeNone.

Functions

func ParseNDStream 

func ParseNDStream(r io.Reader, res chan<- Stream, reuse <-chan *ParsedJson)

ParseNDStream will parse a stream and return parsed JSON to the supplied result channel. The method will return immediately. Each element is contained within a root tag. 

<root>Element 1</root><root>Element 2</root>...

Each result will contain an unspecified number of full elements, so it can be assumed that each result starts and ends with a root tag. The parser will keep parsing until writes to the result stream blocks. A stream is finished when a non-nil Error is returned. If the stream was parsed until the end the Error value will be io.EOF The channel will be closed after an error has been returned. An optional channel for returning consumed results can be provided. There is no guarantee that elements will be consumed, so always use non-blocking writes to the reuse channel.

func SupportedCPU 

func SupportedCPU() bool

SupportedCPU will return whether the CPU is supported.

Types

type Array 

type Array struct {
	// contains filtered or unexported fields
}

Array represents a JSON array. There are methods that allows to get full arrays if the value type is the same. Otherwise an iterator can be retrieved.

func (*Array) AsFloat 

func (a *Array) AsFloat() ([]float64, error)

AsFloat returns the array values as float. Integers are automatically converted to float.

func (*Array) AsInteger 

func (a *Array) AsInteger() ([]int64, error)

AsInteger returns the array values as float. Integers are automatically converted to float.

func (*Array) AsString 

func (a *Array) AsString() ([]string, error)

AsString returns the array values as a slice of strings. No conversion is done.

func (*Array) FirstType 

func (a *Array) FirstType() Type

FirstType will return the type of the first element. If there are no elements, TypeNone is returned.

func (*Array) Interface 

func (a *Array) Interface() ([]interface{}, error)

Interface returns the array as a slice of interfaces. See Iter.Interface() for a reference on value types.

func (*Array) Iter 

func (a *Array) Iter() Iter

Iter returns the array as an iterator. This can be used for parsing mixed content arrays. The first value is ready with a call to Advance. Calling after last element should have TypeNone.

func (*Array) MarshalJSON 

func (a *Array) MarshalJSON() ([]byte, error)

MarshalJSON will marshal the entire remaining scope of the iterator.

func (*Array) MarshalJSONBuffer 

func (a *Array) MarshalJSONBuffer(dst []byte) ([]byte, error)

MarshalJSONBuffer will marshal all elements. An optional buffer can be provided for fewer allocations. Output will be appended to the destination.

type CompressMode 

type CompressMode uint8
const (
	// CompressNone no compression whatsoever.
	CompressNone CompressMode = iota

	// CompressFast will apply light compression,
	// but will not deduplicate strings which may affect deserialization speed.
	CompressFast

	// CompressDefault applies light compression and deduplicates strings.
	CompressDefault

	// CompressBest
	CompressBest
)

type Element 

type Element struct {
	// Name of the element
	Name string
	// Type of the element
	Type Type
	// Iter containing the element
	Iter Iter
}

Element represents an element in an object.

type Elements 

type Elements struct {
	Elements []Element
	Index    map[string]int
}

Elements contains all elements in an object kept in original order. And index contains lookup for object keys.

func (Elements) Lookup 

func (e Elements) Lookup(key string) *Element

Lookup a key in elements and return the element. Returns nil if key doesn't exist. Keys are case sensitive.

func (Elements) MarshalJSON 

func (e Elements) MarshalJSON() ([]byte, error)

MarshalJSON will marshal the entire remaining scope of the iterator.

func (Elements) MarshalJSONBuffer 

func (e Elements) MarshalJSONBuffer(dst []byte) ([]byte, error)

MarshalJSONBuffer will marshal all elements. An optional buffer can be provided for fewer allocations. Output will be appended to the destination.

type Iter 

type Iter struct {
	// contains filtered or unexported fields
}

Iter represents a section of JSON. To start iterating it, use Advance() or AdvanceIter() methods which will queue the first element. If an Iter is copied, the copy will be independent.

func (*Iter) Advance 

func (i *Iter) Advance() Type

Advance will read the type of the next element and queues up the value on the same level.

func (*Iter) AdvanceInto 

func (i *Iter) AdvanceInto() Tag

AdvanceInto will read the tag of the next element and move into and out of arrays , objects and root elements. This should only be used for strictly manual parsing.

func (*Iter) AdvanceIter 

func (i *Iter) AdvanceIter(dst *Iter) (Type, error)

AdvanceIter will read the type of the next element and return an iterator only containing the object. If dst and i are the same, both will contain the value inside.

func (*Iter) Array 

func (i *Iter) Array(dst *Array) (*Array, error)

Array will return the next element as an array. An optional destination can be given.

func (*Iter) Bool 

func (i *Iter) Bool() (bool, error)

Bool() returns the bool value.

func (*Iter) Float 

func (i *Iter) Float() (float64, error)

Float returns the float value of the next element. Integers are automatically converted to float.

func (*Iter) Int 

func (i *Iter) Int() (int64, error)

Int returns the integer value of the next element. Integers and floats within range are automatically converted.

func (*Iter) Interface 

func (i *Iter) Interface() (interface{}, error)

Interface returns the value as an interface. Objects are returned as map[string]interface{}. Arrays are returned as []interface{}. Float values are returned as float64. Integer values are returned as int64 or uint64. String values are returned as string. Boolean values are returned as bool. Null values are returned as nil. Root objects are returned as []interface{}.

func (*Iter) MarshalJSON 

func (i *Iter) MarshalJSON() ([]byte, error)

MarshalJSON will marshal the entire remaining scope of the iterator.

func (*Iter) MarshalJSONBuffer 

func (i *Iter) MarshalJSONBuffer(dst []byte) ([]byte, error)

MarshalJSONBuffer will marshal the remaining scope of the iterator including the current value. An optional buffer can be provided for fewer allocations. Output will be appended to the destination.

func (*Iter) Object 

func (i *Iter) Object(dst *Object) (*Object, error)

Object will return the next element as an object. An optional destination can be given.

func (*Iter) PeekNext 

func (i *Iter) PeekNext() Type

PeekNext will return the next value type. Returns TypeNone if next ends iterator.

func (*Iter) PeekNextTag 

func (i *Iter) PeekNextTag() Tag

PeekNextTag will return the tag at the current offset. Will return TagEnd if at end of iterator.

func (*Iter) Root 

func (i *Iter) Root(dst *Iter) (Type, *Iter, error)

Root() returns the object embedded in root as an iterator along with the type of the content of the first element of the iterator. An optional destination can be supplied to avoid allocations.

func (*Iter) String 

func (i *Iter) String() (string, error)

String() returns a string value.

func (*Iter) StringBytes 

func (i *Iter) StringBytes() ([]byte, error)

StringBytes() returns a byte array.

func (*Iter) StringCvt 

func (i *Iter) StringCvt() (string, error)

StringCvt() returns a string representation of the value. Root, Object and Arrays are not supported.

func (*Iter) Type 

func (i *Iter) Type() Type

Type returns the queued value type from the previous call to Advance.

func (*Iter) Uint 

func (i *Iter) Uint() (uint64, error)

Uint returns the unsigned integer value of the next element. Positive integers and floats within range are automatically converted.

type Object 

type Object struct {
	// contains filtered or unexported fields
}

Object represents a JSON object.

func (*Object) FindKey 

func (o *Object) FindKey(key string, dst *Element) *Element

FindKey will return a single named element. An optional destination can be given. The method will return nil if the element cannot be found. This should only be used to locate a single key where the object is no longer needed. The object will not be advanced.

func (*Object) Map 

func (o *Object) Map(dst map[string]interface{}) (map[string]interface{}, error)

Map will unmarshal into a map[string]interface{} See Iter.Interface() for a reference on value types.

func (*Object) NextElement 

func (o *Object) NextElement(dst *Iter) (name string, t Type, err error)

NextElement sets dst to the next element and returns the name. TypeNone with nil error will be returned if there are no more elements.

func (*Object) NextElementBytes 

func (o *Object) NextElementBytes(dst *Iter) (name []byte, t Type, err error)

NextElementBytes sets dst to the next element and returns the name. TypeNone with nil error will be returned if there are no more elements. Contrary to NextElement this will not cause allocations.

func (*Object) Parse 

func (o *Object) Parse(dst *Elements) (*Elements, error)

Parse will return all elements and iterators. An optional destination can be given. The Object will be consumed.

type ParsedJson 

type ParsedJson struct {
	Message []byte
	Tape    []uint64
	Strings []byte
	// contains filtered or unexported fields
}

func Parse 

func Parse(b []byte, reuse *ParsedJson) (*ParsedJson, error)

Parse a block of data and return the parsed JSON. An optional block of previously parsed json can be supplied to reduce allocations.

func ParseND 

func ParseND(b []byte, reuse *ParsedJson) (*ParsedJson, error)

ParseND will parse newline delimited JSON. An optional block of previously parsed json can be supplied to reduce allocations.

func (*ParsedJson) Iter 

func (pj *ParsedJson) Iter() Iter

Iter returns a new Iter.

func (*ParsedJson) Reset 

func (pj *ParsedJson) Reset()

type Serializer 

type Serializer struct {
	// contains filtered or unexported fields
}

Serializer allows to serialize parsed json and read it back. A Serializer can be reused, but not used concurrently.

func NewSerializer 

func NewSerializer() *Serializer

NewSerializer will create and initialize a serializer.

func (*Serializer) CompressMode 

func (s *Serializer) CompressMode(c CompressMode)

func (*Serializer) Deserialize 

func (s *Serializer) Deserialize(src []byte, dst *ParsedJson) (*ParsedJson, error)

Deserialize the content in src. Only basic sanity checks will be performed. Slight corruption will likely go through unnoticed. And optional destination can be provided.

func (*Serializer) Serialize 

func (s *Serializer) Serialize(dst []byte, pj ParsedJson) []byte

Serialize the data in pj and return the data. An optional destination can be provided.

type Stream 

type Stream struct {
	Value *ParsedJson
	Error error
}

A Stream is used to stream back results. Either Error or Value will be set on returned results.

type Tag 

type Tag uint8

Tag indicates the data type of a tape entry

func (Tag) String 

func (t Tag) String() string

func (Tag) Type 

func (t Tag) Type() Type

Type converts a tag to a type. Only basic types and array+object start match a type.

type Type 

type Type uint8

Type is a JSON value type. 

const (
	TypeNone Type = iota
	TypeNull
	TypeString
	TypeInt
	TypeUint
	TypeFloat
	TypeBool
	TypeObject
	TypeArray
	TypeRoot
)

func (Type) String 

func (t Type) String() string

String returns the type as a string.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.