marshmallow

README

Marshmallow

Marshmallow package provides a simple API to perform flexible and performant JSON unmarshalling in Go.

Marshmallow specializes in dealing with unstructured struct - when some fields are known and some aren't, with zero performance overhead nor extra coding needed. While unmarshalling, marshmallow allows fully retaining the original data and access it via a typed struct and a dynamic map.

Contents

• Install
• Usage
• Performance Benchmark And Alternatives
• When Should I Use Marshmallow
• API

Install

go get -u github.com/perimeterx/marshmallow

Usage

func main() {
	v := struct {
		Foo string `json:"foo"`
		Boo []int  `json:"boo"`
	}{}
	result, err := marshmallow.Unmarshal([]byte(`{"foo":"bar","boo":[1,2,3],"goo":12.6}`), &v)
	fmt.Printf("v=%+v, result=%+v, err=%v", v, result, err)
	// Output: v={Foo:bar Boo:[1 2 3]}, result=map[boo:[1 2 3] foo:bar goo:12.6], err=<nil>
}

Examples can be found here

Performance Benchmark And Alternatives

Marshmallow performs best when dealing with mixed data - when some fields are known and some are unknown. More info below. Other solutions are available for this kind of use case, each solution is explained and documented in the link below. The full benchmark test can be found here.

Benchmark	Iterations	Time/Iteration	Bytes Allocated	Allocations
unmarshall twice	228693	5164 ns/op	1640 B/op	51 allocs/op
raw map	232236	5116 ns/op	2296 B/op	53 allocs/op
go codec	388442	3077 ns/op	2512 B/op	37 allocs/op
marshmallow	626168	1853 ns/op	608 B/op	18 allocs/op
marshmallow without populating struct	678616	1751 ns/op	608 B/op	18 allocs/op

Marshmallow provides the best performance (up to X3 faster) while not requiring any extra coding. In fact, marshmallow performs as fast as normal json.Unmarshal call, however, such a call causes loss of data for all the fields that did not match the given struct. With marshmallow you never lose any data.

Benchmark	Iterations	Time/Iteration	Bytes Allocated	Allocations
marshmallow	626168	1853 ns/op	608 B/op	18 allocs/op
native library	652106	1845 ns/op	304 B/op	11 allocs/op
marshmallow without populating struct	678616	1751 ns/op	608 B/op	18 allocs/op

When Should I Use Marshmallow

Marshmallow is best suited for use cases where you are interested in all the input data, but you have predetermined information only about a subset of it. For instance, if you plan to reference two specific fields from the data, then iterate all the data and apply some generic logic. How does it look with the native library:

func isAllowedToDrive(data []byte) (bool, error) {
	result := make(map[string]interface{})
	err := json.Unmarshal(data, &result)
	if err != nil {
		return false, err
	}

	age, ok := result["age"]
	if !ok {
		return false, nil
	}
	a, ok := age.(float64)
	if !ok {
		return false, nil
	}
	if a < 17 {
		return false, nil
	}

	hasDriversLicense, ok := result["has_drivers_license"]
	if !ok {
		return false, nil
	}
	h, ok := hasDriversLicense.(bool)
	if !ok {
		return false, nil
	}
	if !h {
		return false, nil
	}

	for key := range result {
		if strings.Contains(key, "prior_conviction") {
			return false, nil
		}
	}

	return true, nil
}

And with marshmallow:

func isAllowedToDrive(data []byte) (bool, error) {
	v := struct {
		Age               int  `json:"age"`
		HasDriversLicense bool `json:"has_drivers_license"`
	}{}
	result, err := marshmallow.Unmarshal(data, &v)
	if err != nil {
		return false, err
	}

	if v.Age < 17 || !v.HasDriversLicense {
		return false, nil
	}

	for key := range result {
		if strings.Contains(key, "prior_conviction") {
			return false, nil
		}
	}

	return true, nil
}

API

Marshmallow exposes two main API functions - Unmarshal and UnmarshalFromJSONMap. While unmarshalling, marshmallow supports the following optional options:

• Setting the mode for handling invalid data using the WithMode function.
• Excluding known fields from the result map using the WithExcludeKnownFieldsFromMap function.
• Skipping struct population to boost performance using the WithSkipPopulateStruct function.

In order to capture unknown nested fields, structs must implement JSONDataErrorHandler. More info here.

Marshmallow also supports caching of refection information using EnableCache and EnableCustomCache.

Contact and Contribute

Reporting issues and requesting features may be done in our GitHub issues page. Discussions may be conducted in our GitHub discussions page. For any further questions or comments you can reach us out at open-source@humansecurity.com.

Any type of contribution is warmly welcome and appreciated ❤️ Please read our contribution guide for more info.

If you're looking for something to get started with, tou can always follow our issues page and look for good first issue and help wanted labels.

Marshmallow Logo

Marshmallow logo and assets by Adva Rom are licensed under a Creative Commons Attribution 4.0 International License.

Documentation

Index

• Variables
• func EnableCache()
• func EnableCustomCache(c Cache)
• func Unmarshal(data []byte, v interface{}, options ...UnmarshalOption) (map[string]interface{}, error)
• func UnmarshalFromJSONMap(data map[string]interface{}, v interface{}, options ...UnmarshalOption) (map[string]interface{}, error)
• type Cache
• type JSONDataErrorHandler
• type JSONDataHandler
• type Mode
• type MultipleError
  • func (m *MultipleError) Error() string
• type MultipleLexerError
  • func (m *MultipleLexerError) Error() string
• type ParseError
  • func (p *ParseError) Error() string
• type UnmarshalOption
  • func WithExcludeKnownFieldsFromMap(excludeKnownFields bool) UnmarshalOption
  • func WithMode(mode Mode) UnmarshalOption
  • func WithSkipPopulateStruct(skipPopulateStruct bool) UnmarshalOption
• type UnmarshalerFromJSONMap

Variables

var (
	// ErrInvalidInput indicates the input JSON is invalid
	ErrInvalidInput = errors.New("invalid JSON input")

	// ErrInvalidValue indicates the target struct has invalid type
	ErrInvalidValue = errors.New("unexpected non struct value")
)

Functions

func EnableCache

func EnableCache()

EnableCache enables unmarshalling cache with default implementation. More info at EnableCustomCache.

func EnableCustomCache

func EnableCustomCache(c Cache)

EnableCustomCache enables unmarshalling cache. It allows reuse of refection information about types needed to perform the unmarshalling. A use of such cache can boost up unmarshalling by x1.4. Check out benchmark_test.go for an example.

EnableCustomCache is not thread safe! Do not use it while performing unmarshalling, or it will cause an unsafe race condition. Typically, EnableCustomCache should be called once when the process boots.

Caching is disabled by default. The use of this function allows enabling it and controlling the behavior of the cache. Typically, the use of sync.Map should be good enough. The caching mechanism stores a single map per struct type. If you plan to unmarshal a huge amount of distinct struct it may get to consume a lot of resources, in which case you have the control to choose the caching implementation you like and its setup.

func Unmarshal

func Unmarshal(data []byte, v interface{}, options ...UnmarshalOption) (map[string]interface{}, error)

Unmarshal parses the JSON-encoded object in data and stores the values in the struct pointed to by v and in the returned map. If v is nil or not a pointer to a struct, Unmarshal returns an ErrInvalidValue. If data is not a valid JSON or not a JSON object Unmarshal returns an ErrInvalidInput.

Unmarshal follows the rules of json.Unmarshal with the following exceptions: - All input fields are stored in the resulting map, including fields that do not exist in the struct pointed by v. - Unmarshal only operates on JSON object inputs. It will reject all other types of input by returning ErrInvalidInput. - Unmarshal only operates on struct values. It will reject all other types of v by returning ErrInvalidValue. - Unmarshal supports three types of Mode values. Each mode is self documented and affects how Unmarshal behaves.

func UnmarshalFromJSONMap

func UnmarshalFromJSONMap(data map[string]interface{}, v interface{}, options ...UnmarshalOption) (map[string]interface{}, error)

UnmarshalFromJSONMap parses the JSON map data and stores the values in the struct pointed to by v and in the returned map. If v is nil or not a pointer to a struct, UnmarshalFromJSONMap returns an ErrInvalidValue.

UnmarshalFromJSONMap follows the rules of json.Unmarshal with the following exceptions: - All input fields are stored in the resulting map, including fields that do not exist in the struct pointed by v. - UnmarshalFromJSONMap receive a JSON map instead of raw bytes. The given input map is assumed to be a JSON map, meaning it should only contain the following types: bool, string, float64, []interface, and map[string]interface{}. Other types will cause decoding to return unexpected results. - UnmarshalFromJSONMap only operates on struct values. It will reject all other types of v by returning ErrInvalidValue. - UnmarshalFromJSONMap supports three types of Mode values. Each mode is self documented and affects how UnmarshalFromJSONMap behaves.

Types

type Cache

type Cache interface {
	// Load returns the value stored in the map for a key, or nil if no value is present.
	// The ok result indicates whether value was found in the map.
	Load(key interface{}) (interface{}, bool)
	// Store sets the value for a key.
	Store(key, value interface{})
}

Cache allows unmarshalling to use a cached version of refection information about types. Cache interface follows the implementation of sync.Map, but you may wrap any cache implementation to match it. This allows you to control max cache size, eviction policies and any other caching aspect.

type JSONDataErrorHandler

type JSONDataErrorHandler interface {
	HandleJSONData(data map[string]interface{}) error
}

JSONDataErrorHandler allow types to handle JSON data as maps. Types should implement this interface if they wish to act on the map representation of parsed JSON input. This is mainly used to allow nested objects to capture unknown fields and leverage marshmallow's abilities. If HandleJSONData returns an error, it will be propagated as an unmarshal error

type JSONDataHandler

type JSONDataHandler interface {
	HandleJSONData(data map[string]interface{})
}

Deprecated: use JSONDataErrorHandler instead

type Mode

type Mode uint8

Mode dictates the unmarshalling mode. Each mode is self documented below.

const (
	// ModeFailOnFirstError is the default mode. It makes unmarshalling terminate
	// immediately on any kind of error. This error will then be returned.
	ModeFailOnFirstError Mode = iota

	// ModeAllowMultipleErrors mode makes unmarshalling keep decoding even if
	// errors are encountered. In case of such error, the erroneous value will be omitted from the result.
	// Eventually, all errors will all be returned, alongside the partial result.
	ModeAllowMultipleErrors

	// ModeFailOverToOriginalValue mode makes unmarshalling keep decoding even if
	// errors are encountered. In case of such error, the original external value be placed in the
	// result data, even though it does not meet the schematic requirements.
	// Eventually, all errors will be returned, alongside the full result. Note that the result map
	// will contain values that do not match the struct schema.
	ModeFailOverToOriginalValue
)

type MultipleError

type MultipleError struct {
	Errors []error
}

MultipleError indicates one or more unmarshalling errors during JSON map decode

func (*MultipleError) Error

func (m *MultipleError) Error() string

type MultipleLexerError

type MultipleLexerError struct {
	Errors []*jlexer.LexerError
}

MultipleLexerError indicates one or more unmarshalling errors during JSON bytes decode

func (*MultipleLexerError) Error

func (m *MultipleLexerError) Error() string

type ParseError

type ParseError struct {
	Reason string
	Path   string
}

ParseError indicates a JSON map decode error

func (*ParseError) Error

func (p *ParseError) Error() string

type UnmarshalOption

type UnmarshalOption func(*unmarshalOptions)

func WithExcludeKnownFieldsFromMap

func WithExcludeKnownFieldsFromMap(excludeKnownFields bool) UnmarshalOption

WithExcludeKnownFieldsFromMap is an UnmarshalOption function to set the excludeKnownFieldsFromMap option. Exclude known fields flag is set to false by default. When the flag is set to true, fields specified in the input struct (known fields) will be excluded from the result map

func WithMode

func WithMode(mode Mode) UnmarshalOption

WithMode is an UnmarshalOption function to set the unmarshalling mode.

func WithSkipPopulateStruct

func WithSkipPopulateStruct(skipPopulateStruct bool) UnmarshalOption

WithSkipPopulateStruct is an UnmarshalOption function to set the skipPopulateStruct option. Skipping populate struct is set to false by default. If you do not intend to use the struct value once unmarshalling is finished, set this option to true to boost performance. This would mean the struct fields will not be set with values, but rather it will only be used as the target schema when populating the result map.

type UnmarshalerFromJSONMap

type UnmarshalerFromJSONMap interface {
	UnmarshalJSONFromMap(data interface{}) error
}

UnmarshalerFromJSONMap is the interface implemented by types that can unmarshal a JSON description of themselves. In case you want to implement custom unmarshalling, json.Unmarshaler only supports receiving the data as []byte. However, while unmarshalling from JSON map, the data is not available as a raw []byte and converting to it will significantly hurt performance. Thus, if you wish to implement a custom unmarshalling on a type that is being unmarshalled from a JSON map, you need to implement UnmarshalerFromJSONMap interface.