codec

README

Codecs

The go-ipld-prime/codec package is a grouping package. The subpackages contains some codecs which reside in this repo.

The codecs included here are our "batteries included" codecs, but they are not otherwise special.

It is not necessary for a codec to be a subpackage here to be a valid codec to use with go-ipld; anything that implements the codec.Encoder and codec.Decoder interfaces is fine.

Terminology

We generally refer to "codecs" as having an "encode" function and "decode" function.

We consider "encoding" to be the process of going from {Data Model} to {serial data}, and "decoding" to be the process of going from {serial data} to {Data Model}.

Codec vs Multicodec

A "codec" is any function that goes from {Data Model} to {serial data}, or vice versa.

A "multicodec" is a function which does that and is also specifically recognized and described in the tables in https://github.com/multiformats/multicodec/ .

Multicodecs generally leave no further room for customization and configuration, because their entire behavior is supposed to be specified by a multicodec indicator code number.

Our codecs, in the child packages of this one, usually offer configuration options. They also usually offer exactly one function, which does not allow configuration, which is supplying a multicodec-compatible behavior. You'll see this marked in the docs on those functions.

Marshal vs Encode

It's common to see the terms "marshal" and "unmarshal" used in golang.

Those terms are usually describing when structured data is transformed into linearized, tokenized data (and then, perhaps, all the way to serially encoded data), or vice versa.

We would use the words the same way... except we don't end up using them, because that feature doesn't really come up in our codec layer.

In IPLD, we would describe mapping some typed data into Data Model as "marshalling". (It's one step shy of tokenizing, but barely: Data Model does already have defined ordering for every element of data.) And we do have systems that do this: bindnode and our codegen systems both do this, implicitly, when they give you an ipld.Node of the representation of some data.

We just don't end up talking about it as "marshalling" because of how it's done implicitly by those systems. As a result, all of our features relating to codecs only end up speaking about "encoding" and "decoding".

Legacy code

There are some appearances of the words "marshal" and "unmarshal" in some of our subpackages here.

That verbiage is generally on the way out. For functions and structures with those names, you'll notice their docs marking them as deprecated.

Why have "batteries-included" codecs?

These codecs live in this repo because they're commonly used, highly supported, and general-purpose codecs that we recommend for widespread usage in new developments.

Also, it's just plain nice to have something in-repo for development purposes. It makes sure that if we try to make any API changes, we immediately see if they'd make codecs harder to implement. We also use the batteries-included codecs for debugging, for test fixtures, and for benchmarking.

Further yet, the batteries-included codecs let us offer getting-started APIs. For example, we offer some helper APIs which use codecs like e.g. JSON to give consumers of the libraries one-step helper methods that "do the right thing" with zero config... so long as they happen to use that codec. Even for consumers who don't use those codecs, such functions then serve as natural documentation and examples for what to do to put their codec of choice to work.

Documentation

Index

• type Decoder
• type Encoder
• type ErrBudgetExhausted
  • func (e ErrBudgetExhausted) Error() string
• type MapSortMode

Types

type Decoder

type Decoder func(datamodel.NodeAssembler, io.Reader) error

Decoder defines the shape of a function which produces a Node tree by reading serialized data from an io.Reader. (Decoder doesn't itself return a Node directly, but rather takes a NodeAssembler as an argument, because this allows the caller more control over the Node implementation, as well as some control over allocations.)

The dual of Decoder is an Encoder, which takes a Node and emits its data in a serialized form into an io.Writer. Typically, Decoder and Encoder functions will be found in pairs, and will be expected to be able to round-trip each other's data.

Decoder functions can be used directly. Decoder functions are also often used via a LinkSystem when working with content-addressed storage. LinkSystem methods will helpfully handle the entire process of opening block readers, verifying the hash of the data stream, and applying a Decoder to build Nodes -- all as one step.

A Decoder works with Nodes. If you have a native golang structure, and want to populate it with data using a Decoder, you'll need to either get a NodeAssembler which proxies data into that structure directly, or assemble a Node as intermediate storage and copy the data to the native structure as a separate step.

It may be useful to understand "multicodecs" when working with Decoders. See the documentation on the Encoder function interface for more discussion of multicodecs, the multicodec table, and how this is typically connected to linking.

type Encoder

type Encoder func(datamodel.Node, io.Writer) error

Encoder defines the shape of a function which traverses a Node tree and emits its data in a serialized form into an io.Writer.

The dual of Encoder is a Decoder, which takes a NodeAssembler and fills it with deserialized data consumed from an io.Reader. Typically, Decoder and Encoder functions will be found in pairs, and will be expected to be able to round-trip each other's data.

Encoder functions can be used directly. Encoder functions are also often used via a LinkSystem when working with content-addressed storage. LinkSystem methods will helpfully handle the entire process of traversing a Node tree, encoding this data, hashing it, streaming it to the writer, and committing it -- all as one step.

An Encoder works with Nodes. If you have a native golang structure, and want to serialize it using an Encoder, you'll need to figure out how to transform that golang structure into an ipld.Node tree first.

It may be useful to understand "multicodecs" when working with Encoders. In IPLD, a system called "multicodecs" is typically used to describe encoding foramts. A "multicodec indicator" is a number which describes an encoding; the Link implementations used in IPLD (CIDs) store a multicodec indicator in the Link; and in this library, a multicodec registry exists in the `codec` package, and can be used to associate a multicodec indicator number with an Encoder function. The default EncoderChooser in a LinkSystem will use this multicodec registry to select Encoder functions. However, you can construct a LinkSystem that uses any EncoderChooser you want. It is also possible to have and use Encoder functions that aren't registered as a multicodec at all... we just recommend being cautious of this, because it may make your data less recognizable when working with other systems that use multicodec indicators as part of their communication.

type ErrBudgetExhausted

type ErrBudgetExhausted struct{}

func (ErrBudgetExhausted) Error

func (e ErrBudgetExhausted) Error() string

type MapSortMode

type MapSortMode uint8

const (
	MapSortMode_None MapSortMode = iota
	MapSortMode_Lexical
	MapSortMode_RFC7049
)