data

package
v0.0.0-...-06bacb4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 8, 2024 License: Apache-2.0, MIT Imports: 14 Imported by: 5

Documentation

Overview

Package data supports schema-less serializaiton and deserialization of atproto data

Some restrictions from the data model include: - string sizes - array and object element counts - the "shape" of $bytes and $blob data objects - $type must contain a non-empty string

Details are specified at https://atproto.com/specs/data-model

This package includes types (CIDLink, Bytes, Blob) which are represent the corresponding atproto data model types. These implement JSON and CBOR marshaling in (with whyrusleeping/cbor-gen) the expected way.

Can parse generic atproto records (or other objects) in JSON or CBOR format in to map[string]interface{}, while validating atproto-specific constraints on data (eg, that cid-link objects have only a single field).

Has a helper for serializing generic data (map[string]interface{}) to CBOR, which handles converting JSON-style object types (like $link and $bytes) as needed. There is no "MarshalJSON" method; simply use the standard library's `encoding/json`.

Index

Constants

View Source
const (
	// maximum size of any CBOR data, in any context, in atproto
	MAX_CBOR_SIZE = 5 * 1024 * 1024
	// maximum serialized size of an individual atproto record, in CBOR format
	MAX_CBOR_RECORD_SIZE = 1 * 1024 * 1024
	// maximum serialized size of an individual atproto record, in JSON format
	MAX_JSON_RECORD_SIZE = 2 * 1024 * 1024
	// maximum serialized size of blocks (raw bytes) in an atproto repo stream event (NOT ENFORCED YET)
	MAX_STREAM_REPO_DIFF_SIZE = 4 * 1024 * 1024
	// maximum size of a WebSocket frame in atproto event streams (NOT ENFORCED YET)
	MAX_STREAM_FRAME_SIZE = MAX_CBOR_SIZE
	// maximum size of any individual string inside an atproto record
	MAX_RECORD_STRING_LEN = MAX_CBOR_RECORD_SIZE
	// maximum size of any individual byte array (bytestring) inside an atproto record
	MAX_RECORD_BYTES_LEN = MAX_CBOR_RECORD_SIZE
	// limit on size of CID representation (NOT ENFORCED YET)
	MAX_CID_BYTES = 100
	// limit on depth of nested containers (objects or arrays) for atproto data (NOT ENFORCED YET)
	MAX_CBOR_NESTED_LEVELS = 32
	// maximum number of elements in an object or array in atproto data
	MAX_CBOR_CONTAINER_LEN = 128 * 1024
	// largest integer which can be represented in a float64. integers in atproto "should" not be larger than this. (NOT ENFORCED)
	MAX_SAFE_INTEGER = 9007199254740991
	// largest negative integer which can be represented in a float64. integers in atproto "should" not go below this. (NOT ENFORCED)
	MIN_SAFE_INTEGER = -9007199254740991
	// maximum length of string (UTF-8 bytes) in an atproto object (map)
	MAX_OBJECT_KEY_LEN = 8192
)

Variables

This section is empty.

Functions

func ExtractTypeCBOR

func ExtractTypeCBOR(b []byte) (string, error)

Parses the top-level $type field from generic atproto CBOR data

func ExtractTypeCBORReader

func ExtractTypeCBORReader(r io.Reader) (string, []byte, error)

Parses top-level $type field from generic atproto CBOR.

Returns that string field, and additional bytes (TODO: the parsed bytes, or remaining bytes?)

func ExtractTypeJSON

func ExtractTypeJSON(b []byte) (string, error)

Parses the top-level $type field from generic atproto JSON data

func MarshalCBOR

func MarshalCBOR(obj map[string]any) ([]byte, error)

Serializes generic atproto data (object) to DAG-CBOR bytes

Does not re-validate that data conforms to atproto data model, but does handle Blob, Bytes, and CIDLink as expected.

func UnmarshalCBOR

func UnmarshalCBOR(b []byte) (map[string]any, error)

Parses generic data (object) in CBOR (specifically, IPLD dag-cbor), validating against the atproto data model at the same time.

func UnmarshalJSON

func UnmarshalJSON(b []byte) (map[string]any, error)

Parses generic data (object) in JSON, validating against the atproto data model at the same time.

The standard library's MarshalJSON can be used to invert this function.

func Validate

func Validate(obj map[string]any) error

Checks that generic data (object) complies with the atproto data model.

Types

type Blob

type Blob struct {
	Ref      CIDLink
	MimeType string
	Size     int64
}

Represents the "blob" type from the atproto data model.

This struct does not get marshaled/unmarshaled directly in to JSON or CBOR; see the BlobSchema and LegacyBlobSchema structs. This is the type that should be included in golang struct definitions.

When representing a "legacy" blob (no size field, string CID), size == -1.

func ExtractBlobs

func ExtractBlobs(obj map[string]any) []Blob

Recursively finds all the "blob" objects from generic atproto data (which has already been parsed).

Returns an array with all Blob instances; does not de-dupe.

func (*Blob) MarshalCBOR

func (b *Blob) MarshalCBOR(w io.Writer) error

func (Blob) MarshalJSON

func (b Blob) MarshalJSON() ([]byte, error)

func (*Blob) UnmarshalCBOR

func (lb *Blob) UnmarshalCBOR(r io.Reader) error

func (*Blob) UnmarshalJSON

func (b *Blob) UnmarshalJSON(raw []byte) error

type BlobSchema

type BlobSchema struct {
	LexiconTypeID string  `json:"$type,const=blob" cborgen:"$type,const=blob"`
	Ref           CIDLink `json:"ref" cborgen:"ref"`
	MimeType      string  `json:"mimeType" cborgen:"mimeType"`
	Size          int64   `json:"size" cborgen:"size"`
}

func (*BlobSchema) MarshalCBOR

func (t *BlobSchema) MarshalCBOR(w io.Writer) error

func (*BlobSchema) UnmarshalCBOR

func (t *BlobSchema) UnmarshalCBOR(r io.Reader) (err error)

type Bytes

type Bytes []byte

Represents the "bytes" type from the atproto data model.

In JSON, marshals to an object with $bytes key and base64-encoded data.

In CBOR, marshals to a byte array.

func (*Bytes) MarshalCBOR

func (lb *Bytes) MarshalCBOR(w io.Writer) error

func (Bytes) MarshalJSON

func (lb Bytes) MarshalJSON() ([]byte, error)

func (*Bytes) UnmarshalCBOR

func (lb *Bytes) UnmarshalCBOR(r io.Reader) error

func (*Bytes) UnmarshalJSON

func (lb *Bytes) UnmarshalJSON(raw []byte) error
type CIDLink cid.Cid

Represents the "cid-link" type from the atproto data model.

Implementation is a simple wrapper around the github.com/ipfs/go-cid "cid.Cid" type.

func (CIDLink) CID

func (ll CIDLink) CID() cid.Cid

Unwraps the inner cid.Cid type (github.com/ipfs/go-cid)

func (CIDLink) IsDefined

func (ll CIDLink) IsDefined() bool

Convenience helper, returns false if CID is "undefined" (golang zero value)

func (*CIDLink) MarshalCBOR

func (ll *CIDLink) MarshalCBOR(w io.Writer) error

func (CIDLink) MarshalJSON

func (ll CIDLink) MarshalJSON() ([]byte, error)

func (CIDLink) String

func (ll CIDLink) String() string

Returns string representation.

If the CID is "undefined", returns an empty string (note that this is different from how cid.Cid works).

func (*CIDLink) UnmarshalCBOR

func (ll *CIDLink) UnmarshalCBOR(r io.Reader) error

func (*CIDLink) UnmarshalJSON

func (ll *CIDLink) UnmarshalJSON(raw []byte) error

type GenericRecord

type GenericRecord struct {
	Type string `json:"$type" cborgen:"$type"`
}

Helper type for extracting record $type from CBOR

func (*GenericRecord) MarshalCBOR

func (t *GenericRecord) MarshalCBOR(w io.Writer) error

func (*GenericRecord) UnmarshalCBOR

func (t *GenericRecord) UnmarshalCBOR(r io.Reader) (err error)

type JsonBytes

type JsonBytes struct {
	Bytes string `json:"$bytes"`
}

type LegacyBlobSchema

type LegacyBlobSchema struct {
	Cid      string `json:"cid" cborgen:"cid"`
	MimeType string `json:"mimeType" cborgen:"mimeType"`
}

func (*LegacyBlobSchema) MarshalCBOR

func (t *LegacyBlobSchema) MarshalCBOR(w io.Writer) error

func (*LegacyBlobSchema) UnmarshalCBOR

func (t *LegacyBlobSchema) UnmarshalCBOR(r io.Reader) (err error)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL