data

package
v0.0.0-...-06bacb4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 8, 2024 License: Apache-2.0, MIT Imports: 14 Imported by: 5

Documentation

Overview

Package data supports schema-less serializaiton and deserialization of atproto data

Some restrictions from the data model include: - string sizes - array and object element counts - the "shape" of $bytes and $blob data objects - $type must contain a non-empty string

Details are specified at https://atproto.com/specs/data-model

This package includes types (CIDLink, Bytes, Blob) which are represent the corresponding atproto data model types. These implement JSON and CBOR marshaling in (with whyrusleeping/cbor-gen) the expected way.

Can parse generic atproto records (or other objects) in JSON or CBOR format in to map[string]interface{}, while validating atproto-specific constraints on data (eg, that cid-link objects have only a single field).

Has a helper for serializing generic data (map[string]interface{}) to CBOR, which handles converting JSON-style object types (like $link and $bytes) as needed. There is no "MarshalJSON" method; simply use the standard library's `encoding/json`.

Index

Constants

View Source 
const (
	// maximum size of any CBOR data, in any context, in atproto
	MAX_CBOR_SIZE = 5 * 1024 * 1024
	// maximum serialized size of an individual atproto record, in CBOR format
	MAX_CBOR_RECORD_SIZE = 1 * 1024 * 1024
	// maximum serialized size of an individual atproto record, in JSON format
	MAX_JSON_RECORD_SIZE = 2 * 1024 * 1024
	// maximum serialized size of blocks (raw bytes) in an atproto repo stream event (NOT ENFORCED YET)
	MAX_STREAM_REPO_DIFF_SIZE = 4 * 1024 * 1024
	// maximum size of a WebSocket frame in atproto event streams (NOT ENFORCED YET)
	MAX_STREAM_FRAME_SIZE = MAX_CBOR_SIZE
	// maximum size of any individual string inside an atproto record
	MAX_RECORD_STRING_LEN = MAX_CBOR_RECORD_SIZE
	// maximum size of any individual byte array (bytestring) inside an atproto record
	MAX_RECORD_BYTES_LEN = MAX_CBOR_RECORD_SIZE
	// limit on size of CID representation (NOT ENFORCED YET)
	MAX_CID_BYTES = 100
	// limit on depth of nested containers (objects or arrays) for atproto data (NOT ENFORCED YET)
	MAX_CBOR_NESTED_LEVELS = 32
	// maximum number of elements in an object or array in atproto data
	MAX_CBOR_CONTAINER_LEN = 128 * 1024
	// largest integer which can be represented in a float64. integers in atproto "should" not be larger than this. (NOT ENFORCED)
	MAX_SAFE_INTEGER = 9007199254740991
	// largest negative integer which can be represented in a float64. integers in atproto "should" not go below this. (NOT ENFORCED)
	MIN_SAFE_INTEGER = -9007199254740991
	// maximum length of string (UTF-8 bytes) in an atproto object (map)
	MAX_OBJECT_KEY_LEN = 8192
)

Variables

This section is empty.

Functions

func ExtractTypeCBOR 

func ExtractTypeCBOR(b []byte) (string, error)

Parses the top-level $type field from generic atproto CBOR data

func ExtractTypeCBORReader 

func ExtractTypeCBORReader(r io.Reader) (string, []byte, error)

Parses top-level $type field from generic atproto CBOR.

Returns that string field, and additional bytes (TODO: the parsed bytes, or remaining bytes?)

func ExtractTypeJSON 

func ExtractTypeJSON(b []byte) (string, error)

Parses the top-level $type field from generic atproto JSON data

func MarshalCBOR 

func MarshalCBOR(obj map[string]any) ([]byte, error)

Serializes generic atproto data (object) to DAG-CBOR bytes

Does not re-validate that data conforms to atproto data model, but does handle Blob, Bytes, and CIDLink as expected.

func UnmarshalCBOR 

func UnmarshalCBOR(b []byte) (map[string]any, error)

Parses generic data (object) in CBOR (specifically, IPLD dag-cbor), validating against the atproto data model at the same time.

func UnmarshalJSON 

func UnmarshalJSON(b []byte) (map[string]any, error)

Parses generic data (object) in JSON, validating against the atproto data model at the same time.

The standard library's MarshalJSON can be used to invert this function.

func Validate 

func Validate(obj map[string]any) error

Checks that generic data (object) complies with the atproto data model.

Types

type Blob 

type Blob struct {
	Ref      CIDLink
	MimeType string
	Size     int64
}

Represents the "blob" type from the atproto data model.

This struct does not get marshaled/unmarshaled directly in to JSON or CBOR; see the BlobSchema and LegacyBlobSchema structs. This is the type that should be included in golang struct definitions.

When representing a "legacy" blob (no size field, string CID), size == -1.

func ExtractBlobs 

func ExtractBlobs(obj map[string]any) []Blob

Recursively finds all the "blob" objects from generic atproto data (which has already been parsed).

Returns an array with all Blob instances; does not de-dupe.

func (*Blob) MarshalCBOR 

func (b *Blob) MarshalCBOR(w io.Writer) error

func (Blob) MarshalJSON 

func (b Blob) MarshalJSON() ([]byte, error)

func (*Blob) UnmarshalCBOR 

func (lb *Blob) UnmarshalCBOR(r io.Reader) error

func (*Blob) UnmarshalJSON 

func (b *Blob) UnmarshalJSON(raw []byte) error

type BlobSchema 

type BlobSchema struct {
	LexiconTypeID string  `json:"$type,const=blob" cborgen:"$type,const=blob"`
	Ref           CIDLink `json:"ref" cborgen:"ref"`
	MimeType      string  `json:"mimeType" cborgen:"mimeType"`
	Size          int64   `json:"size" cborgen:"size"`
}

func (*BlobSchema) MarshalCBOR 

func (t *BlobSchema) MarshalCBOR(w io.Writer) error

func (*BlobSchema) UnmarshalCBOR 

func (t *BlobSchema) UnmarshalCBOR(r io.Reader) (err error)

type Bytes 

type Bytes []byte

Represents the "bytes" type from the atproto data model.

In JSON, marshals to an object with $bytes key and base64-encoded data.

In CBOR, marshals to a byte array.

func (*Bytes) MarshalCBOR 

func (lb *Bytes) MarshalCBOR(w io.Writer) error

func (Bytes) MarshalJSON 

func (lb Bytes) MarshalJSON() ([]byte, error)

func (*Bytes) UnmarshalCBOR 

func (lb *Bytes) UnmarshalCBOR(r io.Reader) error

func (*Bytes) UnmarshalJSON 

func (lb *Bytes) UnmarshalJSON(raw []byte) error
type CIDLink cid.Cid

Represents the "cid-link" type from the atproto data model.

Implementation is a simple wrapper around the github.com/ipfs/go-cid "cid.Cid" type.

func (CIDLink) CID 

func (ll CIDLink) CID() cid.Cid

Unwraps the inner cid.Cid type (github.com/ipfs/go-cid)

func (CIDLink) IsDefined 

func (ll CIDLink) IsDefined() bool

Convenience helper, returns false if CID is "undefined" (golang zero value)

func (*CIDLink) MarshalCBOR 

func (ll *CIDLink) MarshalCBOR(w io.Writer) error

func (CIDLink) MarshalJSON 

func (ll CIDLink) MarshalJSON() ([]byte, error)

func (CIDLink) String 

func (ll CIDLink) String() string

Returns string representation.

If the CID is "undefined", returns an empty string (note that this is different from how cid.Cid works).

func (*CIDLink) UnmarshalCBOR 

func (ll *CIDLink) UnmarshalCBOR(r io.Reader) error

func (*CIDLink) UnmarshalJSON 

func (ll *CIDLink) UnmarshalJSON(raw []byte) error

type GenericRecord 

type GenericRecord struct {
	Type string `json:"$type" cborgen:"$type"`
}

Helper type for extracting record $type from CBOR

func (*GenericRecord) MarshalCBOR 

func (t *GenericRecord) MarshalCBOR(w io.Writer) error

func (*GenericRecord) UnmarshalCBOR 

func (t *GenericRecord) UnmarshalCBOR(r io.Reader) (err error)

type JsonBytes 

type JsonBytes struct {
	Bytes string `json:"$bytes"`
}

type LegacyBlobSchema 

type LegacyBlobSchema struct {
	Cid      string `json:"cid" cborgen:"cid"`
	MimeType string `json:"mimeType" cborgen:"mimeType"`
}

func (*LegacyBlobSchema) MarshalCBOR 

func (t *LegacyBlobSchema) MarshalCBOR(w io.Writer) error

func (*LegacyBlobSchema) UnmarshalCBOR 

func (t *LegacyBlobSchema) UnmarshalCBOR(r io.Reader) (err error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.