automergendjsonsync

package module
v1.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 5, 2024 License: MIT Imports: 13 Imported by: 0

README

automerge-ndjson-sync (an automerge-go sync library)

GitHub License GitHub go.mod Go version GitHub Actions Workflow Status

This library is a utility library for synchronising automerge documents over HTTP using a application/x-ndjson protocol:

  1. Both the request and response bodies contain newline-delimited json lines. The content-type is application/x-ndjson; charset=utf-8.
  2. Each line looks like {"event":"sync", "data":"<base64-encoded sync message>"}\n
  3. The server stays connected, continuously receiving messages and sending messages as they are ready on the document via either HTTP2 or well-behaved HTTP1.1 clients.
  4. The client decides when to terminate the connection by observing the messages it receives, either:
    1. The response body is closed after the server detects that the request body is complete and no more messages are available.
    2. The client sees a sync message that meets its "termination check", which may indicate that the server matches the local state or that the local state contains all the remote head nodes. This can be used for local tools that need to perform a "one-shot" synchronisation on startup.
  5. There's a broadcast capability that allows a server to serve changes from multiple clients on the same doc simultaneously or for a client to synchronise with multiple servers.
  6. The client supports HTTP redirect behavior so that servers can implement rudimentary partitioning and balancing of requests.

This library will be used to build a series of small peer-to-peer and distributed state utilities built on Automerge. The protocol above is easy to replicate in most languages, most importantly Go (in this repo) and Javascript.

FAQ: Why not use the Automerge sync-server Websocket protocols?

The Automerge sync server https://github.com/automerge/automerge-repo-sync-server and related https://github.com/automerge/automerge-connection libraries are written in and generally use websockets to communicate. I wanted to try and utilise HTTP2 and concurrent HTTP1.1 to achieve a similar thing with Go.

This gives me a pure-go option with very few dependencies that I can trust to be stable and maintainable for a long time.

FAQ: Can you give me an example over the wire?

By executing go run ./examples/server/ in one terminal, and DOC_ID=example go run ./examples/http2writer in another terminal, I can then execute a raw curl request in a 3rd terminal to follow the stream over HTTPS. I send an empty sync message to start and observe the following:

$ curl -k -v -X PUT https://localhost:8080/example -d '{"event":"sync","data":"QgAAAQAAAA=="}' -H 'Content-Type: application/x-ndjson'
...
> PUT /example HTTP/2
> Host: localhost:8080
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/x-ndjson
> Content-Length: 38

< HTTP/2 200 
< content-type: application/x-ndjson; charset=utf-8
< date: Sat, 26 Oct 2024 09:56:23 GMT
< 
{"event":"sync","data":"QgFbkqa2LT<snip>9CqZQD6wA="}
{"event":"sync","data":"QgEVJTDZHR<snip>F/AH+BAQ=="}
{"event":"sync","data":"QgG3qWH1RH<snip>8BfwB/ggE="}
...

Until I decide to hang up the connection with Ctrl-C. And this works perfectly fine with HTTP1.1 too:

$ curl -k -i -X PUT https://localhost:8080/example -d '{"event":"sync","data":"QgAAAQAAAA=="}' -H 'Content-Type: application/x-ndjson' --http1.1
< HTTP/1.1 200 OK
< Content-Type: application/x-ndjson; charset=utf-8
< Date: Sat, 26 Oct 2024 11:32:52 GMT
< Transfer-Encoding: chunked

{"event":"sync","data":"QgFbkqa2LT<snip>9CqZQD6wA="}
{"event":"sync","data":"QgEVJTDZHR<snip>F/AH+BAQ=="}
{"event":"sync","data":"QgG3qWH1RH<snip>8BfwB/ggE="}
...

FAQ: Why do the examples use HTTPS, do I need to use HTTPS?

The examples include an HTTP2 client, so the server has a self-signed certificate so that it can present HTTP2 over HTTPS. The client can still use HTTP1.1 as seen in the curl example above or the http1follower example.

Dependencies

This is purposefully built with only the Go standard library + github.com/automerge/automerge-go. This is to reduce maintenance burden for me.

Testing

Unit tests, including Server and Client syncing, are executed through either make test or the Github Actions CI.

Documentation

Index

Constants

View Source 
const ContentType = "application/x-ndjson"
View Source 
const ContentTypeWithCharset = ContentType + "; charset=utf-8"
View Source 
const EventSync = "sync"

Variables

This section is empty.

Functions

func CompareHeads 

func CompareHeads(a []automerge.ChangeHash, b []automerge.ChangeHash) (missingInA int, missingInB int)

CompareHeads is a utility function that compares change hash arrays and returns values indicating the intersection or overlap.

func HasAllRemoteHeads 

func HasAllRemoteHeads(doc *automerge.Doc, m *automerge.SyncMessage) bool

HasAllRemoteHeads will continue accepting messages until it confirms that the local doc contains all the remote heads. But the opposite may not be true.

func HeadsEqualCheck 

func HeadsEqualCheck(doc *automerge.Doc, m *automerge.SyncMessage) bool

HeadsEqualCheck will continue accepting messages until both the local doc and remote doc have the same heads.

func Logger added in v1.3.0 

func Logger(ctx context.Context) *slog.Logger

func NoReadPredicate added in v1.2.0 

func NoReadPredicate(doc *automerge.Doc, msg *automerge.SyncMessage) (bool, error)

NoReadPredicate is a ReadPredicate which includes all messages

func NoTerminationCheck 

func NoTerminationCheck(doc *automerge.Doc, m *automerge.SyncMessage) bool

NoTerminationCheck will continue reading all messages and not stop.

func SetContextLogger added in v1.3.0 

func SetContextLogger(ctx context.Context, l *slog.Logger) context.Context

func SkipChangesReadPredicate added in v1.2.0 

func SkipChangesReadPredicate(doc *automerge.Doc, msg *automerge.SyncMessage) (bool, error)

SkipChangesReadPredicate is a ReadPredicate which skips any messages that contain changes. Effectively turning the doc read only since it does not accept incoming changes but is happy to doll them out.

Types

type ClientOption 

type ClientOption func(*clientOptions)

func WithClientRequestEditor 

func WithClientRequestEditor(f func(r *http.Request)) ClientOption

func WithClientSyncState 

func WithClientSyncState(state *automerge.SyncState) ClientOption

func WithClientTerminationCheck 

func WithClientTerminationCheck(check TerminationCheck) ClientOption

func WithHttpClient 

func WithHttpClient(client HttpDoer) ClientOption

type HttpDoer 

type HttpDoer interface {
	Do(req *http.Request) (*http.Response, error)
}

type HttpDoerFunc 

type HttpDoerFunc func(*http.Request) (*http.Response, error)

func (HttpDoerFunc) Do 

func (f HttpDoerFunc) Do(req *http.Request) (*http.Response, error)

type LoggableChangeHashes 

type LoggableChangeHashes []automerge.ChangeHash

LoggableChangeHashes is a type alias that allows the change hash array that represents the document heads to be easily attached to log messages via the slog.LogValuer interface.

func (LoggableChangeHashes) LogValue 

func (l LoggableChangeHashes) LogValue() slog.Value

type NdJson 

type NdJson struct {
	Event string `json:"event"`
	Data  []byte `json:"data,omitempty"`
}

type ReadPredicate added in v1.2.0 

type ReadPredicate func(doc *automerge.Doc, msg *automerge.SyncMessage) (bool, error)

A ReadPredicate is used to filter messages before receiving them in the doc. The function should return a bool (true=include, false=exclude/skip). An error will cause the sync to abort.

type ServerOption 

type ServerOption func(*serverOptions)

func WithReadPredicate added in v1.2.0 

func WithReadPredicate(f ReadPredicate) ServerOption

func WithServerHeaderEditor 

func WithServerHeaderEditor(f func(headers http.Header)) ServerOption

func WithServerSyncState 

func WithServerSyncState(state *automerge.SyncState) ServerOption

func WithTerminationCheck added in v1.2.0 

func WithTerminationCheck(f TerminationCheck) ServerOption

type SharedDoc 

type SharedDoc struct {
	// contains filtered or unexported fields
}

SharedDoc encapsulates a doc with a signalling mechanism that broadcasts an event when new messages changes have been synced into the doc. This event is generally used to wake up other goroutines for generating sync messages to other clients or servers but can also be used to driver other mechanisms like backups or transformers.

func NewSharedDoc 

func NewSharedDoc(doc *automerge.Doc) *SharedDoc

NewSharedDoc returns a new SharedDoc

func (*SharedDoc) Doc 

func (b *SharedDoc) Doc() *automerge.Doc

Doc returns the document held by this SharedDoc.

func (*SharedDoc) HttpPushPullChanges 

func (b *SharedDoc) HttpPushPullChanges(ctx context.Context, url string, opts ...ClientOption) error

HttpPushPullChanges is the HTTP client function to synchronise a local document with a remote server. This uses either HTTP2 or HTTP1.1 depending on the remote server - HTTP2 is preferred since it has better understood bidirectional body capabilities.

func (*SharedDoc) NotifyReceivedChanges 

func (b *SharedDoc) NotifyReceivedChanges()

NotifyReceivedChanges should be called after the shared doc has "received" a message. This allows any goroutines that are generating messages to be preempted and know that new messaged may be available. This is a broadcast because any number of goroutines may be writing changes to the doc to their client.

func (*SharedDoc) ServeChanges 

func (b *SharedDoc) ServeChanges(rw http.ResponseWriter, req *http.Request, opts ...ServerOption) (finalErr error)

func (*SharedDoc) SubscribeToReceivedChanges added in v1.2.0 

func (b *SharedDoc) SubscribeToReceivedChanges() (chan bool, func())

SubscribeToReceivedChanges allows the caller to subscribe to changes received by the doc. Call the finish function to clean up.

type TerminationCheck 

type TerminationCheck func(doc *automerge.Doc, m *automerge.SyncMessage) bool

TerminationCheck can be used on a message reader to stop reading messages when the local document and remote document are suitably in-sync. NoTerminationCheck will never stop reading.

Source Files

View all Source files

Directories

Path Synopsis
examples
http1follower
http2writer
server

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.