deje

README

go-deje

Golang library for DEJE Next, a protocol/technology for decentralized document hosting and concurrent editing.

Please note that this is not the same protocol as the original DEJE, which is vulnerable to a few security issues, and has never been completely implemented, essentially due to overengineering, and a conflict of concurrency models in Python.

go-deje requires you to be running a Bitcoin daemon, or at least point to one on another machine. Otherwise, it is self-contained, because of Golang's static linking (does require some dependencies to build).

DEJE Next

• Uses WAMP as a communications bus - actually more efficient, because of all the many-recipient messages involved in the consensus algorithm.
• Uses Bitcoin blockchain as a distributed timestamping service - with a bit of work, could be swapped out for any comparable service.
• Simpler protocol and serialization format. No subscriptions, no snapshots.
• Two layers of protection: document acceptor consensus, and blockchain-based checkpoint ordering.
• Can bootstrap from multiple download URLs to ensure majority agreement.

This protocol's flagship use will be a decentralized DNS database, but it has many other potential uses, including turn-based network gaming, and a successor technology to Google/Apache Wave called Orchard.

Data model

A document is made of a DAG of events, each of which is an action signed by its author. It also has timestamps, which use an external timestamping service, which bind events to a specific order.

The state structure, which is constructed from the application of a series of events upon an initial starting state, represents the contents of the document at the given point in time/history. This state is a JSON-compatible object, which includes metacontent such as the event handlers and permissions information.

Event

There will be certain built-in event types, like setting/copying/deleting values in the document. Basic stuff. These will have UPPERCASE names, like "SET". There will also, at some point, be a facility for custom event handlers written in Lua.

The primary reasoning for document-custom functions are permissions. Allowing everyone full write access is like letting other people log into your computer as root. Yuck. Custom event handlers allow you to make specific actions, like "edit my own comment", which allow people to interact with the document, without having access to the all-powerful building blocks of those actions. It also provides a mechanism for contextual validation- in a chess game, for example, whether a move with certain arguments is valid depends entirely on the state of the board.

The secondary reasoning is it allows for efficient expression of actions that are specific to the document. A good example - which is more network efficient, replacing a gigantic blob of text with an almost-identical one, or expressing that change in the form of a regex or diff?

Finally, it allows conceptually atomic (indivisible) changes to be atomic in the implementation. If you are expressing one conceptual change in the form of a bunch of low-level events, and someone builds off your halfway-broadcast event chain, and their chain becomes the official one... well, you just orphaned half of something that was intended to be transactional. That's one of the worst kinds of surprises, short of sugar-free gummy bears.

Timestamp

A single timestamp in the external timestamping service. Imposes a mostly-reasonable, somewhat-arbitrary order on when events happened, and this order allows us to pick a single official chain of events.

Timestamps are ordered first by their blockheight (timestamps in earlier blocks always happen before timestamps in later blocks), then by vote count, and then by a string sort of their hashes (as a tiebreaker for multiple timestamps in a single block).

This allows for odd orders of confirmation sometimes - a child event may be confirmed earlier than its parent, for example - but this is harmless, those events will only be applied once.

What is the correct latest event?

We get all the timestamps for the document. Then we apply the following algorithm, iterating through timestamps in serialization order:

• Get event data from peers, including ancestors.
• Is the event a fast-forward of the current tip? If not, drop it.

Then just traverse the chain of the tip's history, ignore any events that are invalid, and there you go.

Documentation

Overview

This package implements the DEJE Next protocol. For more information, read the docs below, or README.md.

This software is currently in a round of refactoring, and as such, the API is *very* subject to change. You can expect it to become simpler in the next few weeks, such that it's fairly easy to create a new document or sync an existing one from the network, and pass all changes on to a UI.

Index

• func GetRouterAndTopic(deje_url string) (router, topic string, err error)
• type Client
  • func NewClient(topic string) Client
  • func (c *Client) Connect(url string) error
  • func (c *Client) Publish(event interface{}) error
  • func (c *Client) SetConnectCallback(callback OnConnectCallback)
  • func (c *Client) SetEventCallback(callback OnEventCallback)
• type OnConnectCallback
• type OnEventCallback
• type OnReTipCallback
• type SimpleClient
  • func NewSimpleClient(topic string, logger *log.Logger) *SimpleClient
  • func Open(deje_url string, logger *log.Logger, cb state.OnPrimitiveCallback) (*SimpleClient, error)
  • func (sc *SimpleClient) Connect(url string) error
  • func (sc *SimpleClient) Export() interface{}
  • func (sc *SimpleClient) GetDoc() *document.Document
  • func (sc *SimpleClient) GetTopic() string
  • func (sc *SimpleClient) Log(args ...interface{})
  • func (sc *SimpleClient) Promote(ev document.Event) error
  • func (sc *SimpleClient) Publish(data interface{}) error
  • func (sc *SimpleClient) PublishEvents() error
  • func (sc *SimpleClient) PublishTimestamps() error
  • func (sc *SimpleClient) ReTip()
  • func (sc *SimpleClient) RequestEvents() error
  • func (sc *SimpleClient) RequestTimestamps() error
  • func (sc *SimpleClient) SetPrimitiveCallback(c state.OnPrimitiveCallback)
  • func (sc *SimpleClient) SetRetipCallback(c OnReTipCallback)

Functions

func GetRouterAndTopic

func GetRouterAndTopic(deje_url string) (router, topic string, err error)

Given a deje:// URL, return router URL and DEJE topic.

Types

type Client

type Client struct {
	Doc   *document.Document
	Topic string
	// contains filtered or unexported fields
}

Contains a document and a WAMP connection.

func NewClient

func NewClient(topic string) Client

func (*Client) Connect

func (c *Client) Connect(url string) error

Connect to a WAMP router. This also calls the 'connect' callback on success.

func (*Client) Publish

func (c *Client) Publish(event interface{}) error

Publish an event to all subscribers. Can be any JSON-compatible value.

func (*Client) SetConnectCallback

func (c *Client) SetConnectCallback(callback OnConnectCallback)

Set callback to be executed when a successful connection has been made to a WAMP router.

func (*Client) SetEventCallback

func (c *Client) SetEventCallback(callback OnEventCallback)

Set callback to be executed when a published event is received.

type OnConnectCallback

type OnConnectCallback func(sessionId string)

When a session connects, this callback is provided with the session ID value. sessionId is just a server-chosen string.

type OnEventCallback

type OnEventCallback func(event interface{})

Called when another peer in the same WAMP topic publishes a JSON-compatible event to all subscribers.

type OnReTipCallback

type OnReTipCallback func(*document.Event)

An optional callback to be called whenever we reanalyze which event is tip.

type SimpleClient

type SimpleClient struct {
	Tip *document.Event
	// contains filtered or unexported fields
}

Wraps the low-level capabilities of the basic Client to provide an easier, more useful API to downstream code.

func NewSimpleClient

func NewSimpleClient(topic string, logger *log.Logger) *SimpleClient

Unless you want to manually specify router URL and topic separately, you should probably use Open() instead of NewSimpleClient().

func Open

func Open(deje_url string, logger *log.Logger, cb state.OnPrimitiveCallback) (*SimpleClient, error)

The preferred way to create SimpleClients. Handles the Connect() call, and uses GetRouterAndTopic() to turn a single deje://... URL into a router URL and topic.

func (*SimpleClient) Connect

func (sc *SimpleClient) Connect(url string) error

Connect and immediately request timestamps.

func (*SimpleClient) Export

func (sc *SimpleClient) Export() interface{}

Return the current contents of the document.

func (*SimpleClient) GetDoc

func (sc *SimpleClient) GetDoc() *document.Document

Get the Document object owned by this Client.

func (*SimpleClient) GetTopic

func (sc *SimpleClient) GetTopic() string

Get the Topic of the underlying Client.

func (*SimpleClient) Log

func (sc *SimpleClient) Log(args ...interface{})

func (*SimpleClient) Promote

func (sc *SimpleClient) Promote(ev document.Event) error

Navigate the Document to an Event, and promote it as the tip.

func (*SimpleClient) Publish

func (sc *SimpleClient) Publish(data interface{}) error

func (*SimpleClient) PublishEvents

func (sc *SimpleClient) PublishEvents() error

func (*SimpleClient) PublishTimestamps

func (sc *SimpleClient) PublishTimestamps() error

func (*SimpleClient) ReTip

func (sc *SimpleClient) ReTip()

func (*SimpleClient) RequestEvents

func (sc *SimpleClient) RequestEvents() error

func (*SimpleClient) RequestTimestamps

func (sc *SimpleClient) RequestTimestamps() error

func (*SimpleClient) SetPrimitiveCallback

func (sc *SimpleClient) SetPrimitiveCallback(c state.OnPrimitiveCallback)

Set a callback for when primitives are applied to the document state.

func (*SimpleClient) SetRetipCallback

func (sc *SimpleClient) SetRetipCallback(c OnReTipCallback)

Set a callback for when we reanalyze which event is tip. This may be called multiple times with the same result tip.