slick

package module
v0.0.0-...-8fcfbba Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 7, 2023 License: LGPL-3.0 Imports: 32 Imported by: 1

README

go-slick

Go Reference

Everything here is subject to change, and it's still in active development. If you'd like to contribute please join us on Discord.

Overview

Slick is a library for building offline-first, e2e encrypted applications for mobile or desktop. It does this by letting you share SQLite databases with all your devices and friends. It uses last-write-win to resolve conflicts, and allows changes to the schema without losing data between users.

An example app uses this framework can be found in the Roost repo.

Installation

This library can be used for either building desktop applications or mobile applications. If you're building for both desktop and mobile, it's recommended you build a library for your app that can be compiled with gomobile, but then imported within your desktop app.

Desktop

To get started on desktop, https://wails.io/ is an easy way to create an HTML/Javascript desktop app that can be shipped on MacOS, Windows and Linux. However, any gui library can be used if one better suits your needs.

Within a go project, install Slick by running go get -u github.com/meow-io/slick within your Go project.

Mobile

For mobile development, use gomobile to compile a library for use within XCode or Android Studio.

For a more detailed example, see roost's mobile build scripts to see how to create an xcframework for iOS.

Usage

Create a Slick instance & define your schema

First, creating an instance and define a schema for your data. This is much like a regular SQLite table, with a few extra columns. For instance, a table named messages can be created using

import (
  "github.com/meow-io/go-slick"
  slick_config "github.com/meow-io/go-slick/config"
)

conf := slick_config.NewConfig(
  slick_config.WithRootDir("./app"), // or some other directory
)
s := slick.NewSlick(conf)
Define your schema

First, define a schema for your data. This is much like a regular SQLite table, with a few extra columns. For instance, a table named messages can be created using


conf := slick_config.NewConfig(
  config.WithRootDir("./app"), // or some other directory
)
s := slick.NewSlick(conf)
_ := s.DB.Migrate("app", []*migration.Migration{
  {
    Name: "Create initial tables",
    Func: func(tx *sql.Tx) error {
      if err := r.slick.EAVCreateTables(map[string]*eav.TableDefinition{
        "messages": {
          Columns: map[string]*eav.ColumnDefinition{
            "body": {
              SourceName: "message_body",
              ColumnType: eav.Text,
              Required:   true,
              Nullable:   false,
            },
            Indexes: [][]string{{"_ctime"}, {"group_id"}},
          },
        }); err != nil {
          return err
        }
      },
    },
  })
})

This table could then be queried using

type Message struct {
	ID            []byte  `db:"id"`
	GroupID       []byte  `db:"group_id"`
	CtimeSec      float64 `db:"_ctime"`          // creation time
	MtimeSec      float64 `db:"_mtime"`          // last modification time
	WtimeSec      float64 `db:"_wtime"`          // last time write took place
	IdentityTag   []byte  `db:"_identity_tag"`   // 4-byte identity prefix
	MembershipTag []byte  `db:"_membership_tag"` // 3-byte membership prefix
	Body          string  `db:"body"`
}

var messages []*Message
_ := s.EAVSelect(&messages, "select * from messages")

IdentityTag and MembershipTag identity who initially created the row, and we'll get back to the later.

Create a group

Data is shared within a group, and we're going to create a group before we read and write from this table. We can do that by calling CreateGroup as follows.

id, _ := s.CreateGroup("new group")
g, _ := s.Group(id)
Write some data

Now that we have a group, we can write data to our table for that specific group.

writer := s.EAVWriter(g)
writer.Insert("messages", map[string]interface{}{
  "body":     "Hello there!",
})
_ := writer.Execute()

Now we can query that newly written data

_ := s.EAVSelect(&messages, "select * from messages")
// messages is
// []*Message{ ... Body: "Hello there!", GroupID: g.ID ... }
Add someone to a group

You can do this by creating an invite and sending it to them, along with the password to access it.

invite, _ := s.Invite(id, "password")

On the other end, accept the invite, along with the password.

_ := s.AcceptInvite(invite, "password")

People added to groups get a backfill of all available data in their group.

Add another device

Slick supports multiple devices. To add devices to your device network, use the following

link, _ := s.DeviceGroup.GetDeviceLink()

And on your other device

_ := s.DeviceGroup.LinkDevice(link)

All devices will automatically be joined to every group within your device network.

Achitecture

Slick is an implementation of the Slick protocol written in Go.

Slick

Overview

Slick uses SQLCipher^sqlcipher to persist its state. Transactions are coordinated from this package between various subsystems to ensure operations are atomic. The Slick package also exposes a high-level API for users to consume. It also defines the device group, which itself uses the API presented by slick.

As well, a small API for generating passwords from a salt file on disk is presented here. This employs argon2id to generate high-entropy keys from passwords.

Messaging

The messaging layer defines an API for creating groups and creating and accepting invites. It also provides an API for sending messages to group. As well, it manages the state for sending messages, groups, and backfills.

Data

This layer passes data down to EAV layer and passes errors back up to slick.

EAV

The EAV layer takes the existing EAV writes and offers a sql-like interface on top of that data. It does this by pivoting data into "horizontal" views. This is accomplished by using SQLite partial indexes to create views of the data.

While SQL reads are supported, writing is performed through the EAV write mechanism. As such, transactions or using queries within writes is not permitted.

Pivot

Transport

This layer is responsible for actually receiving incoming message and sending outgoing messages. For incoming it associates them with a URL, however, it does not guarantee the sender is authentic for the heya transport. It does not have any sort of queue for outgoing messages, that responsibility is part of the messaging layer.

Local

This transport supports the "id" scheme detailed in the specification under the heading "ID".

Local endpoints discover each other using bonjour.

Heya

This transport supports the "heya" scheme detailed in specification under the heading "Heya".

Other packages

Database

All subsystems rely on a shared sqlite instance, specifically SQLCipher^sqlcipher, provided by internal/db. A locking transaction function is provided on top of this database as well.

Bencode

The bencode serialization/deserialization is provided by bencode. Future work could examine other libraries to see if there is a suitable replacement for this homegrown version, however, the critical feature missing is support for byte keys.

Clock

This defines a thin wrapper around the system clock which can then be manipulated during tests.

Config

This defines a common config struct which is used by all subsystems within

Security considerations

In the event an attacker gains access to the salt file and database, the database could be subject to an offline dictionary attack. If a password is stored on the device, such as in a secure enclave, its security properties would be the same as the mechanism used to store the password.

If an attacker gained access to a running process they could extract key material from memory.

References

Documentation

Overview

This package provides a high-level interface to the Slick implementation. It provides functions for defining schemas, writing, and querying data. As well it provides functions for joining, leaving and creating groups as well as the device group.

Index

Constants

View Source 
const (
	// Constants for application state.
	StateNew = iota
	StateInitialized
	StateRunning
	StateClosing
	StateClosed
	// Meta-intro states useful to developers. INTRO_SUCCEEDED indicates a completely backfilled group. IntroFailed indicates something went
	// wrong during the invite process.
	IntroFailed    = 1000
	IntroSucceeded = 1001
)

Variables

This section is empty.

Functions

func DeserializeInvite 

func DeserializeInvite(s string) (*messaging.Jpake1, error)

func NewPin 

func NewPin() (string, error)

Make a random 6-digit pincode for use in invites

func SerializeDeviceInvite 

func SerializeDeviceInvite(i *DeviceGroupInvite) (string, error)

func SerializeInvite 

func SerializeInvite(i *messaging.Jpake1) (string, error)

Types

type AppState 

type AppState struct {
	State int
}

An event indicating a change in the state of Slick.

type Device 

type Device struct {
	Name string `db:"name"`
	Type string `db:"type"`
	// contains filtered or unexported fields
}

type DeviceGroupInvite 

type DeviceGroupInvite struct {
	Invite   *messaging.Jpake1 `bencode:"i"`
	Password string            `bencode:"s"`
}

func DeserializeDeviceInvite 

func DeserializeDeviceInvite(s string) (*DeviceGroupInvite, error)

type EAVWriter 

type EAVWriter struct {
	InsertIDs []ids.ID
	// contains filtered or unexported fields
}

func (*EAVWriter) Execute 

func (w *EAVWriter) Execute() error

func (*EAVWriter) Insert 

func (w *EAVWriter) Insert(tablename string, values map[string]interface{})

func (*EAVWriter) Update 

func (w *EAVWriter) Update(tablename string, id []byte, values map[string]interface{})

type Group 

type Group struct {
	ID            ids.ID
	AuthorTag     [7]byte
	IdentityTag   [4]byte
	MembershipTag [3]byte
	State         int
	Name          string
	// contains filtered or unexported fields
}

A group.

type GroupUpdate 

type GroupUpdate struct {
	ID                   ids.ID
	AckedMemberCount     uint
	GroupState           int
	MemberCount          uint
	ConnectedMemberCount uint
	Seq                  uint64
	PendingMessageCount  uint
}

An event indicating a change in a group.

type IntroUpdate 

type IntroUpdate struct {
	GroupID   ids.ID
	Initiator bool
	Stage     uint32
	Type      uint32
}

An event indicating a change in an intro.

type MessagesFetchedUpdate 

type MessagesFetchedUpdate struct {
}

type Slick 

type Slick struct {
	DeviceGroup *deviceGroup
	DB          *db.Database
	// contains filtered or unexported fields
}

func NewSlick 

func NewSlick(c *config.Config, applicationInit func(s *Slick) error) (*Slick, error)

Create a slick instance

func (*Slick) AcceptInvite 

func (s *Slick) AcceptInvite(invite *messaging.Jpake1, password string) (ids.ID, error)

Accept a password-protected invite.

func (*Slick) AddPushNotificationToken 

func (s *Slick) AddPushNotificationToken(token string) error

Add a push notification token.

func (*Slick) CancelInvites 

func (s *Slick) CancelInvites(groupID ids.ID) error

Cancel all invites for a specific group.

func (*Slick) CreateGroup 

func (s *Slick) CreateGroup(name string) (ids.ID, error)

Create a new group.

func (*Slick) DeletePushNotificationToken 

func (s *Slick) DeletePushNotificationToken(token string) error

Remove a push notification token.

func (*Slick) EAVCreateViews 

func (s *Slick) EAVCreateViews(views map[string]*eav.ViewDefinition) error

Creates multiple views in the EAV database.

func (*Slick) EAVGet 

func (s *Slick) EAVGet(dest interface{}, query string, vars ...interface{}) error

Get a single struct conforming to the sqlx-style tags.

func (*Slick) EAVIndexWhere 

func (s *Slick) EAVIndexWhere(viewName, prefix string) (string, error)

func (*Slick) EAVQuery 

func (s *Slick) EAVQuery(query string, vars ...interface{}) (*eav.Result, error)

Query the EAV database with a SQL statement.

func (*Slick) EAVSelect 

func (s *Slick) EAVSelect(dest interface{}, query string, vars ...interface{}) error

Select multiple structs conforming to the sqlx-style tags.

func (*Slick) EAVSelectors 

func (s *Slick) EAVSelectors(viewName, prefix string, columnNames ...string) (string, error)

func (*Slick) EAVSubscribeAfterEntity 

func (s *Slick) EAVSubscribeAfterEntity(cb func(viewName string, groupID, id ids.ID), includeBackfill bool, views ...string) error

Register callback for entities changes after they are committed

func (*Slick) EAVSubscribeAfterView 

func (s *Slick) EAVSubscribeAfterView(cb func(viewName string), includeBackfill bool, views ...string) error

Register callback for view changes after they are committed

func (*Slick) EAVSubscribeBeforeEntity 

func (s *Slick) EAVSubscribeBeforeEntity(cb func(viewName string, groupID, id ids.ID) error, includeBackfill bool, views ...string) error

Register callback for entities changes before they are committed

func (*Slick) EAVSubscribeBeforeView 

func (s *Slick) EAVSubscribeBeforeView(cb func(viewName string) error, includeBackfill bool, views ...string) error

Register callback for view changes before they are committed

func (*Slick) EAVWrite 

func (s *Slick) EAVWrite(id ids.ID, ops *eav.Operations) error

Write to the EAV database.

func (*Slick) EAVWriter 

func (s *Slick) EAVWriter(g *Group) *EAVWriter

Write to the EAV database.

func (*Slick) GetMessages 

func (s *Slick) GetMessages(key []byte) error

Open an existing slick with a given key, get all messages from it and wait till it's done.

func (*Slick) Group 

func (s *Slick) Group(groupID ids.ID) (*Group, error)

Get a specific group.

func (*Slick) GroupState 

func (s *Slick) GroupState(groupID ids.ID) (*GroupUpdate, error)

Get the current group state for a specific group.

func (*Slick) Groups 

func (s *Slick) Groups() ([]*Group, error)

Get all groups.

func (*Slick) Initialize 

func (s *Slick) Initialize(key []byte) error

Initialize slick with a given key.

func (*Slick) Initialized 

func (s *Slick) Initialized() bool

Returns true is slick is in INITIALIZED state.

func (*Slick) Invite 

func (s *Slick) Invite(groupID ids.ID, password string) (*messaging.Jpake1, error)

Create a password-protected invite for a specific group.

func (*Slick) New 

func (s *Slick) New() bool

Returns true is slick is in NEW state.

func (*Slick) NewID 

func (s *Slick) NewID(authorTag [7]byte) (ids.ID, error)

Makes a new id for use in a write to EAV.

func (*Slick) NewKey 

func (s *Slick) NewKey(password string) ([]byte, error)

Makes a key from a password

func (*Slick) Open 

func (s *Slick) Open(key []byte) error

Open an existing slick with a given key.

func (*Slick) RegisterHeyaTransport 

func (s *Slick) RegisterHeyaTransport(authToken, host string, port int) error

Register a heya transport.

func (*Slick) Running 

func (s *Slick) Running() bool

Returns true is slick is in RUNNING state.

func (*Slick) Shutdown 

func (s *Slick) Shutdown() error

Gracefully stop an existing slick instance.

func (*Slick) TransportStates 

func (s *Slick) TransportStates() map[string]string

Get current transport states

func (*Slick) Transports 

func (s *Slick) Transports() *Transports

func (*Slick) Updates 

func (s *Slick) Updates() <-chan interface{}

Gets various updates which must be dealt with. This will either produce *GroupUpdate, *eav.TableRowUpdate, *eav.TableRowInsert or *eav.TableUpdate

type TableUpdate 

type TableUpdate struct {
	ID        ids.ID
	Name      string
	Tablename string
}

An event indicating an update to a table.

type TransportStateUpdate 

type TransportStateUpdate struct {
	URL   string
	State string
}

type Transports 

type Transports struct {
	// contains filtered or unexported fields
}

func (*Transports) Preflight 

func (st *Transports) Preflight(urls []string) []bool

func (*Transports) ReportGroup 

func (st *Transports) ReportGroup(id ids.ID) error

func (*Transports) StatusChanged 

func (st *Transports) StatusChanged(f func(string, bool))

func (*Transports) URLsForGroup 

func (st *Transports) URLsForGroup(id ids.ID) ([]string, error)

Source Files

View all Source files

Directories

Path Synopsis
This package defines (yet another) bencode encoding/decoding library.
 This package defines (yet another) bencode encoding/decoding library.
A thin wrapper over the system clock which can be implemented for use in tests.
 A thin wrapper over the system clock which can be implemented for use in tests.
This package defines a common config struct which can be used by any subsystem within slick.
 This package defines a common config struct which can be used by any subsystem within slick.
Handles routing application messages to the correct database handler.
 Handles routing application messages to the correct database handler.
eav
This package defines a common id type which is used through out slick.
 This package defines a common id type which is used through out slick.
internal
db
This package defines a SQLCipher database.
 This package defines a SQLCipher database.
test
Package implements reliable messaging layer for parties identified by a URL.
 Package implements reliable messaging layer for parties identified by a URL.
heya
Defines package responsible for performing the actual sending and receiving of messages to URLs.
 Defines package responsible for performing the actual sending and receiving of messages to URLs.
local

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.