dqlite

package module
v0.9.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 17, 2019 License: Apache-2.0 Imports: 20 Imported by: 0

README

go-dqlite Build Status Coverage Status Go Report Card GoDoc

This repository provides the dqlite Go package, which can be used to replicate a SQLite database across a cluster, using the Raft algorithm.

Design higlights

  • No external processes needed: dqlite is just a Go library, you link it it to your application exactly like you would with SQLite.
  • Replication needs a SQLite patch which is not yet included upstream.
  • The Go Raft package from Hashicorp is used internally for replicating the write-ahead log frames of SQLite across all nodes.

How does it compare to rqlite?

The main differences from rqlite are:

  • Full support for transactions
  • No need for statements to be deterministic (e.g. you can use time())
  • Frame-based replication instead of statement-based replication, this means in dqlite there's more data flowing between nodes, so expect lower performance. Should not really matter for most use cases.

Status

This is beta software for now, but we'll get to rc/release soon.

Demo

To see dqlite in action, make sure you have the following dependencies installed:

  • Go (tested on 1.8)
  • gcc
  • any dependency/header that SQLite needs to build from source
  • Python 3

Then run:

go get -d github.com/CanonicalLtd/dqlite
cd $GOPATH/src/github.com/CanonicalLtd/dqlite
make dependencies
./run-demo

This should spawn three dqlite-based nodes, each of one running the code in the demo Go source.

Each node inserts data in a test table and then dies abruptly after a random timeout. Leftover transactions and failover to other nodes should be handled gracefully.

While the demo is running, to get more details about what's going on behind the scenes you can also open another terminal and run a command like:

watch ls -l /tmp/dqlite-demo-*/ /tmp/dqlite-demo-*/snapshots/

and see how the data directories of the three nodes evolve in terms SQLite databases (test.db), write-ahead log files (test.db-wal), raft logs store (raft.db), and raft snapshots.

Documentation

The documentation for this package can be found on Godoc.

FAQ

Q: How does dqlite behave during conflict situations? Does Raft select a winning WAL write and any others in flight are aborted?

A: There can't be a conflict situation. Raft's model is that only the leader can append new log entries, which translated to dqlite means that only the leader can write new WAL frames. So this means that any attempt to perform a write transaction on a non-leader node will fail with a sqlite3x.ErrNotLeader error (and in this case clients are supposed to retry against whoever is the new leader).

Q: When not enough nodes are available, are writes hung until consensus?

A: Yes, however there's a (configurable) timeout. This is a consequence of Raft sitting in the CP spectrum of the CAP theorem: in case of a network partition it chooses consistency and sacrifices availability.

Documentation

Overview

Package dqlite implements a database/sql/driver with raft-based SQLite replication.

Index

Constants

View Source 
const (
	LogDebug = logging.Debug
	LogInfo  = logging.Info
	LogWarn  = logging.Warn
	LogError = logging.Error
)

Available logging levels.

View Source 
const (
	Unavailable = bindings.Unavailable
	Follower    = bindings.Follower
	Candidate   = bindings.Candidate
	Leader      = bindings.Leader
)

States

Variables

View Source 
var ErrNoAvailableLeader = client.ErrNoAvailableLeader

ErrNoAvailableLeader is returned as root cause of Open() if there's no leader available in the cluster.

View Source 
var ErrServerCantBootstrap = bindings.ErrServerCantBootstrap

ErrServerCantBootstrap is returned by Server.Bootstrap() if the server has already a raft configuration.

View Source 
var NewInmemServerStore = client.NewInmemServerStore

NewInmemServerStore creates ServerStore which stores its data in-memory.

Functions

func Leave added in v0.9.2 

func Leave(ctx context.Context, id uint64, store ServerStore, dial DialFunc) error

Leave a cluster.

Types

type Conn 

type Conn struct {
	// contains filtered or unexported fields
}

Conn implements the sql.Conn interface.

func (*Conn) Begin deprecated 

func (c *Conn) Begin() (driver.Tx, error)

Begin starts and returns a new transaction.

Deprecated: Drivers should implement ConnBeginTx instead (or additionally).

func (*Conn) BeginTx 

func (c *Conn) BeginTx(ctx context.Context, opts driver.TxOptions) (driver.Tx, error)

BeginTx starts and returns a new transaction. If the context is canceled by the user the sql package will call Tx.Rollback before discarding and closing the connection.

This must check opts.Isolation to determine if there is a set isolation level. If the driver does not support a non-default level and one is set or if there is a non-default isolation level that is not supported, an error must be returned.

This must also check opts.ReadOnly to determine if the read-only value is true to either set the read-only transaction property if supported or return an error if it is not supported.

func (*Conn) Close 

func (c *Conn) Close() error

Close invalidates and potentially stops any current prepared statements and transactions, marking this connection as no longer in use.

Because the sql package maintains a free pool of connections and only calls Close when there's a surplus of idle connections, it shouldn't be necessary for drivers to do their own connection caching.

func (*Conn) Exec 

func (c *Conn) Exec(query string, args []driver.Value) (driver.Result, error)

Exec is an optional interface that may be implemented by a Conn.

func (*Conn) ExecContext 

func (c *Conn) ExecContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Result, error)

ExecContext is an optional interface that may be implemented by a Conn.

func (*Conn) Prepare 

func (c *Conn) Prepare(query string) (driver.Stmt, error)

Prepare returns a prepared statement, bound to this connection.

func (*Conn) PrepareContext 

func (c *Conn) PrepareContext(ctx context.Context, query string) (driver.Stmt, error)

PrepareContext returns a prepared statement, bound to this connection. context is for the preparation of the statement, it must not store the context within the statement itself.

func (*Conn) Query 

func (c *Conn) Query(query string, args []driver.Value) (driver.Rows, error)

Query is an optional interface that may be implemented by a Conn.

func (*Conn) QueryContext 

func (c *Conn) QueryContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Rows, error)

QueryContext is an optional interface that may be implemented by a Conn.

type DatabaseServerStore 

type DatabaseServerStore struct {
	// contains filtered or unexported fields
}

DatabaseServerStore persists a list addresses of dqlite servers in a SQL table.

func DefaultServerStore 

func DefaultServerStore(filename string) (*DatabaseServerStore, error)

DefaultServerStore creates a new ServerStore using the given filename to open a SQLite database, with default names for the schema, table and column parameters.

It also creates the table if it doesn't exist yet.

func NewServerStore 

func NewServerStore(db *sql.DB, schema, table, column string) *DatabaseServerStore

NewServerStore creates a new ServerStore.

func (*DatabaseServerStore) Get 

func (d *DatabaseServerStore) Get(ctx context.Context) ([]ServerInfo, error)

Get the current servers.

func (*DatabaseServerStore) Set 

func (d *DatabaseServerStore) Set(ctx context.Context, servers []ServerInfo) error

Set the servers addresses.

type DialFunc 

type DialFunc bindings.DialFunc

DialFunc is a function that can be used to establish a network connection.

type Driver 

type Driver struct {
	// contains filtered or unexported fields
}

Driver perform queries against a dqlite server.

func NewDriver 

func NewDriver(store ServerStore, options ...DriverOption) (*Driver, error)

NewDriver creates a new dqlite driver, which also implements the driver.Driver interface.

func (*Driver) Open 

func (d *Driver) Open(uri string) (driver.Conn, error)

Open establishes a new connection to a SQLite database on the dqlite server.

The given name must be a pure file name without any directory segment, dqlite will connect to a database with that name in its data directory.

Query parameters are always valid except for "mode=memory".

If this node is not the leader, or the leader is unknown an ErrNotLeader error is returned.

func (*Driver) SetContextTimeout added in v0.2.2 

func (d *Driver) SetContextTimeout(timeout time.Duration)

SetContextTimeout sets the default client timeout when no context deadline is provided.

type DriverError 

type DriverError = bindings.Error

DriverError is returned in case of database errors.

type DriverOption 

type DriverOption func(*driverOptions)

DriverOption can be used to tweak driver parameters.

func WithConnectionBackoffCap 

func WithConnectionBackoffCap(cap time.Duration) DriverOption

WithConnectionBackoffCap sets the maximum connection retry backoff value, (regardless of the backoff factor) for retrying failed connection attempts.

If not used, the default is 1 second.

func WithConnectionBackoffFactor 

func WithConnectionBackoffFactor(factor time.Duration) DriverOption

WithConnectionBackoffFactor sets the exponential backoff factor for retrying failed connection attempts.

If not used, the default is 50 milliseconds.

func WithConnectionTimeout 

func WithConnectionTimeout(timeout time.Duration) DriverOption

WithConnectionTimeout sets the connection timeout.

If not used, the default is 5 seconds.

func WithContext 

func WithContext(context context.Context) DriverOption

WithContext sets a global cancellation context.

func WithContextTimeout added in v0.2.2 

func WithContextTimeout(timeout time.Duration) DriverOption

WithContextTimeout sets the default client context timeout when no context deadline is provided.

If not used, the default is 5 seconds.

func WithDialFunc 

func WithDialFunc(dial DialFunc) DriverOption

WithDialFunc sets a custom dial function.

func WithLogFunc 

func WithLogFunc(log LogFunc) DriverOption

WithLogFunc sets a custom logging function.

type InmemServerStore 

type InmemServerStore = client.InmemServerStore

InmemServerStore keeps the list of target gRPC SQL servers in memory.

type LogFunc 

type LogFunc = logging.Func

LogFunc is a function that can be used for logging.

type LogLevel 

type LogLevel = logging.Level

LogLevel defines the logging level.

type Result 

type Result struct {
	// contains filtered or unexported fields
}

Result is the result of a query execution.

func (*Result) LastInsertId 

func (r *Result) LastInsertId() (int64, error)

LastInsertId returns the database's auto-generated ID after, for example, an INSERT into a table with primary key.

func (*Result) RowsAffected 

func (r *Result) RowsAffected() (int64, error)

RowsAffected returns the number of rows affected by the query.

type Rows 

type Rows struct {
	// contains filtered or unexported fields
}

Rows is an iterator over an executed query's results.

func (*Rows) Close 

func (r *Rows) Close() error

Close closes the rows iterator.

func (*Rows) ColumnTypeDatabaseTypeName 

func (r *Rows) ColumnTypeDatabaseTypeName(i int) string

ColumnTypeDatabaseTypeName implements RowsColumnTypeDatabaseTypeName.

func (*Rows) ColumnTypeScanType 

func (r *Rows) ColumnTypeScanType(i int) reflect.Type

ColumnTypeScanType implements RowsColumnTypeScanType.

func (*Rows) Columns 

func (r *Rows) Columns() []string

Columns returns the names of the columns. The number of columns of the result is inferred from the length of the slice. If a particular column name isn't known, an empty string should be returned for that entry.

func (*Rows) Next 

func (r *Rows) Next(dest []driver.Value) error

Next is called to populate the next row of data into the provided slice. The provided slice will be the same size as the Columns() are wide.

Next should return io.EOF when there are no more rows.

type Server 

type Server struct {
	// contains filtered or unexported fields
}

Server implements the dqlite network protocol.

func NewServer 

func NewServer(info ServerInfo, dir string, options ...ServerOption) (*Server, error)

NewServer creates a new Server instance.

func (*Server) Bootstrap added in v0.9.0 

func (s *Server) Bootstrap(servers []ServerInfo) error

Bootstrap the server.

func (*Server) Close 

func (s *Server) Close() error

Close the server, releasing all resources it created.

func (*Server) Cluster added in v0.9.2 

func (s *Server) Cluster() ([]ServerInfo, error)

Cluster returns information about all servers in the cluster.

func (*Server) Dump 

func (s *Server) Dump(name string, dir string) error

Dump the files of a database to disk.

func (*Server) Join added in v0.9.2 

func (s *Server) Join(ctx context.Context, store ServerStore, dial DialFunc) error

Join a cluster.

func (*Server) Leader added in v0.9.2 

func (s *Server) Leader() *ServerInfo

Leader returns information about the current leader, if any.

func (*Server) Start added in v0.9.0 

func (s *Server) Start(listener net.Listener) error

Start serving requests.

type ServerInfo 

type ServerInfo = client.ServerInfo

ServerInfo holds information about a single server.

type ServerOption 

type ServerOption func(*serverOptions)

ServerOption can be used to tweak server parameters.

func WithServerDialFunc added in v0.9.2 

func WithServerDialFunc(dial DialFunc) ServerOption

WithServerDialFunc sets a custom dial function for the server.

func WithServerLogFunc 

func WithServerLogFunc(log LogFunc) ServerOption

WithServerLogFunc sets a custom log function for the server.

func WithServerWatchFunc added in v0.9.3 

func WithServerWatchFunc(watch WatchFunc) ServerOption

WithServerWatchFunc sets a function that will be invoked whenever this server acquires leadership.

type ServerStore 

type ServerStore = client.ServerStore

ServerStore is used by a dqlite client to get an initial list of candidate dqlite server addresses that it can dial in order to find a leader dqlite server to use.

Once connected, the client periodically updates the addresses in the store by querying the leader server about changes in the cluster (such as servers being added or removed).

type Stmt 

type Stmt struct {
	// contains filtered or unexported fields
}

Stmt is a prepared statement. It is bound to a Conn and not used by multiple goroutines concurrently.

func (*Stmt) Close 

func (s *Stmt) Close() error

Close closes the statement.

func (*Stmt) Exec 

func (s *Stmt) Exec(args []driver.Value) (driver.Result, error)

Exec executes a query that doesn't return rows, such

func (*Stmt) ExecContext 

func (s *Stmt) ExecContext(ctx context.Context, args []driver.NamedValue) (driver.Result, error)

ExecContext executes a query that doesn't return rows, such as an INSERT or UPDATE.

ExecContext must honor the context timeout and return when it is canceled.

func (*Stmt) NumInput 

func (s *Stmt) NumInput() int

NumInput returns the number of placeholder parameters.

func (*Stmt) Query 

func (s *Stmt) Query(args []driver.Value) (driver.Rows, error)

Query executes a query that may return rows, such as a

func (*Stmt) QueryContext 

func (s *Stmt) QueryContext(ctx context.Context, args []driver.NamedValue) (driver.Rows, error)

QueryContext executes a query that may return rows, such as a SELECT.

QueryContext must honor the context timeout and return when it is canceled.

type Tx 

type Tx struct {
	// contains filtered or unexported fields
}

Tx is a transaction.

func (*Tx) Commit 

func (tx *Tx) Commit() error

Commit the transaction.

func (*Tx) Rollback 

func (tx *Tx) Rollback() error

Rollback the transaction.

type WatchFunc added in v0.9.3 

type WatchFunc = bindings.WatchFunc

WatchFunc notifies about state changes.

Source Files

View all Source files

Directories

Path Synopsis
internal
bindings
client
logging

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.