client

package
v0.0.0-...-3e87057 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 3, 2018 License: Apache-2.0 Imports: 16 Imported by: 4

README

0-stor client godoc

Specifications

API documentation : https://godoc.org/github.com/zero-os/0-stor/client

Motivation

  • Building a secure & fast object store client library with support for big files

Basic idea

  • Splitting a large file into smaller chunks.
  • Compress each chunk using snappy
  • Encrypt each chunk using Hash(content)
    • Hash functions supported: blake2
    • Support for symmetric encryption
  • Save each part in different 0-db server
  • Replicate same chunk into different 0-db servers
  • Save metadata about chunks and where they're in etcd server or other metadata db
  • Assemble smaller chunks for a file upon retrieval
    • Get chunks location from metadata server
    • Assemble file from chunks

Important

  • 0-db server is JUST a simple key/value store
  • splitting, compression, encryption, and replication is the responsibility of client
  • you need at least etcd v3 if you want to use etcd as metadata DB storage

Features

Reference list

The client.Write method takes a third parameter other then the key and value, namely the reference list (refList).
This reference list is also returned as the second value from the client.Read method.

As the 0-db server doesn't do anything with this list, it can be omitted and ignored if the client has no desire of using it.
The reference list for example, can be used to allow the client to do deduplication.

TODO: show example (https://github.com/zero-os/0-stor/issues/216)

Metadata

Metadata format

  • Format for the metadata: 
    type Chunk struct {
	Size   uint64    # Size of the chunk in bytes
	Key    []byte    # key used in the 0-stor
	Shards []string
}
type Meta struct {
	Epoch     int64  # creation epoch
	Key       []byte # key used in the 0-stor
	Chunks    []*Chunk # list of chunks of the files
	Previous  []byte   # Key to the previous metadata entry
	Next      []byte   # Key to the next metadata entry
	ConfigPtr []byte   # Key to the configuration used by the lib to set the data.
}

chunks

Depending on the policy used to create the client, the data could be split into multiple chunks. Which means that the metadata can be composed of minimum one up to n chunks.

Each chunks can then have one or multiple shards.

  • If you use replication, each shards is the location of one of the replicate.
  • If you use distribution, each shards is the location of one of the data or parity block.

metadata linked list

With Previous and Next fields in the metadata, we can build metadata linked list of sequential data such as transaction log/history and block chain. The linked list can then be used to walk over the data in forward or backward fashion.

Metadata linked list will be build if user specify previous meta key when calling WriteWithMeta or WriteFWithMeta methods.

Getting started

Now into some technical details!

Pre processing of data

walk over the metadata

Thanks to our metadata linked list, we can walk over the linked list in forward and backward mode using Walk or WalkBack API.

Those APIs returns channel of WalkResult which then can be iterated. The WalkResult consist of these fields:

  • key in metadata
  • the metadata
  • the data stored on 0-stor server
  • error if exist

It can be used for example to reconstruct the stored sequential data.

Using 0-stor client examples:

Hello World

File: /examples/hello_world/main.go

In this example, when we store the data, the data will be processed as follow: plain data -> compress -> encrypt -> distribution/erasure encoding (which send to 0-stor server and write metadata)

When we get the data from 0-db server, the reverse process will happen: distribution/erasure decoding (which reads metadata & Get data from 0-stor) -> decrypt -> decompress -> plain data.

To run this example, you need to run:

  • 0-db server at port 12345
  • 0-db server at port 12346
  • 0-db server at port 12347
  • etcd server at port 2379

Than you can run the example as follows:

go run examples/hello_world/main.go

Please check out the source code to see how this example works.

Hello World: Config File Edition

File: /examples/hello_config/main.go

In this file we are doing exactly the same, and you'll need to run the same servers as were required before.

However this time we create the client, using a file-based config.

You can run the example as follows:

go run examples/hello_config/main.go

Please check out the source code to see how this example works.

Configuration

Configuration file example can be found on config.yaml.

Libraries

This client includes some components that can be used independently. See components directory for more details.

CLI

A cli can be found in the cli directory.

This command-line client includes a command to spawn it as a daemon.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNilKey is an error returned in case a nil key is given to a client method.
	ErrNilKey = errors.New("Client: nil/empty key given")
	// ErrNilContext is an error returned in case a context given to a client method is nil.
	ErrNilContext = errors.New("Client: nil context given")

	// ErrRepairSupport is returned when data is not stored using replication or distribution
	ErrRepairSupport = errors.New("data is not stored using replication or distribution, repair impossible")

	// ErrInvalidReadRange is returned when given read range is not valid
	ErrInvalidReadRange = errors.New("invalid read range")
)
View Source 
var (
	// ErrInvalidTraverseIterator is an error returned when (meta)data
	// of an iterator is requested, while that iterator is in an invalid state.
	ErrInvalidTraverseIterator = errors.New(
		"TraverseIterator is invalid: did you call (TraverseIterator).Next?")
	// ErrInvalidEpochRange is an error returned when,
	// during the creation of a traverse iterator,
	// the given epoch range is invalid (e.g. start > end).
	ErrInvalidEpochRange = errors.New(
		"cannot create traverse iterator: epoch range is invalid")
)

Functions

func EpochNow 

func EpochNow() int64

EpochNow returns the current time, expressed in nano seconds, within the UTC timezone, in the epoch (unix) format.

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client defines 0-stor client

func NewClient 

func NewClient(metaClient *metastor.Client, dataPipeline pipeline.Pipeline) *Client

NewClient creates a 0-stor client, with the data (zstordb) cluster already created, used to read/write object data, as well as the metastor client, which is used to read/write the metadata of the objects. if metaClient is not nil, the metadata will be written at Write operation using the given metaClient.

func NewClientFromConfig 

func NewClientFromConfig(cfg Config, metastorClient *metastor.Client, jobCount int) (*Client, error)

NewClientFromConfig creates new 0-stor client using the given config.

If JobCount is 0 or negative, the default JobCount will be used, as defined by the pipeline package.

func (*Client) Check 

func (c *Client) Check(meta metatypes.Metadata, fast bool) (storage.CheckStatus, error)

Check gets the status of data stored in a 0-stor cluster. It does so using the chunks stored as metadata CheckStatusInvalid indicates the data is invalid and non-repairable, Any other value indicates the data is readable, but if it's not optimal, it could use a repair.

func (*Client) Close 

func (c *Client) Close() error

Close the client and all its used (internal/indirect) resources.

func (*Client) Delete 

func (c *Client) Delete(meta metatypes.Metadata) error

Delete deletes the data, from the 0-stor cluster, using the reference information fetched from the given metadata (which is linked to the given key).

func (*Client) Read 

func (c *Client) Read(meta metatypes.Metadata, w io.Writer) error

Read reads the data, from the 0-stor cluster, using the reference information fetched from the given metadata.

func (*Client) ReadRange 

func (c *Client) ReadRange(meta metatypes.Metadata, w io.Writer, offset, length int64) error

ReadRange reads data with the given offset & length.

func (*Client) Repair 

func (c *Client) Repair(md metatypes.Metadata) (*metatypes.Metadata, error)

Repair repairs broken data, whether it's needed or not.

If the data is distributed and the amount of corrupted chunks is acceptable, we recreate the missing chunks.

Id the data is replicated and we still have one valid replication, we create the missing replications until we reach the replication number configured in the config.

if the data has not been distributed or replicated, we can't repair it, or if not enough shards are available we cannot repair it either.

func (*Client) Traverse 

func (c *Client) Traverse(startKey []byte, fromEpoch, toEpoch int64) (TraverseIterator, error)

Traverse traverses the stored (meta)data, which is chained together using the (*Client).WriteLinked method. It starts searching from a given startKey and will iterate through all (meta)data, which has a registered CreationEpoch in the given inclusive epoch range.

Both the fromEpoch and toEpoch input parameters are optional and do not have to be given. Any value of 0 or less can be given as to not specify them. Not giving an epoch limit, simply means that this limit won't be used/enforced.

An error will be returned in case no (valid) startKey is given, or in case the given epoch range is invalid (fromEpoch > toEpoch).

The returned TraverseIterator is only valid, as long as the client which created and owns that iterator is valid (e.g. not closed). This traverse iterator is NOT /THREAD-SAFE/.

This method is to be considered EXPERIMENTAL, and might be moved or changed in a future milestone. See https://github.com/zero-os/0-stor/issues/424 for more information.

func (*Client) TraversePostOrder 

func (c *Client) TraversePostOrder(startKey []byte, fromEpoch, toEpoch int64) (TraverseIterator, error)

TraversePostOrder traverses the stored (meta)data, backwards, which is chained together using the (*Client).WriteLinked method. It starts searching from a given startKey and will iterate through all (meta)data, which has a registered CreationEpoch in the given inclusive epoch range.

Both the fromEpoch and toEpoch input parameters are optional and do not have to be given. Any value of 0 or less can be given as to not specify them. Not giving an epoch limit, simply means that this limit won't be used/enforced.

As this method traverses backwards, the startKey is expected to be the newest data as the given fromEpoch should be the most recent time in this chain.

An error will be returned in case no (valid) startKey is given, or in case the given epoch range is invalid (toEpoch > fromEpoch).

The returned TraverseIterator is only valid, as long as the client which created and owns that iterator is valid (e.g. not closed). This traverse iterator is NOT /THREAD-SAFE/.

This method is to be considered EXPERIMENTAL, and might be moved or changed in a future milestone. See https://github.com/zero-os/0-stor/issues/424 for more information.

func (*Client) Write 

func (c *Client) Write(key []byte, r io.Reader) (*metatypes.Metadata, error)

Write writes the data to a 0-stor cluster, storing the metadata using the internal metastor client.

func (*Client) WriteLinked 

func (c *Client) WriteLinked(key, prevKey []byte, r io.Reader) (meta, prevMeta *metatypes.Metadata, err error)

WriteLinked writes the data to a 0-stor cluster, storing the metadata using the internal metastor client, as well as linking the metadata created for this data, to the metadata linked to the given previous key.

This method is to be considered EXPERIMENTAL, and might be moved or changed in a future milestone. See https://github.com/zero-os/0-stor/issues/424 for more information.

func (*Client) WriteWithUserMeta 

func (c *Client) WriteWithUserMeta(key []byte, r io.Reader, userDefined map[string]string) (*metatypes.Metadata, error)

WriteWithUserMeta writes the data to a 0-stor cluster, storing the metadata using the internal metastor client. The given user defined metadata will be stored in the `UserDefined` field of the metadata.

type Config 

type Config struct {
	// Password defines the optional 0-db password.
	Password string `yaml:"password" json:"password"`
	// Namespace defines the label (ID of namespace),
	// to be used for all read/write/delete operations.
	Namespace string `yaml:"namespace" json:"namespace"`

	// DataStor defines the configuration for the zstordb data shards (servers),
	// at least one zstordb shard is given, but more might be required,
	// if you define a distributed storage configuration in the pipeline config.
	DataStor DataStorConfig `yaml:"datastor" json:"datastor"`
}

Config defines the configuration of the 0-stor client. It configures everything from namespaces, permissions, storage clusters, as well as the entire read/write pipeline, used to read and write data.

func ReadConfig 

func ReadConfig(path string) (*Config, error)

ReadConfig reads the configuration from a file. NOTE that it isn't validated, this will be done automatically, when you use the config to create a 0-stor client.

type DataStorConfig 

type DataStorConfig struct {
	// Shards defines the Listed shards, at least one listed shard is required
	Shards []string `yaml:"shards" json:"shards"` // required

	// Pipeline defines the object read/write pipeline configuration
	// for this 0-stor client. It defines how to structure,
	// process, identify and store all data to be written,
	// and that same configuration is required to read the data back.
	Pipeline pipeline.Config `yaml:"pipeline" json:"pipeline"`

	// TLS defines the optional global TLS config,
	// which is used for all lised and unlisted datastor shards, in case it is given.
	TLS DataStorTLSConfig `yaml:"tls" json:"tls"`
}

DataStorConfig is used to configure a zstordb cluster.

type DataStorTLSConfig 

type DataStorTLSConfig struct {
	// has to be true in order to enable this config
	Enabled bool `yaml:"enabled" json:"enabled"`
	// when not given the TLS implemenation will skip certification verification,
	// exposing the client to man-in-the-middle attacks
	ServerName string `yaml:"server" json:"server"`
	// when not given, the system CA will be used
	RootCA string `yaml:"root_ca" json:"root_ca"`

	// optional min/max TLS versions, limiting the
	// accepted TLS version used by the server
	MinVersion TLSVersion `yaml:"min_version" json:"min_version"`
	MaxVersion TLSVersion `yaml:"max_version" json:"max_version"`
}

DataStorTLSConfig is used to config the global TLS config used for all listed and unlisted datastor shards.

type TLSVersion 

type TLSVersion uint8

TLSVersion defines a TLS Version, usable to restrict the possible TLS Versions. 

const (
	// UndefinedTLSVersion defines an undefined TLS Version,
	// which can can be used to signal the desired use of a default TLS Version
	UndefinedTLSVersion TLSVersion = iota
	// TLSVersion12 defines TLS version 1.2,
	// and is also the current default TLS Version.
	TLSVersion12
	// TLSVersion11 defines TLS version 1.1
	TLSVersion11
	// TLSVersion10 defines TLS version 1.0,
	// but should not be used, unless you have no other option.
	TLSVersion10
)

func (TLSVersion) MarshalText 

func (v TLSVersion) MarshalText() (text []byte, err error)

MarshalText implements encoding.TextMarshaler.MarshalText

func (TLSVersion) String 

func (v TLSVersion) String() string

String implements Stringer.String

func (*TLSVersion) UnmarshalText 

func (v *TLSVersion) UnmarshalText(text []byte) error

UnmarshalText implements encoding.TextUnmarshaler.UnmarshalText

func (TLSVersion) VersionTLSOrDefault 

func (v TLSVersion) VersionTLSOrDefault(def uint16) uint16

VersionTLSOrDefault returns the tls.VersionTLS value, which corresponds to this TLSVersion. Or it returns the given default TLS Version in case this TLS Version is undefined.

type TraverseIterator 

type TraverseIterator interface {
	// Next moves the iterator one (valid) position forward,
	// returning false if the iterator has been exhausted.
	//
	// Next has to be called before any (meta)data can be fetched or read.
	Next() bool

	// PeekNextKey returns the next key in line.
	// Note that due to the specified epoch range it
	// might mean that the data of this key will never be available,
	// in case the creation time of the linked (meta)data is
	// not within the specified time range.
	//
	// False is returned in case the iterator has been exhausted,
	// and thus no next key is lined up.
	PeekNextKey() ([]byte, bool)

	// GetMetadata returns the current (and already fetched) metadata.
	//
	// An error is returned in case no metadata is available,
	// due to the iterator being in an invalid state.
	GetMetadata() (*metatypes.Metadata, error)

	// ReadData reads the data available for the current metadata,
	// and writes it to the specified writer.
	//
	// An error is returned in case the iterator is in an invalid state,
	// and thus no data is available to be read.
	ReadData(w io.Writer) error
}

TraverseIterator defines the interface of an iterator, which is returned by a client traverse function.

Source Files

View all Source files

Directories

Path Synopsis
Package datastor defines the clients and other types, to be used to interface with a zstordb server.
 Package datastor defines the clients and other types, to be used to interface with a zstordb server.
pipeline
Package pipeline is used to write/read content from/to a datastor cluster.
 Package pipeline is used to write/read content from/to a datastor cluster.
pipeline/crypto
Package crypto collects common cryptographic components.
 Package crypto collects common cryptographic components.
pipeline/storage
zerodb
zerodb/test
db
db/badger
db/etcd
db/test
db/utils
encoding
encoding/proto
Package proto is a generated protocol buffer package.
 Package proto is a generated protocol buffer package.
metatypes

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.