filedb

README

FileDB - file system configuration reader plugin

The fileDB plugin allows to use the file system of a operating system as a key-value data store. The filesystem plugin watches for pre-defined files or directories, reads a configuration and sends response events according to changes.

All the configuration is resynced in the beginning (as for standard key-value data store). Configuration files then can be added, updated, moved, renamed or removed, plugin makes all the necessary changes.

Important note: fileDB as datastore is read-only from the plugin perspective, changes from within the plugin are not allowed.

Configuration

All files/directories used as a data store must be defined in configuration file. Location of the file can be defined either by the command line flag filedb-config or set via the FILEDB_CONFIG environment variable.

Supported formats

• JSON (*.json)
• YAML (*.yaml)

Data structure

Plugin currently supports only JSON and YAML-formatted data. The format of the file is as follows for JSON:

{
    "data": [
        {
            "key": "<key>",
            "value": {
                <proto-modelled data>
            }
        },
        {
            "key": "<key>",
            "value": {
                <proto-modelled data>
            }
        },
        ...
    ]
}

For YAML:

---
data:
    -
        key: '<key>'
        value: '<modelled data>'

Key has to contain also instance prefix with micro service label, so plugin knows which parts of the configuration are intended for it. All configuration is stored internally in local database. It allows to compare events and respond with correct 'previous' value for a given key.

Data state propagation

Data types supporting status propagation (like interfaces or bridge domains) can store the state in filesystem. There is a field in configuration file called status-path which has to be set in order to store the status. Status data will be stored in the same format as for configuration, type is defined by the file extension (JSON or YAML).

Data will not be propagated if target path directory.

Documentation

Index

• Variables
• type BrokerWatcher
  • func (pdb *BrokerWatcher) Delete(key string, opts ...datasync.DelOption) (existed bool, err error)
  • func (pdb *BrokerWatcher) GetValue(key string) (data []byte, found bool, revision int64, err error)
  • func (pdb *BrokerWatcher) ListKeys(prefix string) (keyval.BytesKeyIterator, error)
  • func (pdb *BrokerWatcher) ListValues(key string) (keyval.BytesKeyValIterator, error)
  • func (pdb *BrokerWatcher) NewTxn() keyval.BytesTxn
  • func (pdb *BrokerWatcher) Put(key string, data []byte, opts ...datasync.PutOption) error
  • func (pdb *BrokerWatcher) Watch(resp func(keyval.BytesWatchResp), closeChan chan string, keys ...string) error
• type Client
  • func NewClient(cfgPaths []string, statusPath string, dcs []decoder.API, fsh filesystem.API, ...) (*Client, error)
  • func (c *Client) Close() error
  • func (c *Client) Delete(key string, opts ...datasync.DelOption) (existed bool, err error)
  • func (c *Client) GetDataForFile(path string) []*decoder.FileDataEntry
  • func (c *Client) GetDataForKey(key string) (*decoder.FileDataEntry, bool)
  • func (c *Client) GetPaths() []string
  • func (c *Client) GetValue(key string) (data []byte, found bool, revision int64, err error)
  • func (c *Client) ListKeys(prefix string) (keyval.BytesKeyIterator, error)
  • func (c *Client) ListValues(prefix string) (keyval.BytesKeyValIterator, error)
  • func (c *Client) NewBroker(prefix string) keyval.BytesBroker
  • func (c *Client) NewTxn() keyval.BytesTxn
  • func (c *Client) NewWatcher(prefix string) keyval.BytesWatcher
  • func (c *Client) Put(key string, data []byte, opts ...datasync.PutOption) error
  • func (c *Client) Watch(resp func(response keyval.BytesWatchResp), closeChan chan string, ...) error
• type Config
• type Deps
• type Option
  • func UseDeps(cb func(*Deps)) Option
• type Plugin
  • func NewPlugin(opts ...Option) *Plugin
  • func (p *Plugin) AfterInit() error
  • func (p *Plugin) Close() error
  • func (p *Plugin) Disabled() bool
  • func (p *Plugin) Init() error
  • func (p *Plugin) NewBroker(keyPrefix string) keyval.ProtoBroker
  • func (p *Plugin) NewWatcher(keyPrefix string) keyval.ProtoWatcher
  • func (p *Plugin) OnConnect(callback func() error)
  • func (p *Plugin) String() string

Variables

var DefaultPlugin = *NewPlugin()

DefaultPlugin is a default instance of Plugin.

Types

type BrokerWatcher

type BrokerWatcher struct {
	*Client
	// contains filtered or unexported fields
}

BrokerWatcher implements CoreBrokerWatcher and provides broker/watcher constructors with client

func (*BrokerWatcher) Delete

func (pdb *BrokerWatcher) Delete(key string, opts ...datasync.DelOption) (existed bool, err error)

Delete calls client's 'Delete' method

func (*BrokerWatcher) GetValue

func (pdb *BrokerWatcher) GetValue(key string) (data []byte, found bool, revision int64, err error)

GetValue calls client's 'GetValue' method

func (*BrokerWatcher) ListKeys

func (pdb *BrokerWatcher) ListKeys(prefix string) (keyval.BytesKeyIterator, error)

ListKeys returns a list of all database keys for given prefix

func (*BrokerWatcher) ListValues

func (pdb *BrokerWatcher) ListValues(key string) (keyval.BytesKeyValIterator, error)

ListValues returns a list of all database values for given key

func (*BrokerWatcher) NewTxn

func (pdb *BrokerWatcher) NewTxn() keyval.BytesTxn

NewTxn calls client's 'NewTxn' method

func (*BrokerWatcher) Put

func (pdb *BrokerWatcher) Put(key string, data []byte, opts ...datasync.PutOption) error

Put calls client's 'Put' method

func (*BrokerWatcher) Watch

func (pdb *BrokerWatcher) Watch(resp func(keyval.BytesWatchResp), closeChan chan string, keys ...string) error

Watch augments watcher's response and removes prefix from it

type Client

type Client struct {
	sync.Mutex
	// contains filtered or unexported fields
}

Client arranges communication between file system and internal database, and is also responsible for upstream events.

func NewClient

func NewClient(cfgPaths []string, statusPath string, dcs []decoder.API, fsh filesystem.API, log logging.Logger) (*Client, error)

NewClient initializes file watcher, database and registers paths provided via plugin configuration file

func (*Client) Close

func (c *Client) Close() error

Close closes all readers

func (*Client) Delete

func (c *Client) Delete(key string, opts ...datasync.DelOption) (existed bool, err error)

Delete is not allowed for fileDB, configuration file is read-only

func (*Client) GetDataForFile

func (c *Client) GetDataForFile(path string) []*decoder.FileDataEntry

GetDataForFile returns data gor given file

func (*Client) GetDataForKey

func (c *Client) GetDataForKey(key string) (*decoder.FileDataEntry, bool)

GetDataForKey returns data gor given file

func (*Client) GetPaths

func (c *Client) GetPaths() []string

GetPaths returns client file paths

func (*Client) GetValue

func (c *Client) GetValue(key string) (data []byte, found bool, revision int64, err error)

GetValue returns a value for given key

func (*Client) ListKeys

func (c *Client) ListKeys(prefix string) (keyval.BytesKeyIterator, error)

ListKeys returns a set of keys for given prefix

func (*Client) ListValues

func (c *Client) ListValues(prefix string) (keyval.BytesKeyValIterator, error)

ListValues returns a list of values for given prefix

func (*Client) NewBroker

func (c *Client) NewBroker(prefix string) keyval.BytesBroker

NewBroker provides BytesBroker object with client and given prefix

func (*Client) NewTxn

func (c *Client) NewTxn() keyval.BytesTxn

NewTxn is not supported, filesystem plugin does not allow to do changes to the configuration

func (*Client) NewWatcher

func (c *Client) NewWatcher(prefix string) keyval.BytesWatcher

NewWatcher provides BytesWatcher object with client and given prefix

func (*Client) Put

func (c *Client) Put(key string, data []byte, opts ...datasync.PutOption) error

Put reads status file, add data to it and performs write

func (*Client) Watch

func (c *Client) Watch(resp func(response keyval.BytesWatchResp), closeChan chan string, keys ...string) error

Watch starts single watcher for every key prefix. Every watcher listens on its own data channel.

type Config

type Config struct {
	ConfigPaths []string `json:"configuration-paths"`
	StatusPath  string   `json:"status-path"`
}

Config is fileDB configuration file structure

type Deps

type Deps struct {
	infra.PluginDeps
}

Deps are fileDB plugin dependencies

type Option

type Option func(*Plugin)

Option is a function that can be used in NewPlugin to customize Plugin.

func UseDeps

func UseDeps(cb func(*Deps)) Option

UseDeps returns Option that can inject custom dependencies.

type Plugin

type Plugin struct {
	Deps
	// contains filtered or unexported fields
}

Plugin filesystem uses host os file system as database to store configuration.

func NewPlugin

func NewPlugin(opts ...Option) *Plugin

NewPlugin creates a new Plugin with the provided Options.

func (*Plugin) AfterInit

func (p *Plugin) AfterInit() error

AfterInit starts file system event watcher

func (*Plugin) Close

func (p *Plugin) Close() error

Close client

func (*Plugin) Disabled

func (p *Plugin) Disabled() bool

Disabled returns flag whether plugin is disabled

func (*Plugin) Init

func (p *Plugin) Init() error

Init reads file config and creates new client to communicate with file system

func (*Plugin) NewBroker

func (p *Plugin) NewBroker(keyPrefix string) keyval.ProtoBroker

NewBroker returns new broker created by proto wrapper

func (*Plugin) NewWatcher

func (p *Plugin) NewWatcher(keyPrefix string) keyval.ProtoWatcher

NewWatcher returns new watcher created by proto wrapper

func (*Plugin) OnConnect

func (p *Plugin) OnConnect(callback func() error)

OnConnect executes datasync callback

func (*Plugin) String

func (p *Plugin) String() string

String returns string-representation of plugin name