gotinydb

package module
v0.5.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 12, 2019 License: Apache-2.0 Imports: 25 Imported by: 0

README

GoTinyDB

GoDoc Build Status FOSSA Status codecov Go Report Card License codebeat badge

The goal is to have a fairly simple database which is light and don't needs to fit in RAM. This part is done by Badger which use SSD capabilities to have low RAM consumption and good performances.

The indexing is done by Bleve.

It's pure Go language, so no issues with CGO.

Features

Collections

Save deferent document type in different collection.

The database can have many collections, see prefix limitations. many collection can be used on the same database.

Index and query is done by Bleve

It's a fully featured indexing package. Indexing is done at the collection level and one collection can have many indexes. See prefix limitations.

Files and media content

In the same database you can save files of any size and many small documents.

Supports for big content with io.Reader and io.Writer interface. It split content into chunks of 5MB.

Confidentiality and data integrity (encryption)

The all database content is encrypted and signed with XChaCha20-Poly1305.

See encryption limitations

Installing

go get -u github.com/alexandrestein/gotinydb

Getting started

The package is supposed to be used inside your software and at this point it is not supposed to be a dedicated database service. Take a look at GoDoc and to the examples folder.

Road Map

  • Make a basic master/slaves replication system for data protection

Contributing

Any contribution will be appreciate. Feedbacks and suggestions are welcome.

Vendoring

We moved to go module. In future Go release this will be the default vendoring behavior. So this will be transparent in the next release.

Versioning

Version v1.0.0 is not ready. It's under development and versions have the form of v0.x.x.

There is no compatibility promise for now.

Limitations

Concurrency

Most of the methods can be run concurrently. But management actions can not:

  • *DB.Backup
  • *DB.Close
  • *DB.DeleteCollection
  • *DB.Load
  • *Collection.DeleteIndex
  • *Collection.SetBleveIndex

Those functions and methods should be run on idle database.

Prefixes

Prefixes to split different parts of the database: collection, files, indexes and documents. The prefixes are only 2 bytes (16bits). It gives 65536 possibilities which much more than enough. But the hash collision are possible so the real limit is much less.

If you are unlucky you will hit this ErrHashCollision.

Encryption

The database keys are not encrypted. But for indexing some of the content are used as keys. For iteration reason the keys can't be encrypted.

For the content which needs to be sealed don't index them. Bleve index mapping provides a very sine control of what are or not indexed.

Author

License

This project is licensed under the "Apache License, Version 2.0" see the LICENSE file for details or follow this link.

FOSSA Status

Documentation

Overview

Package gotinydb implements a simple but useful embedded database.

It supports document insertion and retrieving of golang pointers via the JSON package. Those documents can be indexed with Bleve.

File management is also supported and the all database is encrypted.

It relais on Bleve and Badger to do the job.

Example 
Output:

id
316 jonas-90@tlaloc.com Nov  5 12:20:44

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	ErrNotFound           = fmt.Errorf("not found")
	ErrHashCollision      = fmt.Errorf("the name is in collision with an other element")
	ErrEmptyID            = fmt.Errorf("ID must be provided")
	ErrIndexNotFound      = fmt.Errorf("index not found")
	ErrNameAllreadyExists = fmt.Errorf("element with the same name allready exists")
	ErrGetMultiNotEqual   = fmt.Errorf("you must provied the same number of ids and destinations")

	ErrEndOfQueryResult = fmt.Errorf("there is no more values to retrieve from the query")

	ErrFileInWrite              = fmt.Errorf("this file is already in write mode")
	ErrFileItemIteratorNotValid = fmt.Errorf("item is not valid")
)

This defines most of the package errors

Functions

This section is empty.

Types

type Batch added in v0.3.1 

type Batch struct {
	// contains filtered or unexported fields
}

Batch is a simple struct to manage multiple write in one commit

Example 
Output:

<nil>

func (*Batch) Delete added in v0.3.1 

func (b *Batch) Delete(id string) error

Delete add a delete operation to the existing Transaction pointer

func (*Batch) Put added in v0.3.1 

func (b *Batch) Put(id string, content interface{}) error

Put add a put operation to the existing Transaction pointer

func (*Batch) PutClean added in v0.3.1 

func (b *Batch) PutClean(id string, content interface{}) error

PutClean add a put operation to the existing Transaction pointer but clean existing history of the id

func (*Batch) PutWithTTL added in v0.4.1 

func (b *Batch) PutWithTTL(id string, content interface{}, ttl time.Duration) error

PutWithTTL add a put operation to the existing Transaction pointer. The ttl parameter tels the database who long to keep the record in DB. After the given duration the content and the ID are removed from DB with clean history.

func (*Batch) Write added in v0.3.1 

func (b *Batch) Write() error

Write execute the batch

type BleveIndex added in v0.3.0 

type BleveIndex struct {
	Path string

	BleveIndexAsBytes []byte
	// contains filtered or unexported fields
}

BleveIndex defines for now the only supported index (no plan for other unless it's needed).

type Collection 

type Collection struct {

	// BleveIndexes in public for marshalling reason and should never be used directly
	BleveIndexes []*BleveIndex
	// contains filtered or unexported fields
}

Collection defines the base element for saving objects. It holds the indexes and the values.

func (*Collection) Delete 

func (c *Collection) Delete(id string) (err error)

Delete deletes all references of the given id.

func (*Collection) DeleteIndex 

func (c *Collection) DeleteIndex(name string)

DeleteIndex delete the index and all references

func (*Collection) Get 

func (c *Collection) Get(id string, dest interface{}) (contentAsBytes []byte, err error)

Get returns the saved element. It fills up the given dest pointer if provided. It always returns the content as a stream of bytes and an error if any.

Example 
Output:

{"Name":"testing name"}
&{testing name}

func (*Collection) GetBleveIndex added in v0.3.0 

func (c *Collection) GetBleveIndex(name string) (*BleveIndex, error)

GetBleveIndex gives an easy way to interact directly with bleve

func (*Collection) GetIterator added in v0.3.1 

func (c *Collection) GetIterator() *CollectionIterator

GetIterator provides an easy way to list elements

func (*Collection) GetMulti added in v0.3.1 

func (c *Collection) GetMulti(ids []string, destinations []interface{}) (contentsAsBytes [][]byte, err error)

GetMulti open one badger transaction and get all document concurrently

func (*Collection) GetRevertedIterator added in v0.3.1 

func (c *Collection) GetRevertedIterator() *CollectionIterator

GetRevertedIterator does same as above but work in the opposite way

func (*Collection) History added in v0.3.0 

func (c *Collection) History(id string, limit int) (valuesAsBytes [][]byte, err error)

History returns the previous versions of the given id. The first value is the actual value and more you travel inside the list more the records are old.

func (*Collection) NewBatch added in v0.3.1 

func (c *Collection) NewBatch(ctx context.Context) (*Batch, error)

NewBatch build a new write transaction to do all write operation in one commit

func (*Collection) Put 

func (c *Collection) Put(id string, content interface{}) error

Put sets a new element into the collection. If the content match some of the indexes it will be indexed

Example 
Output:

<nil>

func (*Collection) PutWithCleanHistory added in v0.3.0 

func (c *Collection) PutWithCleanHistory(id string, content interface{}) (err error)

PutWithCleanHistory set the content to the given id but clean all previous records of this id

func (*Collection) PutWithTTL added in v0.4.1 

func (c *Collection) PutWithTTL(id string, content interface{}, ttl time.Duration) error

PutWithTTL does the same as *Collection.Put but removes the content and its ID after the given duration

func (*Collection) Search added in v0.3.0 

func (c *Collection) Search(indexName string, query query.Query) (*SearchResult, error)

Search make a search with the default bleve search request bleve.NewSearchRequest() and returns a local SearchResult pointer

Example 
Output:

&{I'm the example document}
index X document
{"Name":"I'm the example document"}
0.7071067690849304

func (*Collection) SearchWithOptions added in v0.3.0 

func (c *Collection) SearchWithOptions(indexName string, searchRequest *bleve.SearchRequest) (*SearchResult, error)

SearchWithOptions does the same as *Collection.Search but you provide the searchRequest

func (*Collection) SetBleveIndex added in v0.3.0 

func (c *Collection) SetBleveIndex(name string, documentMapping *mapping.DocumentMapping) (err error)

SetBleveIndex adds a bleve index to the collection. It build a new index with the given index mapping.

Example 
Output:

<nil>

type CollectionIterator added in v0.3.1 

type CollectionIterator struct {
	// contains filtered or unexported fields
}

CollectionIterator provides a nice way to list elements

Example 
Output:

{"Name":"testing name"}
{"Name":"I'm the example document"}
{"ID":56548,"Email":"user2@dom.com","LastLogin":"2018-11-05T12:20:44.588809926+01:00"}

func (CollectionIterator) Close added in v0.3.1 

func (i CollectionIterator) Close()

Close closes the current iterator and it's related components. This method needs to be called ones the iterator is no more needed.

func (*CollectionIterator) GetBytes added in v0.3.1 

func (i *CollectionIterator) GetBytes() []byte

GetBytes returns the document as a slice of bytes

func (*CollectionIterator) GetID added in v0.3.1 

func (i *CollectionIterator) GetID() string

GetID returns the collection id if the current element

func (*CollectionIterator) GetValue added in v0.3.1 

func (i *CollectionIterator) GetValue(dest interface{})

GetValue tries to fill-up the dest pointer with the coresponding document

func (*CollectionIterator) Next added in v0.3.1 

func (i *CollectionIterator) Next()

Next moves the cursor to the next position. If the iterator is in regular mode it will move to the smallest bigger key than the current one. If the iterator is in reverted mode it will move to the biggest smaller key than the current one.

func (*CollectionIterator) Seek added in v0.3.1 

func (i *CollectionIterator) Seek(id string)

Seek would seek to the provided key if present. If absent, it would seek to the next smallest key greater than provided if iterating in the forward direction. Behavior would be reversed is iterating backwards.

func (*CollectionIterator) Valid added in v0.3.1 

func (i *CollectionIterator) Valid() bool

Valid returns true if the cursor still on valid value. It returns false if the iteration is done

type DB 

type DB struct {

	// PrivateKey is public for marshaling reason and should never by used or changes.
	// This is the primary key used to derive every records.
	PrivateKey [32]byte

	// Collection is public for marshaling reason and should never be used.
	// It contains the collections pointers used to manage the documents.
	Collections []*Collection

	// FileStore provides all accessibility to the file storage facilities
	FileStore *FileStore
	// contains filtered or unexported fields
}

DB is the base struct of the package. It provides the collection and manage all writes to the database.

func Open 

func Open(path string, configKey [32]byte) (db *DB, err error)

Open initialize a new database or open an existing one. The path defines the place the data will be saved and the configuration key permit to decrypt existing configuration and to encrypt new one.

Example 
Output:

<nil>

func (*DB) Backup 

func (d *DB) Backup(w io.Writer) error

Backup perform a full backup of the database. It fills up the io.Writer with all data indexes and configurations.

func (*DB) Close 

func (d *DB) Close() (err error)

Close close the database and all subcomposants. It returns the error if any

func (*DB) DeleteCollection 

func (d *DB) DeleteCollection(colName string)

DeleteCollection removes every document and indexes and the collection itself

func (*DB) Load 

func (d *DB) Load(r io.Reader) error

Load recover an existing database from a backup generated with *DB.Backup

func (*DB) Use 

func (d *DB) Use(colName string) (col *Collection, err error)

Use build a new collection or open an existing one.

Example 
Output:

collection name
[1 20 101]

type FileIterator added in v0.3.1 

type FileIterator struct {
	// contains filtered or unexported fields
}

FileIterator provides easy access to all written files

Example 
Output:

read file txt
file example test.txt

func (FileIterator) Close added in v0.3.1 

func (i FileIterator) Close()

Close closes the current iterator and it's related components. This method needs to be called ones the iterator is no more needed.

func (*FileIterator) GetMeta added in v0.3.1 

func (i *FileIterator) GetMeta() *FileMeta

GetMeta returns the metadata of the actual cursor position

func (*FileIterator) Next added in v0.3.1 

func (i *FileIterator) Next() error

Next moves to the next valid metadata element

func (*FileIterator) Seek added in v0.3.1 

func (i *FileIterator) Seek(id string)

Seek moves to the meta coresponding to the given id

func (*FileIterator) Valid added in v0.3.1 

func (i *FileIterator) Valid() bool

Valid checks if the cursor point a valid metadata document

type FileMeta added in v0.3.1 

type FileMeta struct {
	ID                        string
	Name                      string
	Size                      int64
	LastModified              time.Time
	ChuckSize                 int
	RelatedDocumentID         string
	RelatedDocumentCollection string
	// contains filtered or unexported fields
}

FileMeta defines some file metadata informations

type FileStore added in v0.5.0 

type FileStore struct {
	// contains filtered or unexported fields
}

FileStore defines database file storage object

func (*FileStore) DeleteFile added in v0.5.0 

func (fs *FileStore) DeleteFile(id string) (err error)

DeleteFile deletes every chunks of the given file ID

func (*FileStore) GetFileIterator added in v0.5.0 

func (fs *FileStore) GetFileIterator() *FileIterator

GetFileIterator returns a file iterator which help to list existing files

func (*FileStore) GetFileReader added in v0.5.0 

func (fs *FileStore) GetFileReader(id string) (Reader, error)

GetFileReader returns a struct to provide simple reading partial of big files. The default position is at the begining of the file.

func (*FileStore) GetFileWriter added in v0.5.0 

func (fs *FileStore) GetFileWriter(id, name string) (Writer, error)

GetFileWriter returns a struct to provide simple partial write of big files. The default position is at the end of the file.

func (*FileStore) GetFileWriterRelated added in v0.5.0 

func (fs *FileStore) GetFileWriterRelated(id, name string, colName, documentID string) (Writer, error)

GetFileWriterRelated does the same as GetFileWriter but with related document

func (*FileStore) GetFileWriterWithTTL added in v0.5.0 

func (fs *FileStore) GetFileWriterWithTTL(id, name string, ttl time.Duration) (Writer, error)

GetFileWriterWithTTL does the same as GetFileWriter but it's automatically removed after the given duration

func (*FileStore) PutFile added in v0.5.0 

func (fs *FileStore) PutFile(id string, name string, reader io.Reader) (n int, err error)

PutFile let caller insert large element into the database via a reader interface

func (*FileStore) PutFileRelated added in v0.5.0 

func (fs *FileStore) PutFileRelated(id string, name string, reader io.Reader, colName, documentID string) (n int, err error)

PutFileRelated does the same as *DB.PutFile but the file is automatically removed when the related document is removed. The use case can be for blog post for example. You have posts which has images and medias in it. Ones the post is removed the images and the medias are not needed anymore. This provide a easy way remove files automatically based on collection documents.

func (*FileStore) PutFileWithTTL added in v0.5.0 

func (fs *FileStore) PutFileWithTTL(id string, name string, reader io.Reader, ttl time.Duration) (n int, err error)

PutFileWithTTL let caller insert large element into the database via a reader interface

func (*FileStore) ReadFile added in v0.5.0 

func (fs *FileStore) ReadFile(id string, writer io.Writer) error

ReadFile write file content into the given writer

type Reader added in v0.3.1 

type Reader interface {
	io.ReadCloser
	io.Seeker
	io.ReaderAt

	GetMeta() *FileMeta
}

Reader define a simple object to read parts of the file

Example 
Output:

39 GFvOPRqvWxfHUmAFAksELPTpV0lmPvwjMwdqq5i

type Response 

type Response struct {
	ID            string
	Content       []byte
	DocumentMatch *search.DocumentMatch
}

Response are returned by *SearchResult.NextResponse if the caller needs to have access to the byte stream

type SearchResult added in v0.3.0 

type SearchResult struct {
	BleveSearchResult *bleve.SearchResult
	// contains filtered or unexported fields
}

SearchResult is returned bu *Collection.Search or *Collection.SearchWithOptions. It provides a easy listing of the result.

func (*SearchResult) Next added in v0.3.0 

func (s *SearchResult) Next(dest interface{}) (id string, err error)

Next fills up the destination by marshaling the saved byte stream. It returns an error if any and the coresponding id of the element.

func (*SearchResult) NextResponse added in v0.3.0 

func (s *SearchResult) NextResponse(dest interface{}) (resp *Response, _ error)

NextResponse fills up the destination by marshaling the saved byte stream. It returns the byte stream and the id of the document inside a Response pointer or an error if any.

type Writer added in v0.3.1 

type Writer interface {
	Reader

	io.Writer
	io.WriterAt
}

Writer define a simple object to write parts of the file

Example 
Output:

writtenBytes 50
50 this is a text file
and then the second is written

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.