docstore

README

DocStore Service

This service provides a simple, indexed key/value store for both JSON documents and Binaries. Documents are stored in a BoltDB database, and indexed (and thus searchable) using Bleve full-text search engine.

API

GRPC Handler provides the following service

service DocStore {
    rpc PutDocument (PutDocumentRequest) returns (PutDocumentResponse) {};
    rpc GetDocument (GetDocumentRequest) returns (GetDocumentResponse) {};
    rpc DeleteDocuments (DeleteDocumentsRequest) returns (DeleteDocumentsResponse) {};
    rpc CountDocuments(ListDocumentsRequest) returns (CountDocumentsResponse) {};
    rpc ListDocuments(ListDocumentsRequest) returns (stream ListDocumentsResponse) {};
}

Documents objects are use the following model:

message Document {
    string ID = 2;
    DocumentType Type = 3;
    string Owner = 4;
    bytes Data = 5;
    bytes IndexableMeta = 6;
}

Data can contain a JSON serialized string that will be actually stored, whereas IndexableMeta contains JSON that will be indexed by the search engine. This metadata can have many level depth, and keys can then be searched with Bleve query string like "Key1: value" or "+Key1.SubKey:value*"

Binaries

Binary documents are redirected at the gateway level to a dedicated S3 bucket defined in the configuration. Binary are then served directly via S3.

Documentation

Overview

Docstore provides an indexed JSON document store.

It is used by various services to store their data instead of implementing yet-another persistence layer. It is also used directly by the php frontend to store and retrieve some specific data.

It uses a combination of Bolt for storage and Bleve for indexation.

Index

• type BleveServer
  • func NewBleveEngine(bleveIndexPath string, deleteOnClose ...bool) (*BleveServer, error)
  • func (s *BleveServer) ClearIndex(ctx context.Context) error
  • func (s *BleveServer) Close() error
  • func (s *BleveServer) DeleteDocument(storeID string, docID string) error
  • func (s *BleveServer) IndexDocument(storeID string, doc *docstore.Document) error
  • func (s *BleveServer) SearchDocuments(storeID string, query *docstore.DocumentQuery, countOnly bool) ([]string, int64, error)
• type BoltStore
  • func NewBoltStore(fileName string, deleteOnClose ...bool) (*BoltStore, error)
  • func (b *BoltStore) Close() error
  • func (s *BoltStore) DeleteDocument(storeID string, docID string) error
  • func (s *BoltStore) GetDocument(storeID string, docId string) (*docstore.Document, error)
  • func (s *BoltStore) GetStore(tx *bolt.Tx, storeID string, mode string) (*bolt.Bucket, error)
  • func (s *BoltStore) ListDocuments(storeID string, query *docstore.DocumentQuery) (chan *docstore.Document, chan bool, error)
  • func (s *BoltStore) PutDocument(storeID string, doc *docstore.Document) error
• type Indexer
• type Store

Types

type BleveServer

type BleveServer struct {
	// Internal Bleve database
	Engine bleve.Index
	// For Testing purpose : delete file after closing
	DeleteOnClose bool
	// Path to the DB file
	IndexPath string
}

func NewBleveEngine

func NewBleveEngine(bleveIndexPath string, deleteOnClose ...bool) (*BleveServer, error)

func (*BleveServer) ClearIndex

func (s *BleveServer) ClearIndex(ctx context.Context) error

func (*BleveServer) Close

func (s *BleveServer) Close() error

func (*BleveServer) DeleteDocument

func (s *BleveServer) DeleteDocument(storeID string, docID string) error

func (*BleveServer) IndexDocument

func (s *BleveServer) IndexDocument(storeID string, doc *docstore.Document) error

func (*BleveServer) SearchDocuments

func (s *BleveServer) SearchDocuments(storeID string, query *docstore.DocumentQuery, countOnly bool) ([]string, int64, error)

type BoltStore

type BoltStore struct {

	// For Testing purpose : delete file after closing
	DeleteOnClose bool
	// Path to the DB file
	DbPath string
	// contains filtered or unexported fields
}

func NewBoltStore

func NewBoltStore(fileName string, deleteOnClose ...bool) (*BoltStore, error)

func (*BoltStore) Close

func (b *BoltStore) Close() error

func (*BoltStore) DeleteDocument

func (s *BoltStore) DeleteDocument(storeID string, docID string) error

func (*BoltStore) GetDocument

func (s *BoltStore) GetDocument(storeID string, docId string) (*docstore.Document, error)

func (*BoltStore) GetStore

func (s *BoltStore) GetStore(tx *bolt.Tx, storeID string, mode string) (*bolt.Bucket, error)

func (*BoltStore) ListDocuments

func (s *BoltStore) ListDocuments(storeID string, query *docstore.DocumentQuery) (chan *docstore.Document, chan bool, error)

func (*BoltStore) PutDocument

func (s *BoltStore) PutDocument(storeID string, doc *docstore.Document) error

type Indexer

type Indexer interface {
	IndexDocument(storeID string, doc *docstore.Document) error
	DeleteDocument(storeID string, docID string) error
	SearchDocuments(storeID string, query *docstore.DocumentQuery, countOnly bool) ([]string, int64, error)
	Close() error
}

type Store

type Store interface {
	PutDocument(storeID string, doc *docstore.Document) error
	GetDocument(storeID string, docId string) (*docstore.Document, error)
	DeleteDocument(storeID string, docID string) error
	ListDocuments(storeID string, query *docstore.DocumentQuery) (chan *docstore.Document, chan bool, error)
	Close() error
}