watertower

README

watertower

`

© Copyright Steve Daniels and licensed under CC BY-SA 2.0

Search Engine for Serverless environment.

• Search via words in documents
• Filter by tag

Architecture

This API uses standard inverted index to achieve search. When searching, calculate score by TFIDF algorithm and sort.

This tool is using gocloud.dev's docstore as an storage. So it can run completely managed environment (DynamoDB, Google Firestore, Azure CosmosDB) and MongoDB.

Elasticsearch can use flexible document, but this watertower can use only the following structure. "title" and "content" are processed (tokenize, stemming, remove stop words) by packages in github.com/future-architect/nlp. Now only English and Japanese are implemented.

{
  "unique_key": "100",
  "lang": "en",
  "title": "100 Continue",
  "tags": ["100", "no-error"],
  "content": "This interim response indicates that everything so far is OK and that the client should continue the request, or ignore the response if the request is already finished.",
  "metadata": {"this is not":  "searchable, but you can store any strings"}
}

Go API

Basically, this is an document storage. All API you should handle is WaterTower struct. To create instance, you can pass URL.

Before using this search engine, you should import several packages:

import (
    // Natural languages you want to use
	_ "github.com/future-architect/watertower/nlp/english"
	_ "github.com/future-architect/watertower/nlp/japanese"

    // The storage backends you want to use.
	_ "gocloud.dev/docstore/memdocstore"
	_ "gocloud.dev/docstore/awsdynamodb"
	_ "gocloud.dev/docstore/gcpfirestore"
	_ "gocloud.dev/docstore/mongodocstore"
)

Each storage should be initialized table with "id" as a primary key(partition key for DynamoDB). Index will be a table name.

wt, err := watertower.NewWaterTower(ctx, watertower.Option{
    DocumentUrl: "dynamodb://", // default is "mem://"
    Index:       "docs",        // default is "index"
})

To store document, call PostDocument() method.

docID, err := wt.PostDocument("unique-document-key", &watertower.Document{
    Language: "en",                               // required
    Title: "title",                               // document title
    Content: "content",                           // document body
    Tags: []string{"tag"},                        // tags
    Metadata: map[string]string{"extra": "data"}, // extra data
})

To get document, you can use by unique-key or document-id.

doc, err := wt.FindDocumentByKey(uniqueKey)
docs, err := wt.FindDocuments(id1, id2)

To search document, use Search() method. First parameter is natural language search word for title and content. Second parameter is tags to filter result. Third parameter is natural language name to process search word.

docs, err := wt.Search(searchWord, tags, lang)

To use local index file, open watertower instance without DocumentUrl option.

Then call ReadIndex(r io.Reader) for/and WriteIndex(w io.Writer) to read/store index:

f, err := os.Open("watertower.idx")
if err != nil {
	panic(err)
}

wt, err := watertower.NewWaterTower(ctx, watertower.Option{
    DefaultLanguage: *defaultLanguage,
})
wt.ReadIndex(f)

Sample Codes

httpstatus in /samples/httpstatus

CLI tool to search http status code. It reads from bundled documents and registeres them when booting. It uses memdocstore as a backend datastore.

HTTP interface

watertower-server in /cmd/watertower-server implements Elasticsearch inspired API.

./watertower-server --port=8888

# Register document
$ curl -X POST "http://127.0.0.1:8888/index/_doc/"
　　-H "content-type: application/json"
　　-d '{ "unique_key": "id1", "title": "hello watertower",
　　　　　"content": "watertower is a full text search engine with tag filtering", "lang": "en" }'
{"_id":"d1","_index":"index","_type":"_doc","result":"created"}

# Get document by unique-key
$ curl -X GET "http://127.0.0.1:8888/index/_search?q=unique_key%3Aid1"
    -H"content-type: application/json"
{"hits":{"hits":[{"_id":"d1","_index":"index","_source":{"content":"watertower is a full text search engine with tag filtering","lang":"en","metadata":{},"tags":null,"title":"hello watertower","unique_key":"id1"},"_type":"_doc","sort":null}],"total":{"total":1}}}

# Get document by document ID
$ curl -X GET "http://127.0.0.1:8888/index/_source/d1"
(...)

# Search
$ curl -X GET "http://127.0.0.1:8888/index/_search"
  -H "content-type: application/json"
  -d '{"query": {"bool": {"must": {"match_phrase": {"content": {"query": "stay", "analyzer": "en"}}}}}}'
(...)

CLI tool

CLI tool watertower-cli in /cmd/watertower-cli.

$ ./watertower-cli --help
usage: watertower-cli [<flags>] <command> [<args> ...]

Flags:
  --help  Show context-sensitive help (also try --help-long and --help-man).
  --default-language=DEFAULT-LANGUAGE  
          Default language

Commands:
  help [<command>...]
    Show help.

  create-index [<flags>] <INPUT>
    Generate Index

  search [<flags>] [<WORDS>...]
    Search

It can create single file index by using create-index sub command.

You can try searching the index file with search sub command.

License

Apache2

Documentation

Index

• Variables
• func DefaultCollectionOpener(ctx context.Context, opt Option) (*docstore.Collection, error)
• func DefaultCollectionURL(opt ...Option) (string, error)
• type CombinedError
  • func (c CombinedError) Error() string
• type Document
  • func (d Document) DocumentID() (uint32, error)
• type Option
• type Storage
• type WaterTower
  • func NewWaterTower(ctx context.Context, opt ...Option) (*WaterTower, error)
  • func (wt *WaterTower) AddTagToDocument(tag, uniqueKey string) error
  • func (wt *WaterTower) Close() (err error)
  • func (wt *WaterTower) FindDocumentByKey(uniqueKey string) (*Document, error)
  • func (wt *WaterTower) FindDocuments(ids ...uint32) ([]*Document, error)
  • func (wt *WaterTower) FindTags(tagNames ...string) ([]*tag, error)
  • func (wt *WaterTower) FindTagsWithContext(ctx context.Context, tagNames ...string) ([]*tag, error)
  • func (wt *WaterTower) FindTokens(words ...string) ([]*token, error)
  • func (wt *WaterTower) FindTokensWithContext(ctx context.Context, words ...string) ([]*token, error)
  • func (wt *WaterTower) PostDocument(uniqueKey string, document *Document) (uint32, error)
  • func (wt *WaterTower) ReadIndex(w io.Reader) error
  • func (wt *WaterTower) RemoveDocumentByID(docID uint32) error
  • func (wt *WaterTower) RemoveDocumentByKey(uniqueKey string) error
  • func (wt *WaterTower) RemoveDocumentFromTag(tag string, docID uint32) error
  • func (wt *WaterTower) Search(searchWord string, tags []string, lang string) ([]*Document, error)
  • func (wt *WaterTower) WriteIndex(w io.Writer) error

Variables

var ErrBatchGet = errors.New("batch get error")

Functions

func DefaultCollectionOpener

func DefaultCollectionOpener(ctx context.Context, opt Option) (*docstore.Collection, error)

func DefaultCollectionURL

func DefaultCollectionURL(opt ...Option) (string, error)

DefaultCollectionURL returns collection URL. This function is for help message or debugging

Types

type CombinedError

type CombinedError struct {
	Message string
	Errors  []error
}

func (CombinedError) Error

func (c CombinedError) Error() string

type Document

type Document struct {
	ID             string            `json:"-" docstore:"id"`
	UniqueKey      string            `json:"unique_key" docstore:"unique_key"`
	Language       string            `json:"lang" docstore:"lang"`
	Title          string            `json:"title" docstore:"title"`
	UpdatedAt      time.Time         `json:"updated_at,omitempty" docstore:"updated_at"`
	Tags           []string          `json:"tags,omitempty" docstore:"tags"`
	Content        string            `json:"content" docstore:"content"`
	WordCount      int               `json:"-" docstore:"wc"`
	Metadata       map[string]string `json:"metadata,omitempty" docstore:"metadata"`
	TitleWordCount int               `json:"-" docstore:"twc"`
	Score          float64           `json:"score,omitempty" docstore:"-"`

	Schema  string `json:"$schema,omitempty" docstore:"-"`
	Comment string `json:"$comment,omitempty" docstore:"-"`
}

func (Document) DocumentID

func (d Document) DocumentID() (uint32, error)

type Option

type Option struct {
	DocumentUrl        string
	CollectionOpener   func(ctx context.Context, opt Option) (*docstore.Collection, error)
	LocalFolder        string
	Index              string
	CounterConcurrency int
	TitleScoreRatio    float64
	DefaultLanguage    string
}

type Storage

type Storage interface {
	IncrementDocID() (int, error)
	IncrementDocCount() (int, error)
	DecrementDocCount() error
	DocCount() (int, error)
	Context() context.Context
	Create(id string, doc interface{}) error
	Replace(id string, doc interface{}) error
	GetDoc(doc *Document) error
	GetDocKey(docKey *documentKey) error
	GetTag(tag *tagEntity) error
	GetToken(token *tokenEntity) error
	BatchDocGet(ctx context.Context, docs []*Document) (map[int]bool, error)
	BatchTagGet(ctx context.Context, tags []*tagEntity) (map[int]bool, error)
	BatchTokenGet(ctx context.Context, tokens []*tokenEntity) (map[int]bool, error)
	DeleteDoc(doc *Document) error
	DeleteDocKey(docKey *documentKey) error
	DeleteTag(tag *tagEntity) error
	DeleteToken(token *tokenEntity) error
	Close() error
	WriteIndex(w io.Writer) error
	ReadIndex(r io.Reader) error
}

type WaterTower

type WaterTower struct {
	// contains filtered or unexported fields
}

func NewWaterTower

func NewWaterTower(ctx context.Context, opt ...Option) (*WaterTower, error)

NewWaterTower initialize WaterTower instance

func (*WaterTower) AddTagToDocument

func (wt *WaterTower) AddTagToDocument(tag, uniqueKey string) error

AddTagToDocument adds tag to existing document.

func (*WaterTower) Close

func (wt *WaterTower) Close() (err error)

Close closes document store connection. Some docstore (at least memdocstore) needs Close() to store file

func (*WaterTower) FindDocumentByKey

func (wt *WaterTower) FindDocumentByKey(uniqueKey string) (*Document, error)

FindDocumentByKey looks up document by uniqueKey.

func (*WaterTower) FindDocuments

func (wt *WaterTower) FindDocuments(ids ...uint32) ([]*Document, error)

FindDocuments returns documents by id list.

func (*WaterTower) FindTags

func (wt *WaterTower) FindTags(tagNames ...string) ([]*tag, error)

func (*WaterTower) FindTagsWithContext

func (wt *WaterTower) FindTagsWithContext(ctx context.Context, tagNames ...string) ([]*tag, error)

func (*WaterTower) FindTokens

func (wt *WaterTower) FindTokens(words ...string) ([]*token, error)

func (*WaterTower) FindTokensWithContext

func (wt *WaterTower) FindTokensWithContext(ctx context.Context, words ...string) ([]*token, error)

func (*WaterTower) PostDocument

func (wt *WaterTower) PostDocument(uniqueKey string, document *Document) (uint32, error)

PostDocument registers single document to storage and update index

uniqueKey is a key of document like URL.

document's title and content fields are indexed and searched via natural language algorithms (tokenize, stemming).

document's tags field contains texts and filter documents via complete match algorithm.

func (*WaterTower) ReadIndex

func (wt *WaterTower) ReadIndex(w io.Reader) error

func (*WaterTower) RemoveDocumentByID

func (wt *WaterTower) RemoveDocumentByID(docID uint32) error

RemoveDocumentByID removes document via ID

func (*WaterTower) RemoveDocumentByKey

func (wt *WaterTower) RemoveDocumentByKey(uniqueKey string) error

RemoveDocumentByKey removes document via uniqueKey

func (*WaterTower) RemoveDocumentFromTag

func (wt *WaterTower) RemoveDocumentFromTag(tag string, docID uint32) error

func (*WaterTower) Search

func (wt *WaterTower) Search(searchWord string, tags []string, lang string) ([]*Document, error)

Search searches documents.

searchWord is for title, content. This is processed by NLP logic specified by lang parameter.

tags is for filter search result.

func (*WaterTower) WriteIndex

func (wt *WaterTower) WriteIndex(w io.Writer) error