lens

README

🔍 Lens

Search engine for the distributed web

Lens is an opt-in search engine and data collection tool to aid content discovery of the distributed web. It exposes a simple, minimal API for intelligently indexing and querying content on IPFS.

Features and Usage

Initially integrated with Temporal, Lens will allow users to optionally have the data they upload be searched and indexed and be awarded with RTC for participating in the data collection process. Users can then search for content using a simple-to-use API.

Searching through Lens will be facilitated through Temporal web. Optionally, we will have a service independent from Temporal which users can submit content to have it be indexed. This however, is not compensated with RTC. In order to receive the RTC, you must participate through Lens indexing within the Temporal web interface.

API

Lens exposes a simple API via gRPC. The definitions are in RTradeLtd/grpc.

The Lens API, summarized, currently consists of three core RPCs:

service LensV2 {
  rpc Index(IndexReq)   returns (IndexResp)  {}
  rpc Search(SearchReq) returns (SearchResp) {}
  rpc Remove(RemoveReq) returns (RemoveResp) {}
}

Golang bindings for the Lens API can be found in RTradeLtd/grpc.

Supported Formats

Only IPFS CIDs are supported, and they must be plaintext files. We attempt to determine the content type via mime type sniffing, and use that to determine whether or not we can analyze the content.

Please see the following table for supported content types that we can index. Note if the type is listed as <type>/* it means that any "sub type" of that mime type is supported.

Mime Type	Support Level	Tested Types
text/*	Beta	text/plain, text/html
image/*	Beta	image/jpeg
application/pdf	Beta	application/pdf

Deployment

The recommended way to deploy a Lens instance is via the rtradetech/lens Docker image.

$> docker pull rtradetech/lens:latest

A docker-compose configuration is available that also starts up other prerequisites:

$> wget -O lens.yml https://raw.githubusercontent.com/RTradeLtd/Lens/master/lens.yml
$> LENS=latest BASE=/my/dir docker-compose -f lens.yml up

Development

This project requires:

• Go 1.11+
• dep
• Tesseract
• Tensorflow
• go-fitz

To fetch the codebase, use go get:

$> go get github.com/RTradeLtd/Lens

A rudimentary Makefile target make dep is available for installing the required dependencies.

Documentation

Index

• type V2
  • func NewV2(opts V2Options, ipfs rtfs.Manager, ia images.TensorflowAnalyzer, ...) (*V2, error)
  • func NewV2WithEngine(opts V2Options, ipfs rtfs.Manager, ia images.TensorflowAnalyzer, ...) *V2
  • func (v *V2) Close()
  • func (v *V2) Index(ctx context.Context, req *lensv2.IndexReq) (*lensv2.IndexResp, error)
  • func (v *V2) Remove(ctx context.Context, req *lensv2.RemoveReq) (*lensv2.RemoveResp, error)
  • func (v *V2) Search(ctx context.Context, req *lensv2.SearchReq) (*lensv2.SearchResp, error)
• type V2Options

Types

type V2

type V2 struct {
	// contains filtered or unexported fields
}

V2 is the new Lens API, and implements the LensV2 gRPC interface directly.

func NewV2

func NewV2(
	opts V2Options,
	ipfs rtfs.Manager,
	ia images.TensorflowAnalyzer,
	logger *zap.SugaredLogger,
) (*V2, error)

NewV2 instantiates a new V2 API

func NewV2WithEngine

func NewV2WithEngine(
	opts V2Options,
	ipfs rtfs.Manager,
	ia images.TensorflowAnalyzer,
	se engine.Searcher,
	logger *zap.SugaredLogger,
) *V2

NewV2WithEngine instantiates a Lens V2 service with the given engine

func (*V2) Close

func (v *V2) Close()

Close releases Lens resources

func (*V2) Index

func (v *V2) Index(ctx context.Context, req *lensv2.IndexReq) (*lensv2.IndexResp, error)

Index analyzes and stores the given object

func (*V2) Remove

func (v *V2) Remove(ctx context.Context, req *lensv2.RemoveReq) (*lensv2.RemoveResp, error)

Remove unindexes and deletes the requested object

func (*V2) Search

func (v *V2) Search(ctx context.Context, req *lensv2.SearchReq) (*lensv2.SearchResp, error)

Search executes a query against the Lens index

type V2Options

type V2Options struct {
	TesseractConfigPath string

	Engine engine.Opts
}

V2Options denotes options for the V2 Lens API