index

package
v0.0.0-...-4b1b879 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 29, 2013 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Overview

Package index provides a generic indexing system on top of the abstract Storage interface.

The following keys & values are populated by receiving blobs and queried for search operations:

  • Recent Permanodes "recpn|<pgp-keyid>|<reverse-modtime>|<claim-blobref>" == "<permanode-blobref>" where reverse-modtime flips each digit to '9'-<digit> and prepends "rt" (for reverse time) "2011-11-27T01:23:45Z" ==> "rt7988-88-72T98:76:54Z"

  • signer blobref of ascii public key -> gpg key id "signerkeyid:sha1-ad87ca5c78bd0ce1195c46f7c98e6025abbaf007" = "2931A67C26F5ABDA"

  • PermanodeOfSignerAttrValue: "signerattrvalue|<keyid>|<URLEscape(attr)>|<URLEscape(value)>|<reverse-claimtime>|<claim-blobref>" == "<permanode>" e.g. "signerattrvalue|2931A67C26F5ABDA|camliRoot|rootval|"+ "rt7988-88-71T98:67:60.999876543Z|sha1-bf115940641f1aae2e007edcf36b3b18c17256d9" => "sha1-7a14cce982aa73ab519e63050f82e2a2adfcf039"

  • Other: "meta:<blobref>" == "<size>|<mimetype>" "have:<blobref>" == "<size>" (used for enumeration, which doesn't need mime type)

  • For GetOwnerClaims(permanode, signer): "claim|<permanode-blobref>|<keyid>|<date>|<claim-blobref>" => "<URL:type>|<URL:attr>|<URL:value>"

Index

Constants

This section is empty.

Variables

View Source 
var ErrNotFound = errors.New("index: key not found")

Functions

This section is empty.

Types

type BatchMutation 

type BatchMutation interface {
	Set(key, value string)
	Delete(key string)
}

func NewBatchMutation 

func NewBatchMutation() BatchMutation

type BlobSniffer 

type BlobSniffer struct {
	// contains filtered or unexported fields
}

func NewBlobSniffer 

func NewBlobSniffer(ref *blobref.BlobRef) *BlobSniffer

func (*BlobSniffer) Body 

func (sn *BlobSniffer) Body() ([]byte, error)

func (*BlobSniffer) IsTruncated 

func (sn *BlobSniffer) IsTruncated() bool

func (*BlobSniffer) MIMEType 

func (sn *BlobSniffer) MIMEType() string

MIMEType returns the sniffed blob's content-type or the empty string if unknown. If the blob is a Camlistore schema metadata blob, the MIME type will be of the form "application/json; camliType=foo".

func (*BlobSniffer) Parse 

func (sn *BlobSniffer) Parse()

func (*BlobSniffer) SchemaBlob 

func (sn *BlobSniffer) SchemaBlob() (meta *schema.Blob, ok bool)

func (*BlobSniffer) Size 

func (sn *BlobSniffer) Size() int64

func (*BlobSniffer) Write 

func (sn *BlobSniffer) Write(d []byte) (int, error)

type Index 

type Index struct {
	*blobserver.SimpleBlobHubPartitionMap
	*blobserver.NoImplStorage

	KeyFetcher blobref.StreamingFetcher // for verifying claims

	// Used for fetching blobs to find the complete sha1s of file & bytes
	// schema blobs.
	BlobSource blobref.StreamingFetcher
	// contains filtered or unexported fields
}

func New 

func New(s Storage) *Index

func NewMemoryIndex 

func NewMemoryIndex() *Index

NewMemoryIndex returns an Index backed only by memory, for use in tests.

func (*Index) EdgesTo 

func (x *Index) EdgesTo(ref *blobref.BlobRef, opts *search.EdgesToOpts) (edges []*search.Edge, err error)

func (*Index) EnumerateBlobs 

func (ix *Index) EnumerateBlobs(dest chan<- blobref.SizedBlobRef, after string, limit int, wait time.Duration) error

func (*Index) ExistingFileSchemas 

func (x *Index) ExistingFileSchemas(wholeRef *blobref.BlobRef) (schemaRefs []*blobref.BlobRef, err error)

func (*Index) GetBlobHub 

func (ix *Index) GetBlobHub() blobserver.BlobHub

func (*Index) GetBlobMIMEType 

func (x *Index) GetBlobMIMEType(blob *blobref.BlobRef) (mime string, size int64, err error)

func (*Index) GetFileInfo 

func (x *Index) GetFileInfo(fileRef *blobref.BlobRef) (*search.FileInfo, error)

func (*Index) GetImageInfo 

func (x *Index) GetImageInfo(fileRef *blobref.BlobRef) (*search.ImageInfo, error)

func (*Index) GetOwnerClaims 

func (x *Index) GetOwnerClaims(permaNode, owner *blobref.BlobRef) (cl search.ClaimList, err error)

func (*Index) GetRecentPermanodes 

func (x *Index) GetRecentPermanodes(dest chan *search.Result, owner *blobref.BlobRef, limit int) (err error)

func (*Index) PathLookup 

func (x *Index) PathLookup(signer, base *blobref.BlobRef, suffix string, at time.Time) (*search.Path, error)

func (*Index) PathsLookup 

func (x *Index) PathsLookup(signer, base *blobref.BlobRef, suffix string) (paths []*search.Path, err error)

func (*Index) PathsOfSignerTarget 

func (x *Index) PathsOfSignerTarget(signer, target *blobref.BlobRef) (paths []*search.Path, err error)

func (*Index) PermanodeOfSignerAttrValue 

func (x *Index) PermanodeOfSignerAttrValue(signer *blobref.BlobRef, attr, val string) (permaNode *blobref.BlobRef, err error)

func (*Index) ReceiveBlob 

func (ix *Index) ReceiveBlob(blobRef *blobref.BlobRef, source io.Reader) (retsb blobref.SizedBlobRef, err error)

func (*Index) SearchPermanodesWithAttr 

func (x *Index) SearchPermanodesWithAttr(dest chan<- *blobref.BlobRef, request *search.PermanodeByAttrRequest) (err error)

This is just like PermanodeOfSignerAttrValue except we return multiple and dup-suppress. If request.Query is "", it is not used in the prefix search.

func (*Index) StatBlobs 

func (ix *Index) StatBlobs(dest chan<- blobref.SizedBlobRef, blobs []*blobref.BlobRef, wait time.Duration) error

func (*Index) Storage 

func (x *Index) Storage() Storage

type Iterator 

type Iterator interface {
	// Next moves the iterator to the next key/value pair.
	// It returns false when the iterator is exhausted.
	Next() bool

	// Key returns the key of the current key/value pair.
	// Only valid after a call to Next returns true.
	Key() string

	// Value returns the value of the current key/value pair.
	// Only valid after a call to Next returns true.
	Value() string

	// Close closes the iterator and returns any accumulated error. Exhausting
	// all the key/value pairs in a table is not considered to be an error.
	// It is valid to call Close multiple times. Other methods should not be
	// called after the iterator has been closed.
	Close() error
}

Iterator iterates over an index Storage's key/value pairs in key order.

An iterator must be closed after use, but it is not necessary to read an iterator until exhaustion.

An iterator is not necessarily goroutine-safe, but it is safe to use multiple iterators concurrently, with each in a dedicated goroutine.

type Mutation 

type Mutation interface {
	Key() string
	Value() string
	IsDelete() bool
}

type Storage 

type Storage interface {
	// Get gets the value for the given key. It returns ErrNotFound if the DB
	// does not contain the key.
	Get(key string) (string, error)

	Set(key, value string) error
	Delete(key string) error

	BeginBatch() BatchMutation
	CommitBatch(b BatchMutation) error

	// Find returns an iterator positioned before the first key/value pair
	// whose key is 'greater than or equal to' the given key. There may be no
	// such pair, in which case the iterator will return false on Next.
	//
	// Any error encountered will be implicitly returned via the iterator. An
	// error-iterator will yield no key/value pairs and closing that iterator
	// will return that error.
	Find(key string) Iterator
}

Storage is the minimal interface that must be implemented by index storage implementations. (e.g. mysql, postgres, mongo, sqlite, leveldb, dynamo)

func NewMemoryStorage 

func NewMemoryStorage() Storage

NewMemoryStorage returns an index Storage implementation that's backed only by memory, for use in tests.

Source Files

View all Source files

Directories

Path Synopsis
Package indextest contains the unit tests for the indexer so they can be re-used for each specific implementation of the index Storage interface.
 Package indextest contains the unit tests for the indexer so they can be re-used for each specific implementation of the index Storage interface.
Package mongo implements the Camlistore index storage abstraction on top of MongoDB.
 Package mongo implements the Camlistore index storage abstraction on top of MongoDB.
Package mysql implements the Camlistore index storage abstraction on top of MySQL.
 Package mysql implements the Camlistore index storage abstraction on top of MySQL.
Package postgres implements the Camlistore index storage abstraction on top of Postgres.
 Package postgres implements the Camlistore index storage abstraction on top of Postgres.
Package sqlindex implements the index.Storage interface using an *sql.DB.
 Package sqlindex implements the index.Storage interface using an *sql.DB.
Package sqlite implements the Camlistore index storage abstraction using an SQLite database file.
 Package sqlite implements the Camlistore index storage abstraction using an SQLite database file.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.