walrus

package module
v0.0.0-...-247491a Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 21, 2023 License: Apache-2.0 Imports: 25 Imported by: 3

README

GoDoc License

Walrus

Walrus is a Go library that provides a tiny implementation of the Bucket API from the sg-bucket package.

If you have Go code that talks to a Couchbase Server bucket, you can fairly easily switch it to use an in-process Walrus bucket instead. This is handy for development and testing, since you don't have to run a full Couchbase Server instance, configure buckets, flush them before running tests, etc.

Walrus is not fast or scalable. It's not supposed to be. It's supposed to be the simplest thing that could possibly implement the Bucket API.

Walrus supports

  • Basic CRUD operations on both JavaScript and raw documents
  • Atomic increment
  • Design documents and views with JavaScript map functions
  • Key ranges and limits in queries
  • Persistence (by archiving the in-memory data structures to a file)

Walrus does not yet support

  • Direct CAS operations (only the higher-level Update call)
  • Document expiration
  • Reduce functions in views
  • Offsets or reverse order in queries

Patches gratefully accepted :)

Walrus intentionally does not support

  • Network access -- it's a library, not a server
  • Incremental updates of the persistent store
  • Multiple processes sharing a persistent store

Using Walrus

It couldn't be easier:

import "github.com/couchbaselabs/walrus"

Now you can change any of your couchbase.GetBucket calls to the equivalent Walrus call:

bucket := walrus.GetBucket("http://localhost:8091", "default", "bucketName")

This creates an in-memory (non-persistent) bucket you can use as though it were a couchbase.Bucket. The HTTP URL and pool name are ignored, except that if you call GetBucket with the same three parameter values again, it will return the same Bucket instance.

In the simplest case, if you just want to create a new non-persistent bucket, you can just call:

bucket := walrus.NewBucket("bucketname")
Persistent buckets

You can create a persistent bucket by using a "file:" or "walrus:" URL, or just an absolute directory path. Any of these three calls will create a persistent bucket, whose backing file will be /tmp/bucketName.walrus:

walrus.GetBucket("file:///tmp", "default", "bucketName")
walrus.GetBucket("walrus:/tmp", "default", "bucketName")
walrus.GetBucket("/tmp", "default", "bucketName")

Buckets persist themselves simply by archiving the Bucket object to a data blob using the Gob format, then writing that blob to a file. When a bucket's contents are changed, it schedules a save for two seconds in the future, which will (atomically) rewrite the entire file. Again, this is obviously not scalable, but it's simple and works fine for small data sets.

Documentation

Index

Constants

View Source 
const (
	SimulatedVBucketCount = 1024 // Used when hashing doc id -> vbno
)

Variables

View Source 
var Logging bool

Set this to true to enable logging

View Source 
var MaxDocSize = 0 // Used during the write function

Functions

func CheckDDoc 

func CheckDDoc(design *sgbucket.DesignDoc) error

Validates a design document.

func CollateJSON 

func CollateJSON(key1, key2 interface{}) int

func Parallelize 

func Parallelize(f PipelineFunc, parallelism int, input <-chan jsMapFunctionInput) <-chan interface{}

Feeds the input channel through a number of copies of the function in parallel. This call is asynchronous. Output can be read from the returned channel.

Types

type CollectionBucket 

type CollectionBucket struct {
	// contains filtered or unexported fields
}

CollectionBucket is an in-memory implementation of a BucketStore. Individual collections are implemented as standard WalrusBuckets (in-memory DataStore implementations).

func GetCollectionBucket 

func GetCollectionBucket(url, bucketName string) (*CollectionBucket, error)

func NewCollectionBucket 

func NewCollectionBucket(bucketName string) *CollectionBucket

func (*CollectionBucket) Close 

func (cb *CollectionBucket) Close(ctx context.Context)

func (*CollectionBucket) CloseAndDelete 

func (cb *CollectionBucket) CloseAndDelete() error

CloseAndDelete calls closeAndDelete on the underlying buckets when using persisted buckets

func (*CollectionBucket) CreateDataStore 

func (wh *CollectionBucket) CreateDataStore(_ context.Context, name sgbucket.DataStoreName) error

func (*CollectionBucket) DefaultDataStore 

func (wh *CollectionBucket) DefaultDataStore() sgbucket.DataStore

func (*CollectionBucket) DropDataStore 

func (wh *CollectionBucket) DropDataStore(name sgbucket.DataStoreName) error

func (*CollectionBucket) GetCollectionID 

func (wh *CollectionBucket) GetCollectionID(scope, collection string) (uint32, error)

func (*CollectionBucket) GetMaxVbno 

func (wh *CollectionBucket) GetMaxVbno() (uint16, error)

func (*CollectionBucket) GetName 

func (wh *CollectionBucket) GetName() string

func (*CollectionBucket) IsError 

func (wh *CollectionBucket) IsError(err error, errorType sgbucket.DataStoreErrorType) bool

func (*CollectionBucket) IsSupported 

func (wh *CollectionBucket) IsSupported(feature sgbucket.BucketStoreFeature) bool

func (*CollectionBucket) ListDataStores 

func (wh *CollectionBucket) ListDataStores() ([]sgbucket.DataStoreName, error)

func (*CollectionBucket) NamedDataStore 

func (wh *CollectionBucket) NamedDataStore(name sgbucket.DataStoreName) (sgbucket.DataStore, error)

func (*CollectionBucket) StartDCPFeed 

func (wh *CollectionBucket) StartDCPFeed(ctx context.Context, args sgbucket.FeedArguments, callback sgbucket.FeedEventCallbackFunc, dbStats *expvar.Map) error

StartDCPFeed implements a multi-collection feed by calling StartDCPFeed for each requested collection. Each collection's DCP feed runs its own goroutine, callback may be invoked concurrently by these goroutines.

func (*CollectionBucket) StartTapFeed 

func (wh *CollectionBucket) StartTapFeed(args sgbucket.FeedArguments, dbStats *expvar.Map) (sgbucket.MutationFeed, error)

func (*CollectionBucket) UUID 

func (wh *CollectionBucket) UUID() (string, error)

type DocTooBigErr 

type DocTooBigErr struct{}

func (DocTooBigErr) Error 

func (err DocTooBigErr) Error() string

type JSONCollator 

type JSONCollator struct {
	// contains filtered or unexported fields
}

Context for JSON collation. This struct is not thread-safe (or rather, its embedded string collator isn't) so it should only be used on one goroutine at a time.

func (*JSONCollator) Clear 

func (c *JSONCollator) Clear()

func (*JSONCollator) Collate 

func (c *JSONCollator) Collate(key1, key2 interface{}) int

CouchDB-compatible collation/comparison of JSON values. See: http://wiki.apache.org/couchdb/View_collation#Collation_Specification

func (*JSONCollator) CollateRaw 

func (c *JSONCollator) CollateRaw(key1, key2 []byte) int

Collates raw JSON data without unmarshaling it. THE INPUTS MUST BE VALID JSON, WITH NO WHITESPACE! Invalid input will result in a panic, or perhaps just bogus output.

type Pipeline 

type Pipeline struct {
	// contains filtered or unexported fields
}

type PipelineFunc 

type PipelineFunc func(input jsMapFunctionInput, output chan<- interface{})

type WalrusBucket 

type WalrusBucket struct {
	// contains filtered or unexported fields
}

Simple, inefficient in-memory implementation of Bucket interface. http://ihasabucket.com

func GetBucket 

func GetBucket(url, poolName, bucketName string) (*WalrusBucket, error)

Returns a Walrus-based Bucket specific to the given (url, pool, bucketname) tuple. That is, passing the same parameters will return the same Bucket.

If the urlStr has any of the forms below it will be considered a filesystem directory path, and the bucket will use a persistent backing file in that directory. 

walrus:/foo/bar
walrus:bar
file:///foo/bar
/foo/bar
./bar

The bucket's filename will be "bucketName.walrus", or if the poolName is not "default", "poolName-bucketName.walrus".

If the URL isn't considered a directory (e.g. "walrus:" or ""), the bucket will just be created in memory using NewBucket.

func NewBucket 

func NewBucket(bucketName string) *WalrusBucket

Creates a simple in-memory Bucket, suitable only for amusement purposes & testing. The Bucket is created empty. There is no way to save it persistently.

func NewPersistentBucket 

func NewPersistentBucket(dir, poolName, bucketName string) (*WalrusBucket, error)

Loads or creates a persistent bucket in the given filesystem directory. The bucket's backing file will be named "bucketName.walrus", or if the poolName is not empty "default", "poolName-bucketName.walrus".

func (*WalrusBucket) Add 

func (bucket *WalrusBucket) Add(k string, exp uint32, v interface{}) (added bool, err error)

func (*WalrusBucket) AddRaw 

func (bucket *WalrusBucket) AddRaw(k string, exp uint32, v []byte) (added bool, err error)

func (*WalrusBucket) Append 

func (bucket *WalrusBucket) Append(k string, data []byte) error

func (*WalrusBucket) Close 

func (bucket *WalrusBucket) Close(_ context.Context)

func (*WalrusBucket) CloseAndDelete 

func (bucket *WalrusBucket) CloseAndDelete() error

func (*WalrusBucket) CouchbaseServerVersion 

func (bucket *WalrusBucket) CouchbaseServerVersion() (major uint64, minor uint64, micro string)

func (*WalrusBucket) DefaultDataStore 

func (bucket *WalrusBucket) DefaultDataStore() sgbucket.DataStore

func (*WalrusBucket) Delete 

func (bucket *WalrusBucket) Delete(k string) error

func (*WalrusBucket) DeleteDDoc 

func (bucket *WalrusBucket) DeleteDDoc(docname string) error

func (*WalrusBucket) DeleteWithXattr 

func (bucket *WalrusBucket) DeleteWithXattr(_ context.Context, k string, xattrKey string) error

func (*WalrusBucket) DeleteXattrs 

func (bucket *WalrusBucket) DeleteXattrs(_ context.Context, k string, xattrKeys ...string) error

func (*WalrusBucket) Dump 

func (bucket *WalrusBucket) Dump()

func (*WalrusBucket) Exists 

func (bucket *WalrusBucket) Exists(k string) (ok bool, err error)

func (*WalrusBucket) Get 

func (bucket *WalrusBucket) Get(k string, rv interface{}) (cas uint64, err error)

func (*WalrusBucket) GetAndTouchRaw 

func (bucket *WalrusBucket) GetAndTouchRaw(k string, exp uint32) (rv []byte, cas uint64, err error)

func (*WalrusBucket) GetBulkRaw 

func (bucket *WalrusBucket) GetBulkRaw(keys []string) (map[string][]byte, error)

func (*WalrusBucket) GetDDoc 

func (bucket *WalrusBucket) GetDDoc(docname string) (ddoc sgbucket.DesignDoc, err error)

func (*WalrusBucket) GetDDocs 

func (bucket *WalrusBucket) GetDDocs() (ddocs map[string]sgbucket.DesignDoc, err error)

func (*WalrusBucket) GetExpiry 

func (bucket *WalrusBucket) GetExpiry(_ context.Context, k string) (expiry uint32, getMetaError error)

func (*WalrusBucket) GetMaxVbno 

func (bucket *WalrusBucket) GetMaxVbno() (uint16, error)

func (*WalrusBucket) GetName 

func (bucket *WalrusBucket) GetName() string

func (*WalrusBucket) GetRaw 

func (bucket *WalrusBucket) GetRaw(k string) (rv []byte, cas uint64, err error)

func (*WalrusBucket) GetStatsVbSeqno 

func (bucket *WalrusBucket) GetStatsVbSeqno(maxVbno uint16, useAbsHighSeqNo bool) (uuids map[uint16]uint64, highSeqnos map[uint16]uint64, seqErr error)

func (*WalrusBucket) GetSubDocRaw 

func (bucket *WalrusBucket) GetSubDocRaw(_ context.Context, k string, subdocKey string) (value []byte, casOut uint64, err error)

GetSubDocRaw Walrus implementation only works with a top-level subdocKey

func (*WalrusBucket) GetWithXattr 

func (bucket *WalrusBucket) GetWithXattr(_ context.Context, k string, xattrKey string, userXattrKey string, rv interface{}, xv interface{}, uxv interface{}) (cas uint64, err error)

func (*WalrusBucket) GetXattr 

func (bucket *WalrusBucket) GetXattr(_ context.Context, k string, xattrKey string, xv interface{}) (casOut uint64, err error)

func (*WalrusBucket) Incr 

func (bucket *WalrusBucket) Incr(k string, amt, def uint64, exp uint32) (uint64, error)

func (*WalrusBucket) IsError 

func (bucket *WalrusBucket) IsError(err error, errorType sgbucket.DataStoreErrorType) bool

func (*WalrusBucket) IsSupported 

func (bucket *WalrusBucket) IsSupported(feature sgbucket.BucketStoreFeature) bool

func (*WalrusBucket) ListDataStores 

func (bucket *WalrusBucket) ListDataStores() ([]sgbucket.DataStoreName, error)

func (*WalrusBucket) NamedDataStore 

func (bucket *WalrusBucket) NamedDataStore(name sgbucket.DataStoreName) (sgbucket.DataStore, error)

func (*WalrusBucket) PutDDoc 

func (bucket *WalrusBucket) PutDDoc(_ context.Context, docname string, design *sgbucket.DesignDoc) error

func (*WalrusBucket) Refresh 

func (bucket *WalrusBucket) Refresh() error

func (*WalrusBucket) Remove 

func (bucket *WalrusBucket) Remove(k string, cas uint64) (casOut uint64, err error)

func (*WalrusBucket) RemoveXattr 

func (bucket *WalrusBucket) RemoveXattr(_ context.Context, k string, xattrKey string, cas uint64) error

func (*WalrusBucket) Set 

func (bucket *WalrusBucket) Set(k string, exp uint32, _ *sgbucket.UpsertOptions, v interface{}) error

func (*WalrusBucket) SetBulk 

func (bucket *WalrusBucket) SetBulk(entries []*sgbucket.BulkSetEntry) (err error)

func (*WalrusBucket) SetRaw 

func (bucket *WalrusBucket) SetRaw(k string, exp uint32, _ *sgbucket.UpsertOptions, v []byte) error

func (*WalrusBucket) SetVbAndSeq 

func (bucket *WalrusBucket) SetVbAndSeq(doc *walrusDoc, k string) (err error)

func (*WalrusBucket) SetXattr 

func (bucket *WalrusBucket) SetXattr(_ context.Context, k string, xattrKey string, xv []byte) (casOut uint64, err error)

func (*WalrusBucket) StartDCPFeed 

func (bucket *WalrusBucket) StartDCPFeed(ctx context.Context, args sgbucket.FeedArguments, callback sgbucket.FeedEventCallbackFunc, dbStats *expvar.Map) error

Until a full DCP implementation is available, walrus wraps tap feed to invoke callback

func (*WalrusBucket) StartTapFeed 

func (bucket *WalrusBucket) StartTapFeed(args sgbucket.FeedArguments, dbStats *expvar.Map) (sgbucket.MutationFeed, error)

Starts a TAP feed on a client connection. The events can be read from the returned channel. To stop receiving events, call Close() on the feed.

func (*WalrusBucket) SubdocInsert 

func (bucket *WalrusBucket) SubdocInsert(_ context.Context, docID string, fieldPath string, cas uint64, value interface{}) error

func (*WalrusBucket) Touch 

func (bucket *WalrusBucket) Touch(k string, exp uint32) (casOut uint64, err error)

func (*WalrusBucket) UUID 

func (bucket *WalrusBucket) UUID() (string, error)

func (*WalrusBucket) Update 

func (bucket *WalrusBucket) Update(k string, exp uint32, callback sgbucket.UpdateFunc) (casOut uint64, err error)

func (*WalrusBucket) VBHash 

func (bucket *WalrusBucket) VBHash(docID string) uint32

func (*WalrusBucket) View 

func (bucket *WalrusBucket) View(_ context.Context, docName, viewName string, params map[string]interface{}) (sgbucket.ViewResult, error)

func (*WalrusBucket) ViewCustom 

func (bucket *WalrusBucket) ViewCustom(ctx context.Context, ddoc, name string, params map[string]interface{}, vres interface{}) error

func (*WalrusBucket) ViewQuery 

func (bucket *WalrusBucket) ViewQuery(ctx context.Context, ddoc, name string, params map[string]interface{}) (sgbucket.QueryResultIterator, error)

func (*WalrusBucket) Write 

func (bucket *WalrusBucket) Write(k string, flags int, exp uint32, v interface{}, opt sgbucket.WriteOptions) (err error)

func (*WalrusBucket) WriteCas 

func (bucket *WalrusBucket) WriteCas(k string, flags int, exp uint32, cas uint64, v interface{}, opt sgbucket.WriteOptions) (casOut uint64, err error)

func (*WalrusBucket) WriteCasWithXattr 

func (bucket *WalrusBucket) WriteCasWithXattr(_ context.Context, k string, xattrKey string, exp uint32, cas uint64, opts *sgbucket.MutateInOptions, v interface{}, xv interface{}) (casOut uint64, err error)

func (*WalrusBucket) WriteSubDoc 

func (bucket *WalrusBucket) WriteSubDoc(_ context.Context, k string, subdocKey string, cas uint64, value []byte) (casOut uint64, err error)

WriteSubDoc Walrus implementation only works with a top-level subdocKey

func (*WalrusBucket) WriteUpdate 

func (bucket *WalrusBucket) WriteUpdate(k string, exp uint32, callback sgbucket.WriteUpdateFunc) (casOut uint64, err error)

func (*WalrusBucket) WriteUpdateWithXattr 

func (bucket *WalrusBucket) WriteUpdateWithXattr(_ context.Context, k string, xattrKey string, userXattrKey string, exp uint32, opts *sgbucket.MutateInOptions, previous *sgbucket.BucketDocument, callback sgbucket.WriteUpdateWithXattrFunc) (casOut uint64, err error)

func (*WalrusBucket) WriteWithXattr 

func (bucket *WalrusBucket) WriteWithXattr(_ context.Context, k string, xattrKey string, exp uint32, cas uint64, opts *sgbucket.MutateInOptions, value []byte, xattrValue []byte, isDelete bool, deleteBody bool) (casOut uint64, err error)

type WalrusCollection 

type WalrusCollection struct {
	*WalrusBucket
	FQName       scopeAndCollection // Fully qualified collection name (scope and collection)
	CollectionID uint32             // Unique collectionID
}

A WalrusCollection wraps a walrus single key value store to add metadata

func (*WalrusCollection) CollectionName 

func (wh *WalrusCollection) CollectionName() string

func (*WalrusCollection) ScopeName 

func (wh *WalrusCollection) ScopeName() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.