cbft

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 18, 2015 License: Apache-2.0 Imports: 46 Imported by: 0

README

cbft

Couchbase Full Text server

The cbft project integrates the bleve full-text search engine and Couchbase Server.

GoDoc Build Status Coverage Status

A cbft process creates and maintains full-text indexes using the bleve full-text indexing engine.

Data sources for indexing include Couchbase 3.0+ and Couchbase 4.0+ buckets.

Full-text indexes can be optionally partitioned across multiple cbft processes.

Queries on a cbft index will be scatter/gather'ed across the relevant, distributed cbft processes.

Licenses

Getting started and documentation

Please see the getting started guide for cbft, available at http://labs.couchbase.com/cbft.

For developers / contributors

Please see the README for developers

Documentation

Index

Constants

View Source 
const BLEVE_DEST_INITIAL_BUF_SIZE_BYTES = 40 * 1024 // 40K.
View Source 
const DEST_EXTRAS_TYPE_DCP = DestExtrasType(0x0002)

DEST_EXTRAS_TYPE_DCP represents the extras that comes from DCP protocol.

View Source 
const DEST_EXTRAS_TYPE_NIL = DestExtrasType(0)

DEST_EXTRAS_TYPE_NIL means there are no extras as part of a Dest.DataUpdate/DataDelete invocation.

View Source 
const DEST_EXTRAS_TYPE_TAP = DestExtrasType(0x0001)

DEST_EXTRAS_TYPE_TAP represents the extras that comes from TAP protocol.

View Source 
const FEED_BACKOFF_FACTOR = 1.5
View Source 
const FEED_SLEEP_INIT_MS = 100
View Source 
const FEED_SLEEP_MAX_MS = 10000

Default values for feed parameters.

View Source 
const FILES_FEED_BACKOFF_FACTOR = 1.5
View Source 
const FILES_FEED_MAX_SLEEP_MS = 1000 * 60 * 5 // 5 minutes.
View Source 
const FILES_FEED_SLEEP_START_MS = 5000
View Source 
const INDEX_DEFS_KEY = "indexDefs"

INDEX_DEFS_KEY is the key used for Cfg access.

View Source 
const INDEX_NAME_REGEXP = `^[A-Za-z][0-9A-Za-z_\-]*$`

INDEX_NAME_REGEXP is used to validate index definition names.

View Source 
const JANITOR_CLOSE_PINDEX = "janitor_close_pindex"
View Source 
const JANITOR_REMOVE_PINDEX = "janitor_remove_pindex"
View Source 
const MANAGER_MAX_EVENTS = 10

MANAGER_MAX_EVENTS limits the number of events tracked by a Manager for diagnosis/debugging.

View Source 
const NODE_DEFS_KEY = "nodeDefs" // NODE_DEFS_KEY is used for Cfg access.
View Source 
const NODE_DEFS_KNOWN = "known" // NODE_DEFS_KNOWN is used for Cfg access.
View Source 
const NODE_DEFS_WANTED = "wanted" // NODE_DEFS_WANTED is used for Cfg access.
View Source 
const PINDEX_META_FILENAME string = "PINDEX_META"
View Source 
const PLAN_PINDEXES_KEY = "planPIndexes"

PLAN_PINDEXES_KEY is used for Cfg access.

View Source 
const QUERY_CTL_DEFAULT_TIMEOUT_MS = int64(10000)

QUERY_CTL_DEFAULT_TIMEOUT_MS is the default query timeout.

View Source 
const VERSION = "3.1.0"

The cbft.VERSION tracks persistence versioning (schema/format of persisted data and configuration). The main.VERSION (see cmd/cbft/...), in contrast, is an overall "product" version. For example, we might introduce new UI-only features or fix a UI typo, in which case we'd bump the main.VERSION number; but, if the persisted data/config format was unchanged, then the cbft.VERSION number should remain unchanged.

NOTE: You *must* update cbft.VERSION if you change what's stored in the Cfg (such as the JSON/struct definitions or the planning algorithms).

View Source 
const VERSION_KEY = "version"
View Source 
const WORK_KICK = "kick"
View Source 
const WORK_NOOP = ""

Variables

View Source 
var EMPTY_BYTES = []byte{}
View Source 
var FeedTypes = make(map[string]*FeedType) // Key is sourceType.

FeedTypes is a global registry of available feed types and is initialized on startup. It should be immutable after startup time.

View Source 
var FileNotReadable = errors.New("file is not open for reading")
View Source 
var FileNotWritable = errors.New("file is not open for writing")
View Source 
var PINDEX_STORE_MAX_ERRORS = 40

PINDEX_STORE_MAX_ERRORS is the max number of errors that a PIndexStoreStats will track.

View Source 
var PIndexImplTypes = make(map[string]*PIndexImplType)

PIndexImplTypes is a global registry of pindex type backends or implementations. It is keyed by indexType and should be treated as immutable/read-only after process init/startup.

Functions

func Asset added in v0.0.1 

func Asset(name string) ([]byte, error)

Asset loads and returns the asset for the given name. It returns an error if the asset could not be found or could not be loaded.

func AssetDir added in v0.0.1 

func AssetDir(name string) ([]string, error)

AssetDir returns the file names below a certain directory embedded in the file by go-bindata. For example if you run go-bindata on data/... and data contains the following hierarchy: 

data/
  foo.txt
  img/
    a.png
    b.png

then AssetDir("data") would return []string{"foo.txt", "img"} AssetDir("data/img") would return []string{"a.png", "b.png"} AssetDir("foo.txt") and AssetDir("notexist") would return an error AssetDir("") will return []string{"data"}.

func AssetFS added in v0.0.5 

func AssetFS() *assetfs.AssetFS

AssetFS returns the assetfs.AssetFS "filesystem" that holds static HTTP resources (css/html/js/images, etc) for the web UI.

Users might introduce their own static HTTP resources and override resources from AssetFS() with their own resource lookup chaining.

func AssetInfo added in v0.0.1 

func AssetInfo(name string) (os.FileInfo, error)

AssetInfo loads and returns the asset info for the given name. It returns an error if the asset could not be found or could not be loaded.

func AssetNames added in v0.0.1 

func AssetNames() []string

AssetNames returns the names of the assets.

func AtomicCopyMetrics added in v0.0.1 

func AtomicCopyMetrics(s, r interface{},
	fn func(sv uint64, rv uint64) uint64)

AtomicCopyMetrics copies uint64 metrics from s to r (from source to result), and also applies an optional fn function to each metric. The fn is invoked with metrics from s and r, and can be used to compute additions, subtractions, etc. When fn is nil, AtomicCopyTo defaults to just a straight copier.

func BleveMetaExtra added in v0.1.0 

func BleveMetaExtra(m map[string]interface{})

func BlevePIndexImplInitRouter added in v0.1.0 

func BlevePIndexImplInitRouter(r *mux.Router, phase string)

func CalcPIndexesDelta 

func CalcPIndexesDelta(mgrUUID string,
	currPIndexes map[string]*PIndex,
	wantedPlanPIndexes *PlanPIndexes) (
	addPlanPIndexes []*PlanPIndex,
	removePIndexes []*PIndex)

Functionally determine the delta of which pindexes need creation and which should be shut down on our local node (mgrUUID).

func CfgNodeDefsKey added in v0.0.1 

func CfgNodeDefsKey(kind string) string

CfgNodeDefsKey returns the Cfg access key for a NodeDef kind.

func CfgSetIndexDefs 

func CfgSetIndexDefs(cfg Cfg, indexDefs *IndexDefs, cas uint64) (uint64, error)

Updates index definitions on a Cfg provider.

func CfgSetNodeDefs 

func CfgSetNodeDefs(cfg Cfg, kind string, nodeDefs *NodeDefs,
	cas uint64) (uint64, error)

Updates node definitions on a Cfg provider.

func CfgSetPlanPIndexes 

func CfgSetPlanPIndexes(cfg Cfg, planPIndexes *PlanPIndexes, cas uint64) (
	uint64, error)

Updates PlanPIndexes on a Cfg provider.

func CheckVersion 

func CheckVersion(cfg Cfg, myVersion string) (bool, error)

Returns true if a given version is modern enough to modify the Cfg. Older versions (which are running with older JSON/struct defintions or planning algorithms) will see false from their CheckVersion()'s.

func ConsistencyWaitDone added in v0.0.1 

func ConsistencyWaitDone(partition string,
	cancelCh <-chan bool,
	doneCh chan error,
	currSeq func() uint64) error

ConsistencyWaitDone() waits for either the cancelCh or doneCh to finish, and provides the partition's seq if it was the cancelCh.

func ConsistencyWaitGroup added in v0.0.1 

func ConsistencyWaitGroup(indexName string,
	consistencyParams *ConsistencyParams, cancelCh <-chan bool,
	localPIndexes []*PIndex,
	addLocalPIndex func(*PIndex) error) error

ConsistencyWaitGroup waits for all the partitions from a group of pindexes to reach a required consistency level.

func ConsistencyWaitPIndex added in v0.0.1 

func ConsistencyWaitPIndex(pindex *PIndex, t ConsistencyWaiter,
	consistencyParams *ConsistencyParams, cancelCh <-chan bool) error

ConsistencyWaitPIndex waits for all the partitions in a pindex to reach the required consistency level.

func ConsistencyWaitPartitions added in v0.0.1 

func ConsistencyWaitPartitions(
	t ConsistencyWaiter,
	partitions map[string]bool,
	consistencyLevel string,
	consistencyVector map[string]uint64,
	cancelCh <-chan bool) error

ConsistencyWaitPartitions waits for the given partitions to reach the required consistency level.

func CouchbaseParseSourceName added in v0.0.5 

func CouchbaseParseSourceName(
	serverURLDefault, poolNameDefault, sourceName string) (
	string, string, string)

CouchbaseParseSourceName parses a sourceName, if it's a couchbase REST/HTTP URL, into a server URL, poolName and bucketName. Otherwise, returns the serverURLDefault, poolNameDefault, and treat the sourceName as a bucketName.

func CouchbasePartitions added in v0.0.1 

func CouchbasePartitions(sourceType, sourceName, sourceUUID, sourceParams,
	serverIn string) (partitions []string, err error)

CouchbasePartitions parses a sourceParams for a couchbase data-source/feed.

func CountAlias added in v0.0.1 

func CountAlias(mgr *Manager, indexName, indexUUID string) (uint64, error)

func CountBlevePIndexImpl added in v0.0.1 

func CountBlevePIndexImpl(mgr *Manager, indexName, indexUUID string) (
	uint64, error)

func DataSourcePartitions added in v0.0.1 

func DataSourcePartitions(sourceType, sourceName, sourceUUID, sourceParams,
	server string) ([]string, error)

DataSourcePartitions is a helper function that returns the data source partitions for a named data source or feed type.

func ErrorToString added in v0.0.1 

func ErrorToString(e error) string

ErrorToString is a helper func that returns e.Error(), but also returns "" for nil error.

func ExponentialBackoffLoop 

func ExponentialBackoffLoop(name string,
	f func() int,
	startSleepMS int,
	backoffFactor float32,
	maxSleepMS int)

Calls f() in a loop, sleeping in an exponential backoff if needed. The provided f() function should return < 0 to stop the loop; >= 0 to continue the loop, where > 0 means there was progress which allows an immediate retry of f() with no sleeping. A return of < 0 is useful when f() will never make any future progress.

func FeedName 

func FeedName(pindex *PIndex) string

FeedName functionally computes the name of a feed given a pindex.

NOTE: We're depending on the IndexName/IndexUUID to "cover" the SourceType, SourceName, SourceUUID, SourceParams values, so we don't need to encode those source parts into the feed name.

func FilesFeedPartitions added in v0.1.0 

func FilesFeedPartitions(sourceType, sourceName, sourceUUID, sourceParams,
	server string) ([]string, error)

FilesFeedPartitions returns the partitions, controlled by FilesFeedParams.NumPartitions, for a FilesFeed instance.

func FilesFindMatches added in v0.1.0 

func FilesFindMatches(dataDir, sourceName string,
	regExps []string, modTimeGTE time.Time, maxSize int64) (
	[]string, error)

FilesFindMatches finds all leaf file paths in a subdirectory tree that match any in an optional array of regExps (regular expression strings). If regExps is nil, though, then all leaf file paths are considered as a potential candidate match. The regExps are with respect to a path from filepath.Walk().

Additionally, a candidate file must have been modified since a modTimeGTE and (if maxSize is > 0) should have size that's <= maxSize.

func FilesPathToPartition added in v0.1.0 

func FilesPathToPartition(h hash.Hash32,
	partitions []string, path string) string

FilesPathToPartition hashes a file path to a partition.

func IndentJSON added in v0.0.4 

func IndentJSON(x interface{}, prefix, indent string) string

IndentJSON is a helper func that returns indented JSON for its interface{} x parameter.

func InitManagerRESTRouter added in v0.0.5 

func InitManagerRESTRouter(r *mux.Router, versionMain string,
	mgr *Manager, staticDir, staticETag string, mr *MsgRing) (
	*mux.Router, map[string]RESTMeta, error)

InitManagerRESTRouter initializes a mux.Router with REST API routes.

func InitStaticFileRouter added in v0.1.0 

func InitStaticFileRouter(r *mux.Router, staticDir, staticETag string,
	pages []string) *mux.Router

InitStaticFileRouter adds static HTTP resource routes to a router.

func MustAsset added in v0.0.1 

func MustAsset(name string) []byte

MustAsset is like Asset but panics when Asset would return an error. It simplifies safe initialization of global variables.

func NewBlackHolePIndexImpl added in v0.0.1 

func NewBlackHolePIndexImpl(indexType, indexParams,
	path string, restart func()) (PIndexImpl, Dest, error)

func NewBlevePIndexImpl added in v0.0.1 

func NewBlevePIndexImpl(indexType, indexParams, path string,
	restart func()) (PIndexImpl, Dest, error)

func NewManagerRESTRouter 

func NewManagerRESTRouter(versionMain string, mgr *Manager,
	staticDir, staticETag string, mr *MsgRing) (
	*mux.Router, map[string]RESTMeta, error)

NewManagerRESTRouter creates a mux.Router initialized with the REST API and web UI routes. See also InitStaticFileRouter and InitManagerRESTRouter if you need finer control of the router initialization.

func NewPIndexImpl added in v0.0.1 

func NewPIndexImpl(indexType, indexParams, path string, restart func()) (
	PIndexImpl, Dest, error)

NewPIndexImpl creates an index partition of the given, registered index type.

func NewUUID 

func NewUUID() string

func OpenBlackHolePIndexImpl added in v0.0.1 

func OpenBlackHolePIndexImpl(indexType, path string, restart func()) (
	PIndexImpl, Dest, error)

func OpenBlevePIndexImpl added in v0.0.1 

func OpenBlevePIndexImpl(indexType, path string,
	restart func()) (PIndexImpl, Dest, error)

func OpenPIndexImpl added in v0.0.1 

func OpenPIndexImpl(indexType, path string, restart func()) (
	PIndexImpl, Dest, error)

OpenPIndexImpl loads an index partition of the given, registered index type from a given path.

func PIndexMatchesPlan 

func PIndexMatchesPlan(pindex *PIndex, planPIndex *PlanPIndex) bool

Returns true if both the PIndex meets the PlanPIndex, ignoring UUID.

func PIndexPath 

func PIndexPath(dataDir, pindexName string) string

Computes the storage path for a pindex.

func PIndexTypesInitRouter added in v0.1.0 

func PIndexTypesInitRouter(r *mux.Router, phase string)

PIndexTypesInitRouter initializes a mux.Router with the REST API routes provided by registered pindex types.

func ParsePIndexPath 

func ParsePIndexPath(dataDir, pindexPath string) (string, bool)

Retrieves a pindex name from a pindex path.

func ParsePartitionsToVBucketIds added in v0.0.1 

func ParsePartitionsToVBucketIds(dests map[string]Dest) ([]uint16, error)

ParsePartitionsToVBucketIds is specific to couchbase data-sources/feeds, converting a set of partition strings from a dests map to vbucketId numbers.

func PlanPIndexName added in v0.0.1 

func PlanPIndexName(indexDef *IndexDef, sourcePartitions string) string

NOTE: PlanPIndex.Name must be unique across the cluster and ideally functionally based off of the indexDef so that the SamePlanPIndex() comparison works even if concurrent planners are racing to calculate plans.

NOTE: We can't use sourcePartitions directly as part of a PlanPIndex.Name suffix because in vbucket/hash partitioning the string would be too long -- since PIndexes might use PlanPIndex.Name for filesystem paths.

func PlanPIndexNodeCanRead added in v0.0.1 

func PlanPIndexNodeCanRead(p *PlanPIndexNode) bool

PlanPIndexNodeCanRead returns true if PlanPIndexNode.CanRead is true; it's useful as a filter arg for Manager.CoveringPIndexes().

func PlanPIndexNodeCanWrite added in v0.0.1 

func PlanPIndexNodeCanWrite(p *PlanPIndexNode) bool

PlanPIndexNodeCanWrite returns true if PlanPIndexNode.CanWrite is true; it's useful as a filter arg for Manager.CoveringPIndexes().

func PlanPIndexNodeOk added in v0.0.3 

func PlanPIndexNodeOk(p *PlanPIndexNode) bool

PlanPIndexNodeOk always returns true; it's useful as a filter arg for Manager.CoveringPIndexes().

func PlannerCheckVersion added in v0.0.1 

func PlannerCheckVersion(cfg Cfg, version string) error

PlannerCheckVersion errors if a version string is too low.

func PrimaryFeedPartitions added in v0.0.1 

func PrimaryFeedPartitions(sourceType, sourceName, sourceUUID, sourceParams,
	server string) ([]string, error)

PrimaryFeedPartitions generates partition strings based on a PrimarySourceParams.NumPartitions parameter.

func QueryAlias added in v0.0.1 

func QueryAlias(mgr *Manager, indexName, indexUUID string,
	req []byte, res io.Writer) error

func QueryBlevePIndexImpl added in v0.0.1 

func QueryBlevePIndexImpl(mgr *Manager, indexName, indexUUID string,
	req []byte, res io.Writer) error

func RegisterFeedType added in v0.0.1 

func RegisterFeedType(sourceType string, f *FeedType)

RegisterFeedType is invoked at init/startup time to register a FeedType.

func RegisterPIndexImplType added in v0.0.1 

func RegisterPIndexImplType(indexType string, t *PIndexImplType)

RegisterPIndexImplType registers a index type into the system.

func RestoreAsset added in v0.0.1 

func RestoreAsset(dir, name string) error

Restore an asset under the given directory

func RestoreAssets added in v0.0.1 

func RestoreAssets(dir, name string) error

Restore assets under the given directory recursively

func RewriteURL 

func RewriteURL(to string, h http.Handler) http.Handler

RewriteURL is a helper function that returns a URL path rewriter HandlerFunc, rewriting the URL path to a provided "to" string.

func SamePlanPIndex 

func SamePlanPIndex(a, b *PlanPIndex) bool

Returns true if both PlanPIndex are the same, ignoring PlanPIndex.UUID.

func SamePlanPIndexes 

func SamePlanPIndexes(a, b *PlanPIndexes) bool

Returns true if both PlanPIndexes are the same, where we ignore any differences in UUID or ImplVersion.

func StartDCPFeed added in v0.0.1 

func StartDCPFeed(mgr *Manager, feedName, indexName, indexUUID,
	sourceType, sourceName, bucketUUID, params string,
	dests map[string]Dest) error

StartDCPFeed starts a DCP related feed and is registered at init/startup time with the system via RegisterFeedType().

func StartFilesFeed added in v0.1.0 

func StartFilesFeed(mgr *Manager, feedName, indexName, indexUUID,
	sourceType, sourceName, sourceUUID, params string,
	dests map[string]Dest) error

StartFilesFeed starts a FilesFeed and is the the callback function registered at init/startup time.

func StartTAPFeed added in v0.0.1 

func StartTAPFeed(mgr *Manager, feedName, indexName, indexUUID,
	sourceType, sourceName, bucketUUID, params string,
	dests map[string]Dest) error

StartDCPFeed starts a TAP related feed and is registered at init/startup time with the system via RegisterFeedType().

func StringsIntersectStrings added in v0.0.1 

func StringsIntersectStrings(a, b []string) []string

StringsIntersectStrings returns a brand new array that has the intersection of a and b.

func StringsRemoveStrings added in v0.0.1 

func StringsRemoveStrings(stringArr, removeArr []string) []string

StringsRemoveStrings returns a copy of stringArr, but with some strings removed, keeping the same order as stringArr.

func StringsToMap added in v0.0.1 

func StringsToMap(strsArr []string) map[string]bool

StringsToMap connverts an array of (perhaps duplicated) strings into a map with key of those strings and values of true, and is useful for simple set-like operations.

func SubsetPlanPIndexes 

func SubsetPlanPIndexes(a, b *PlanPIndexes) bool

Returns true if PlanPIndex children in a are a subset of those in b, using SamePlanPIndex() for sameness comparion.

func Time added in v0.0.1 

func Time(f func() error,
	totalDuration, totalCount, maxDuration *uint64) error

Time invokes a func f and updates the totalDuration, totalCount and maxDuration metrics. See also Timer() for a metrics based alternative.

func TimeoutCancelChan added in v0.0.1 

func TimeoutCancelChan(timeout int64) <-chan bool

TimeoutCancelChan creates a channel that closes after a given timeout in milliseconds.

func Timer added in v0.0.1 

func Timer(f func() error, t metrics.Timer) error

Timer updates a metrics.Timer. Unlike metrics.Timer.Time(), this version also captures any error return value.

func ValidateAlias added in v0.0.1 

func ValidateAlias(indexType, indexName, indexParams string) error

func ValidateBlevePIndexImpl added in v0.0.1 

func ValidateBlevePIndexImpl(indexType, indexName, indexParams string) error

func VersionGTE 

func VersionGTE(x, y string) bool

Compares two dotted versioning strings, like "1.0.1" and "1.2.3". Returns true when x >= y.

TODO: Need to handle non-numeric parts?

func WriteTimerJSON added in v0.0.1 

func WriteTimerJSON(w io.Writer, timer metrics.Timer)

WriteTimerJSON writes a metrics.Timer instance as JSON to a io.Writer.

Types

type AliasParams added in v0.0.1 

type AliasParams struct {
	Targets map[string]*AliasParamsTarget `json:"targets"` // Keyed by indexName.
}

AliasParams holds the definition for a user-defined index alias. A user-defined index alias can be used as a level of indirection (the "LastQuartersSales" alias points currently to the "2014-Q3-Sales" index, but the administrator might repoint it in the future without changing the application) or to scatter-gather or fan-out a query across multiple real indexes (e.g., to query across customer records, product catalog, call-center records, etc, in one shot).

type AliasParamsTarget added in v0.0.1 

type AliasParamsTarget struct {
	IndexUUID string `json:"indexUUID"` // Optional.
}

type BlackHole added in v0.0.1 

type BlackHole struct {
	// contains filtered or unexported fields
}

Implements both Dest and PIndexImpl interfaces.

func (*BlackHole) Close added in v0.0.1 

func (t *BlackHole) Close() error

func (*BlackHole) ConsistencyWait added in v0.0.1 

func (t *BlackHole) ConsistencyWait(partition, partitionUUID string,
	consistencyLevel string,
	consistencySeq uint64,
	cancelCh <-chan bool) error

func (*BlackHole) Count added in v0.0.1 

func (t *BlackHole) Count(pindex *PIndex,
	cancelCh <-chan bool) (uint64, error)

func (*BlackHole) DataDelete added in v0.0.3 

func (t *BlackHole) DataDelete(partition string,
	key []byte, seq uint64,
	cas uint64,
	extrasType DestExtrasType, extras []byte) error

func (*BlackHole) DataUpdate added in v0.0.3 

func (t *BlackHole) DataUpdate(partition string,
	key []byte, seq uint64, val []byte,
	cas uint64,
	extrasType DestExtrasType, extras []byte) error

func (*BlackHole) OpaqueGet added in v0.0.3 

func (t *BlackHole) OpaqueGet(partition string) (
	value []byte, lastSeq uint64, err error)

func (*BlackHole) OpaqueSet added in v0.0.3 

func (t *BlackHole) OpaqueSet(partition string, value []byte) error

func (*BlackHole) Query added in v0.0.1 

func (t *BlackHole) Query(pindex *PIndex, req []byte, w io.Writer,
	cancelCh <-chan bool) error

func (*BlackHole) Rollback added in v0.0.1 

func (t *BlackHole) Rollback(partition string, rollbackSeq uint64) error

func (*BlackHole) SnapshotStart added in v0.0.3 

func (t *BlackHole) SnapshotStart(partition string,
	snapStart, snapEnd uint64) error

func (*BlackHole) Stats added in v0.0.1 

func (t *BlackHole) Stats(w io.Writer) error

type BleveDest added in v0.0.1 

type BleveDest struct {
	// contains filtered or unexported fields
}

func NewBleveDest added in v0.0.1 

func NewBleveDest(path string, bindex bleve.Index,
	restart func()) *BleveDest

func (*BleveDest) AddError added in v0.0.1 

func (t *BleveDest) AddError(op, partition string,
	key []byte, seq uint64, val []byte, err error)

func (*BleveDest) Close added in v0.0.1 

func (t *BleveDest) Close() error

func (*BleveDest) ConsistencyWait added in v0.0.1 

func (t *BleveDest) ConsistencyWait(partition, partitionUUID string,
	consistencyLevel string,
	consistencySeq uint64,
	cancelCh <-chan bool) error

func (*BleveDest) Count added in v0.0.1 

func (t *BleveDest) Count(pindex *PIndex, cancelCh <-chan bool) (uint64, error)

func (*BleveDest) Dest added in v0.0.1 

func (t *BleveDest) Dest(partition string) (Dest, error)

func (*BleveDest) Query added in v0.0.1 

func (t *BleveDest) Query(pindex *PIndex, req []byte, res io.Writer,
	cancelCh <-chan bool) error

func (*BleveDest) Rollback added in v0.0.1 

func (t *BleveDest) Rollback(partition string, rollbackSeq uint64) error

func (*BleveDest) Stats added in v0.0.1 

func (t *BleveDest) Stats(w io.Writer) (err error)

type BleveDestPartition added in v0.0.1 

type BleveDestPartition struct {
	// contains filtered or unexported fields
}

Used to track state for a single partition.

func (*BleveDestPartition) Close added in v0.0.1 

func (t *BleveDestPartition) Close() error

func (*BleveDestPartition) ConsistencyWait added in v0.0.1 

func (t *BleveDestPartition) ConsistencyWait(
	partition, partitionUUID string,
	consistencyLevel string,
	consistencySeq uint64,
	cancelCh <-chan bool) error

func (*BleveDestPartition) Count added in v0.0.1 

func (t *BleveDestPartition) Count(pindex *PIndex,
	cancelCh <-chan bool) (
	uint64, error)

func (*BleveDestPartition) DataDelete added in v0.0.3 

func (t *BleveDestPartition) DataDelete(partition string,
	key []byte, seq uint64,
	cas uint64,
	extrasType DestExtrasType, extras []byte) error

func (*BleveDestPartition) DataUpdate added in v0.0.3 

func (t *BleveDestPartition) DataUpdate(partition string,
	key []byte, seq uint64, val []byte,
	cas uint64,
	extrasType DestExtrasType, extras []byte) error

func (*BleveDestPartition) OpaqueGet added in v0.0.3 

func (t *BleveDestPartition) OpaqueGet(partition string) ([]byte, uint64, error)

func (*BleveDestPartition) OpaqueSet added in v0.0.3 

func (t *BleveDestPartition) OpaqueSet(partition string, value []byte) error

func (*BleveDestPartition) Query added in v0.0.1 

func (t *BleveDestPartition) Query(pindex *PIndex,
	req []byte, res io.Writer,
	cancelCh <-chan bool) error

func (*BleveDestPartition) Rollback added in v0.0.1 

func (t *BleveDestPartition) Rollback(partition string,
	rollbackSeq uint64) error

func (*BleveDestPartition) SnapshotStart added in v0.0.3 

func (t *BleveDestPartition) SnapshotStart(partition string,
	snapStart, snapEnd uint64) error

func (*BleveDestPartition) Stats added in v0.0.1 

func (t *BleveDestPartition) Stats(w io.Writer) error

type BleveParams added in v0.0.1 

type BleveParams struct {
	Mapping bleve.IndexMapping     `json:"mapping"`
	Store   map[string]interface{} `json:"store"`
}

func NewBleveParams added in v0.0.1 

func NewBleveParams() *BleveParams

type CBFeedParams added in v0.0.1 

type CBFeedParams struct {
	AuthUser     string `json:"authUser"` // May be "" for no auth.
	AuthPassword string `json:"authPassword"`
}

CBFeedParams are common couchbase data-source/feed specific connection parameters that may be part of a sourceParams JSON.

type Cfg 

type Cfg interface {
	// Get retrieves an entry from the Cfg.  A zero cas means don't do
	// a CAS match on Get(), and a non-zero cas value means the Get()
	// will succeed only if the CAS matches.
	Get(key string, cas uint64) (val []byte, casSuccess uint64, err error)

	// Set creates or updates an entry in the Cfg.  A non-zero cas
	// that does not match will result in an error.  A zero cas means
	// the Set() operation must be an entry creation, where a zero cas
	// Set() will error if the entry already exists.
	Set(key string, val []byte, cas uint64) (casSuccess uint64, err error)

	// Del removes an entry from the Cfg.  A non-zero cas that does
	// not match will result in an error.  A zero cas means a CAS
	// match will be skipped, so that clients can perform a
	// "don't-care, out-of-the-blue" deletion.
	Del(key string, cas uint64) error

	// Subscribe allows clients to receive events on changes to a key.
	// During a deletion event, the CfgEvent.CAS field will be 0.
	Subscribe(key string, ch chan CfgEvent) error

	// Refresh forces the Cfg implementation to reload from its
	// backend-specific data source, clearing any locally cached data.
	// Any subscribers will receive events on a Refresh, where it's up
	// to subscribers to detect if there were actual changes or not.
	Refresh() error
}

Cfg is the interface that configuration providers must implement.

type CfgCASError 

type CfgCASError struct{}

The error used on mismatches of CAS (compare and set/swap) values.

func (*CfgCASError) Error 

func (e *CfgCASError) Error() string

type CfgCB added in v0.0.1 

type CfgCB struct {
	// contains filtered or unexported fields
}

CfgCB is an implementation of Cfg that uses a couchbase bucket, and uses DCP to get change notifications.

TODO: This current implementation is race-y! Instead of storing everything as a single uber key/value, we should instead be storing individual key/value's on every get/set/del operation.

func NewCfgCB added in v0.0.1 

func NewCfgCB(url, bucket string) (*CfgCB, error)

func (*CfgCB) DataDelete added in v0.0.1 

func (r *CfgCB) DataDelete(vbucketId uint16, key []byte, seq uint64,
	req *gomemcached.MCRequest) error

func (*CfgCB) DataUpdate added in v0.0.1 

func (r *CfgCB) DataUpdate(vbucketId uint16, key []byte, seq uint64,
	req *gomemcached.MCRequest) error

func (*CfgCB) Del added in v0.0.1 

func (c *CfgCB) Del(key string, cas uint64) error

func (*CfgCB) Get added in v0.0.1 

func (c *CfgCB) Get(key string, cas uint64) (
	[]byte, uint64, error)

func (*CfgCB) GetCredentials added in v0.0.1 

func (a *CfgCB) GetCredentials() (string, string, string)

func (*CfgCB) GetMetaData added in v0.0.1 

func (r *CfgCB) GetMetaData(vbucketId uint16) (
	value []byte, lastSeq uint64, err error)

func (*CfgCB) Load added in v0.0.1 

func (c *CfgCB) Load() error

func (*CfgCB) OnError added in v0.0.1 

func (r *CfgCB) OnError(err error)

func (*CfgCB) Refresh added in v0.0.1 

func (c *CfgCB) Refresh() error

func (*CfgCB) Rollback added in v0.0.1 

func (r *CfgCB) Rollback(vbucketId uint16, rollbackSeq uint64) error

func (*CfgCB) Set added in v0.0.1 

func (c *CfgCB) Set(key string, val []byte, cas uint64) (
	uint64, error)

func (*CfgCB) SetMetaData added in v0.0.1 

func (r *CfgCB) SetMetaData(vbucketId uint16, value []byte) error

func (*CfgCB) SnapshotStart added in v0.0.1 

func (r *CfgCB) SnapshotStart(vbucketId uint16,
	snapStart, snapEnd uint64, snapType uint32) error

func (*CfgCB) Subscribe added in v0.0.1 

func (c *CfgCB) Subscribe(key string, ch chan CfgEvent) error

type CfgEvent added in v0.0.1 

type CfgEvent struct {
	Key string
	CAS uint64
}

See the Cfg.Subscribe() method.

type CfgGetHandler added in v0.0.1 

type CfgGetHandler struct {
	// contains filtered or unexported fields
}

CfgGetHandler is a REST handler that retrieves the contents of the Cfg system.

func NewCfgGetHandler added in v0.0.1 

func NewCfgGetHandler(mgr *Manager) *CfgGetHandler

func (*CfgGetHandler) ServeHTTP added in v0.0.1 

func (h *CfgGetHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type CfgMem 

type CfgMem struct {
	CASNext uint64
	Entries map[string]*CfgMemEntry
	// contains filtered or unexported fields
}

CfgMem is a local-only, memory-only implementation of Cfg interface that's useful for development and testing.

func NewCfgMem 

func NewCfgMem() *CfgMem

NewCfgMem returns an empty CfgMem instance.

func (*CfgMem) Del 

func (c *CfgMem) Del(key string, cas uint64) error

func (*CfgMem) Get 

func (c *CfgMem) Get(key string, cas uint64) (
	[]byte, uint64, error)

func (*CfgMem) Refresh added in v0.0.1 

func (c *CfgMem) Refresh() error

func (*CfgMem) Set 

func (c *CfgMem) Set(key string, val []byte, cas uint64) (
	uint64, error)

func (*CfgMem) Subscribe added in v0.0.1 

func (c *CfgMem) Subscribe(key string, ch chan CfgEvent) error

type CfgMemEntry 

type CfgMemEntry struct {
	CAS uint64
	Val []byte
}

CfgMemEntry is a CAS-Val pairing tracked by CfgMem.

type CfgRefreshHandler added in v0.0.1 

type CfgRefreshHandler struct {
	// contains filtered or unexported fields
}

CfgRefreshHandler is a REST handler that processes a request for the manager/node to refresh its cached snapshot of the Cfg system contents.

func NewCfgRefreshHandler added in v0.0.1 

func NewCfgRefreshHandler(mgr *Manager) *CfgRefreshHandler

func (*CfgRefreshHandler) ServeHTTP added in v0.0.1 

func (h *CfgRefreshHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type CfgSimple 

type CfgSimple struct {
	// contains filtered or unexported fields
}

CfgSimple is a local-only, persisted (in a single file) implementation of the Cfg interface that's useful for non-clustered, single-node instances of cbft for developers.

func NewCfgSimple 

func NewCfgSimple(path string) *CfgSimple

NewCfgSimple returns a CfgSimple that reads and stores its single configuration file in the provided file path.

func (*CfgSimple) Del 

func (c *CfgSimple) Del(key string, cas uint64) error

func (*CfgSimple) Get 

func (c *CfgSimple) Get(key string, cas uint64) (
	[]byte, uint64, error)

func (*CfgSimple) Load 

func (c *CfgSimple) Load() error

func (*CfgSimple) Refresh added in v0.0.1 

func (c *CfgSimple) Refresh() error

func (*CfgSimple) Set 

func (c *CfgSimple) Set(key string, val []byte, cas uint64) (
	uint64, error)

func (*CfgSimple) Subscribe added in v0.0.1 

func (c *CfgSimple) Subscribe(key string, ch chan CfgEvent) error

type ConsistencyParams added in v0.0.1 

type ConsistencyParams struct {
	// A Level value of "" means stale is ok; "at_plus" means we need
	// consistency at least at or beyond the consistency vector but
	// not before.
	Level string `json:"level"`

	// Keyed by indexName.
	Vectors map[string]ConsistencyVector `json:"vectors"`
}

ConsistencyParams represent the consistency requirements of a client's request.

type ConsistencyVector added in v0.0.1 

type ConsistencyVector map[string]uint64

Key is partition or partition/partitionUUID. Value is seq. For example, a DCP data source might have the key as either "vbucketId" or "vbucketId/vbucketUUID".

type ConsistencyWaitReq added in v0.0.1 

type ConsistencyWaitReq struct {
	PartitionUUID    string
	ConsistencyLevel string
	ConsistencySeq   uint64
	CancelCh         <-chan bool
	DoneCh           chan error
}

A ConsistencyWaitReq represents a runtime consistency wait request for a partition.

type ConsistencyWaiter added in v0.0.1 

type ConsistencyWaiter interface {
	ConsistencyWait(partition, partitionUUID string,
		consistencyLevel string,
		consistencySeq uint64,
		cancelCh <-chan bool) error
}

ConsistencyWaiter interface represents a service that can wait for consistency.

type CountHandler added in v0.0.1 

type CountHandler struct {
	// contains filtered or unexported fields
}

CountHandler is a REST handler for counting documents/entries in an index.

func NewCountHandler added in v0.0.1 

func NewCountHandler(mgr *Manager) *CountHandler

func (*CountHandler) RESTOpts added in v0.0.1 

func (h *CountHandler) RESTOpts(opts map[string]string)

func (*CountHandler) ServeHTTP added in v0.0.1 

func (h *CountHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type CountPIndexHandler added in v0.0.1 

type CountPIndexHandler struct {
	// contains filtered or unexported fields
}

CountPIndexHandler is a REST handler for counting the documents/entries in a pindex.

func NewCountPIndexHandler added in v0.0.1 

func NewCountPIndexHandler(mgr *Manager) *CountPIndexHandler

func (*CountPIndexHandler) ServeHTTP added in v0.0.1 

func (h *CountPIndexHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type CreateIndexHandler 

type CreateIndexHandler struct {
	// contains filtered or unexported fields
}

CreateIndexHandler is a REST handler that processes an index creation request.

func NewCreateIndexHandler added in v0.0.1 

func NewCreateIndexHandler(mgr *Manager) *CreateIndexHandler

func (*CreateIndexHandler) RESTOpts added in v0.0.1 

func (h *CreateIndexHandler) RESTOpts(opts map[string]string)

func (*CreateIndexHandler) ServeHTTP 

func (h *CreateIndexHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type CwrQueue added in v0.1.0 

type CwrQueue []*ConsistencyWaitReq

A CwrQueue is a consistency wait request queue, implementing the heap.Interface for ConsistencyWaitReq's, and is heap ordered by sequence number.

func (CwrQueue) Len added in v0.1.0 

func (pq CwrQueue) Len() int

func (CwrQueue) Less added in v0.1.0 

func (pq CwrQueue) Less(i, j int) bool

func (*CwrQueue) Pop added in v0.1.0 

func (pq *CwrQueue) Pop() interface{}

func (*CwrQueue) Push added in v0.1.0 

func (pq *CwrQueue) Push(x interface{})

func (CwrQueue) Swap added in v0.1.0 

func (pq CwrQueue) Swap(i, j int)

type DCPFeed 

type DCPFeed struct {
	// contains filtered or unexported fields
}

A DCPFeed implements both Feed and cbdatasource.Receiver interfaces, and forwards any incoming cbdatasource.Receiver callbacks to the relevant, hooked-up Dest instances.

func NewDCPFeed 

func NewDCPFeed(name, indexName, url, poolName,
	bucketName, bucketUUID, paramsStr string,
	pf DestPartitionFunc, dests map[string]Dest,
	disable bool) (*DCPFeed, error)

NewDCPFeed creates a new, ready-to-be-started DCP feed.

func (*DCPFeed) Close 

func (t *DCPFeed) Close() error

func (*DCPFeed) DataDelete added in v0.0.1 

func (r *DCPFeed) DataDelete(vbucketId uint16, key []byte, seq uint64,
	req *gomemcached.MCRequest) error

func (*DCPFeed) DataUpdate added in v0.0.1 

func (r *DCPFeed) DataUpdate(vbucketId uint16, key []byte, seq uint64,
	req *gomemcached.MCRequest) error

func (*DCPFeed) Dests added in v0.0.1 

func (t *DCPFeed) Dests() map[string]Dest

func (*DCPFeed) GetMetaData added in v0.0.1 

func (r *DCPFeed) GetMetaData(vbucketId uint16) (
	value []byte, lastSeq uint64, err error)

func (*DCPFeed) IndexName added in v0.0.1 

func (t *DCPFeed) IndexName() string

func (*DCPFeed) Name 

func (t *DCPFeed) Name() string

func (*DCPFeed) OnError added in v0.0.1 

func (r *DCPFeed) OnError(err error)

func (*DCPFeed) Rollback added in v0.0.1 

func (r *DCPFeed) Rollback(vbucketId uint16, rollbackSeq uint64) error

func (*DCPFeed) SetMetaData added in v0.0.1 

func (r *DCPFeed) SetMetaData(vbucketId uint16, value []byte) error

func (*DCPFeed) SnapshotStart added in v0.0.1 

func (r *DCPFeed) SnapshotStart(vbucketId uint16,
	snapStart, snapEnd uint64, snapType uint32) error

func (*DCPFeed) Start 

func (t *DCPFeed) Start() error

func (*DCPFeed) Stats added in v0.0.1 

func (t *DCPFeed) Stats(w io.Writer) error

type DCPFeedParams added in v0.0.1 

type DCPFeedParams struct {
	AuthUser     string `json:"authUser"` // May be "" for no auth.
	AuthPassword string `json:"authPassword"`

	// Factor (like 1.5) to increase sleep time between retries
	// in connecting to a cluster manager node.
	ClusterManagerBackoffFactor float32 `json:"clusterManagerBackoffFactor"`

	// Initial sleep time (millisecs) before first retry to cluster manager.
	ClusterManagerSleepInitMS int `json:"clusterManagerSleepInitMS"`

	// Maximum sleep time (millisecs) between retries to cluster manager.
	ClusterManagerSleepMaxMS int `json:"clusterManagerSleepMaxMS"`

	// Factor (like 1.5) to increase sleep time between retries
	// in connecting to a data manager node.
	DataManagerBackoffFactor float32 `json:"dataManagerBackoffFactor"`

	// Initial sleep time (millisecs) before first retry to data manager.
	DataManagerSleepInitMS int `json:"dataManagerSleepInitMS"`

	// Maximum sleep time (millisecs) between retries to data manager.
	DataManagerSleepMaxMS int `json:"dataManagerSleepMaxMS"`

	// Buffer size in bytes provided for UPR flow control.
	FeedBufferSizeBytes uint32 `json:"feedBufferSizeBytes"`

	// Used for UPR flow control and buffer-ack messages when this
	// percentage of FeedBufferSizeBytes is reached.
	FeedBufferAckThreshold float32 `json:"feedBufferAckThreshold"`
}

DCPFeedParams are DCP data-source/feed specific connection parameters that may be part of a sourceParams JSON and is a superset of CBFeedParams. DCPFeedParams holds the information used to populate a cbdatasource.BucketDataSourceOptions on calls to cbdatasource.NewBucketDataSource(). DCPFeedParams also implements the couchbase.AuthHandler interface.

func NewDCPFeedParams added in v0.0.1 

func NewDCPFeedParams() *DCPFeedParams

NewDCPFeedParams returns a DCPFeedParams initialized with default values.

func (*DCPFeedParams) GetCredentials added in v0.0.1 

func (d *DCPFeedParams) GetCredentials() (string, string, string)

GetCredentials is part of the couchbase.AuthHandler interface.

type DeleteIndexHandler 

type DeleteIndexHandler struct {
	// contains filtered or unexported fields
}

DeleteIndexHandler is a REST handler that processes an index deletion request.

func NewDeleteIndexHandler 

func NewDeleteIndexHandler(mgr *Manager) *DeleteIndexHandler

func (*DeleteIndexHandler) RESTOpts added in v0.0.1 

func (h *DeleteIndexHandler) RESTOpts(opts map[string]string)

func (*DeleteIndexHandler) ServeHTTP 

func (h *DeleteIndexHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type Dest added in v0.0.1 

type Dest interface {
	// Invoked by PIndex.Close().
	Close() error

	// Invoked when there's a new mutation from a data source for a
	// partition.  Dest implementation is responsible for making its
	// own copies of the key, val and extras data.
	DataUpdate(partition string, key []byte, seq uint64, val []byte,
		cas uint64,
		extrasType DestExtrasType, extras []byte) error

	// Invoked by the data source when there's a data deletion in a
	// partition.  Dest implementation is responsible for making its
	// own copies of the key and extras data.
	DataDelete(partition string, key []byte, seq uint64,
		cas uint64,
		extrasType DestExtrasType, extras []byte) error

	// An callback invoked by the data source when there's a start of
	// a new snapshot for a partition.  The Receiver implementation,
	// for example, might choose to optimize persistence perhaps by
	// preparing a batch write to application-specific storage.
	SnapshotStart(partition string, snapStart, snapEnd uint64) error

	// OpaqueGet() should return the opaque value previously
	// provided by an earlier call to OpaqueSet().  If there was no
	// previous call to OpaqueSet(), such as in the case of a brand
	// new instance of a Dest (as opposed to a restarted or reloaded
	// Dest), the Dest should return (nil, 0, nil) for (value,
	// lastSeq, err), respectively.  The lastSeq should be the last
	// sequence number received and persisted during calls to the
	// Dest's DataUpdate() & DataDelete() methods.
	OpaqueGet(partition string) (value []byte, lastSeq uint64, err error)

	// The Dest implementation should persist the value parameter of
	// OpaqueSet() for retrieval during some future call to
	// OpaqueGet() by the system.  The metadata value should be
	// considered "in-stream", or as part of the sequence history of
	// mutations.  That is, a later Rollback() to some previous
	// sequence number for a particular partition should rollback
	// both persisted metadata and regular data.  The Dest
	// implementation should make its own copy of the value data.
	OpaqueSet(partition string, value []byte) error

	// Invoked by when the datasource signals a rollback during dest
	// initialization.  Note that both regular data and opaque data
	// should be rolled back to at a maximum of the rollbackSeq.  Of
	// note, the Dest is allowed to rollback even further, even all
	// the way back to the start or to zero.
	Rollback(partition string, rollbackSeq uint64) error

	// Blocks until the Dest has reached the desired consistency for
	// the partition or until the cancelCh is readable or closed by
	// some goroutine related to the calling goroutine.  The error
	// response might be a ErrorConsistencyWait instance, which has
	// StartEndSeqs information.  The seqStart is the seq number when
	// the operation started waiting and the seqEnd is the seq number
	// at the end of operation (even when cancelled or error), so that
	// the caller might get a rough idea of ingest velocity.
	ConsistencyWait(partition, partitionUUID string,
		consistencyLevel string,
		consistencySeq uint64,
		cancelCh <-chan bool) error

	// Counts the underlying pindex implementation.
	Count(pindex *PIndex, cancelCh <-chan bool) (uint64, error)

	// Queries the underlying pindex implementation, blocking if
	// needed for the Dest to reach the desired consistency.
	Query(pindex *PIndex, req []byte, w io.Writer,
		cancelCh <-chan bool) error

	Stats(io.Writer) error
}

Dest interface defines the data sink or destination for data that cames from a data-source. In other words, a data-source (or a Feed instance) is hooked up to one or more Dest instances. As a Feed receives incoming data, the Feed will invoke methods on its Dest instances.

func BasicPartitionFunc added in v0.0.1 

func BasicPartitionFunc(partition string, key []byte,
	dests map[string]Dest) (Dest, error)

This basic partition func first tries a direct lookup by partition string, else it tries the "" partition.

func VBucketIdToPartitionDest added in v0.0.1 

func VBucketIdToPartitionDest(pf DestPartitionFunc,
	dests map[string]Dest, vbucketId uint16, key []byte) (
	partition string, dest Dest, err error)

VBucketIdToPartitionDest is specific to couchbase data-sources/feeds, choosing the right Dest based on a vbucketId.

type DestExtrasType added in v0.1.0 

type DestExtrasType uint16

DestExtrasType represents the encoding for the Dest.DataUpdate/DataDelete() extras parameter.

type DestForwarder added in v0.0.1 

type DestForwarder struct {
	DestProvider DestProvider
}

A DestForwarder implements the Dest interface by forwarding method calls to the Dest returned by a DestProvider.

It is useful for pindex backend implementations that have their own level-of-indirection features. One example would be pindex backends that track a separate batch per partition (ex: see the bleve pindex backend).

func (*DestForwarder) Close added in v0.0.1 

func (t *DestForwarder) Close() error

func (*DestForwarder) ConsistencyWait added in v0.0.1 

func (t *DestForwarder) ConsistencyWait(partition, partitionUUID string,
	consistencyLevel string,
	consistencySeq uint64,
	cancelCh <-chan bool) error

func (*DestForwarder) Count added in v0.0.1 

func (t *DestForwarder) Count(pindex *PIndex, cancelCh <-chan bool) (
	uint64, error)

func (*DestForwarder) DataDelete added in v0.0.3 

func (t *DestForwarder) DataDelete(partition string,
	key []byte, seq uint64,
	cas uint64,
	extrasType DestExtrasType, extras []byte) error

func (*DestForwarder) DataUpdate added in v0.0.3 

func (t *DestForwarder) DataUpdate(partition string,
	key []byte, seq uint64, val []byte,
	cas uint64,
	extrasType DestExtrasType, extras []byte) error

func (*DestForwarder) OpaqueGet added in v0.0.3 

func (t *DestForwarder) OpaqueGet(partition string) (
	value []byte, lastSeq uint64, err error)

func (*DestForwarder) OpaqueSet added in v0.0.3 

func (t *DestForwarder) OpaqueSet(partition string, value []byte) error

func (*DestForwarder) Query added in v0.0.1 

func (t *DestForwarder) Query(pindex *PIndex, req []byte, res io.Writer,
	cancelCh <-chan bool) error

func (*DestForwarder) Rollback added in v0.0.1 

func (t *DestForwarder) Rollback(partition string, rollbackSeq uint64) error

func (*DestForwarder) SnapshotStart added in v0.0.3 

func (t *DestForwarder) SnapshotStart(partition string,
	snapStart, snapEnd uint64) error

func (*DestForwarder) Stats added in v0.0.1 

func (t *DestForwarder) Stats(w io.Writer) error

type DestPartitionFunc added in v0.0.1 

type DestPartitionFunc func(partition string, key []byte,
	dests map[string]Dest) (Dest, error)

A DestPartitionFunc allows a level of indirection/abstraction for the Feed-to-Dest relationship. A Feed is hooked up in a one-to-many relationship with multiple Dest instances. The DestPartitionFunc provided to a Feed instance defines the mapping of which Dest the Feed should invoke when the Feed receives an incoming data item.

The partition parameter is encoded as a string, instead of a uint16 or number, to allow for future range partitioning functionality.

type DestProvider added in v0.0.1 

type DestProvider interface {
	Dest(partition string) (Dest, error)

	Count(pindex *PIndex, cancelCh <-chan bool) (uint64, error)

	Query(pindex *PIndex, req []byte, res io.Writer,
		cancelCh <-chan bool) error

	Stats(io.Writer) error

	Close() error
}

A DestProvider returns the Dest to use for different kinds of operations and is used in conjunction with a DestForwarder.

type DestStats added in v0.0.1 

type DestStats struct {
	TotError uint64

	TimerDataUpdate    metrics.Timer
	TimerDataDelete    metrics.Timer
	TimerSnapshotStart metrics.Timer
	TimerOpaqueGet     metrics.Timer
	TimerOpaqueSet     metrics.Timer
	TimerRollback      metrics.Timer
}

DestStats holds the common stats or metrics for a Dest.

func NewDestStats added in v0.0.1 

func NewDestStats() *DestStats

NewDestStats creates a new, ready-to-use DestStats.

func (*DestStats) WriteJSON added in v0.0.1 

func (d *DestStats) WriteJSON(w io.Writer)

type DiagGetHandler added in v0.0.1 

type DiagGetHandler struct {
	// contains filtered or unexported fields
}

DiagGetHandler is a REST handler that retrieves diagnostic information for a node.

func NewDiagGetHandler added in v0.0.1 

func NewDiagGetHandler(versionMain string,
	mgr *Manager, mr *MsgRing) *DiagGetHandler

func (*DiagGetHandler) ServeHTTP added in v0.0.1 

func (h *DiagGetHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type DiagHandler added in v0.1.0 

type DiagHandler struct {
	Name        string
	Handler     http.Handler
	HandlerFunc http.HandlerFunc
}

DiagHandler represents a part of the DiagGetHandler response. Different modules can provide their parts of a DiagGetHandler response via their own DiagHandler's.

type Documentation added in v0.2.0 

type Documentation struct {
	Text string      // Optional documentation text (markdown).
	JSON interface{} // Optional marshall'able to JSON.
}

Documentation is used for auto-generated documentation.

func BlevePIndexQuerySamples added in v0.2.0 

func BlevePIndexQuerySamples() []Documentation

type ErrorConsistencyWait added in v0.0.1 

type ErrorConsistencyWait struct {
	Err    error  // The underlying, wrapped error.
	Status string // Short status reason, like "timeout", "cancelled", etc.

	// Keyed by partitionId, value is pair of start/end seq's.
	StartEndSeqs map[string][]uint64
}

An ErrorConsistencyWait represents an error or timeout while waiting for a partition to reach some consistency requirements.

func (*ErrorConsistencyWait) Error added in v0.0.1 

func (e *ErrorConsistencyWait) Error() string

type Feed 

type Feed interface {
	Name() string
	IndexName() string
	Start() error
	Close() error
	Dests() map[string]Dest // Key is partition identifier.

	// Writes stats as JSON to the given writer.
	Stats(io.Writer) error
}

A Feed interface represents an abstract data source. A Feed instance is hooked up to one-or-more Dest instances. When incoming data is received by a Feed, the Feed will invoke relvate methods on the relevant Dest instances.

In this codebase, the words "index source", "source" and "data source" are often associated with and used roughly as synonyms with "feed".

func CalcFeedsDelta 

func CalcFeedsDelta(nodeUUID string, planPIndexes *PlanPIndexes,
	currFeeds map[string]Feed, pindexes map[string]*PIndex) (
	addFeeds [][]*PIndex, removeFeeds []Feed)

Functionally determine the delta of which feeds need creation and which should be shut down.

type FeedPartitionsFunc added in v0.0.1 

type FeedPartitionsFunc func(sourceType, sourceName, sourceUUID, sourceParams,
	server string) ([]string, error)

Each Feed or data-source type knows of the data partitions for a data source.

type FeedStartFunc added in v0.0.1 

type FeedStartFunc func(mgr *Manager, feedName, indexName, indexUUID string,
	sourceType, sourceName, sourceUUID, sourceParams string,
	dests map[string]Dest) error

A FeedStartFunc is part of a FeedType registration as is invoked by a Manager when a new feed instance needs to be started.

type FeedType added in v0.0.1 

type FeedType struct {
	Start           FeedStartFunc
	Partitions      FeedPartitionsFunc
	Public          bool
	Description     string
	StartSample     interface{}
	StartSampleDocs map[string]string
}

A FeedType represents an immutable registration of a single feed type or data source type.

type FileDoc added in v0.1.0 

type FileDoc struct {
	Name     string `json:"name"`
	Path     string `json:"path"` // Path relative to the source name.
	Contents string `json:"contents"`
}

FileDoc represents the JSON for each file/document that will be emitted by a FilesFeed as a data source.

type FileLike added in v0.0.1 

type FileLike interface {
	io.Closer
	io.ReaderAt
	io.WriterAt
	Stat() (os.FileInfo, error)
	Truncate(size int64) error
}

A FileLike looks like a File, but file opening/closing is delayed until the actual read/write/etc operation. See also FileService.

type FileService added in v0.0.1 

type FileService struct {
	// contains filtered or unexported fields
}

A FileService represents filesystem resources and provides a level a indirection that trades off performance with the ability to avoid running out of OS file descriptors.

func NewFileService added in v0.0.1 

func NewFileService(concurrency int) *FileService

NewFileService creates FileService that services N concurrent FileLike requests.

func (*FileService) Close added in v0.0.1 

func (f *FileService) Close() error

func (*FileService) Do added in v0.0.1 

func (f *FileService) Do(path string, flags int,
	fn func(*os.File) error) error

func (*FileService) OpenFile added in v0.0.1 

func (fs *FileService) OpenFile(path string, mode int) (FileLike, error)

Open a FileLike instance that works within this FileService.

type FilesFeed added in v0.1.0 

type FilesFeed struct {
	// contains filtered or unexported fields
}

FilesFeed is a Feed interface implementation that that emits file contents from a local subdirectory tree.

The subdirectory tree lives under the dataDir... 

<dataDir>/<sourceName/**

FilesFeed supports optional regexp patterns to allow you to filter for only the file paths that you want.

Limitations:

- Only a small number of files will work well (hundreds to low thousands, not millions).

- FilesFeed polls for file modification timestamp changes as a poor-man's approach instead of properly tracking sequence numbers. That has implications such as whenever a FilesFeed (re-)starts (e.g., the process restarts), the FilesFeed will re-emits all files and then track the max modification timestamp going forwards as it regularly polls for file changes.

func NewFilesFeed added in v0.1.0 

func NewFilesFeed(mgr *Manager, name, indexName, sourceName,
	paramsStr string, dests map[string]Dest, disable bool) (
	*FilesFeed, error)

NewFilesFeed creates a ready-to-be-started FilesFeed.

func (*FilesFeed) Close added in v0.1.0 

func (t *FilesFeed) Close() error

func (*FilesFeed) Dests added in v0.1.0 

func (t *FilesFeed) Dests() map[string]Dest

func (*FilesFeed) IndexName added in v0.1.0 

func (t *FilesFeed) IndexName() string

func (*FilesFeed) Name added in v0.1.0 

func (t *FilesFeed) Name() string

func (*FilesFeed) Start added in v0.1.0 

func (t *FilesFeed) Start() error

func (*FilesFeed) Stats added in v0.1.0 

func (t *FilesFeed) Stats(w io.Writer) error

type FilesFeedParams added in v0.1.0 

type FilesFeedParams struct {
	RegExps       []string `json:"regExps"`
	MaxFileSize   int64    `json:"maxFileSize"`
	NumPartitions int      `json:"numPartitions"`
	SleepStartMS  int      `json:"sleepStartMS"`
	BackoffFactor float32  `json:"backoffFactor"`
	MaxSleepMS    int      `json:"maxSleepMS"`
}

FilesFeedParams represents the JSON expected as the sourceParams for a FilesFeed.

type GetIndexHandler added in v0.0.1 

type GetIndexHandler struct {
	// contains filtered or unexported fields
}

GetIndexHandler is a REST handler for retrieving an index definition.

func NewGetIndexHandler added in v0.0.1 

func NewGetIndexHandler(mgr *Manager) *GetIndexHandler

func (*GetIndexHandler) RESTOpts added in v0.0.1 

func (h *GetIndexHandler) RESTOpts(opts map[string]string)

func (*GetIndexHandler) ServeHTTP added in v0.0.1 

func (h *GetIndexHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type GetPIndexHandler added in v0.0.1 

type GetPIndexHandler struct {
	// contains filtered or unexported fields
}

GetPIndexHandler is a REST handler for retrieving information on a pindex.

func NewGetPIndexHandler added in v0.0.1 

func NewGetPIndexHandler(mgr *Manager) *GetPIndexHandler

func (*GetPIndexHandler) ServeHTTP added in v0.0.1 

func (h *GetPIndexHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type IndexClient added in v0.0.1 

type IndexClient struct {
	QueryURL    string
	CountURL    string
	Consistency *ConsistencyParams
}

IndexClient implements the Search() and DocCount() subset of the bleve.Index interface by accessing a remote cbft server via REST protocol. This allows callers to add a IndexClient as a target of a bleve.IndexAlias, and implements cbft protocol features like query consistency and auth.

TODO: Implement propagating auth info in IndexClient.

func (*IndexClient) Advanced added in v0.0.1 

func (r *IndexClient) Advanced() (index.Index, store.KVStore, error)

func (*IndexClient) Batch added in v0.0.1 

func (r *IndexClient) Batch(b *bleve.Batch) error

func (*IndexClient) Close added in v0.0.1 

func (r *IndexClient) Close() error

func (*IndexClient) Count added in v0.0.1 

func (r *IndexClient) Count() (uint64, error)

func (*IndexClient) Delete added in v0.0.1 

func (r *IndexClient) Delete(id string) error

func (*IndexClient) DeleteInternal added in v0.0.1 

func (r *IndexClient) DeleteInternal(key []byte) error

func (*IndexClient) DocCount added in v0.0.1 

func (r *IndexClient) DocCount() (uint64, error)

func (*IndexClient) Document added in v0.0.1 

func (r *IndexClient) Document(id string) (*document.Document, error)

func (*IndexClient) DumpAll added in v0.0.1 

func (r *IndexClient) DumpAll() chan interface{}

func (*IndexClient) DumpDoc added in v0.0.1 

func (r *IndexClient) DumpDoc(id string) chan interface{}

func (*IndexClient) DumpFields added in v0.0.1 

func (r *IndexClient) DumpFields() chan interface{}

func (*IndexClient) FieldDict added in v0.0.1 

func (r *IndexClient) FieldDict(field string) (index.FieldDict, error)

func (*IndexClient) FieldDictPrefix added in v0.0.1 

func (r *IndexClient) FieldDictPrefix(field string,
	termPrefix []byte) (index.FieldDict, error)

func (*IndexClient) FieldDictRange added in v0.0.1 

func (r *IndexClient) FieldDictRange(field string,
	startTerm []byte, endTerm []byte) (index.FieldDict, error)

func (*IndexClient) Fields added in v0.0.1 

func (r *IndexClient) Fields() ([]string, error)

func (*IndexClient) GetInternal added in v0.0.1 

func (r *IndexClient) GetInternal(key []byte) ([]byte, error)

func (*IndexClient) Index added in v0.0.1 

func (r *IndexClient) Index(id string, data interface{}) error

func (*IndexClient) Mapping added in v0.0.1 

func (r *IndexClient) Mapping() *bleve.IndexMapping

func (*IndexClient) NewBatch added in v0.0.1 

func (r *IndexClient) NewBatch() *bleve.Batch

func (*IndexClient) Query added in v0.0.1 

func (r *IndexClient) Query(buf []byte) ([]byte, error)

func (*IndexClient) Search added in v0.0.1 

func (r *IndexClient) Search(req *bleve.SearchRequest) (
	*bleve.SearchResult, error)

func (*IndexClient) SetInternal added in v0.0.1 

func (r *IndexClient) SetInternal(key, val []byte) error

func (*IndexClient) Stats added in v0.0.1 

func (r *IndexClient) Stats() *bleve.IndexStat

type IndexControlHandler added in v0.0.1 

type IndexControlHandler struct {
	// contains filtered or unexported fields
}

IndexControlHandler is a REST handler for processing admin control requests on an index.

func NewIndexControlHandler added in v0.0.1 

func NewIndexControlHandler(mgr *Manager, control string,
	allowedOps map[string]bool) *IndexControlHandler

func (*IndexControlHandler) RESTOpts added in v0.0.1 

func (h *IndexControlHandler) RESTOpts(opts map[string]string)

func (*IndexControlHandler) ServeHTTP added in v0.0.1 

func (h *IndexControlHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type IndexDef 

type IndexDef struct {
	Type         string     `json:"type"` // Ex: "bleve", "alias", "blackhole", etc.
	Name         string     `json:"name"`
	UUID         string     `json:"uuid"`
	Params       string     `json:"params"`
	SourceType   string     `json:"sourceType"`
	SourceName   string     `json:"sourceName"`
	SourceUUID   string     `json:"sourceUUID"`
	SourceParams string     `json:"sourceParams"` // Optional connection info.
	PlanParams   PlanParams `json:"planParams"`
}

An IndexDef is a logical index definition.

type IndexDefs 

type IndexDefs struct {
	// IndexDefs.UUID changes whenever any child IndexDef changes.
	UUID        string               `json:"uuid"`
	IndexDefs   map[string]*IndexDef `json:"indexDefs"`   // Key is IndexDef.Name.
	ImplVersion string               `json:"implVersion"` // See VERSION.
}

An IndexDefs is zero or more index definitions.

func CfgGetIndexDefs 

func CfgGetIndexDefs(cfg Cfg) (*IndexDefs, uint64, error)

Returns index definitions from a Cfg provider.

func NewIndexDefs 

func NewIndexDefs(version string) *IndexDefs

Returns an intiialized IndexDefs.

func PlannerGetIndexDefs added in v0.0.1 

func PlannerGetIndexDefs(cfg Cfg, version string) (*IndexDefs, error)

PlannerGetIndexDefs retrives index definitions from a Cfg.

type JSONStatsWriter added in v0.0.1 

type JSONStatsWriter interface {
	WriteJSON(w io.Writer)
}

type ListIndexHandler added in v0.0.1 

type ListIndexHandler struct {
	// contains filtered or unexported fields
}

ListIndexHandler is a REST handler for list indexes.

func NewListIndexHandler added in v0.0.1 

func NewListIndexHandler(mgr *Manager) *ListIndexHandler

func (*ListIndexHandler) ServeHTTP added in v0.0.1 

func (h *ListIndexHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type ListPIndexHandler added in v0.0.1 

type ListPIndexHandler struct {
	// contains filtered or unexported fields
}

ListPIndexHandler is a REST handler for listing pindexes.

func NewListPIndexHandler added in v0.0.1 

func NewListPIndexHandler(mgr *Manager) *ListPIndexHandler

func (*ListPIndexHandler) ServeHTTP added in v0.0.1 

func (h *ListPIndexHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type LogGetHandler added in v0.0.1 

type LogGetHandler struct {
	// contains filtered or unexported fields
}

LogGetHandler is a REST handler that retrieves recent log messages.

func NewLogGetHandler added in v0.0.1 

func NewLogGetHandler(mgr *Manager, mr *MsgRing) *LogGetHandler

func (*LogGetHandler) ServeHTTP added in v0.0.1 

func (h *LogGetHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type Manager 

type Manager struct {
	// contains filtered or unexported fields
}

A Manager represents a runtime node in a cluster.

Although often used like a singleton, multiple Manager instances can be instantiated in a process to simulate a cluster of nodes.

A Manager has two related child, actor-like goroutines: - planner - janitor

A planner splits index definitions into index partitions (pindexes) and assigns those pindexes to nodes. A planner wakes up and runs whenever the index definitions change or the set of nodes changes (which are both read from the Cfg system). A planner stores the latest plans into the Cfg system.

A janitor running on each node maintains runtime PIndex and Feed instances, creating, deleting & hooking them up as necessary to try to match to latest plans from the planner. A janitor wakes up and runs whenever it sees that latest plans in the Cfg have changed.

func NewManager 

func NewManager(version string, cfg Cfg, uuid string, tags []string,
	container string, weight int, extras, bindHttp, dataDir, server string,
	meh ManagerEventHandlers) *Manager

NewManager returns a new, ready-to-be-started Manager instance.

func (*Manager) Cfg added in v0.0.1 

func (mgr *Manager) Cfg() Cfg

Returns the configured Cfg of a Manager.

func (*Manager) ClosePIndex added in v0.0.1 

func (mgr *Manager) ClosePIndex(pindex *PIndex) error

ClosePIndex synchronously has the janitor close a pindex.

func (*Manager) CoveringPIndexes added in v0.0.1 

func (mgr *Manager) CoveringPIndexes(indexName, indexUUID string,
	wantNode PlanPIndexFilter, wantKind string) (
	localPIndexes []*PIndex,
	remotePlanPIndexes []*RemotePlanPIndex,
	err error)

CoveringPIndexes returns a non-overlapping, disjoint set (or cut) of PIndexes (either local or remote) that cover all the partitons of an index so that the caller can perform scatter/gather queries, etc. Only PlanPIndexes on wanted nodes that pass the wantNode filter will be returned.

TODO: Perhaps need a tighter check around indexUUID, as the current implementation might have a race where old pindexes with a matching (but outdated) indexUUID might be chosen.

TODO: This implementation currently always favors the local node's pindex, but should it? Perhaps a remote node is more up-to-date than the local pindex?

TODO: We should favor the most up-to-date node rather than the first one that we run into here? But, perhaps the most up-to-date node is also the most overloaded? Or, perhaps the planner may be trying to rebalance away the most up-to-date node and hitting it with load just makes the rebalance take longer?

func (*Manager) CreateIndex 

func (mgr *Manager) CreateIndex(sourceType, sourceName, sourceUUID, sourceParams,
	indexType, indexName, indexParams string, planParams PlanParams,
	prevIndexUUID string) error

Creates a logical index definition. A non-"" prevIndexUUID means an update to an existing index.

func (*Manager) CurrentMaps 

func (mgr *Manager) CurrentMaps() (map[string]Feed, map[string]*PIndex)

Returns a snapshot copy of the current feeds and pindexes.

func (*Manager) DataDir 

func (mgr *Manager) DataDir() string

Returns the configured data dir of a Manager.

func (*Manager) DeleteIndex 

func (mgr *Manager) DeleteIndex(indexName string) error

Deletes a logical index definition.

TODO: DeleteIndex should check an optional index UUID?

func (*Manager) GetIndexDefs added in v0.0.1 

func (mgr *Manager) GetIndexDefs(refresh bool) (
	*IndexDefs, map[string]*IndexDef, error)

Returns read-only snapshot of the IndexDefs, also with IndexDef's organized by name. Use refresh of true to force a read from Cfg.

func (*Manager) GetPIndex added in v0.0.1 

func (mgr *Manager) GetPIndex(pindexName string) *PIndex

GetPIndex retrieves a named pindex instance.

func (*Manager) GetPlanPIndexes added in v0.0.1 

func (mgr *Manager) GetPlanPIndexes(refresh bool) (
	*PlanPIndexes, map[string][]*PlanPIndex, error)

Returns read-only snapshot of the PlanPIndexes, also with PlanPIndex's organized by IndexName. Use refresh of true to force a read from Cfg.

func (*Manager) IndexControl added in v0.0.1 

func (mgr *Manager) IndexControl(indexName, indexUUID, readOp, writeOp,
	planFreezeOp string) error

IndexControl is used to change runtime properties of an index definition.

func (*Manager) JanitorKick 

func (mgr *Manager) JanitorKick(msg string)

JanitorKick synchronously kicks the manager's janitor, if any.

func (*Manager) JanitorLoop 

func (mgr *Manager) JanitorLoop()

JanitorLoop is the main loop for the janitor.

func (*Manager) JanitorNOOP added in v0.0.1 

func (mgr *Manager) JanitorNOOP(msg string)

JanitorNOOP sends a synchronous NOOP request to the manager's janitor, if any.

func (*Manager) JanitorOnce 

func (mgr *Manager) JanitorOnce(reason string) error

JanitorOnce is the main body of a JanitorLoop.

func (*Manager) Kick added in v0.0.1 

func (mgr *Manager) Kick(msg string)

Schedule kicks of the planner and janitor of a Manager.

func (*Manager) LoadDataDir 

func (mgr *Manager) LoadDataDir() error

Walk the data dir and register pindexes for a Manager instance.

func (*Manager) PIndexPath 

func (mgr *Manager) PIndexPath(pindexName string) string

PIndexPath returns the filesystem path for a given named pindex. See also ParsePIndexPath().

func (*Manager) ParsePIndexPath 

func (mgr *Manager) ParsePIndexPath(pindexPath string) (string, bool)

ParsePIndexPath returns the name for a pindex given a filesystem path. See also PIndexPath().

func (*Manager) PlannerKick 

func (mgr *Manager) PlannerKick(msg string)

PlannerKick synchronously kicks the manager's planner, if any.

func (*Manager) PlannerLoop 

func (mgr *Manager) PlannerLoop()

PlannerLoop is the main loop for the planner.

func (*Manager) PlannerNOOP added in v0.0.1 

func (mgr *Manager) PlannerNOOP(msg string)

PlannerNOOP sends a synchronous NOOP request to the manager's planner, if any.

func (*Manager) PlannerOnce 

func (mgr *Manager) PlannerOnce(reason string) (bool, error)

PlannerOnce is the main body of a PlannerLoop.

func (*Manager) RemoveNodeDef added in v0.0.1 

func (mgr *Manager) RemoveNodeDef(kind string) error

SaveNodeDef removes the NodeDef registrations in the Cfg system for this Manager node instance.

func (*Manager) RemovePIndex added in v0.0.1 

func (mgr *Manager) RemovePIndex(pindex *PIndex) error

ClosePIndex synchronously has the janitor remove a pindex.

func (*Manager) SaveNodeDef 

func (mgr *Manager) SaveNodeDef(kind string, force bool) error

SaveNodeDef updates the NodeDef registrations in the Cfg system for this Manager node instance.

func (*Manager) Start 

func (mgr *Manager) Start(register string) error

Start will start and register a Manager instance with its configured Cfg system, based on the register parameter. See Manager.StartRegister().

func (*Manager) StartRegister added in v0.0.1 

func (mgr *Manager) StartRegister(register string) error

StartRegister will register or unregister a Manager with its configured Cfg system, based on the register parameter, which can have these values: * wanted - register this node as wanted * wantedForce - same as wanted, but force a Cfg update * known - register this node as known * knownForce - same as unknown, but force a Cfg update * unwanted - unregister this node no longer wanted * unknown - unregister this node no longer wanted and no longer known * unchanged - don't change any Cfg registrations for this node

func (*Manager) UUID added in v0.0.1 

func (mgr *Manager) UUID() string

Returns the UUID (the "node UUID") of a Manager.

type ManagerEventHandlers 

type ManagerEventHandlers interface {
	OnRegisterPIndex(pindex *PIndex)
	OnUnregisterPIndex(pindex *PIndex)
}

ManagerEventHandlers represents the callback interface where an application can receive important event callbacks from a Manager.

type ManagerKickHandler added in v0.0.1 

type ManagerKickHandler struct {
	// contains filtered or unexported fields
}

ManagerKickHandler is a REST handler that processes a request to kick a manager.

func NewManagerKickHandler added in v0.0.1 

func NewManagerKickHandler(mgr *Manager) *ManagerKickHandler

func (*ManagerKickHandler) ServeHTTP added in v0.0.1 

func (h *ManagerKickHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type ManagerMetaHandler added in v0.0.1 

type ManagerMetaHandler struct {
	// contains filtered or unexported fields
}

ManagerMetaHandler is a REST handler that returns metadata about a manager/node.

func NewManagerMetaHandler added in v0.0.1 

func NewManagerMetaHandler(mgr *Manager,
	meta map[string]RESTMeta) *ManagerMetaHandler

func (*ManagerMetaHandler) ServeHTTP added in v0.0.1 

func (h *ManagerMetaHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type ManagerStats added in v0.0.1 

type ManagerStats struct {
	TotKick uint64

	TotSaveNodeDef             uint64
	TotSaveNodeDefGetErr       uint64
	TotSaveNodeDefSetErr       uint64
	TotSaveNodeDefUUIDTakenErr uint64
	TotSaveNodeDefUUIDErr      uint64
	TotSaveNodeDefOk           uint64

	TotCreateIndex    uint64
	TotCreateIndexOk  uint64
	TotDeleteIndex    uint64
	TotDeleteIndexOk  uint64
	TotIndexControl   uint64
	TotIndexControlOk uint64

	TotPlannerNOOP              uint64
	TotPlannerNOOPOk            uint64
	TotPlannerKick              uint64
	TotPlannerKickStart         uint64
	TotPlannerKickChanged       uint64
	TotPlannerKickErr           uint64
	TotPlannerKickOk            uint64
	TotPlannerUnknownErr        uint64
	TotPlannerSubscriptionEvent uint64

	TotJanitorNOOP              uint64
	TotJanitorNOOPOk            uint64
	TotJanitorKick              uint64
	TotJanitorKickStart         uint64
	TotJanitorKickErr           uint64
	TotJanitorKickOk            uint64
	TotJanitorClosePIndex       uint64
	TotJanitorRemovePIndex      uint64
	TotJanitorUnknownErr        uint64
	TotJanitorSubscriptionEvent uint64
}

ManagerStats represents the stats/metrics tracked by a Manager instance.

func (*ManagerStats) AtomicCopyTo added in v0.0.1 

func (s *ManagerStats) AtomicCopyTo(r *ManagerStats)

AtomicCopyTo copies metrics from s to r (from source to result).

type MetaDesc added in v0.0.1 

type MetaDesc struct {
	Description     string            `json:"description"`
	StartSample     interface{}       `json:"startSample"`
	StartSampleDocs map[string]string `json:"startSampleDocs"`
}

MetaDesc represents a part of the JSON of a ManagerMetaHandler REST response.

type MetaDescIndex added in v0.0.3 

type MetaDescIndex struct {
	MetaDesc

	CanCount bool `json:"canCount"`
	CanQuery bool `json:"canQuery"`

	QuerySamples interface{} `json:"querySamples"`
	QueryHelp    string      `json:"queryHelp"`
}

MetaDescSource represents the index-type parts of the JSON of a ManagerMetaHandler REST response.

type MetaDescSource added in v0.0.3 

type MetaDescSource MetaDesc

MetaDescSource represents the source-type/feed-type parts of the JSON of a ManagerMetaHandler REST response.

type MsgRing added in v0.0.1 

type MsgRing struct {
	Next int      `json:"next"`
	Msgs [][]byte `json:"msgs"`
	// contains filtered or unexported fields
}

A MsgRing wraps an io.Writer, and remembers a ring of previous writes to the io.Writer. It is concurrent safe and is useful, for example, for remembering recent log messages.

func NewMsgRing added in v0.0.1 

func NewMsgRing(inner io.Writer, ringSize int) (*MsgRing, error)

NewMsgRing returns a MsgRing of a given ringSize.

func (*MsgRing) Messages added in v0.0.1 

func (m *MsgRing) Messages() [][]byte

Retrieves the recent writes to the MsgRing.

func (*MsgRing) Write added in v0.0.1 

func (m *MsgRing) Write(p []byte) (n int, err error)

Implements the io.Writer interface.

type NILFeed added in v0.0.1 

type NILFeed struct {
	// contains filtered or unexported fields
}

A NILFeed implements the Feed interface and never feeds any data to its Dest instances. It's useful for testing and for pindexes that are actually primary data sources.

See also the "blackhole" pindex type for the "opposite equivalent" of a NILFeed.

func NewNILFeed added in v0.0.1 

func NewNILFeed(name, indexName string, dests map[string]Dest) *NILFeed

NewNILFeed creates a ready-to-be-started NILFeed instance.

func (*NILFeed) Close added in v0.0.1 

func (t *NILFeed) Close() error

func (*NILFeed) Dests added in v0.0.1 

func (t *NILFeed) Dests() map[string]Dest

func (*NILFeed) IndexName added in v0.0.1 

func (t *NILFeed) IndexName() string

func (*NILFeed) Name added in v0.0.1 

func (t *NILFeed) Name() string

func (*NILFeed) Start added in v0.0.1 

func (t *NILFeed) Start() error

func (*NILFeed) Stats added in v0.0.1 

func (t *NILFeed) Stats(w io.Writer) error

type NodeDef 

type NodeDef struct {
	HostPort    string   `json:"hostPort"`
	UUID        string   `json:"uuid"`
	ImplVersion string   `json:"implVersion"` // See VERSION.
	Tags        []string `json:"tags"`
	Container   string   `json:"container"`
	Weight      int      `json:"weight"`
	Extras      string   `json:"extras"`
}

A NodeDef is a node definition.

type NodeDefs 

type NodeDefs struct {
	// NodeDefs.UUID changes whenever any child NodeDef changes.
	UUID        string              `json:"uuid"`
	NodeDefs    map[string]*NodeDef `json:"nodeDefs"`    // Key is NodeDef.HostPort.
	ImplVersion string              `json:"implVersion"` // See VERSION.
}

A NodeDefs is comprised of zero or more node definitions.

func CfgGetNodeDefs 

func CfgGetNodeDefs(cfg Cfg, kind string) (*NodeDefs, uint64, error)

Retrieves node definitions from a Cfg provider.

func NewNodeDefs 

func NewNodeDefs(version string) *NodeDefs

Returns an initialized NodeDefs.

func PlannerGetNodeDefs added in v0.0.1 

func PlannerGetNodeDefs(cfg Cfg, version, uuid, bindHttp string) (
	*NodeDefs, error)

PlannerGetNodeDefs retrives node definitions from a Cfg.

type NodePlanParam added in v0.0.1 

type NodePlanParam struct {
	CanRead  bool `json:"canRead"`
	CanWrite bool `json:"canWrite"`
}

A NodePlanParam defines whether a particular node can service a particular index indefinition.

func GetNodePlanParam added in v0.0.1 

func GetNodePlanParam(nodePlanParams map[string]map[string]*NodePlanParam,
	nodeUUID, indexDefName, planPIndexName string) *NodePlanParam

GetNodePlanParam returns a relevant NodePlanParam for a given node from a nodePlanParams, defaulting to a less-specific NodePlanParam if needed.

type PIndex 

type PIndex struct {
	Name             string     `json:"name"`
	UUID             string     `json:"uuid"`
	IndexType        string     `json:"indexType"`
	IndexName        string     `json:"indexName"`
	IndexUUID        string     `json:"indexUUID"`
	IndexParams      string     `json:"indexParams"`
	SourceType       string     `json:"sourceType"`
	SourceName       string     `json:"sourceName"`
	SourceUUID       string     `json:"sourceUUID"`
	SourceParams     string     `json:"sourceParams"`
	SourcePartitions string     `json:"sourcePartitions"`
	Path             string     `json:"-"` // Transient, not persisted.
	Impl             PIndexImpl `json:"-"` // Transient, not persisted.
	Dest             Dest       `json:"-"` // Transient, not persisted.
	// contains filtered or unexported fields
}

A PIndex represents a partition of an index, or an "index partition". A logical index definition will be split into one or more pindexes.

func NewPIndex 

func NewPIndex(mgr *Manager, name, uuid,
	indexType, indexName, indexUUID, indexParams,
	sourceType, sourceName, sourceUUID, sourceParams, sourcePartitions string,
	path string) (*PIndex, error)

Creates a pindex, including its backend implementation structures, and its files.

func OpenPIndex 

func OpenPIndex(mgr *Manager, path string) (*PIndex, error)

OpenPIndex reopens a previously created pindex. The path argument must be a directory for the pindex.

func (*PIndex) Close added in v0.0.1 

func (p *PIndex) Close(remove bool) error

Close down a pindex, optionally removing its stored files.

type PIndexImpl added in v0.0.1 

type PIndexImpl interface{}

PIndexImpl represents a runtime pindex implementation instance, whose runtime type depends on the pindex's type.

type PIndexImplType added in v0.0.1 

type PIndexImplType struct {
	// Invoked by the manager when it wants validate indef definition
	// inputs before doing the actual creation.
	Validate func(indexType, indexName, indexParams string) error

	// Invoked by the manager when it wants to create an index
	// partition.  The pindex implementation should persist enough
	// info into the path subdirectory so that it can reconstitute the
	// pindex during restart and Open().
	New func(indexType, indexParams, path string, restart func()) (
		PIndexImpl, Dest, error)

	// Invoked by the manager when it wants a pindex implementation to
	// reconstitute and reload a pindex instance back into the
	// process, such as when the process has re-started.
	Open func(indexType, path string, restart func()) (
		PIndexImpl, Dest, error)

	// Invoked by the manager when it wants a count of documents from
	// an index.  The registered Count() function can be nil.
	Count func(mgr *Manager, indexName, indexUUID string) (
		uint64, error)

	// Invoked by the manager when it wants to query an index.  The
	// registered Query() function can be nil.
	Query func(mgr *Manager, indexName, indexUUID string,
		req []byte, res io.Writer) error

	// Description is used to populate docs, UI, etc, such as index
	// type drop-down control in the web admin UI.  Format of the
	// description string:
	//
	//    $categoryName/$indexType - short descriptive string
	//
	// The $categoryName is something like "advanced", or "general".
	Description string

	// A prototype instance of indexParams JSON that is usable for
	// Validate() and New().
	StartSample interface{}

	// Example instances of JSON that are usable for Query requests().
	// These are used to help generate API documentation.
	QuerySamples func() []Documentation

	// Displayed in docs, web admin UI, etc, and often might be a link
	// to even further help.
	QueryHelp string

	// Invoked during startup to allow pindex implementation to affect
	// the REST API with its own endpoint.
	InitRouter func(r *mux.Router, phase string)

	// Optional, additional handlers a pindex implementation may have
	// for /api/diag output.
	DiagHandlers []DiagHandler

	// Optional, allows pindex implementaiton to add more information
	// to the REST /api/managerMeta output.
	MetaExtra func(map[string]interface{})
}

PIndexImplType defines the functions that every pindex implementation type must register on startup.

func PIndexImplTypeForIndex added in v0.0.1 

func PIndexImplTypeForIndex(cfg Cfg, indexName string) (
	*PIndexImplType, error)

PIndexImplTypeForIndex retrieves from the Cfg provider the index type for a given index.

type PIndexStoreStats added in v0.0.1 

type PIndexStoreStats struct {
	TimerBatchStore metrics.Timer
	Errors          *list.List // Capped list of string (json).
}

PIndexStoreStats provides some common stats/metrics and error tracking that some pindex type backends can reuse.

func (*PIndexStoreStats) WriteJSON added in v0.0.1 

func (d *PIndexStoreStats) WriteJSON(w io.Writer)

type PlanPIndex 

type PlanPIndex struct {
	Name             string `json:"name"` // Stable & unique cluster wide.
	UUID             string `json:"uuid"`
	IndexType        string `json:"indexType"`   // See IndexDef.Type.
	IndexName        string `json:"indexName"`   // See IndexDef.Name.
	IndexUUID        string `json:"indexUUID"`   // See IndefDef.UUID.
	IndexParams      string `json:"indexParams"` // See IndexDef.Params.
	SourceType       string `json:"sourceType"`
	SourceName       string `json:"sourceName"`
	SourceUUID       string `json:"sourceUUID"`
	SourceParams     string `json:"sourceParams"` // Optional connection info.
	SourcePartitions string `json:"sourcePartitions"`

	Nodes map[string]*PlanPIndexNode `json:"nodes"` // Keyed by NodeDef.UUID.
}

A PlanPIndex represents the plan for a particular index partition, including on what nodes that the index partition is assigned to. An index partition might be assigned to more than one node if the "plan params" has a replica count > 0.

type PlanPIndexFilter added in v0.0.3 

type PlanPIndexFilter func(*PlanPIndexNode) bool

PlanPIndexFilter is used to filter out nodes being considered by CoveringPIndexes().

type PlanPIndexNode added in v0.0.1 

type PlanPIndexNode struct {
	CanRead  bool `json:"canRead"`
	CanWrite bool `json:"canWrite"`
	Priority int  `json:"priority"` // Lower is higher priority, 0 is highest.
}

A PlanPIndexNode represents the kind of service a node has been assigned to provide for an index partition.

type PlanPIndexNodeRef added in v0.0.1 

type PlanPIndexNodeRef struct {
	UUID string
	Node *PlanPIndexNode
}

PlanPIndexNodeRef represents an assignment of a pindex to a node.

type PlanPIndexNodeRefs added in v0.0.1 

type PlanPIndexNodeRefs []*PlanPIndexNodeRef

PlanPIndexNodeRefs represents assignments of pindexes to nodes.

func (PlanPIndexNodeRefs) Len added in v0.0.1 

func (pms PlanPIndexNodeRefs) Len() int

func (PlanPIndexNodeRefs) Less added in v0.0.1 

func (pms PlanPIndexNodeRefs) Less(i, j int) bool

func (PlanPIndexNodeRefs) Swap added in v0.0.1 

func (pms PlanPIndexNodeRefs) Swap(i, j int)

type PlanPIndexes 

type PlanPIndexes struct {
	// PlanPIndexes.UUID changes whenever any child PlanPIndex changes.
	UUID         string                 `json:"uuid"`
	PlanPIndexes map[string]*PlanPIndex `json:"planPIndexes"` // Key is PlanPIndex.Name.
	ImplVersion  string                 `json:"implVersion"`  // See VERSION.
	Warnings     map[string][]string    `json:"warnings"`     // Key is IndexDef.Name.
}

A PlanPIndexes is comprised of zero or more planPIndexes.

func CalcPlan 

func CalcPlan(indexDefs *IndexDefs, nodeDefs *NodeDefs,
	planPIndexesPrev *PlanPIndexes, version, server string) (
	*PlanPIndexes, error)

Split logical indexes into PIndexes and assign PIndexes to nodes.

func CfgGetPlanPIndexes 

func CfgGetPlanPIndexes(cfg Cfg) (*PlanPIndexes, uint64, error)

Retrieves PlanPIndexes from a Cfg provider.

func NewPlanPIndexes 

func NewPlanPIndexes(version string) *PlanPIndexes

Returns an initialized PlanPIndexes.

func PlannerGetPlanPIndexes added in v0.0.1 

func PlannerGetPlanPIndexes(cfg Cfg, version string) (
	*PlanPIndexes, uint64, error)

PlannerGetPlanPIndexes retrieves the planned pindexes from a Cfg.

type PlanParams added in v0.0.1 

type PlanParams struct {
	// MaxPartitionsPerPIndex controls the maximum number of source
	// partitions the planner can assign to or clump into a PIndex (or
	// index partition).
	MaxPartitionsPerPIndex int `json:"maxPartitionsPerPIndex"`

	// NumReplicas controls the number of replicas for a PIndex, over
	// the first copy.  The first copy is not counted as a replica.
	// For example, a NumReplicas setting of 2 means there should be a
	// primary and 2 replicas... so 3 copies in total.  A NumReplicas
	// of 0 means just the first, primary copy only.
	NumReplicas int `json:"numReplicas"`

	// HierarchyRules defines the policy the planner should follow
	// when assigning PIndexes to nodes, especially for replica
	// placement.  Through the HierarchyRules, a user can specify, for
	// example, that the first replica should be not on the same rack
	// and zone as the first copy.  Some examples:
	// Try to put the first replica on the same rack...
	// {"replica":[{"includeLevel":1,"excludeLevel":0}]}
	// Try to put the first replica on a different rack...
	// {"replica":[{"includeLevel":2,"excludeLevel":1}]}
	HierarchyRules blance.HierarchyRules `json:"hierarchyRules"`

	// NodePlanParams allows users to specify per-node input to the
	// planner, such as whether PIndexes assigned to different nodes
	// can be readable or writable.  Keyed by node UUID.  Value is
	// keyed by planPIndex.Name or indexDef.Name.  The empty string
	// ("") is used to represent any node UUID and/or any planPIndex
	// and/or any indexDef.
	NodePlanParams map[string]map[string]*NodePlanParam `json:"nodePlanParams"`

	// PlanFrozen means the planner should not change the previous
	// plan for an index, even if as nodes join or leave and even if
	// there was no previous plan.  Defaults to false (allow
	// re-planning).
	PlanFrozen bool `json:"planFrozen"`
}

A PlanParams holds input parameters to the planner, that control how the planner should split an index definition into one or more index partitions, and how the planner should assign those index partitions to nodes.

type PrimaryFeed added in v0.0.1 

type PrimaryFeed struct {
	// contains filtered or unexported fields
}

A PrimaryFeed implements both the Feed and Dest interfaces, for chainability; and is also useful for testing.

One motivation for a PrimaryFeed implementation is from the realization that some pindex backends might not actually be secondary indexes, but are instead better considered as primary data sources in their own right. For example, you can imagine some kind of KeyValuePIndex backend. The system design, however, still requires hooking such "primary pindexes" up to a feed. Instead of using a NILFeed, you might instead use a PrimaryFeed, as unlike a NILFeed the PrimaryFeed provides a "NumPartitions" functionality.

func NewPrimaryFeed added in v0.0.1 

func NewPrimaryFeed(name, indexName string, pf DestPartitionFunc,
	dests map[string]Dest) *PrimaryFeed

func (*PrimaryFeed) Close added in v0.0.1 

func (t *PrimaryFeed) Close() error

func (*PrimaryFeed) ConsistencyWait added in v0.0.1 

func (t *PrimaryFeed) ConsistencyWait(partition, partitionUUID string,
	consistencyLevel string,
	consistencySeq uint64,
	cancelCh <-chan bool) error

func (*PrimaryFeed) Count added in v0.0.1 

func (t *PrimaryFeed) Count(pindex *PIndex, cancelCh <-chan bool) (
	uint64, error)

func (*PrimaryFeed) DataDelete added in v0.0.3 

func (t *PrimaryFeed) DataDelete(partition string,
	key []byte, seq uint64,
	cas uint64,
	extrasType DestExtrasType, extras []byte) error

func (*PrimaryFeed) DataUpdate added in v0.0.3 

func (t *PrimaryFeed) DataUpdate(partition string,
	key []byte, seq uint64, val []byte,
	cas uint64,
	extrasType DestExtrasType, extras []byte) error

func (*PrimaryFeed) Dests added in v0.0.1 

func (t *PrimaryFeed) Dests() map[string]Dest

func (*PrimaryFeed) IndexName added in v0.0.1 

func (t *PrimaryFeed) IndexName() string

func (*PrimaryFeed) Name added in v0.0.1 

func (t *PrimaryFeed) Name() string

func (*PrimaryFeed) OpaqueGet added in v0.0.3 

func (t *PrimaryFeed) OpaqueGet(partition string) (
	value []byte, lastSeq uint64, err error)

func (*PrimaryFeed) OpaqueSet added in v0.0.3 

func (t *PrimaryFeed) OpaqueSet(partition string,
	value []byte) error

func (*PrimaryFeed) Query added in v0.0.1 

func (t *PrimaryFeed) Query(pindex *PIndex, req []byte, w io.Writer,
	cancelCh <-chan bool) error

func (*PrimaryFeed) Rollback added in v0.0.1 

func (t *PrimaryFeed) Rollback(partition string,
	rollbackSeq uint64) error

func (*PrimaryFeed) SnapshotStart added in v0.0.3 

func (t *PrimaryFeed) SnapshotStart(partition string,
	snapStart, snapEnd uint64) error

func (*PrimaryFeed) Start added in v0.0.1 

func (t *PrimaryFeed) Start() error

func (*PrimaryFeed) Stats added in v0.0.1 

func (t *PrimaryFeed) Stats(w io.Writer) error

type PrimarySourceParams added in v0.1.0 

type PrimarySourceParams struct {
	NumPartitions int `json:"numPartitions"`
}

PrimarySourceParams represents the JSON for the sourceParams for a primary feed.

type QueryCtl added in v0.2.0 

type QueryCtl struct {
	Timeout     int64              `json:"timeout"`
	Consistency *ConsistencyParams `json:"consistency"`
}

QueryCtl defines the JSON parameters that control query execution and which are independent of any specific pindex type.

type QueryCtlParams added in v0.2.0 

type QueryCtlParams struct {
	Ctl QueryCtl `json:"ctl"`
}

QueryCtlParams defines the JSON that includes the "ctl" part of a query request. These "ctl" query request parameters are independent of any specific pindex type.

type QueryHandler added in v0.0.1 

type QueryHandler struct {
	// contains filtered or unexported fields
}

QueryHandler is a REST handler for querying an index.

func NewQueryHandler added in v0.0.1 

func NewQueryHandler(mgr *Manager) *QueryHandler

func (*QueryHandler) RESTOpts added in v0.0.1 

func (h *QueryHandler) RESTOpts(opts map[string]string)

func (*QueryHandler) ServeHTTP added in v0.0.1 

func (h *QueryHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type QueryPIndexHandler added in v0.0.1 

type QueryPIndexHandler struct {
	// contains filtered or unexported fields
}

QueryPIndexHandler is a REST handler for querying a pindex.

func NewQueryPIndexHandler added in v0.0.1 

func NewQueryPIndexHandler(mgr *Manager) *QueryPIndexHandler

func (*QueryPIndexHandler) ServeHTTP added in v0.0.1 

func (h *QueryPIndexHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type RESTMeta added in v0.0.1 

type RESTMeta struct {
	Path   string
	Method string
	Opts   map[string]string
}

RESTMeta represents the metadata of a REST API endpoint and is used for auto-generated REST API documentation.

type RESTOpts added in v0.0.1 

type RESTOpts interface {
	RESTOpts(map[string]string)
}

RESTOpts interface may be optionally implemented by REST API handlers to provide even more information for auto-generated REST API documentation.

type RemotePlanPIndex added in v0.0.1 

type RemotePlanPIndex struct {
	PlanPIndex *PlanPIndex
	NodeDef    *NodeDef
}

RemotePlanPIndex associations are returned by CoveringPIndexes().

type RuntimeGetHandler added in v0.0.1 

type RuntimeGetHandler struct {
	// contains filtered or unexported fields
}

RuntimeGetHandler is a REST handler for runtime GET endpoint.

func NewRuntimeGetHandler added in v0.0.1 

func NewRuntimeGetHandler(versionMain string, mgr *Manager) *RuntimeGetHandler

func (*RuntimeGetHandler) ServeHTTP added in v0.0.1 

func (h *RuntimeGetHandler) ServeHTTP(w http.ResponseWriter, r *http.Request)

type ScanCursor added in v0.0.1 

type ScanCursor interface {
	Done() bool
	Key() []byte
	Val() []byte
	Next() bool
}

A ScanCursor represents a scan of a resultset.

type ScanCursors added in v0.0.1 

type ScanCursors []ScanCursor

ScanCursors implements the heap.Interface for easy merging.

func (ScanCursors) Len added in v0.0.1 

func (pq ScanCursors) Len() int

func (ScanCursors) Less added in v0.0.1 

func (pq ScanCursors) Less(i, j int) bool

func (*ScanCursors) Pop added in v0.0.1 

func (pq *ScanCursors) Pop() interface{}

func (*ScanCursors) Push added in v0.0.1 

func (pq *ScanCursors) Push(x interface{})

func (ScanCursors) Swap added in v0.0.1 

func (pq ScanCursors) Swap(i, j int)

type StatsHandler added in v0.0.1 

type StatsHandler struct {
	// contains filtered or unexported fields
}

StatsHandler is a REST handler that provides stats/metrics for a node.

func NewStatsHandler added in v0.0.1 

func NewStatsHandler(mgr *Manager) *StatsHandler

func (*StatsHandler) ServeHTTP added in v0.0.1 

func (h *StatsHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)

type TAPFeed 

type TAPFeed struct {
	// contains filtered or unexported fields
}

A TAPFeed implements the Feed interface and handles the TAP protocol to receive data from a couchbase data source.

func NewTAPFeed 

func NewTAPFeed(name, indexName, url, poolName, bucketName, bucketUUID,
	paramsStr string, pf DestPartitionFunc, dests map[string]Dest,
	disable bool) (*TAPFeed, error)

NewTAPFeed creates a new, ready-to-be-started TAPFeed.

func (*TAPFeed) Close 

func (t *TAPFeed) Close() error

func (*TAPFeed) Dests added in v0.0.1 

func (t *TAPFeed) Dests() map[string]Dest

func (*TAPFeed) IndexName added in v0.0.1 

func (t *TAPFeed) IndexName() string

func (*TAPFeed) Name 

func (t *TAPFeed) Name() string

func (*TAPFeed) Start 

func (t *TAPFeed) Start() error

func (*TAPFeed) Stats added in v0.0.1 

func (t *TAPFeed) Stats(w io.Writer) error

type TAPFeedParams added in v0.0.1 

type TAPFeedParams struct {
	BackoffFactor float32 `json:"backoffFactor"`
	SleepInitMS   int     `json:"sleepInitMS"`
	SleepMaxMS    int     `json:"sleepMaxMS"`
}

TAPFeedParams represents the JSON of the sourceParams for a TAP feed.

Source Files

View all Source files

Directories

Path Synopsis
cmd
cbft
cbft_docs

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.