pilosa

package module
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 4, 2018 License: Apache-2.0 Imports: 49 Imported by: 0

README

Build Status GoDoc Go Report Card license CLA Assistant GitHub release

An open source, distributed bitmap index.

Want to contribute? One of the easiest ways is to tell us how you're using (or want to use) Pilosa. We learn from every discussion!

Docs

See our Documentation for information about installing and working with Pilosa.

Getting Started

  1. Install Pilosa.

  2. Start Pilosa with the default configuration:

    pilosa server

    and verify that it's running:

    curl localhost:10101/nodes

  3. Follow along with the Sample Project to get a better understanding of Pilosa's capabilities.

Data Model

Check out how the Pilosa Data Model works.

Query Language

You can interact with Pilosa directly in the console using the Pilosa Query Language (PQL).

Client Libraries

There are supported libraries for the following languages:

Get Support

There are several channels available for you to reach out to us for support.

Contributing

Pilosa is an open source project. Please see our Contributing Guide for information about how to get involved.

Documentation

Overview

Package pilosa implements the core of the Pilosa distributed bitmap index. It contains all the domain objects, interfaces, and logic that defines pilosa.

Index

Constants

View Source 
const (
	AttrTypeString = 1
	AttrTypeInt    = 2
	AttrTypeBool   = 3
	AttrTypeFloat  = 4
)

Attribute data type enum.

View Source 
const (
	MessageTypeCreateSlice = iota
	MessageTypeCreateIndex
	MessageTypeDeleteIndex
	MessageTypeCreateFrame
	MessageTypeDeleteFrame
	MessageTypeCreateView
	MessageTypeDeleteView
	MessageTypeCreateField
	MessageTypeDeleteField
	MessageTypeCreateInputDefinition
	MessageTypeDeleteInputDefinition
	MessageTypeClusterStatus
	MessageTypeResizeInstruction
	MessageTypeResizeInstructionComplete
	MessageTypeSetCoordinator
	MessageTypeUpdateCoordinator
	MessageTypeNodeState
	MessageTypeRecalculateCaches
	MessageTypeNodeEvent
)

Broadcast message types.

View Source 
const (
	// DefaultPartitionN is the default number of partitions in a cluster.
	DefaultPartitionN = 256

	// ClusterState represents the state returned in the /status endpoint.
	ClusterStateStarting = "STARTING"
	ClusterStateNormal   = "NORMAL"
	ClusterStateResizing = "RESIZING"

	// NodeState represents the state of a node during startup.
	NodeStateLoading = "LOADING"
	NodeStateReady   = "READY"

	// ResizeJob states.
	ResizeJobStateRunning = "RUNNING"
	// Final states.
	ResizeJobStateDone    = "DONE"
	ResizeJobStateAborted = "ABORTED"

	ResizeJobActionAdd    = "ADD"
	ResizeJobActionRemove = "REMOVE"
)
View Source 
const (
	DefaultFrame = "general"

	// MinThreshold is the lowest count to use in a Top-N operation when
	// looking for additional id/count pairs.
	MinThreshold = 1
)

DefaultFrame is the frame used if one is not specified.

View Source 
const (
	// SliceWidth is the number of column IDs in a slice.
	SliceWidth = 1048576

	// SnapshotExt is the file extension used for an in-process snapshot.
	SnapshotExt = ".snapshotting"

	// CopyExt is the file extension used for the temp file used while copying.
	CopyExt = ".copying"

	// CacheExt is the file extension for persisted cache ids.
	CacheExt = ".cache"

	// HashBlockSize is the number of rows in a merkle hash block.
	HashBlockSize = 100
)
View Source 
const (
	DefaultCacheType      = CacheTypeRanked
	DefaultInverseEnabled = false

	// Default ranked frame cache
	DefaultCacheSize = 50000
)

Default frame settings.

View Source 
const (
	CacheTypeLRU    = "lru"
	CacheTypeRanked = "ranked"
	CacheTypeNone   = "none"
)

Cache types.

View Source 
const (
	QueryResultTypeNil uint32 = iota
	QueryResultTypeBitmap
	QueryResultTypePairs
	QueryResultTypeValCount
	QueryResultTypeUint64
	QueryResultTypeBool
)

QueryResult types.

View Source 
const (
	// DefaultCacheFlushInterval is the default value for Fragment.CacheFlushInterval.
	DefaultCacheFlushInterval = 1 * time.Minute

	// FileLimit is the maximum open file limit (ulimit -n) to automatically set.
	FileLimit = 262144 // (512^2)
)
View Source 
const (
	InputMapping       = "mapping"
	InputValueToRow    = "value-to-row"
	InputSingleRowBool = "single-row-boolean"
	InputSetTimestamp  = "set-timestamp"
)

Action types.

View Source 
const (
	ViewStandard = "standard"
	ViewInverse  = "inverse"

	ViewFieldPrefix = "field_"
)

View layout modes.

View Source 
const (
	DefaultDiagnosticServer = "https://diagnostics.pilosa.com/v0/diagnostics"
)

Default server settings.

View Source 
const (
	// DefaultFragmentMaxOpN is the default value for Fragment.MaxOpN.
	DefaultFragmentMaxOpN = 2000
)
View Source 
const (
	FieldTypeInt = "int"
)

List of field data types.

View Source 
const (
	InputDefinitionDir = ".input-definitions"
)

Default index settings.

View Source 
const (
	// ThresholdFactor is used to calculate the threshold for new items entering the cache
	ThresholdFactor = 1.1
)
View Source 
const TimeFormat = "2006-01-02T15:04"

TimeFormat is the go-style time format used to parse string dates.

Variables

View Source 
var (
	ErrHostRequired = errors.New("host required")

	ErrIndexRequired = errors.New("index required")
	ErrIndexExists   = errors.New("index already exists")
	ErrIndexNotFound = errors.New("index not found")

	// ErrFrameRequired is returned when no frame is specified.
	ErrFrameRequired        = errors.New("frame required")
	ErrFrameExists          = errors.New("frame already exists")
	ErrFrameNotFound        = errors.New("frame not found")
	ErrFrameInverseDisabled = errors.New("frame inverse disabled")

	ErrInputDefinitionExists         = errors.New("input-definition already exists")
	ErrInputDefinitionHasPrimaryKey  = errors.New("input-definition must contain one PrimaryKey")
	ErrInputDefinitionDupePrimaryKey = errors.New("input-definition can only contain one PrimaryKey")
	ErrInputDefinitionNameRequired   = errors.New("input-definition name required")
	ErrInputDefinitionAttrsRequired  = errors.New("frames and fields are required")
	ErrInputDefinitionValueMap       = errors.New("valueMap required for map")
	ErrInputDefinitionActionRequired = errors.New("field definitions require an action")
	ErrInputDefinitionNotFound       = errors.New("input-definition not found")

	ErrFieldNotFound         = errors.New("field not found")
	ErrFieldExists           = errors.New("field already exists")
	ErrFieldNameRequired     = errors.New("field name required")
	ErrInvalidFieldType      = errors.New("invalid field type")
	ErrInvalidFieldRange     = errors.New("invalid field range")
	ErrInvalidFieldValueType = errors.New("invalid field value type")
	ErrFieldValueTooLow      = errors.New("field value too low")
	ErrFieldValueTooHigh     = errors.New("field value too high")
	ErrInvalidRangeOperation = errors.New("invalid range operation")
	ErrInvalidBetweenValue   = errors.New("invalid value for between operation")

	ErrInvalidView      = errors.New("invalid view")
	ErrInvalidCacheType = errors.New("invalid cache type")

	ErrName  = errors.New("invalid index or frame's name, must match [a-z0-9_-]")
	ErrLabel = errors.New("invalid row or column label, must match [A-Za-z0-9_-]")

	// ErrFragmentNotFound is returned when a fragment does not exist.
	ErrFragmentNotFound = errors.New("fragment not found")
	ErrQueryRequired    = errors.New("query required")
	ErrTooManyWrites    = errors.New("too many write commands")

	ErrClusterDoesNotOwnSlice = errors.New("cluster does not own slice")

	ErrNodeIDNotExists    = errors.New("node with provided ID does not exist")
	ErrNodeNotCoordinator = errors.New("node is not the coordinator")
	ErrResizeNotRunning   = errors.New("no resize job currently running")
)

System errors.

View Source 
var BuildTime = "not recorded"
View Source 
var ErrInvalidTimeQuantum = errors.New("invalid time quantum")

ErrInvalidTimeQuantum is returned when parsing a time quantum.

View Source 
var Expvar = expvar.NewMap("index")

Expvar global expvar map.

View Source 
var NopBroadcastReceiver = &nopBroadcastReceiver{}

NopBroadcastReceiver is a no-op implementation of the BroadcastReceiver.

View Source 
var NopEventReceiver = &nopEventReceiver{}

NopEventReceiver is a no-op implementation of the EventReceiver.

View Source 
var Version = "v0.0.0"

Functions

func ContainsSubstring added in v0.5.0 

func ContainsSubstring(a string, list []string) bool

ContainsSubstring checks to see if substring a is contained in any string in the slice.

func CountOpenFiles added in v0.5.0 

func CountOpenFiles() (int, error)

CountOpenFiles on operating systems that support lsof.

func DecodeAttrs added in v0.9.0 

func DecodeAttrs(v []byte) (map[string]interface{}, error)

DecodeAttrs decodes a byte slice into an attribute map.

func EncodeAttrs added in v0.9.0 

func EncodeAttrs(attr map[string]interface{}) ([]byte, error)

EncodeAttrs encodes an attribute map into a byte slice.

func EncodeIndexes added in v0.4.0 

func EncodeIndexes(a []*Index) []*internal.Index

EncodeIndexes converts a into its internal representation.

func EncodeNode added in v0.9.0 

func EncodeNode(n *Node) *internal.Node

EncodeNode converts a Node into its internal representation.

func EncodeNodes added in v0.9.0 

func EncodeNodes(a []*Node) []*internal.Node

EncodeNodes converts a slice of Nodes into its internal representation.

func GetTimeStamp added in v0.5.0 

func GetTimeStamp(data map[string]interface{}, timeField string) (int64, error)

GetTimeStamp retrieves unix timestamp from Input data.

func HostToIP added in v0.6.0 

func HostToIP(host string) string

HostToIP converts host to an IP4 address based on net.LookupIP().

func IsInverseView 

func IsInverseView(name string) bool

IsInverseView returns true if the view is used for storing an inverted representation.

func IsValidCacheType 

func IsValidCacheType(v string) bool

IsValidCacheType returns true if v is a valid cache type.

func IsValidFieldType added in v0.5.0 

func IsValidFieldType(v string) bool

func IsValidView 

func IsValidView(name string) bool

IsValidView returns true if name is valid.

func MarshalMessage 

func MarshalMessage(m proto.Message) ([]byte, error)

MarshalMessage encodes the protobuf message into a byte slice.

func NewRouter 

func NewRouter(handler *Handler) *mux.Router

NewRouter creates a new mux http router.

func Pos 

func Pos(rowID, columnID uint64) uint64

Pos returns the row position of a row/column pair.

func SliceDiff added in v0.9.0 

func SliceDiff(a, b []uint64) []uint64

SliceDiff returns the difference between two uint64 slices.

func StringInSlice added in v0.5.0 

func StringInSlice(a string, list []string) bool

StringInSlice checks for substring a in the slice.

func StringSlicesAreEqual added in v0.9.0 

func StringSlicesAreEqual(a, b []string) bool

StringSlicesAreEqual determines if two string slices are equal.

func UnionStringSlice 

func UnionStringSlice(a, b []string) []string

UnionStringSlice returns a sorted set of tags which combine a & b.

func UnmarshalMessage 

func UnmarshalMessage(buf []byte) (proto.Message, error)

UnmarshalMessage decodes the byte slice into a protobuf message.

func ValidateField added in v0.5.0 

func ValidateField(f *Field) error

func ValidateName 

func ValidateName(name string) error

ValidateName ensures that the name is a valid format.

func ViewByTimeUnit 

func ViewByTimeUnit(name string, t time.Time, unit rune) string

ViewByTimeUnit returns the view name for time with a given quantum unit.

func ViewsByTime 

func ViewsByTime(name string, t time.Time, q TimeQuantum) []string

ViewsByTime returns a list of views for a given timestamp.

func ViewsByTimeRange 

func ViewsByTimeRange(name string, start, end time.Time, q TimeQuantum) []string

ViewsByTimeRange returns a list of views to traverse to query a time range.

Types

type API added in v0.9.0 

type API struct {
	Holder *Holder
	// The execution engine for running queries.
	Executor interface {
		Execute(context context.Context, index string, query *pql.Query, slices []uint64, opt *ExecOptions) ([]interface{}, error)
	}
	Broadcaster      Broadcaster
	BroadcastHandler BroadcastHandler
	StatusHandler    StatusHandler
	Cluster          *Cluster
	URI              URI
	RemoteClient     *http.Client
	Logger           Logger
}

API provides the top level programmatic interface to Pilosa. It is usually wrapped by a handler which provides an external interface (e.g. HTTP).

func NewAPI added in v0.9.0 

func NewAPI() *API

NewAPI returns a new API instance.

func (*API) ClusterMessage added in v0.9.0 

func (api *API) ClusterMessage(ctx context.Context, reqBody io.Reader) error

PostClusterMessage is for internal use. It decodes a protobuf message out of the body and forwards it to the BroadcastHandler.

func (*API) CreateField added in v0.9.0 

func (api *API) CreateField(ctx context.Context, indexName string, frameName string, field *Field) error

CreateField creates a new BSI field in the given index and frame.

func (*API) CreateFrame added in v0.9.0 

func (api *API) CreateFrame(ctx context.Context, indexName string, frameName string, options FrameOptions) (*Frame, error)

CreateFrame makes the named frame in the named index with the given options.

func (*API) CreateIndex added in v0.9.0 

func (api *API) CreateIndex(ctx context.Context, indexName string, options IndexOptions) (*Index, error)

CreateIndex makes a new Pilosa index.

func (*API) CreateInputDefinition added in v0.9.0 

func (api *API) CreateInputDefinition(ctx context.Context, indexName string, inputDefName string, inputDef InputDefinitionInfo) error

CreateInputDefinition is deprecated and will be removed. Do not use it.

func (*API) DeleteField added in v0.9.0 

func (api *API) DeleteField(ctx context.Context, indexName string, frameName string, fieldName string) error

DeleteField deletes the given field.

func (*API) DeleteFrame added in v0.9.0 

func (api *API) DeleteFrame(ctx context.Context, indexName string, frameName string) error

DeleteFrame removes the named frame from the named index. If the index is not found, an error is returned. If the frame is not found, it is ignored and no action is taken.

func (*API) DeleteIndex added in v0.9.0 

func (api *API) DeleteIndex(ctx context.Context, indexName string) error

DeleteIndex removes the named index. If the index is not found it does nothing and returns no error.

func (*API) DeleteInputDefinition added in v0.9.0 

func (api *API) DeleteInputDefinition(ctx context.Context, indexName string, inputDefName string) error

DeleteInputDefinition is deprecated and will be removed.

func (*API) DeleteView added in v0.9.0 

func (api *API) DeleteView(ctx context.Context, indexName string, frameName string, viewName string) error

DeleteView removes the given view.

func (*API) ExportCSV added in v0.9.0 

func (api *API) ExportCSV(ctx context.Context, indexName string, frameName string, viewName string, slice uint64, w io.Writer) error

ExportCSV encodes the fragment designated by the index,frame,view,slice as CSV of the form <row>,<col>

func (*API) Fields added in v0.9.0 

func (api *API) Fields(ctx context.Context, indexName string, frameName string) ([]*Field, error)

Fields returns the fields in the given frame.

func (*API) FragmentBlockData added in v0.9.0 

func (api *API) FragmentBlockData(ctx context.Context, body io.Reader) ([]byte, error)

FragmentBlockData is an endpoint for internal usage. It is not guaranteed to return anything useful. Currently it returns protobuf encoded row and column ids from a "block" which is a subdivision of a fragment.

func (*API) FragmentBlocks added in v0.9.0 

func (api *API) FragmentBlocks(ctx context.Context, indexName string, frameName string, viewName string, slice uint64) ([]FragmentBlock, error)

FragmentBlocks returns the checksums and block ids for all blocks in the specified fragment.

func (*API) FrameAttrDiff added in v0.9.0 

func (api *API) FrameAttrDiff(ctx context.Context, indexName string, frameName string, blocks []AttrBlock) (map[uint64]map[string]interface{}, error)

func (*API) Hosts added in v0.9.0 

func (api *API) Hosts(ctx context.Context) []*Node

Hosts returns a list of the hosts in the cluster including their ID, URL, and which is the coordinator.

func (*API) Import added in v0.9.0 

func (api *API) Import(ctx context.Context, req internal.ImportRequest) error

Import bulk imports data into a particular index,frame,slice.

func (*API) ImportValue added in v0.9.0 

func (api *API) ImportValue(ctx context.Context, req internal.ImportValueRequest) error

ImportValue bulk imports values into a particular field.

func (*API) Index added in v0.9.0 

func (api *API) Index(ctx context.Context, indexName string) (*Index, error)

Index retrieves the named index.

func (*API) IndexAttrDiff added in v0.9.0 

func (api *API) IndexAttrDiff(ctx context.Context, indexName string, blocks []AttrBlock) (map[uint64]map[string]interface{}, error)

IndexAttrDiff

func (*API) InputDefinition added in v0.9.0 

func (api *API) InputDefinition(ctx context.Context, indexName string, inputDefName string) (*InputDefinition, error)

InputDefinition is deprecated and will be removed.

func (*API) LocalID added in v0.9.0 

func (api *API) LocalID() string

LocalID returns the current node's ID.

func (*API) LongQueryTime added in v0.9.0 

func (api *API) LongQueryTime() time.Duration

LongQueryTime returns the configured threshold for logging/statting long running queries.

func (*API) MarshalFragment added in v0.9.0 

func (api *API) MarshalFragment(ctx context.Context, indexName string, frameName string, viewName string, slice uint64) (io.WriterTo, error)

MarshalFragment returns an object which can write the specified fragment's data to an io.Writer. The serialized data can be read back into a fragment with the UnmarshalFragment API call.

func (*API) MaxInverseSlices added in v0.9.0 

func (api *API) MaxInverseSlices(ctx context.Context) map[string]uint64

MaxInverseSlices returns the maximum inverse slice number for each index in a map.

func (*API) MaxSlices added in v0.9.0 

func (api *API) MaxSlices(ctx context.Context) map[string]uint64

MaxSlices returns the maximum slice number for each index in a map.

func (*API) Query added in v0.9.0 

func (api *API) Query(ctx context.Context, req *QueryRequest) (QueryResponse, error)

Query parses a PQL query out of the request and executes it.

func (*API) RecalculateCaches added in v0.9.0 

func (api *API) RecalculateCaches(ctx context.Context) error

RecalculateCaches forces all TopN caches to be updated. Used mainly for integration tests.

func (*API) RemoveNode added in v0.9.0 

func (api *API) RemoveNode(id string) (*Node, error)

RemoveNode puts the cluster into the "RESIZING" state and begins the job of removing the given node.

func (*API) ResizeAbort added in v0.9.0 

func (api *API) ResizeAbort() error

ResizeAbort stops the current resize job.

func (*API) RestoreFrame added in v0.9.0 

func (api *API) RestoreFrame(ctx context.Context, indexName string, frameName string, host *URI) error

RestoreFrame reads all the data that this host should have for a given frame from replicas in the cluster and restores that data to it.

func (*API) Schema added in v0.9.0 

func (api *API) Schema(ctx context.Context) []*IndexInfo

Schema returns information about each index in Pilosa including which frames and views they contain.

func (*API) SetCoordinator added in v0.9.0 

func (api *API) SetCoordinator(ctx context.Context, id string) (oldNode, newNode *Node, err error)

SetCoordinator makes a new Node the cluster coordinator.

func (*API) SliceNodes added in v0.9.0 

func (api *API) SliceNodes(ctx context.Context, indexName string, slice uint64) ([]*Node, error)

SliceNodes returns the node and all replicas which should contain a slice's data.

func (*API) State added in v0.9.0 

func (api *API) State() string

State returns the cluster state which is usually "NORMAL", but could be "STARTING", "RESIZING", or potentially others. See cluster.go for more details.

func (*API) StatsWithTags added in v0.9.0 

func (api *API) StatsWithTags(tags []string) StatsClient

StatsWithTags returns an instance of whatever implementation of StatsClient pilosa is using with the given tags.

func (*API) UnmarshalFragment added in v0.9.0 

func (api *API) UnmarshalFragment(ctx context.Context, indexName string, frameName string, viewName string, slice uint64, reader io.ReadCloser) error

UnmarshalFragment creates a new fragment (if necessary) and reads data from a Reader which was previously written by MarshalFragment to populate the fragment's data.

func (*API) Version added in v0.9.0 

func (api *API) Version() string

Version returns the Pilosa version.

func (*API) Views added in v0.9.0 

func (api *API) Views(ctx context.Context, indexName string, frameName string) ([]*View, error)

Views returns the views in the given frame.

func (*API) WriteInput added in v0.9.0 

func (api *API) WriteInput(ctx context.Context, indexName string, inputDefName string, reqs []interface{}) error

WriteInput is deprecated and will be removed.

type Action added in v0.5.0 

type Action struct {
	Frame            string            `json:"frame,omitempty"`
	ValueDestination string            `json:"valueDestination,omitempty"`
	ValueMap         map[string]uint64 `json:"valueMap,omitempty"`
	RowID            *uint64           `json:"rowID,omitempty"`
}

Action describes the mapping method for the field in the InputDefinition.

func (*Action) Encode added in v0.5.0 

func (a *Action) Encode() *internal.InputDefinitionAction

Encode converts Action into its internal representation.

func (*Action) Validate added in v0.5.0 

func (a *Action) Validate() error

Validate ensures the input definition action conforms to our specification.

type ApiMethodNotAllowedError added in v0.9.0 

type ApiMethodNotAllowedError struct {
	// contains filtered or unexported fields
}

ApiMethodNotAllowedError wraps an error value indicating that a particular API method is not allowed in the current cluster state.

type AttrBlock 

type AttrBlock struct {
	ID       uint64 `json:"id"`
	Checksum []byte `json:"checksum"`
}

AttrBlock represents a checksummed block of the attribute store.

type AttrBlocks 

type AttrBlocks []AttrBlock

AttrBlocks represents a list of blocks.

func (AttrBlocks) Diff 

func (a AttrBlocks) Diff(other []AttrBlock) []uint64

Diff returns a list of block ids that are different or are new in other. Block lists must be in sorted order.

type AttrStore 

type AttrStore interface {
	Path() string
	Open() error
	Close() error
	Attrs(id uint64) (m map[string]interface{}, err error)
	SetAttrs(id uint64, m map[string]interface{}) error
	SetBulkAttrs(m map[uint64]map[string]interface{}) error
	Blocks() ([]AttrBlock, error)
	BlockData(i uint64) (map[uint64]map[string]interface{}, error)
}

AttrStore represents an interface for handling row/column attributes. 

var NopAttrStore AttrStore

NopAttrStore represents an AttrStore that doesn't do anything.

func NewNopAttrStore added in v0.9.0 

func NewNopAttrStore(string) AttrStore

type BadRequestError added in v0.9.0 

type BadRequestError struct {
	// contains filtered or unexported fields
}

BadRequestError wraps an error value to signify that a request could not be read, decoded, or parsed such that in an HTTP scenario, http.StatusBadRequest would be returned.

type Bit 

type Bit struct {
	RowID     uint64
	ColumnID  uint64
	RowKey    string
	ColumnKey string
	Timestamp int64
}

Bit represents the location of a single bit.

func HandleAction added in v0.5.0 

func HandleAction(a Action, value interface{}, colID uint64, timestamp int64) (*Bit, error)

HandleAction Process the input data with its action and return a bit to be imported later Note: if the Bit should not be set then nil is returned with no error From the JSON marshalling the possible types are: float64, boolean, string

type Bitmap 

type Bitmap struct {

	// Attributes associated with the bitmap.
	Attrs map[string]interface{}
	// contains filtered or unexported fields
}

Bitmap represents a set of bits.

func NewBitmap 

func NewBitmap(bits ...uint64) *Bitmap

NewBitmap returns a new instance of Bitmap.

func Union 

func Union(bitmaps []*Bitmap) *Bitmap

Union performs a union on a slice of bitmaps.

func (*Bitmap) Bits 

func (b *Bitmap) Bits() []uint64

Bits returns the bits in b as a slice of ints.

func (*Bitmap) ClearBit 

func (b *Bitmap) ClearBit(i uint64) (changed bool)

ClearBit clears the i-th bit of the bitmap.

func (*Bitmap) Count 

func (b *Bitmap) Count() uint64

Count returns the number of set bits in the bitmap.

func (*Bitmap) DecrementCount 

func (b *Bitmap) DecrementCount(i uint64)

DecrementCount decrements the bitmap cached counter.

func (*Bitmap) Difference 

func (b *Bitmap) Difference(other *Bitmap) *Bitmap

Difference returns the diff of b and other.

func (*Bitmap) IncrementCount 

func (b *Bitmap) IncrementCount(i uint64)

IncrementCount increments the bitmap cached counter, note this is an optimization that assumes that the caller is aware the size increased.

func (*Bitmap) Intersect 

func (b *Bitmap) Intersect(other *Bitmap) *Bitmap

Intersect returns the itersection of b and other.

func (*Bitmap) IntersectionCount 

func (b *Bitmap) IntersectionCount(other *Bitmap) uint64

IntersectionCount returns the number of intersections between b and other.

func (*Bitmap) InvalidateCount 

func (b *Bitmap) InvalidateCount()

InvalidateCount updates the cached count in the bitmap.

func (*Bitmap) MarshalJSON 

func (b *Bitmap) MarshalJSON() ([]byte, error)

MarshalJSON returns a JSON-encoded byte slice of b.

func (*Bitmap) Merge 

func (b *Bitmap) Merge(other *Bitmap)

Merge merges data from other into b.

func (*Bitmap) SetBit 

func (b *Bitmap) SetBit(i uint64) (changed bool)

SetBit sets the i-th bit of the bitmap.

func (*Bitmap) Union 

func (b *Bitmap) Union(other *Bitmap) *Bitmap

Union returns the bitwise union of b and other.

func (*Bitmap) Xor added in v0.7.0 

func (b *Bitmap) Xor(other *Bitmap) *Bitmap

Xor returns the xor of b and other.

type BitmapCache 

type BitmapCache interface {
	Fetch(id uint64) (*Bitmap, bool)
	Add(id uint64, b *Bitmap)
}

BitmapCache provides an interface for caching full bitmaps.

type BitmapPair 

type BitmapPair struct {
	ID    uint64
	Count uint64
}

BitmapPair represents a id/count pair with an associated identifier.

type BitmapPairs 

type BitmapPairs []BitmapPair

BitmapPairs is a sortable list of BitmapPair objects.

func (BitmapPairs) Len 

func (p BitmapPairs) Len() int

func (BitmapPairs) Less 

func (p BitmapPairs) Less(i, j int) bool

func (BitmapPairs) Swap 

func (p BitmapPairs) Swap(i, j int)

type BitmapSegment 

type BitmapSegment struct {
	// contains filtered or unexported fields
}

BitmapSegment holds a subset of a bitmap. This could point to a mmapped roaring bitmap or an in-memory bitmap. The width of the segment will always match the slice width.

func (*BitmapSegment) Bits 

func (s *BitmapSegment) Bits() []uint64

Bits returns a list of all bits set in the segment.

func (*BitmapSegment) ClearBit 

func (s *BitmapSegment) ClearBit(i uint64) (changed bool)

ClearBit clears the i-th bit of the bitmap.

func (*BitmapSegment) Count 

func (s *BitmapSegment) Count() uint64

Count returns the number of set bits in the bitmap.

func (*BitmapSegment) Difference 

func (s *BitmapSegment) Difference(other *BitmapSegment) *BitmapSegment

Difference returns the diff of s and other.

func (*BitmapSegment) Intersect 

func (s *BitmapSegment) Intersect(other *BitmapSegment) *BitmapSegment

Intersect returns the itersection of s and other.

func (*BitmapSegment) IntersectionCount 

func (s *BitmapSegment) IntersectionCount(other *BitmapSegment) uint64

IntersectionCount returns the number of intersections between s and other.

func (*BitmapSegment) InvalidateCount 

func (s *BitmapSegment) InvalidateCount()

InvalidateCount updates the cached count in the bitmap.

func (*BitmapSegment) Merge 

func (s *BitmapSegment) Merge(other *BitmapSegment)

Merge adds chunks from other to s. Chunks in s are overwritten if they exist in other.

func (*BitmapSegment) SetBit 

func (s *BitmapSegment) SetBit(i uint64) (changed bool)

SetBit sets the i-th bit of the bitmap.

func (*BitmapSegment) Union 

func (s *BitmapSegment) Union(other *BitmapSegment) *BitmapSegment

Union returns the bitwise union of s and other.

func (*BitmapSegment) Xor added in v0.7.0 

func (s *BitmapSegment) Xor(other *BitmapSegment) *BitmapSegment

Xor returns the xor of s and other.

type Bits 

type Bits []Bit

Bits represents a slice of bits.

func (Bits) ColumnIDs 

func (p Bits) ColumnIDs() []uint64

ColumnIDs returns a slice of all the column IDs.

func (Bits) ColumnKeys added in v0.9.0 

func (p Bits) ColumnKeys() []string

ColumnKeys returns a slice of all the column keys.

func (Bits) GroupBySlice 

func (p Bits) GroupBySlice() map[uint64][]Bit

GroupBySlice returns a map of bits by slice.

func (Bits) Len 

func (p Bits) Len() int

func (Bits) Less 

func (p Bits) Less(i, j int) bool

func (Bits) RowIDs 

func (p Bits) RowIDs() []uint64

RowIDs returns a slice of all the row IDs.

func (Bits) RowKeys added in v0.9.0 

func (p Bits) RowKeys() []string

RowKeys returns a slice of all the row keys.

func (Bits) Swap 

func (p Bits) Swap(i, j int)

func (Bits) Timestamps 

func (p Bits) Timestamps() []int64

Timestamps returns a slice of all the timestamps.

type BitsByPos 

type BitsByPos []Bit

BitsByPos represents a slice of bits sorted by internal position.

func (BitsByPos) Len 

func (p BitsByPos) Len() int

func (BitsByPos) Less 

func (p BitsByPos) Less(i, j int) bool

func (BitsByPos) Swap 

func (p BitsByPos) Swap(i, j int)

type BroadcastHandler 

type BroadcastHandler interface {
	ReceiveMessage(pb proto.Message) error
}

BroadcastHandler is the interface for the pilosa object which knows how to handle broadcast messages. (Hint: this is implemented by pilosa.Server)

type BroadcastReceiver 

type BroadcastReceiver interface {
	// Start starts listening for broadcast messages - it should return
	// immediately, spawning a goroutine if necessary.
	Start(BroadcastHandler) error
}

BroadcastReceiver is the interface for the object which will listen for and decode broadcast messages before passing them to pilosa to handle. The implementation of this could be an http server which listens for messages, gets the protobuf payload, and then passes it to BroadcastHandler.ReceiveMessage.

type Broadcaster 

type Broadcaster interface {
	SendSync(pb proto.Message) error
	SendAsync(pb proto.Message) error
	SendTo(to *Node, pb proto.Message) error
}

Broadcaster is an interface for broadcasting messages. 

var NopBroadcaster Broadcaster

NopBroadcaster represents a Broadcaster that doesn't do anything.

type BufIterator 

type BufIterator struct {
	// contains filtered or unexported fields
}

BufIterator wraps an iterator to provide the ability to unread values.

func NewBufIterator 

func NewBufIterator(itr Iterator) *BufIterator

NewBufIterator returns a buffered iterator that wraps itr.

func (*BufIterator) Next 

func (itr *BufIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next pair in the row. If a value has been buffered then it is returned and the buffer is cleared.

func (*BufIterator) Peek 

func (itr *BufIterator) Peek() (rowID, columnID uint64, eof bool)

Peek reads the next value but leaves it on the buffer.

func (*BufIterator) Seek 

func (itr *BufIterator) Seek(rowID, columnID uint64)

Seek moves to the first pair equal to or greater than pseek/bseek.

func (*BufIterator) Unread 

func (itr *BufIterator) Unread()

Unread pushes previous pair on to the buffer. Panics if the buffer is already full.

type Cache 

type Cache interface {
	Add(id uint64, n uint64)
	BulkAdd(id uint64, n uint64)
	Get(id uint64) uint64
	Len() int

	// Returns a list of all IDs.
	IDs() []uint64

	// Updates the cache, if necessary.
	Invalidate()

	// Rebuilds the cache
	Recalculate()

	// Returns an ordered list of the top ranked bitmaps.
	Top() []BitmapPair

	// SetStats defines the stats client used in the cache.
	SetStats(s StatsClient)
}

Cache represents a cache of counts.

type ClientOptions added in v0.8.0 

type ClientOptions struct {
	TLS *tls.Config
}

ClientOptions represents the configuration for a InternalHTTPClient

type Cluster 

type Cluster struct {
	ID        string
	Node      *Node
	Nodes     []*Node // TODO phase this out?
	MemberSet MemberSet

	// Hashing algorithm used to assign partitions to nodes.
	Hasher Hasher

	// The number of partitions in the cluster.
	PartitionN int

	// The number of replicas a partition has.
	ReplicaN int

	// Threshold for logging long-running queries
	LongQueryTime time.Duration

	// Maximum number of SetBit() or ClearBit() commands per request.
	MaxWritesPerRequest int

	// EventReceiver receives NodeEvents pertaining to node membership.
	EventReceiver EventReceiver

	// Data directory path.
	Path     string
	Topology *Topology

	// Required for cluster Resize.
	Static bool // Static is primarily used for testing in a non-gossip environment.

	Coordinator string
	Holder      *Holder
	Broadcaster Broadcaster

	Logger Logger

	//
	RemoteClient *http.Client
	// contains filtered or unexported fields
}

Cluster represents a collection of nodes.

func NewCluster 

func NewCluster() *Cluster

NewCluster returns a new instance of Cluster with defaults.

func (*Cluster) AddNode added in v0.9.0 

func (c *Cluster) AddNode(node *Node) error

AddNode adds a node to the Cluster and updates and saves the new topology.

func (*Cluster) Close added in v0.9.0 

func (c *Cluster) Close() error

func (*Cluster) CompleteCurrentJob added in v0.9.0 

func (c *Cluster) CompleteCurrentJob(state string) error

CompleteCurrentJob sets the state of the current ResizeJob then removes the pointer to currentJob.

func (*Cluster) ContainsSlices added in v0.9.0 

func (c *Cluster) ContainsSlices(index string, maxSlice uint64, node *Node) []uint64

ContainsSlices is like OwnsSlices, but it includes replicas.

func (*Cluster) CoordinatorNode added in v0.9.0 

func (c *Cluster) CoordinatorNode() *Node

Coordinator returns the coordinator node.

func (*Cluster) FollowResizeInstruction added in v0.9.0 

func (c *Cluster) FollowResizeInstruction(instr *internal.ResizeInstruction) error

FollowResizeInstruction is run by any node that receives a ResizeInstruction.

func (*Cluster) IsCoordinator added in v0.9.0 

func (c *Cluster) IsCoordinator() bool

IsCoordinator is true if this node is the coordinator.

func (*Cluster) Job added in v0.9.0 

func (c *Cluster) Job(id int64) *ResizeJob

Job returns a ResizeJob by id.

func (*Cluster) ListenForJoins added in v0.9.0 

func (c *Cluster) ListenForJoins()

ListenForJoins handles cluster-resize events.

func (*Cluster) MarkResizeInstructionComplete added in v0.9.0 

func (c *Cluster) MarkResizeInstructionComplete(complete *internal.ResizeInstructionComplete) error

func (*Cluster) MergeClusterStatus added in v0.9.0 

func (c *Cluster) MergeClusterStatus(cs *internal.ClusterStatus) error

func (*Cluster) NodeByID added in v0.9.0 

func (c *Cluster) NodeByID(id string) *Node

func (*Cluster) NodeIDs added in v0.9.0 

func (c *Cluster) NodeIDs() []string

NodeIDs returns the list of IDs in the cluster.

func (*Cluster) NodeLeave added in v0.9.0 

func (c *Cluster) NodeLeave(node *Node) error

NodeLeave initiates the removal of a node from the cluster.

func (*Cluster) Open added in v0.9.0 

func (c *Cluster) Open() error

func (*Cluster) OwnsSlice added in v0.9.0 

func (c *Cluster) OwnsSlice(nodeID string, index string, slice uint64) bool

OwnsSlice returns true if a host owns a fragment.

func (*Cluster) OwnsSlices 

func (c *Cluster) OwnsSlices(index string, maxSlice uint64, uri URI) []uint64

OwnsSlices finds the set of slices owned by the node per Index

func (*Cluster) Partition 

func (c *Cluster) Partition(index string, slice uint64) int

Partition returns the partition that a slice belongs to.

func (*Cluster) PartitionNodes 

func (c *Cluster) PartitionNodes(partitionID int) []*Node

PartitionNodes returns a list of nodes that own a partition.

func (*Cluster) ReceiveEvent added in v0.9.0 

func (c *Cluster) ReceiveEvent(e *NodeEvent) error

ReceiveEvent represents an implementation of EventHandler.

func (*Cluster) ReceiveNodeState added in v0.9.0 

func (c *Cluster) ReceiveNodeState(nodeID string, state string) error

ReceiveNodeState sets node state in Topology in order for the Coordinator to keep track of, during startup, which nodes have finished opening their Holder.

func (*Cluster) RemoveNode added in v0.9.0 

func (c *Cluster) RemoveNode(node *Node) error

RemoveNode removes a node from the Cluster and updates and saves the new topology.

func (*Cluster) SetCoordinator added in v0.9.0 

func (c *Cluster) SetCoordinator(n *Node) error

SetCoordinator tells the current node to become the Coordinator. In response to this, the current node will consider itself coordinator and update the other nodes with its version of Cluster.Status.

func (*Cluster) SetNodeState added in v0.9.0 

func (c *Cluster) SetNodeState(state string) error

func (*Cluster) SetState added in v0.9.0 

func (c *Cluster) SetState(state string)

func (*Cluster) SliceNodes added in v0.9.0 

func (c *Cluster) SliceNodes(index string, slice uint64) []*Node

SliceNodes returns a list of nodes that own a fragment.

func (*Cluster) State added in v0.9.0 

func (c *Cluster) State() string

func (*Cluster) Status 

func (c *Cluster) Status() *internal.ClusterStatus

Status returns the internal ClusterStatus representation.

func (*Cluster) UpdateCoordinator added in v0.9.0 

func (c *Cluster) UpdateCoordinator(n *Node) bool

UpdateCoordinator updates this nodes Coordinator value as well as changing the corresponding node's IsCoordinator value to true, and sets all other nodes to false. Returns true if the value changed.

type CmdIO 

type CmdIO struct {
	Stdin  io.Reader
	Stdout io.Writer
	Stderr io.Writer
}

CmdIO holds standard unix inputs and outputs.

func NewCmdIO 

func NewCmdIO(stdin io.Reader, stdout, stderr io.Writer) *CmdIO

NewCmdIO returns a new instance of CmdIO with inputs and outputs set to the arguments.

type ColumnAttrSet 

type ColumnAttrSet struct {
	ID    uint64                 `json:"id"`
	Attrs map[string]interface{} `json:"attrs,omitempty"`
}

ColumnAttrSet represents a set of attributes for a vertical column in an index. Can have a set of attributes attached to it.

type DiagnosticsCollector added in v0.9.0 

type DiagnosticsCollector struct {
	VersionURL string

	Logger Logger
	// contains filtered or unexported fields
}

DiagnosticsCollector represents a collector/sender of diagnostics data.

func NewDiagnosticsCollector added in v0.9.0 

func NewDiagnosticsCollector(host string) *DiagnosticsCollector

NewDiagnosticsCollector returns a new DiagnosticsCollector given an addr in the format "hostname:port".

func (*DiagnosticsCollector) CheckVersion added in v0.9.0 

func (d *DiagnosticsCollector) CheckVersion() error

CheckVersion of the local build against Pilosa master.

func (*DiagnosticsCollector) EnrichWithMemoryInfo added in v0.9.0 

func (d *DiagnosticsCollector) EnrichWithMemoryInfo()

EnrichWithMemoryInfo adds memory information to the diagnostics payload.

func (*DiagnosticsCollector) EnrichWithOSInfo added in v0.9.0 

func (d *DiagnosticsCollector) EnrichWithOSInfo()

EnrichWithOSInfo adds OS information to the diagnostics payload.

func (*DiagnosticsCollector) EnrichWithSchemaProperties added in v0.9.0 

func (d *DiagnosticsCollector) EnrichWithSchemaProperties()

EnrichWithSchemaProperties adds schema info to the diagnostics payload.

func (*DiagnosticsCollector) Flush added in v0.9.0 

func (d *DiagnosticsCollector) Flush() error

Flush sends the current metrics.

func (*DiagnosticsCollector) Set added in v0.9.0 

func (d *DiagnosticsCollector) Set(name string, value interface{})

Set adds a key value metric.

func (*DiagnosticsCollector) SetVersion added in v0.9.0 

func (d *DiagnosticsCollector) SetVersion(v string)

SetVersion of locally running Pilosa Cluster to check against master.

type EventHandler added in v0.9.0 

type EventHandler interface {
	ReceiveEvent(e *NodeEvent) error
}

EventHandler is the interface for the pilosa object which knows how to handle broadcast messages. (Hint: this is implemented by pilosa.Server)

type EventReceiver added in v0.9.0 

type EventReceiver interface {
	// Start starts listening for broadcast messages - it should return
	// immediately, spawning a goroutine if necessary.
	Start(EventHandler) error
}

EventReceiver is the interface for the object which will listen for and decode broadcast messages before passing them to pilosa to handle. The implementation of this could be an http server which listens for messages, gets the protobuf payload, and then passes it to EventHandler.ReceiveMessage.

type ExecOptions 

type ExecOptions struct {
	Remote       bool
	ExcludeAttrs bool
	ExcludeBits  bool
}

ExecOptions represents an execution context for a single Execute() call.

type Executor 

type Executor struct {
	Holder *Holder

	// Local hostname & cluster configuration.
	Node    *Node
	Cluster *Cluster

	// Maximum number of SetBit() or ClearBit() commands per request.
	MaxWritesPerRequest int
	// contains filtered or unexported fields
}

Executor recursively executes calls in a PQL query across all slices.

func NewExecutor 

func NewExecutor(remoteClient *http.Client) *Executor

NewExecutor returns a new instance of Executor.

func (*Executor) Execute 

func (e *Executor) Execute(ctx context.Context, index string, q *pql.Query, slices []uint64, opt *ExecOptions) ([]interface{}, error)

Execute executes a PQL query.

type ExpvarStatsClient 

type ExpvarStatsClient struct {
	// contains filtered or unexported fields
}

ExpvarStatsClient writes stats out to expvars.

func NewExpvarStatsClient 

func NewExpvarStatsClient() *ExpvarStatsClient

NewExpvarStatsClient returns a new instance of ExpvarStatsClient. This client points at the root of the expvar index map.

func (*ExpvarStatsClient) Close added in v0.8.0 

func (c *ExpvarStatsClient) Close() error

Close no-op.

func (*ExpvarStatsClient) Count 

func (c *ExpvarStatsClient) Count(name string, value int64, rate float64)

Count tracks the number of times something occurs.

func (*ExpvarStatsClient) CountWithCustomTags added in v0.4.0 

func (c *ExpvarStatsClient) CountWithCustomTags(name string, value int64, rate float64, tags []string)

CountWithCustomTags Tracks the number of times something occurs per second with custom tags

func (*ExpvarStatsClient) Gauge 

func (c *ExpvarStatsClient) Gauge(name string, value float64, rate float64)

Gauge sets the value of a metric.

func (*ExpvarStatsClient) Histogram 

func (c *ExpvarStatsClient) Histogram(name string, value float64, rate float64)

Histogram tracks statistical distribution of a metric. This works the same as gauge for this client.

func (*ExpvarStatsClient) Open added in v0.8.0 

func (c *ExpvarStatsClient) Open()

Open no-op.

func (*ExpvarStatsClient) Set 

func (c *ExpvarStatsClient) Set(name string, value string, rate float64)

Set tracks number of unique elements.

func (*ExpvarStatsClient) SetLogger added in v0.4.0 

func (c *ExpvarStatsClient) SetLogger(logger Logger)

SetLogger has no logger.

func (*ExpvarStatsClient) Tags 

func (c *ExpvarStatsClient) Tags() []string

Tags returns a sorted list of tags on the client.

func (*ExpvarStatsClient) Timing 

func (c *ExpvarStatsClient) Timing(name string, value time.Duration, rate float64)

Timing tracks timing information for a metric.

func (*ExpvarStatsClient) WithTags 

func (c *ExpvarStatsClient) WithTags(tags ...string) StatsClient

WithTags returns a new client with additional tags appended.

type Field added in v0.5.0 

type Field struct {
	Name string `json:"name,omitempty"`
	Type string `json:"type,omitempty"`
	Min  int64  `json:"min,omitempty"`
	Max  int64  `json:"max,omitempty"`
}

Field represents a range field on a frame.

func (*Field) BaseValue added in v0.7.0 

func (f *Field) BaseValue(op pql.Token, value int64) (baseValue uint64, outOfRange bool)

BaseValue adjusts the value to align with the range for Field for a certain operation type. Note: There is an edge case for GT and LT where this returns a baseValue that does not fully encompass the range. ex: Field.Min = 0, Field.Max = 1023 BaseValue(LT, 2000) returns 1023, which will perform "LT 1023" and effectively exclude any columns with value = 1023. Note that in this case (because the range uses the full BitDepth 0 to 1023), we can't simply return 1024. In order to make this work, we effectively need to change the operator to LTE. Executor.executeFieldRangeSlice() takes this into account and returns `frag.FieldNotNull(field.BitDepth())` in such instances.

func (*Field) BaseValueBetween added in v0.7.0 

func (f *Field) BaseValueBetween(min, max int64) (baseValueMin, baseValueMax uint64, outOfRange bool)

BaseValueBetween adjusts the min/max value to align with the range for Field.

func (*Field) BitDepth added in v0.5.0 

func (f *Field) BitDepth() uint

BitDepth returns the number of bits required to store a value between min & max.

type FieldValue added in v0.7.0 

type FieldValue struct {
	ColumnID uint64
	Value    int64
}

FieldValues represents the value for a column within a range-encoded frame.

type FieldValues added in v0.7.0 

type FieldValues []FieldValue

FieldValues represents a slice of field values.

func (FieldValues) ColumnIDs added in v0.7.0 

func (p FieldValues) ColumnIDs() []uint64

ColumnIDs returns a slice of all the column IDs.

func (FieldValues) GroupBySlice added in v0.7.0 

func (p FieldValues) GroupBySlice() map[uint64][]FieldValue

GroupBySlice returns a map of field values by slice.

func (FieldValues) Len added in v0.7.0 

func (p FieldValues) Len() int

func (FieldValues) Less added in v0.7.0 

func (p FieldValues) Less(i, j int) bool

func (FieldValues) Swap added in v0.7.0 

func (p FieldValues) Swap(i, j int)

func (FieldValues) Values added in v0.7.0 

func (p FieldValues) Values() []int64

Values returns a slice of all the values.

type FileSystem added in v0.9.0 

type FileSystem interface {
	New() (http.FileSystem, error)
}

FileSystem represents an interface for a WebUI file system. 

var NopFileSystem FileSystem

NopFileSystem represents a FileSystem that returns an error if called.

type Fragment 

type Fragment struct {

	// Cache for row counts.
	CacheType string // passed in by frame

	CacheSize uint32

	// Number of operations performed before performing a snapshot.
	// This limits the size of fragments on the heap and flushes them to disk
	// so that they can be mmapped and heap utilization can be kept low.
	MaxOpN int

	// Logger used for out-of-band log entries.
	Logger Logger

	// Row attribute storage.
	// This is set by the parent frame unless overridden for testing.
	RowAttrStore AttrStore
	// contains filtered or unexported fields
}

Fragment represents the intersection of a frame and slice in an index.

func NewFragment 

func NewFragment(path, index, frame, view string, slice uint64) *Fragment

NewFragment returns a new instance of Fragment.

func (*Fragment) BlockData 

func (f *Fragment) BlockData(id int) (rowIDs, columnIDs []uint64)

BlockData returns bits in a block as row & column ID pairs.

func (*Fragment) BlockN 

func (f *Fragment) BlockN() int

BlockN returns the number of blocks in the fragment.

func (*Fragment) Blocks 

func (f *Fragment) Blocks() []FragmentBlock

Blocks returns info for all blocks containing data.

func (*Fragment) Cache 

func (f *Fragment) Cache() Cache

Cache returns the fragment's cache. This is not safe for concurrent use.

func (*Fragment) CachePath 

func (f *Fragment) CachePath() string

CachePath returns the path to the fragment's cache data.

func (*Fragment) Checksum 

func (f *Fragment) Checksum() []byte

Checksum returns a checksum for the entire fragment. If two fragments have the same checksum then they have the same data.

func (*Fragment) ClearBit 

func (f *Fragment) ClearBit(rowID, columnID uint64) (bool, error)

ClearBit clears a bit for a given column & row within the fragment. This updates both the on-disk storage and the in-cache bitmap.

func (*Fragment) Close 

func (f *Fragment) Close() error

Close flushes the underlying storage, closes the file and unlocks it.

func (*Fragment) FieldMax added in v0.9.0 

func (f *Fragment) FieldMax(filter *Bitmap, bitDepth uint) (max, count uint64, err error)

FieldMax returns the max of a given field as well as the number of columns involved. A bitmap can be passed in to optionally filter the computed columns.

func (*Fragment) FieldMin added in v0.9.0 

func (f *Fragment) FieldMin(filter *Bitmap, bitDepth uint) (min, count uint64, err error)

FieldMin returns the min of a given field as well as the number of columns involved. A bitmap can be passed in to optionally filter the computed columns.

func (*Fragment) FieldNotNull added in v0.7.1 

func (f *Fragment) FieldNotNull(bitDepth uint) (*Bitmap, error)

FieldNotNull returns the not-null row (stored at bitDepth).

func (*Fragment) FieldRange added in v0.6.0 

func (f *Fragment) FieldRange(op pql.Token, bitDepth uint, predicate uint64) (*Bitmap, error)

FieldRange returns bitmaps with a field value encoding matching the predicate.

func (*Fragment) FieldRangeBetween added in v0.7.0 

func (f *Fragment) FieldRangeBetween(bitDepth uint, predicateMin, predicateMax uint64) (*Bitmap, error)

FieldRangeBetween returns bitmaps with a field value encoding matching any value between predicateMin and predicateMax.

func (*Fragment) FieldSum added in v0.7.0 

func (f *Fragment) FieldSum(filter *Bitmap, bitDepth uint) (sum, count uint64, err error)

FieldSum returns the sum of a given field as well as the number of columns involved. A bitmap can be passed in to optionally filter the computed columns.

func (*Fragment) FieldValue added in v0.5.0 

func (f *Fragment) FieldValue(columnID uint64, bitDepth uint) (value uint64, exists bool, err error)

FieldValue uses a column of bits to read a multi-bit value.

func (*Fragment) FlushCache 

func (f *Fragment) FlushCache() error

FlushCache writes the cache data to disk.

func (*Fragment) ForEachBit 

func (f *Fragment) ForEachBit(fn func(rowID, columnID uint64) error) error

ForEachBit executes fn for every bit set in the fragment. Errors returned from fn are passed through.

func (*Fragment) Frame 

func (f *Fragment) Frame() string

Frame returns the frame the fragment was initialized with.

func (*Fragment) Import 

func (f *Fragment) Import(rowIDs, columnIDs []uint64) error

Import bulk imports a set of bits and then snapshots the storage. This does not affect the fragment's cache.

func (*Fragment) ImportValue added in v0.7.0 

func (f *Fragment) ImportValue(columnIDs, values []uint64, bitDepth uint) error

ImportValue bulk imports a set of range-encoded values.

func (*Fragment) Index 

func (f *Fragment) Index() string

Index returns the index that the fragment was initialized with.

func (*Fragment) InvalidateChecksums 

func (f *Fragment) InvalidateChecksums()

InvalidateChecksums clears all cached block checksums.

func (*Fragment) MergeBlock 

func (f *Fragment) MergeBlock(id int, data []PairSet) (sets, clears []PairSet, err error)

MergeBlock compares the block's bits and computes a diff with another set of block bits. The state of a bit is determined by consensus from all blocks being considered.

For example, if 3 blocks are compared and two have a set bit and one has a cleared bit then the bit is considered cleared. The function returns the diff per incoming block so that all can be in sync.

func (*Fragment) Open 

func (f *Fragment) Open() error

Open opens the underlying storage.

func (*Fragment) Path 

func (f *Fragment) Path() string

Path returns the path the fragment was initialized with.

func (*Fragment) ReadFrom 

func (f *Fragment) ReadFrom(r io.Reader) (n int64, err error)

ReadFrom reads a data file from r and loads it into the fragment.

func (*Fragment) RecalculateCache 

func (f *Fragment) RecalculateCache()

RecalculateCache rebuilds the cache regardless of invalidate time delay.

func (*Fragment) Row 

func (f *Fragment) Row(rowID uint64) *Bitmap

Row returns a row by ID.

func (*Fragment) SetBit 

func (f *Fragment) SetBit(rowID, columnID uint64) (changed bool, err error)

SetBit sets a bit for a given column & row within the fragment. This updates both the on-disk storage and the in-cache bitmap.

func (*Fragment) SetFieldValue added in v0.5.0 

func (f *Fragment) SetFieldValue(columnID uint64, bitDepth uint, value uint64) (changed bool, err error)

SetFieldValue uses a column of bits to set a multi-bit value.

func (*Fragment) Slice 

func (f *Fragment) Slice() uint64

Slice returns the slice the fragment was initialized with.

func (*Fragment) Snapshot 

func (f *Fragment) Snapshot() error

Snapshot writes the storage bitmap to disk and reopens it.

func (*Fragment) Top 

func (f *Fragment) Top(opt TopOptions) ([]Pair, error)

Top returns the top rows from the fragment. If opt.Src is specified then only rows which intersect src are returned. If opt.FilterValues exist then the row attribute specified by field is matched.

func (*Fragment) View 

func (f *Fragment) View() string

View returns the view the fragment was initialized with.

func (*Fragment) WriteTo 

func (f *Fragment) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the fragment's data to w.

type FragmentBlock 

type FragmentBlock struct {
	ID       int    `json:"id"`
	Checksum []byte `json:"checksum"`
}

FragmentBlock represents info about a subsection of the rows in a block. This is used for comparing data in remote blocks for active anti-entropy.

type FragmentSyncer 

type FragmentSyncer struct {
	Fragment *Fragment

	Node         *Node
	Cluster      *Cluster
	RemoteClient *http.Client

	Closing <-chan struct{}
}

FragmentSyncer syncs a local fragment to one on a remote host.

func (*FragmentSyncer) SyncFragment 

func (s *FragmentSyncer) SyncFragment() error

SyncFragment compares checksums for the local and remote fragments and then merges any blocks which have differences.

type Frame 

type Frame struct {
	Stats StatsClient

	Logger Logger
	// contains filtered or unexported fields
}

Frame represents a container for views.

func NewFrame 

func NewFrame(path, index, name string) (*Frame, error)

NewFrame returns a new instance of frame.

func (*Frame) CacheSize 

func (f *Frame) CacheSize() uint32

CacheSize returns the ranked frame cache size.

func (*Frame) CacheType 

func (f *Frame) CacheType() string

CacheType returns the caching mode for the frame.

func (*Frame) ClearBit 

func (f *Frame) ClearBit(name string, rowID, colID uint64, t *time.Time) (changed bool, err error)

ClearBit clears a bit within the frame.

func (*Frame) Close 

func (f *Frame) Close() error

Close closes the frame and its views.

func (*Frame) CreateField added in v0.7.0 

func (f *Frame) CreateField(field *Field) error

CreateField creates a new field on the frame.

func (*Frame) CreateViewIfNotExists 

func (f *Frame) CreateViewIfNotExists(name string) (*View, error)

CreateViewIfNotExists returns the named view, creating it if necessary. Additionally, a CreateViewMessage is sent to the cluster.

func (*Frame) DeleteField added in v0.7.0 

func (f *Frame) DeleteField(name string) error

DeleteField deletes an existing field on the schema.

func (*Frame) DeleteView added in v0.7.0 

func (f *Frame) DeleteView(name string) error

DeleteView removes the view from the frame.

func (*Frame) Field added in v0.5.0 

func (f *Frame) Field(name string) *Field

Field returns a field by name.

func (*Frame) FieldMax added in v0.9.0 

func (f *Frame) FieldMax(filter *Bitmap, name string) (max, count int64, err error)

FieldMax returns the max for a field. An optional filtering bitmap can be provided.

func (*Frame) FieldMin added in v0.9.0 

func (f *Frame) FieldMin(filter *Bitmap, name string) (min, count int64, err error)

FieldMin returns the min for a field. An optional filtering bitmap can be provided.

func (*Frame) FieldRange added in v0.6.0 

func (f *Frame) FieldRange(name string, op pql.Token, predicate int64) (*Bitmap, error)

func (*Frame) FieldRangeBetween added in v0.7.0 

func (f *Frame) FieldRangeBetween(name string, predicateMin, predicateMax int64) (*Bitmap, error)

func (*Frame) FieldSum added in v0.7.0 

func (f *Frame) FieldSum(filter *Bitmap, name string) (sum, count int64, err error)

FieldSum returns the sum and count for a field. An optional filtering bitmap can be provided.

func (*Frame) FieldValue added in v0.5.0 

func (f *Frame) FieldValue(columnID uint64, name string) (value int64, exists bool, err error)

FieldValue reads a field value for a column.

func (*Frame) Fields added in v0.9.0 

func (f *Frame) Fields() []*Field

Fields returns the fields on the frame.

func (*Frame) GetFields added in v0.7.0 

func (f *Frame) GetFields() ([]*Field, error)

GetFields returns a list of all the fields in the frame.

func (*Frame) HasField added in v0.9.0 

func (f *Frame) HasField(name string) bool

HasField returns true if a field exists on the frame.

func (*Frame) Import 

func (f *Frame) Import(rowIDs, columnIDs []uint64, timestamps []*time.Time) error

Import bulk imports data.

func (*Frame) ImportValue added in v0.7.0 

func (f *Frame) ImportValue(fieldName string, columnIDs []uint64, values []int64) error

ImportValue bulk imports range-encoded value data.

func (*Frame) Index 

func (f *Frame) Index() string

Index returns the index name the frame was initialized with.

func (*Frame) InverseEnabled 

func (f *Frame) InverseEnabled() bool

InverseEnabled returns true if an inverse view is available.

func (*Frame) MaxInverseSlice 

func (f *Frame) MaxInverseSlice() uint64

MaxInverseSlice returns the max inverse slice in the frame.

func (*Frame) MaxSlice 

func (f *Frame) MaxSlice() uint64

MaxSlice returns the max slice in the frame.

func (*Frame) Name 

func (f *Frame) Name() string

Name returns the name the frame was initialized with.

func (*Frame) Open 

func (f *Frame) Open() error

Open opens and initializes the frame.

func (*Frame) Options 

func (f *Frame) Options() FrameOptions

Options returns all options for this frame.

func (*Frame) Path 

func (f *Frame) Path() string

Path returns the path the frame was initialized with.

func (*Frame) RecalculateCaches added in v0.8.0 

func (f *Frame) RecalculateCaches()

RecalculateCaches recalculates caches on every view in the frame.

func (*Frame) RowAttrStore 

func (f *Frame) RowAttrStore() AttrStore

RowAttrStore returns the attribute storage.

func (*Frame) SetBit 

func (f *Frame) SetBit(name string, rowID, colID uint64, t *time.Time) (changed bool, err error)

SetBit sets a bit on a view within the frame.

func (*Frame) SetCacheSize 

func (f *Frame) SetCacheSize(v uint32) error

SetCacheSize sets the cache size for ranked fames. Persists to meta file on update. defaults to DefaultCacheSize 50000

func (*Frame) SetFieldValue added in v0.5.0 

func (f *Frame) SetFieldValue(columnID uint64, name string, value int64) (changed bool, err error)

SetFieldValue sets a field value for a column.

func (*Frame) SetTimeQuantum 

func (f *Frame) SetTimeQuantum(q TimeQuantum) error

SetTimeQuantum sets the time quantum for the frame.

func (*Frame) TimeQuantum 

func (f *Frame) TimeQuantum() TimeQuantum

TimeQuantum returns the time quantum for the frame.

func (*Frame) View 

func (f *Frame) View(name string) *View

View returns a view in the frame by name.

func (*Frame) ViewPath 

func (f *Frame) ViewPath(name string) string

ViewPath returns the path to a view in the frame.

func (*Frame) Views 

func (f *Frame) Views() []*View

Views returns a list of all views in the frame.

type FrameInfo 

type FrameInfo struct {
	Name    string       `json:"name"`
	Options FrameOptions `json:"options"`
	Views   []*ViewInfo  `json:"views,omitempty"`
}

FrameInfo represents schema information for a frame.

type FrameOptions 

type FrameOptions struct {
	InverseEnabled bool        `json:"inverseEnabled,omitempty"`
	RangeEnabled   bool        `json:"rangeEnabled,omitempty"` // deprecated, will be removed
	CacheType      string      `json:"cacheType,omitempty"`
	CacheSize      uint32      `json:"cacheSize,omitempty"`
	TimeQuantum    TimeQuantum `json:"timeQuantum,omitempty"`
	Fields         []*Field    `json:"fields,omitempty"`
}

FrameOptions represents options to set when initializing a frame.

func (*FrameOptions) Encode 

func (o *FrameOptions) Encode() *internal.FrameMeta

Encode converts o into its internal representation.

type GCNotifier added in v0.9.0 

type GCNotifier interface {
	Close()
	AfterGC() <-chan struct{}
}

GCNotifier represents an interface for garbage collection notificationss. 

var NopGCNotifier GCNotifier

NopGCNotifier represents a GCNotifier that doesn't do anything.

type Gossiper added in v0.9.0 

type Gossiper interface {
	SendAsync(pb proto.Message) error
}

Gossiper is an interface for sharing messages via gossip. 

var NopGossiper Gossiper

NopBroadcaster represents a Broadcaster that doesn't do anything.

type Handler 

type Handler struct {
	Router *mux.Router

	FileSystem FileSystem

	Logger Logger

	API *API
	// contains filtered or unexported fields
}

Handler represents an HTTP handler.

func NewHandler 

func NewHandler() *Handler

NewHandler returns a new instance of Handler with a default logger.

func (*Handler) ServeHTTP 

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP handles an HTTP request.

type Hasher 

type Hasher interface {
	// Hashes the key into a number between [0,N).
	Hash(key uint64, n int) int
}

Hasher represents an interface to hash integers into buckets.

func NewHasher 

func NewHasher() Hasher

NewHasher returns a new instance of the default hasher.

type Holder 

type Holder struct {
	Broadcaster Broadcaster

	NewAttrStore func(string) AttrStore

	// Stats
	Stats StatsClient

	// Data directory path.
	Path string

	// The interval at which the cached row ids are persisted to disk.
	CacheFlushInterval time.Duration

	Logger Logger
	// contains filtered or unexported fields
}

Holder represents a container for indexes.

func NewHolder 

func NewHolder() *Holder

NewHolder returns a new instance of Holder.

func (*Holder) ApplySchema added in v0.9.0 

func (h *Holder) ApplySchema(schema *internal.Schema) error

ApplySchema applies an internal Schema to Holder.

func (*Holder) Close 

func (h *Holder) Close() error

Close closes all open fragments.

func (*Holder) CreateIndex 

func (h *Holder) CreateIndex(name string, opt IndexOptions) (*Index, error)

CreateIndex creates an index. An error is returned if the index already exists.

func (*Holder) CreateIndexIfNotExists 

func (h *Holder) CreateIndexIfNotExists(name string, opt IndexOptions) (*Index, error)

CreateIndexIfNotExists returns an index by name. The index is created if it does not already exist.

func (*Holder) DeleteIndex 

func (h *Holder) DeleteIndex(name string) error

DeleteIndex removes an index from the holder.

func (*Holder) EncodeMaxSlices added in v0.9.0 

func (h *Holder) EncodeMaxSlices() *internal.MaxSlices

EncodeMaxSlices creates and internal representation of max slices.

func (*Holder) EncodeSchema added in v0.9.0 

func (h *Holder) EncodeSchema() *internal.Schema

EncodeSchema creates an internal representation of schema.

func (*Holder) Fragment 

func (h *Holder) Fragment(index, frame, view string, slice uint64) *Fragment

Fragment returns the fragment for an index, frame & slice.

func (*Holder) Frame 

func (h *Holder) Frame(index, name string) *Frame

Frame returns the frame for an index and name.

func (*Holder) HasData added in v0.9.0 

func (h *Holder) HasData() (bool, error)

HasData returns true if Holder contains at least one index. This is used to determine if the rebalancing of data is necessary when a node joins the cluster.

func (*Holder) Index 

func (h *Holder) Index(name string) *Index

Index returns the index by name.

func (*Holder) IndexPath 

func (h *Holder) IndexPath(name string) string

IndexPath returns the path where a given index is stored.

func (*Holder) Indexes 

func (h *Holder) Indexes() []*Index

Indexes returns a list of all indexes in the holder.

func (*Holder) MaxInverseSlices 

func (h *Holder) MaxInverseSlices() map[string]uint64

MaxInverseSlices returns MaxInverseSlice map for all indexes.

func (*Holder) MaxSlices 

func (h *Holder) MaxSlices() map[string]uint64

MaxSlices returns MaxSlice map for all indexes.

func (*Holder) Open 

func (h *Holder) Open() error

Open initializes the root data directory for the holder.

func (*Holder) RecalculateCaches added in v0.8.0 

func (h *Holder) RecalculateCaches()

RecalculateCaches recalculates caches on every index in the holder. This is probably not practical to call in real-world workloads, but makes writing integration tests much eaiser, since one doesn't have to wait 10 seconds after setting bits to get expected response.

func (*Holder) Schema 

func (h *Holder) Schema() []*IndexInfo

Schema returns schema information for all indexes, frames, and views.

func (*Holder) View 

func (h *Holder) View(index, frame, name string) *View

View returns the view for an index, frame, and name.

type HolderCleaner added in v0.9.0 

type HolderCleaner struct {
	Node *Node

	Holder  *Holder
	Cluster *Cluster

	// Signals that the sync should stop.
	Closing <-chan struct{}
}

HolderCleaner removes fragments and data files that are no longer used.

func (*HolderCleaner) CleanHolder added in v0.9.0 

func (c *HolderCleaner) CleanHolder() error

CleanHolder compares the holder with the cluster state and removes any unnecessary fragments and files.

func (*HolderCleaner) IsClosing added in v0.9.0 

func (c *HolderCleaner) IsClosing() bool

IsClosing returns true if the cleaner has been marked to close.

type HolderSyncer 

type HolderSyncer struct {
	Holder *Holder

	Node         *Node
	Cluster      *Cluster
	RemoteClient *http.Client

	// Stats
	Stats StatsClient

	// Signals that the sync should stop.
	Closing <-chan struct{}
}

HolderSyncer is an active anti-entropy tool that compares the local holder with a remote holder based on block checksums and resolves differences.

func (*HolderSyncer) IsClosing 

func (s *HolderSyncer) IsClosing() bool

IsClosing returns true if the syncer has been marked to close.

func (*HolderSyncer) SyncHolder 

func (s *HolderSyncer) SyncHolder() error

SyncHolder compares the holder on host with the local holder and resolves differences.

type Index 

type Index struct {
	NewAttrStore func(string) AttrStore

	Stats StatsClient

	Logger Logger
	// contains filtered or unexported fields
}

Index represents a container for frames.

func NewIndex 

func NewIndex(path, name string) (*Index, error)

NewIndex returns a new instance of Index.

func (*Index) Close 

func (i *Index) Close() error

Close closes the index and its frames.

func (*Index) ColumnAttrStore 

func (i *Index) ColumnAttrStore() AttrStore

ColumnAttrStore returns the storage for column attributes.

func (*Index) CreateFrame 

func (i *Index) CreateFrame(name string, opt FrameOptions) (*Frame, error)

CreateFrame creates a frame.

func (*Index) CreateFrameIfNotExists 

func (i *Index) CreateFrameIfNotExists(name string, opt FrameOptions) (*Frame, error)

CreateFrameIfNotExists creates a frame with the given options if it doesn't exist.

func (*Index) CreateInputDefinition added in v0.5.0 

func (i *Index) CreateInputDefinition(pb *internal.InputDefinition) (*InputDefinition, error)

CreateInputDefinition creates a new input definition.

func (*Index) DeleteFrame 

func (i *Index) DeleteFrame(name string) error

DeleteFrame removes a frame from the index.

func (*Index) DeleteInputDefinition added in v0.5.0 

func (i *Index) DeleteInputDefinition(name string) error

DeleteInputDefinition removes an input definition from the index.

func (*Index) Frame 

func (i *Index) Frame(name string) *Frame

Frame returns a frame in the index by name.

func (*Index) FramePath 

func (i *Index) FramePath(name string) string

FramePath returns the path to a frame in the index.

func (*Index) Frames 

func (i *Index) Frames() []*Frame

Frames returns a list of all frames in the index.

func (*Index) InputBits added in v0.5.0 

func (i *Index) InputBits(frame string, bits []*Bit) error

InputBits Process the []Bit though the Frame import process

func (*Index) InputDefinition added in v0.5.0 

func (i *Index) InputDefinition(name string) (*InputDefinition, error)

InputDefinition returns an input definition in the index by name.

func (*Index) InputDefinitionPath added in v0.5.0 

func (i *Index) InputDefinitionPath() string

InputDefinitionPath returns the path to the input definition directory for the index.

func (*Index) InputDefinitions added in v0.9.0 

func (i *Index) InputDefinitions() []*InputDefinition

InputDefinitions returns a list of all inputDefinitions in the index.

func (*Index) MaxInverseSlice 

func (i *Index) MaxInverseSlice() uint64

MaxInverseSlice returns the max inverse slice in the index according to this node.

func (*Index) MaxSlice 

func (i *Index) MaxSlice() uint64

MaxSlice returns the max slice in the index according to this node.

func (*Index) Name 

func (i *Index) Name() string

Name returns name of the index.

func (*Index) Open 

func (i *Index) Open() error

Open opens and initializes the index.

func (*Index) Options added in v0.7.0 

func (i *Index) Options() IndexOptions

Options returns all options for this index.

func (*Index) Path 

func (i *Index) Path() string

Path returns the path the index was initialized with.

func (*Index) RecalculateCaches added in v0.8.0 

func (i *Index) RecalculateCaches()

RecalculateCaches recalculates caches on every frame in the index.

func (*Index) SetRemoteMaxInverseSlice 

func (i *Index) SetRemoteMaxInverseSlice(v uint64)

SetRemoteMaxInverseSlice sets the remote max inverse slice value received from another node.

func (*Index) SetRemoteMaxSlice 

func (i *Index) SetRemoteMaxSlice(newmax uint64)

SetRemoteMaxSlice sets the remote max slice value received from another node.

type IndexInfo 

type IndexInfo struct {
	Name   string       `json:"name"`
	Frames []*FrameInfo `json:"frames"`
}

IndexInfo represents schema information for an index.

type IndexOptions 

type IndexOptions struct{}

IndexOptions represents options to set when initializing an index.

func (*IndexOptions) Encode 

func (i *IndexOptions) Encode() *internal.IndexMeta

Encode converts i into its internal representation.

type InputDefinition added in v0.5.0 

type InputDefinition struct {
	// contains filtered or unexported fields
}

InputDefinition represents a container for the data input definition.

func NewInputDefinition added in v0.5.0 

func NewInputDefinition(path, index, name string) (*InputDefinition, error)

NewInputDefinition returns a new instance of InputDefinition.

func (*InputDefinition) AddFrame added in v0.5.0 

func (i *InputDefinition) AddFrame(frame InputFrame) error

AddFrame manually add frame to input definition.

func (*InputDefinition) Fields added in v0.5.0 

func (i *InputDefinition) Fields() []InputDefinitionField

Fields returns fields of the input definition was initialized with.

func (*InputDefinition) Frames added in v0.5.0 

func (i *InputDefinition) Frames() []InputFrame

Frames returns frames of the input definition was initialized with.

func (*InputDefinition) LoadDefinition added in v0.5.0 

func (i *InputDefinition) LoadDefinition(pb *internal.InputDefinition) error

LoadDefinition loads the protobuf format of a definition.

func (*InputDefinition) Open added in v0.5.0 

func (i *InputDefinition) Open() error

Open opens and initializes the InputDefinition from file.

type InputDefinitionField added in v0.5.0 

type InputDefinitionField struct {
	Name       string   `json:"name,omitempty"`
	PrimaryKey bool     `json:"primaryKey,omitempty"`
	Actions    []Action `json:"actions,omitempty"`
}

InputDefinitionField descripes a single field mapping in the InputDefinition.

func (*InputDefinitionField) Encode added in v0.5.0 

func (o *InputDefinitionField) Encode() *internal.InputDefinitionField

Encode converts InputDefinitionField into its internal representation.

type InputDefinitionInfo added in v0.5.0 

type InputDefinitionInfo struct {
	Frames []InputFrame           `json:"frames"`
	Fields []InputDefinitionField `json:"fields"`
}

InputDefinitionInfo represents the json message format needed to create an InputDefinition.

func (*InputDefinitionInfo) Encode added in v0.5.0 

func (i *InputDefinitionInfo) Encode() *internal.InputDefinition

Encode converts InputDefinitionInfo into its internal representation.

func (*InputDefinitionInfo) Validate added in v0.5.0 

func (i *InputDefinitionInfo) Validate() error

Validate the InputDefinitionInfo data.

type InputFrame added in v0.5.0 

type InputFrame struct {
	Name    string       `json:"name,omitempty"`
	Options FrameOptions `json:"options,omitempty"`
}

InputFrame defines the frame used in the input definition.

func (*InputFrame) Encode added in v0.5.0 

func (i *InputFrame) Encode() *internal.Frame

Encode converts InputFrame into its internal representation.

func (*InputFrame) Validate added in v0.5.0 

func (i *InputFrame) Validate() error

Validate the InputFrame data.

type InternalClient added in v0.8.0 

type InternalClient interface {
	MaxSliceByIndex(ctx context.Context) (map[string]uint64, error)
	MaxInverseSliceByIndex(ctx context.Context) (map[string]uint64, error)
	Schema(ctx context.Context) ([]*IndexInfo, error)
	CreateIndex(ctx context.Context, index string, opt IndexOptions) error
	FragmentNodes(ctx context.Context, index string, slice uint64) ([]*Node, error)
	ExecuteQuery(ctx context.Context, index string, queryRequest *internal.QueryRequest) (*internal.QueryResponse, error)
	Import(ctx context.Context, index, frame string, slice uint64, bits []Bit) error
	ImportK(ctx context.Context, index, frame string, bits []Bit) error
	EnsureIndex(ctx context.Context, name string, options IndexOptions) error
	EnsureFrame(ctx context.Context, indexName string, frameName string, options FrameOptions) error
	ImportValue(ctx context.Context, index, frame, field string, slice uint64, vals []FieldValue) error
	ExportCSV(ctx context.Context, index, frame, view string, slice uint64, w io.Writer) error
	BackupTo(ctx context.Context, w io.Writer, index, frame, view string) error
	BackupSlice(ctx context.Context, index, frame, view string, slice uint64) (io.ReadCloser, error)
	RestoreFrom(ctx context.Context, r io.Reader, index, frame, view string) error
	CreateFrame(ctx context.Context, index, frame string, opt FrameOptions) error
	RestoreFrame(ctx context.Context, host, index, frame string) error
	FrameViews(ctx context.Context, index, frame string) ([]string, error)
	FragmentBlocks(ctx context.Context, index, frame, view string, slice uint64) ([]FragmentBlock, error)
	BlockData(ctx context.Context, index, frame, view string, slice uint64, block int) ([]uint64, []uint64, error)
	ColumnAttrDiff(ctx context.Context, index string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)
	RowAttrDiff(ctx context.Context, index, frame string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)
	SendMessage(ctx context.Context, pb proto.Message) error
	NodeID(uri *URI) (string, error)
}

InternalClient should be implemented by any struct that enables any transport between nodes TODO: Refactor Note from Travis: Typically an interface containing more than two or three methods is an indication that something hasn't been architected correctly. While I understand that putting the entire Client behind an interface might require this many methods, I don't want to let it go unquestioned.

type InternalHTTPClient added in v0.8.0 

type InternalHTTPClient struct {

	// The client to use for HTTP communication.
	HTTPClient *http.Client
	// contains filtered or unexported fields
}

InternalHTTPClient represents a client to the Pilosa cluster.

func NewInternalHTTPClient added in v0.8.0 

func NewInternalHTTPClient(host string, remoteClient *http.Client) (*InternalHTTPClient, error)

NewInternalHTTPClient returns a new instance of InternalHTTPClient to connect to host.

func NewInternalHTTPClientFromURI added in v0.8.0 

func NewInternalHTTPClientFromURI(defaultURI *URI, remoteClient *http.Client) *InternalHTTPClient

func (*InternalHTTPClient) BackupSlice added in v0.8.0 

func (c *InternalHTTPClient) BackupSlice(ctx context.Context, index, frame, view string, slice uint64) (io.ReadCloser, error)

BackupSlice retrieves a streaming backup from a single slice. This function tries slice owners until one succeeds.

func (*InternalHTTPClient) BackupTo added in v0.8.0 

func (c *InternalHTTPClient) BackupTo(ctx context.Context, w io.Writer, index, frame, view string) error

BackupTo backs up an entire frame from a cluster to w.

func (*InternalHTTPClient) BlockData added in v0.8.0 

func (c *InternalHTTPClient) BlockData(ctx context.Context, index, frame, view string, slice uint64, block int) ([]uint64, []uint64, error)

BlockData returns row/column id pairs for a block.

func (*InternalHTTPClient) ColumnAttrDiff added in v0.8.0 

func (c *InternalHTTPClient) ColumnAttrDiff(ctx context.Context, index string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)

ColumnAttrDiff returns data from differing blocks on a remote host.

func (*InternalHTTPClient) CreateFrame added in v0.8.0 

func (c *InternalHTTPClient) CreateFrame(ctx context.Context, index, frame string, opt FrameOptions) error

CreateFrame creates a new frame on the server.

func (*InternalHTTPClient) CreateIndex added in v0.8.0 

func (c *InternalHTTPClient) CreateIndex(ctx context.Context, index string, opt IndexOptions) error

CreateIndex creates a new index on the server.

func (*InternalHTTPClient) EnsureFrame added in v0.8.0 

func (c *InternalHTTPClient) EnsureFrame(ctx context.Context, indexName string, frameName string, options FrameOptions) error

func (*InternalHTTPClient) EnsureIndex added in v0.8.0 

func (c *InternalHTTPClient) EnsureIndex(ctx context.Context, name string, options IndexOptions) error

func (*InternalHTTPClient) ExecuteQuery added in v0.8.0 

func (c *InternalHTTPClient) ExecuteQuery(ctx context.Context, index string, queryRequest *internal.QueryRequest) (*internal.QueryResponse, error)

ExecuteQuery executes query against index on the server.

func (*InternalHTTPClient) ExportCSV added in v0.8.0 

func (c *InternalHTTPClient) ExportCSV(ctx context.Context, index, frame, view string, slice uint64, w io.Writer) error

ExportCSV bulk exports data for a single slice from a host to CSV format.

func (*InternalHTTPClient) FragmentBlocks added in v0.8.0 

func (c *InternalHTTPClient) FragmentBlocks(ctx context.Context, index, frame, view string, slice uint64) ([]FragmentBlock, error)

FragmentBlocks returns a list of block checksums for a fragment on a host. Only returns blocks which contain data.

func (*InternalHTTPClient) FragmentNodes added in v0.8.0 

func (c *InternalHTTPClient) FragmentNodes(ctx context.Context, index string, slice uint64) ([]*Node, error)

FragmentNodes returns a list of nodes that own a slice.

func (*InternalHTTPClient) FrameViews added in v0.8.0 

func (c *InternalHTTPClient) FrameViews(ctx context.Context, index, frame string) ([]string, error)

FrameViews returns a list of view names for a frame.

func (*InternalHTTPClient) Host added in v0.8.0 

func (c *InternalHTTPClient) Host() *URI

Host returns the host the client was initialized with.

func (*InternalHTTPClient) Import added in v0.8.0 

func (c *InternalHTTPClient) Import(ctx context.Context, index, frame string, slice uint64, bits []Bit) error

Import bulk imports bits for a single slice to a host.

func (*InternalHTTPClient) ImportK added in v0.9.0 

func (c *InternalHTTPClient) ImportK(ctx context.Context, index, frame string, bits []Bit) error

ImportK bulk imports bits to a host.

func (*InternalHTTPClient) ImportValue added in v0.8.0 

func (c *InternalHTTPClient) ImportValue(ctx context.Context, index, frame, field string, slice uint64, vals []FieldValue) error

ImportValue bulk imports field values for a single slice to a host.

func (*InternalHTTPClient) MaxInverseSliceByIndex added in v0.8.0 

func (c *InternalHTTPClient) MaxInverseSliceByIndex(ctx context.Context) (map[string]uint64, error)

MaxInverseSliceByIndex returns the number of inverse slices on a server by index.

func (*InternalHTTPClient) MaxSliceByIndex added in v0.8.0 

func (c *InternalHTTPClient) MaxSliceByIndex(ctx context.Context) (map[string]uint64, error)

MaxSliceByIndex returns the number of slices on a server by index.

func (*InternalHTTPClient) NodeID added in v0.9.0 

func (c *InternalHTTPClient) NodeID(uri *URI) (string, error)

func (*InternalHTTPClient) RestoreFrame added in v0.8.0 

func (c *InternalHTTPClient) RestoreFrame(ctx context.Context, host, index, frame string) error

RestoreFrame restores an entire frame from a host in another cluster.

func (*InternalHTTPClient) RestoreFrom added in v0.8.0 

func (c *InternalHTTPClient) RestoreFrom(ctx context.Context, r io.Reader, index, frame, view string) error

RestoreFrom restores a frame from a backup file to an entire cluster.

func (*InternalHTTPClient) RetrieveSliceFromURI added in v0.9.0 

func (c *InternalHTTPClient) RetrieveSliceFromURI(ctx context.Context, index, frame, view string, slice uint64, uri URI) (io.ReadCloser, error)

func (*InternalHTTPClient) RowAttrDiff added in v0.8.0 

func (c *InternalHTTPClient) RowAttrDiff(ctx context.Context, index, frame string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)

RowAttrDiff returns data from differing blocks on a remote host.

func (*InternalHTTPClient) Schema added in v0.8.0 

func (c *InternalHTTPClient) Schema(ctx context.Context) ([]*IndexInfo, error)

Schema returns all index and frame schema information.

func (*InternalHTTPClient) SendMessage added in v0.9.0 

func (c *InternalHTTPClient) SendMessage(ctx context.Context, pb proto.Message) error

SendMessage posts a message synchronously.

type Iterator 

type Iterator interface {
	Seek(rowID, columnID uint64)
	Next() (rowID, columnID uint64, eof bool)
}

Iterator is an interface for looping over row/column pairs.

type LRUCache 

type LRUCache struct {
	// contains filtered or unexported fields
}

LRUCache represents a least recently used Cache implementation.

func NewLRUCache 

func NewLRUCache(maxEntries uint32) *LRUCache

NewLRUCache returns a new instance of LRUCache.

func (*LRUCache) Add 

func (c *LRUCache) Add(id, n uint64)

Add adds a count to the cache.

func (*LRUCache) BulkAdd 

func (c *LRUCache) BulkAdd(id, n uint64)

BulkAdd adds a count to the cache unsorted. You should Invalidate after completion.

func (*LRUCache) Get 

func (c *LRUCache) Get(id uint64) uint64

Get returns a count for a given id.

func (*LRUCache) IDs 

func (c *LRUCache) IDs() []uint64

IDs returns a list of all IDs in the cache.

func (*LRUCache) Invalidate 

func (c *LRUCache) Invalidate()

Invalidate is a no-op.

func (*LRUCache) Len 

func (c *LRUCache) Len() int

Len returns the number of items in the cache.

func (*LRUCache) Recalculate 

func (c *LRUCache) Recalculate()

Recalculate is a no-op.

func (*LRUCache) SetStats added in v0.4.0 

func (c *LRUCache) SetStats(s StatsClient)

SetStats defines the stats client used in the cache.

func (*LRUCache) Top 

func (c *LRUCache) Top() []BitmapPair

Top returns all counts in the cache.

type LimitIterator 

type LimitIterator struct {
	// contains filtered or unexported fields
}

LimitIterator wraps an Iterator and limits it to a max column/row pair.

func NewLimitIterator 

func NewLimitIterator(itr Iterator, maxRowID, maxColumnID uint64) *LimitIterator

NewLimitIterator returns a new LimitIterator.

func (*LimitIterator) Next 

func (itr *LimitIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next row/column ID pair. If the underlying iterator returns a pair higher than the max then EOF is returned.

func (*LimitIterator) Seek 

func (itr *LimitIterator) Seek(rowID, columnID uint64)

Seek moves the underlying iterator to a column/row pair.

type Logger added in v0.9.0 

type Logger interface {
	Printf(format string, v ...interface{})
	Debugf(format string, v ...interface{})
}

Logger represents an interface for a shared logger. 

var NopLogger Logger

NopLogger represents a Logger that doesn't do anything.

type MemberSet added in v0.9.0 

type MemberSet interface {
	// Open starts any network activity implemented by the MemberSet
	// Node is the local node, used for membership broadcasts.
	Open(n *Node) error
}

MemberSet represents an interface for Node membership and inter-node communication.

type MultiStatsClient 

type MultiStatsClient []StatsClient

MultiStatsClient joins multiple stats clients together.

func (MultiStatsClient) Close added in v0.8.0 

func (a MultiStatsClient) Close() error

Close shuts down the stats clients.

func (MultiStatsClient) Count 

func (a MultiStatsClient) Count(name string, value int64, rate float64)

Count tracks the number of times something occurs per second on all clients.

func (MultiStatsClient) CountWithCustomTags added in v0.4.0 

func (a MultiStatsClient) CountWithCustomTags(name string, value int64, rate float64, tags []string)

CountWithCustomTags Tracks the number of times something occurs per second with custom tags

func (MultiStatsClient) Gauge 

func (a MultiStatsClient) Gauge(name string, value float64, rate float64)

Gauge sets the value of a metric on all clients.

func (MultiStatsClient) Histogram 

func (a MultiStatsClient) Histogram(name string, value float64, rate float64)

Histogram tracks statistical distribution of a metric on all clients.

func (MultiStatsClient) Open added in v0.8.0 

func (a MultiStatsClient) Open()

Open starts the stat service.

func (MultiStatsClient) Set 

func (a MultiStatsClient) Set(name string, value string, rate float64)

Set tracks number of unique elements on all clients.

func (MultiStatsClient) SetLogger added in v0.4.0 

func (a MultiStatsClient) SetLogger(logger Logger)

SetLogger Sets the StatsD logger output type.

func (MultiStatsClient) Tags 

func (a MultiStatsClient) Tags() []string

Tags returns tags from the first client.

func (MultiStatsClient) Timing 

func (a MultiStatsClient) Timing(name string, value time.Duration, rate float64)

Timing tracks timing information for a metric on all clients.

func (MultiStatsClient) WithTags 

func (a MultiStatsClient) WithTags(tags ...string) StatsClient

WithTags returns a new set of clients with the additional tags.

type Node 

type Node struct {
	ID            string `json:"id"`
	URI           URI    `json:"uri"`
	IsCoordinator bool   `json:"isCoordinator"`
}

Node represents a node in the cluster.

func DecodeNode added in v0.9.0 

func DecodeNode(node *internal.Node) *Node

DecodeNode converts a proto message into a Node.

func DecodeNodes added in v0.9.0 

func DecodeNodes(a []*internal.Node) []*Node

DecodeNodes converts a proto message into a slice of Nodes.

func (Node) String added in v0.9.0 

func (n Node) String() string

type NodeEvent added in v0.9.0 

type NodeEvent struct {
	Event NodeEventType
	Node  *Node
}

NodeEvent is a single event related to node activity in the cluster.

func DecodeNodeEvent added in v0.9.0 

func DecodeNodeEvent(ne *internal.NodeEventMessage) *NodeEvent

type NodeEventType added in v0.9.0 

type NodeEventType int

NodeEventType are the types of events that can be sent from the ChannelEventDelegate. 

const (
	NodeJoin NodeEventType = iota
	NodeLeave
	NodeUpdate
)

type NodeIDs added in v0.9.0 

type NodeIDs []string

func (NodeIDs) ContainsID added in v0.9.0 

func (n NodeIDs) ContainsID(id string) bool

ContainsID returns true if idi matches one of the nodesets's IDs.

func (NodeIDs) Len added in v0.9.0 

func (n NodeIDs) Len() int

func (NodeIDs) Less added in v0.9.0 

func (n NodeIDs) Less(i, j int) bool

func (NodeIDs) Swap added in v0.9.0 

func (n NodeIDs) Swap(i, j int)

type Nodes 

type Nodes []*Node

Nodes represents a list of nodes.

func (Nodes) Clone 

func (a Nodes) Clone() []*Node

Clone returns a shallow copy of nodes.

func (Nodes) Contains 

func (a Nodes) Contains(n *Node) bool

Contains returns true if a node exists in the list.

func (Nodes) ContainsID added in v0.9.0 

func (a Nodes) ContainsID(id string) bool

ContainsID returns true if host matches one of the node's id.

func (Nodes) Filter 

func (a Nodes) Filter(n *Node) []*Node

Filter returns a new list of nodes with node removed.

func (Nodes) FilterID added in v0.9.0 

func (a Nodes) FilterID(id string) []*Node

FilterID returns a new list of nodes with ID removed.

func (Nodes) FilterURI added in v0.9.0 

func (a Nodes) FilterURI(uri URI) []*Node

FilterURI returns a new list of nodes with URI removed.

func (Nodes) IDs added in v0.9.0 

func (a Nodes) IDs() []string

IDs returns a list of all node IDs.

func (Nodes) URIs added in v0.9.0 

func (a Nodes) URIs() []URI

URIs returns a list of all uris.

type NopCache added in v0.5.0 

type NopCache struct {
	// contains filtered or unexported fields
}

NopCache represents a no-op Cache implementation.

func NewNopCache added in v0.5.0 

func NewNopCache() *NopCache

NewNopCache returns a new instance of NopCache.

func (*NopCache) Add added in v0.5.0 

func (c *NopCache) Add(id uint64, n uint64)

func (*NopCache) BulkAdd added in v0.5.0 

func (c *NopCache) BulkAdd(id uint64, n uint64)

func (*NopCache) Get added in v0.5.0 

func (c *NopCache) Get(id uint64) uint64

func (*NopCache) IDs added in v0.5.0 

func (c *NopCache) IDs() []uint64

func (*NopCache) Invalidate added in v0.5.0 

func (c *NopCache) Invalidate()

func (*NopCache) Len added in v0.5.0 

func (c *NopCache) Len() int

func (*NopCache) Recalculate added in v0.5.0 

func (c *NopCache) Recalculate()

func (*NopCache) SetStats added in v0.5.0 

func (c *NopCache) SetStats(s StatsClient)

func (*NopCache) Top added in v0.5.0 

func (c *NopCache) Top() []BitmapPair

type NopSystemInfo added in v0.9.0 

type NopSystemInfo struct {
}

NopSystemInfo is a no-op implementation of SystemInfo.

func NewNopSystemInfo added in v0.9.0 

func NewNopSystemInfo() *NopSystemInfo

NewNopSystemInfo creates a no-op implementation of SystemInfo.

func (*NopSystemInfo) Family added in v0.9.0 

func (n *NopSystemInfo) Family() (string, error)

Family is a no-op implementation of SystemInfo.Family.

func (*NopSystemInfo) KernelVersion added in v0.9.0 

func (n *NopSystemInfo) KernelVersion() (string, error)

KernelVersion is a no-op implementation of SystemInfo.KernelVersion.

func (*NopSystemInfo) MemFree added in v0.9.0 

func (n *NopSystemInfo) MemFree() (uint64, error)

MemFree is a no-op implementation of SystemInfo.MemFree.

func (*NopSystemInfo) MemTotal added in v0.9.0 

func (n *NopSystemInfo) MemTotal() (uint64, error)

MemTotal is a no-op implementation of SystemInfo.MemTotal.

func (*NopSystemInfo) MemUsed added in v0.9.0 

func (n *NopSystemInfo) MemUsed() (uint64, error)

MemUsed is a no-op implementation of SystemInfo.MemUsed.

func (*NopSystemInfo) OSVersion added in v0.9.0 

func (n *NopSystemInfo) OSVersion() (string, error)

OSVersion is a no-op implementation of SystemInfo.OSVersion.

func (*NopSystemInfo) Platform added in v0.9.0 

func (n *NopSystemInfo) Platform() (string, error)

Platform is a no-op implementation of SystemInfo.Platform.

func (*NopSystemInfo) Uptime added in v0.9.0 

func (n *NopSystemInfo) Uptime() (uint64, error)

Uptime is a no-op implementation of SystemInfo.Uptime.

type Pair 

type Pair struct {
	ID    uint64 `json:"id"`
	Key   string `json:"key,omitempty"`
	Count uint64 `json:"count"`
}

Pair holds an id/count pair.

type PairHeap 

type PairHeap struct {
	Pairs
}

PairHeap is a heap implementation over a group of Pairs.

func (PairHeap) Less 

func (p PairHeap) Less(i, j int) bool

Less implemets the Sort interface. reports whether the element with index i should sort before the element with index j.

type PairSet 

type PairSet struct {
	RowIDs    []uint64
	ColumnIDs []uint64
}

PairSet is a list of equal length row and column id lists.

type Pairs 

type Pairs []Pair

Pairs is a sortable slice of Pair objects.

func (Pairs) Add 

func (p Pairs) Add(other []Pair) []Pair

Add merges other into p and returns a new slice.

func (Pairs) Keys 

func (p Pairs) Keys() []uint64

Keys returns a slice of all keys in p.

func (Pairs) Len 

func (p Pairs) Len() int

func (Pairs) Less 

func (p Pairs) Less(i, j int) bool

func (*Pairs) Pop 

func (p *Pairs) Pop() interface{}

Pop removes the minimum element from the Pair slice.

func (*Pairs) Push 

func (p *Pairs) Push(x interface{})

Push appends the element onto the Pair slice.

func (Pairs) String 

func (p Pairs) String() string

func (Pairs) Swap 

func (p Pairs) Swap(i, j int)

type QueryRequest 

type QueryRequest struct {
	// Index to execute query against.
	Index string

	// The query string to parse and execute.
	Query string

	// The slices to include in the query execution.
	// If empty, all slices are included.
	Slices []uint64

	// Return column attributes, if true.
	ColumnAttrs bool

	// Do not return row attributes, if true.
	ExcludeAttrs bool

	// Do not return bits, if true.
	ExcludeBits bool

	// If true, indicates that query is part of a larger distributed query.
	// If false, this request is on the originating node.
	Remote bool
}

QueryRequest represent a request to process a query.

type QueryResponse 

type QueryResponse struct {
	// Result for each top-level query call.
	// Can be a Bitmap, Pairs, or uint64.
	Results []interface{}

	// Set of column attribute objects matching IDs returned in Result.
	ColumnAttrSets []*ColumnAttrSet

	// Error during parsing or execution.
	Err error
}

QueryResponse represent a response from a processed query.

func (*QueryResponse) MarshalJSON 

func (resp *QueryResponse) MarshalJSON() ([]byte, error)

MarshalJSON marshals QueryResponse into a JSON-encoded byte slice

type RankCache 

type RankCache struct {
	// contains filtered or unexported fields
}

RankCache represents a cache with sorted entries.

func NewRankCache 

func NewRankCache(maxEntries uint32) *RankCache

NewRankCache returns a new instance of RankCache.

func (*RankCache) Add 

func (c *RankCache) Add(id uint64, n uint64)

Add adds a count to the cache.

func (*RankCache) BulkAdd 

func (c *RankCache) BulkAdd(id uint64, n uint64)

BulkAdd adds a count to the cache unsorted. You should Invalidate after completion.

func (*RankCache) Get 

func (c *RankCache) Get(id uint64) uint64

Get returns a count for a given id.

func (*RankCache) IDs 

func (c *RankCache) IDs() []uint64

IDs returns a list of all IDs in the cache.

func (*RankCache) Invalidate 

func (c *RankCache) Invalidate()

Invalidate recalculates the entries by rank.

func (*RankCache) Len 

func (c *RankCache) Len() int

Len returns the number of items in the cache.

func (*RankCache) ReadFrom 

func (c *RankCache) ReadFrom(r io.Reader) (n int64, err error)

ReadFrom read from r into the cache.

func (*RankCache) Recalculate 

func (c *RankCache) Recalculate()

Recalculate rebuilds the cache.

func (*RankCache) SetStats added in v0.4.0 

func (c *RankCache) SetStats(s StatsClient)

SetStats defines the stats client used in the cache.

func (*RankCache) Top 

func (c *RankCache) Top() []BitmapPair

Top returns an ordered list of pairs.

func (*RankCache) WriteTo 

func (c *RankCache) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the cache to w.

type ResizeJob added in v0.9.0 

type ResizeJob struct {
	ID           int64
	IDs          map[string]bool
	Instructions []*internal.ResizeInstruction
	Broadcaster  Broadcaster

	Logger Logger
	// contains filtered or unexported fields
}

func NewResizeJob added in v0.9.0 

func NewResizeJob(existingNodes []*Node, node *Node, action string) *ResizeJob

NewResizeJob returns a new instance of ResizeJob.

func (*ResizeJob) Run added in v0.9.0 

func (j *ResizeJob) Run() error

Run distributes ResizeInstructions.

func (*ResizeJob) SetState added in v0.9.0 

func (j *ResizeJob) SetState(state string)

func (*ResizeJob) State added in v0.9.0 

func (j *ResizeJob) State() string

type RoaringIterator 

type RoaringIterator struct {
	// contains filtered or unexported fields
}

RoaringIterator converts a roaring.Iterator to output column/row pairs.

func NewRoaringIterator 

func NewRoaringIterator(itr *roaring.Iterator) *RoaringIterator

NewRoaringIterator returns a new iterator wrapping itr.

func (*RoaringIterator) Next 

func (itr *RoaringIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next column/row ID pair.

func (*RoaringIterator) Seek 

func (itr *RoaringIterator) Seek(bseek, pseek uint64)

Seek moves the cursor to a pair matching bseek/pseek. If the pair is not found then it moves to the next pair.

type Server 

type Server struct {

	// Internal
	Holder  *Holder
	Cluster *Cluster

	Broadcaster       Broadcaster
	BroadcastReceiver BroadcastReceiver
	Gossiper          Gossiper

	NewAttrStore func(string) AttrStore

	NodeID string
	URI    URI
	// contains filtered or unexported fields
}

Server represents a holder wrapped by a running HTTP server.

func NewServer 

func NewServer(opts ...ServerOption) (*Server, error)

NewServer returns a new instance of Server.

func (*Server) Addr 

func (s *Server) Addr() net.Addr

Addr returns the address of the listener.

func (*Server) Close 

func (s *Server) Close() error

Close closes the server and waits for it to shutdown.

func (*Server) ClusterStatus 

func (s *Server) ClusterStatus() (proto.Message, error)

ClusterStatus returns the ClusterState and NodeSet for the cluster.

func (*Server) HandleRemoteStatus 

func (s *Server) HandleRemoteStatus(pb proto.Message) error

HandleRemoteStatus receives incoming NodeStatus from remote nodes.

func (*Server) LoadNodeID added in v0.9.0 

func (s *Server) LoadNodeID() string

LoadNodeID gets NodeID from disk, or creates a new value. If server.NodeID is already set, a new ID is not created.

func (*Server) LocalStatus 

func (s *Server) LocalStatus() (proto.Message, error)

Server implements StatusHandler. LocalStatus is used to periodically sync information between nodes. Under normal conditions, nodes should remain in sync through Broadcast messages. For cases where a node fails to receive a Broadcast message, or when a new (empty) node needs to get in sync with the rest of the cluster, two things are shared via gossip: - MaxSlice/MaxInverseSlice by Index - Schema In a gossip implementation, memberlist.Delegate.LocalState() uses this.

func (*Server) Open 

func (s *Server) Open() error

Open opens and initializes the server.

func (*Server) ReceiveMessage 

func (s *Server) ReceiveMessage(pb proto.Message) error

ReceiveMessage represents an implementation of BroadcastHandler.

func (*Server) SendAsync added in v0.9.0 

func (s *Server) SendAsync(pb proto.Message) error

SendAsync represents an implementation of Broadcaster.

func (*Server) SendSync added in v0.9.0 

func (s *Server) SendSync(pb proto.Message) error

SendSync represents an implementation of Broadcaster.

func (*Server) SendTo added in v0.9.0 

func (s *Server) SendTo(to *Node, pb proto.Message) error

SendTo represents an implementation of Broadcaster.

type ServerOption added in v0.9.0 

type ServerOption func(s *Server) error

ServerOption is a functional option type for pilosa.Server

func OptServerAntiEntropyInterval added in v0.9.0 

func OptServerAntiEntropyInterval(interval time.Duration) ServerOption

func OptServerAttrStoreFunc added in v0.9.0 

func OptServerAttrStoreFunc(af func(string) AttrStore) ServerOption

func OptServerDataDir added in v0.9.0 

func OptServerDataDir(dir string) ServerOption

func OptServerDiagnosticsInterval added in v0.9.0 

func OptServerDiagnosticsInterval(dur time.Duration) ServerOption

func OptServerGCNotifier added in v0.9.0 

func OptServerGCNotifier(gcn GCNotifier) ServerOption

func OptServerHandler added in v0.9.0 

func OptServerHandler(h *Handler) ServerOption

func OptServerListener added in v0.9.0 

func OptServerListener(ln net.Listener) ServerOption

func OptServerLogger added in v0.9.0 

func OptServerLogger(l Logger) ServerOption

func OptServerLongQueryTime added in v0.9.0 

func OptServerLongQueryTime(dur time.Duration) ServerOption

func OptServerMaxWritesPerRequest added in v0.9.0 

func OptServerMaxWritesPerRequest(n int) ServerOption

func OptServerMetricInterval added in v0.9.0 

func OptServerMetricInterval(dur time.Duration) ServerOption

func OptServerRemoteClient added in v0.9.0 

func OptServerRemoteClient(c *http.Client) ServerOption

func OptServerReplicaN added in v0.9.0 

func OptServerReplicaN(n int) ServerOption

func OptServerStatsClient added in v0.9.0 

func OptServerStatsClient(sc StatsClient) ServerOption

func OptServerSystemInfo added in v0.9.0 

func OptServerSystemInfo(si SystemInfo) ServerOption

func OptServerURI added in v0.9.0 

func OptServerURI(uri *URI) ServerOption

type SimpleCache 

type SimpleCache struct {
	// contains filtered or unexported fields
}

SimpleCache implements BitmapCache it is meant to be a short-lived cache for cases where writes are continuing to access the same bit within a short time frame (i.e. good for write-heavy loads) A read-heavy use case would cause the cache to get bigger, potentially causing the node to run out of memory.

func (*SimpleCache) Add 

func (s *SimpleCache) Add(id uint64, b *Bitmap)

Add adds the bitmap to the cache, keyed on the id.

func (*SimpleCache) Fetch 

func (s *SimpleCache) Fetch(id uint64) (*Bitmap, bool)

Fetch retrieves the bitmap at the id in the cache.

type SliceIterator 

type SliceIterator struct {
	// contains filtered or unexported fields
}

SliceIterator iterates over a pair of row/column ID slices.

func NewSliceIterator 

func NewSliceIterator(rowIDs, columnIDs []uint64) *SliceIterator

NewSliceIterator returns an iterator to iterate over a set of row/column ID pairs. Both slices MUST have an equal length. Otherwise the function will panic.

func (*SliceIterator) Next 

func (itr *SliceIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next row/column ID pair.

func (*SliceIterator) Seek 

func (itr *SliceIterator) Seek(bseek, pseek uint64)

Seek moves the cursor to a given pair. If the pair is not found, the iterator seeks to the next pair.

type StandardLogger added in v0.9.0 

type StandardLogger struct {
	// contains filtered or unexported fields
}

StandardLogger is a basic implementation of pilosa.Logger based on log.Logger.

func NewStandardLogger added in v0.9.0 

func NewStandardLogger(w io.Writer) *StandardLogger

func (*StandardLogger) Debugf added in v0.9.0 

func (s *StandardLogger) Debugf(format string, v ...interface{})

func (*StandardLogger) Logger added in v0.9.0 

func (s *StandardLogger) Logger() *log.Logger

func (*StandardLogger) Printf added in v0.9.0 

func (s *StandardLogger) Printf(format string, v ...interface{})

type StaticMemberSet added in v0.9.0 

type StaticMemberSet struct {
	// contains filtered or unexported fields
}

StaticMemberSet represents a basic MemberSet for testing.

func NewStaticMemberSet added in v0.9.0 

func NewStaticMemberSet(nodes []*Node) *StaticMemberSet

NewStaticMemberSet creates a statically defined MemberSet.

func (*StaticMemberSet) Open added in v0.9.0 

func (s *StaticMemberSet) Open(n *Node) error

Open implements the MemberSet interface to start network activity, but for a static MemberSet it does nothing.

type StatsClient 

type StatsClient interface {
	// Returns a sorted list of tags on the client.
	Tags() []string

	// Returns a new client with additional tags appended.
	WithTags(tags ...string) StatsClient

	// Tracks the number of times something occurs per second.
	Count(name string, value int64, rate float64)

	// Tracks the number of times something occurs per second with custom tags
	CountWithCustomTags(name string, value int64, rate float64, tags []string)

	// Sets the value of a metric.
	Gauge(name string, value float64, rate float64)

	// Tracks statistical distribution of a metric.
	Histogram(name string, value float64, rate float64)

	// Tracks number of unique elements.
	Set(name string, value string, rate float64)

	// Tracks timing information for a metric.
	Timing(name string, value time.Duration, rate float64)

	// SetLogger Set the logger output type
	SetLogger(logger Logger)

	// Starts the service
	Open()

	// Closes the client
	Close() error
}

StatsClient represents a client to a stats server. 

var NopStatsClient StatsClient

NopStatsClient represents a client that doesn't do anything.

type StatusHandler 

type StatusHandler interface {
	LocalStatus() (proto.Message, error)
	ClusterStatus() (proto.Message, error)
	HandleRemoteStatus(proto.Message) error
}

StatusHandler specifies the methods which an object must implement to share state in the cluster. These are used by the GossipMemberSet to implement the LocalState and MergeRemoteState methods of memberlist.Delegate

type SystemInfo added in v0.9.0 

type SystemInfo interface {
	Uptime() (uint64, error)
	Platform() (string, error)
	Family() (string, error)
	OSVersion() (string, error)
	KernelVersion() (string, error)
	MemFree() (uint64, error)
	MemTotal() (uint64, error)
	MemUsed() (uint64, error)
}

SystemInfo collects information about the host OS.

type TimeQuantum 

type TimeQuantum string

TimeQuantum represents a time granularity for time-based bitmaps.

func ParseTimeQuantum 

func ParseTimeQuantum(v string) (TimeQuantum, error)

ParseTimeQuantum parses v into a time quantum.

func (TimeQuantum) HasDay 

func (q TimeQuantum) HasDay() bool

HasDay returns true if the quantum contains a 'D' unit.

func (TimeQuantum) HasHour 

func (q TimeQuantum) HasHour() bool

HasHour returns true if the quantum contains a 'H' unit.

func (TimeQuantum) HasMonth 

func (q TimeQuantum) HasMonth() bool

HasMonth returns true if the quantum contains a 'M' unit.

func (TimeQuantum) HasYear 

func (q TimeQuantum) HasYear() bool

HasYear returns true if the quantum contains a 'Y' unit.

func (*TimeQuantum) Set added in v0.7.0 

func (q *TimeQuantum) Set(value string) error

Set sets the time quantum value.

func (TimeQuantum) String added in v0.7.0 

func (q TimeQuantum) String() string

func (TimeQuantum) Type added in v0.7.0 

func (q TimeQuantum) Type() string

Type returns the type of a time quantum value.

func (TimeQuantum) Valid 

func (q TimeQuantum) Valid() bool

Valid returns true if q is a valid time quantum value.

type TopOptions 

type TopOptions struct {
	// Number of rows to return.
	N int

	// Bitmap to intersect with.
	Src *Bitmap

	// Specific rows to filter against.
	RowIDs       []uint64
	MinThreshold uint64

	// Filter field name & values.
	FilterField       string
	FilterValues      []interface{}
	TanimotoThreshold uint64
}

TopOptions represents options passed into the Top() function.

type Topology added in v0.9.0 

type Topology struct {
	NodeIDs []string

	ClusterID string
	// contains filtered or unexported fields
}

Topology represents the list of hosts in the cluster.

func NewTopology added in v0.9.0 

func NewTopology() *Topology

func (*Topology) AddID added in v0.9.0 

func (t *Topology) AddID(nodeID string) bool

AddID adds the node ID to the topology and returns true if added.

func (*Topology) ContainsID added in v0.9.0 

func (t *Topology) ContainsID(id string) bool

ContainsID returns true if id matches one of the topology's IDs.

func (*Topology) Encode added in v0.9.0 

func (t *Topology) Encode() *internal.Topology

Encode converts t into its internal representation.

func (*Topology) RemoveID added in v0.9.0 

func (t *Topology) RemoveID(nodeID string) bool

RemoveID removes the node ID from the topology and returns true if removed.

type URI added in v0.8.0 

type URI struct {
	// contains filtered or unexported fields
}

URI represents a Pilosa URI. A Pilosa URI consists of three parts: 1) Scheme: Protocol of the URI. Default: http. 2) Host: Hostname or IP URI. Default: localhost. IPv6 addresses should be written in brackets, e.g., `[fd42:4201:f86b:7e09:216:3eff:fefa:ed80]`. 3) Port: Port of the URI. Default: 10101.

All parts of the URI are optional. The following are equivalent: 

http://localhost:10101
http://localhost
http://:10101
localhost:10101
localhost
:10101

func AddressWithDefaults added in v0.6.0 

func AddressWithDefaults(addr string) (*URI, error)

AddressWithDefaults converts addr into a valid address, using defaults when necessary.

func DecodeURI added in v0.9.0 

func DecodeURI(i *internal.URI) URI

func DefaultURI added in v0.8.0 

func DefaultURI() *URI

DefaultURI creates and returns the default URI.

func NewURIFromAddress added in v0.8.0 

func NewURIFromAddress(address string) (*URI, error)

NewURIFromAddress parses the passed address and returns a URI.

func NewURIFromHostPort added in v0.8.0 

func NewURIFromHostPort(host string, port uint16) (*URI, error)

NewURIFromHostPort returns a URI with specified host and port.

func (URI) Encode added in v0.9.0 

func (u URI) Encode() *internal.URI

Encode converts o into its internal representation.

func (URI) Equals added in v0.8.0 

func (u URI) Equals(other *URI) bool

Equals returns true if the checked URI is equivalent to this URI.

func (*URI) Host added in v0.8.0 

func (u *URI) Host() string

Host returns the host of this URI.

func (*URI) HostPort added in v0.8.0 

func (u *URI) HostPort() string

HostPort returns `Host:Port`

func (*URI) MarshalJSON added in v0.9.0 

func (u *URI) MarshalJSON() ([]byte, error)

MarshalJSON marshals URI into a JSON-encoded byte slice.

func (*URI) Normalize added in v0.8.0 

func (u *URI) Normalize() string

Normalize returns the address in a form usable by a HTTP client.

func (*URI) Path added in v0.8.0 

func (u *URI) Path(path string) string

Path returns URI with path

func (*URI) Port added in v0.8.0 

func (u *URI) Port() uint16

Port returns the port of this URI.

func (*URI) Scheme added in v0.8.0 

func (u *URI) Scheme() string

Scheme returns the scheme of this URI.

func (*URI) Set added in v0.8.0 

func (u *URI) Set(value string) error

Set sets the time quantum value.

func (*URI) SetHost added in v0.8.0 

func (u *URI) SetHost(host string) error

SetHost sets the host of this URI.

func (*URI) SetPort added in v0.8.0 

func (u *URI) SetPort(port uint16)

SetPort sets the port of this URI.

func (*URI) SetScheme added in v0.8.0 

func (u *URI) SetScheme(scheme string) error

SetScheme sets the scheme of this URI.

func (URI) String added in v0.8.0 

func (u URI) String() string

String returns the address as a string.

func (URI) Type added in v0.8.0 

func (u URI) Type() string

Type returns the type of a time quantum value.

func (*URI) UnmarshalJSON added in v0.9.0 

func (u *URI) UnmarshalJSON(b []byte) error

type URIs added in v0.9.0 

type URIs []URI

func (URIs) HostPortStrings added in v0.9.0 

func (u URIs) HostPortStrings() []string

type ValCount added in v0.9.0 

type ValCount struct {
	Val   int64 `json:"value"`
	Count int64 `json:"count"`
}

ValCount represents a grouping of sum & count for Sum() and Average() calls.

func (*ValCount) Add added in v0.9.0 

func (vc *ValCount) Add(other ValCount) ValCount

func (*ValCount) Larger added in v0.9.0 

func (vc *ValCount) Larger(other ValCount) ValCount

Larger returns the larger of the two ValCounts.

func (*ValCount) Smaller added in v0.9.0 

func (vc *ValCount) Smaller(other ValCount) ValCount

Smaller returns the smaller of the two ValCounts.

type VerboseLogger added in v0.9.0 

type VerboseLogger struct {
	// contains filtered or unexported fields
}

VerboseLogger is an implementation of pilosa.Logger which includes debug messages.

func NewVerboseLogger added in v0.9.0 

func NewVerboseLogger(w io.Writer) *VerboseLogger

func (*VerboseLogger) Debugf added in v0.9.0 

func (vb *VerboseLogger) Debugf(format string, v ...interface{})

func (*VerboseLogger) Logger added in v0.9.0 

func (vb *VerboseLogger) Logger() *log.Logger

func (*VerboseLogger) Printf added in v0.9.0 

func (vb *VerboseLogger) Printf(format string, v ...interface{})

type View 

type View struct {
	RowAttrStore AttrStore
	Logger       Logger
	// contains filtered or unexported fields
}

View represents a container for frame data.

func NewView 

func NewView(path, index, frame, name string, cacheSize uint32) *View

NewView returns a new instance of View.

func (*View) ClearBit 

func (v *View) ClearBit(rowID, columnID uint64) (changed bool, err error)

ClearBit clears a bit within the view.

func (*View) Close 

func (v *View) Close() error

Close closes the view and its fragments.

func (*View) CreateFragmentIfNotExists 

func (v *View) CreateFragmentIfNotExists(slice uint64) (*Fragment, error)

CreateFragmentIfNotExists returns a fragment in the view by slice.

func (*View) DeleteFragment added in v0.9.0 

func (v *View) DeleteFragment(slice uint64) error

DeleteFragment removes the fragment from the view.

func (*View) FieldMax added in v0.9.0 

func (v *View) FieldMax(filter *Bitmap, bitDepth uint) (max, count uint64, err error)

FieldMax returns the max and count of a field.

func (*View) FieldMin added in v0.9.0 

func (v *View) FieldMin(filter *Bitmap, bitDepth uint) (min, count uint64, err error)

FieldMin returns the min and count of a field.

func (*View) FieldRange added in v0.6.0 

func (v *View) FieldRange(op pql.Token, bitDepth uint, predicate uint64) (*Bitmap, error)

FieldRange returns bitmaps with a field value encoding matching the predicate.

func (*View) FieldRangeBetween added in v0.7.0 

func (v *View) FieldRangeBetween(bitDepth uint, predicateMin, predicateMax uint64) (*Bitmap, error)

FieldRangeBetween returns bitmaps with a field value encoding matching any value between predicateMin and predicateMax.

func (*View) FieldSum added in v0.7.0 

func (v *View) FieldSum(filter *Bitmap, bitDepth uint) (sum, count uint64, err error)

FieldSum returns the sum & count of a field.

func (*View) FieldValue added in v0.5.0 

func (v *View) FieldValue(columnID uint64, bitDepth uint) (value uint64, exists bool, err error)

FieldValue uses a column of bits to read a multi-bit value.

func (*View) Fragment 

func (v *View) Fragment(slice uint64) *Fragment

Fragment returns a fragment in the view by slice.

func (*View) FragmentPath 

func (v *View) FragmentPath(slice uint64) string

FragmentPath returns the path to a fragment in the view.

func (*View) Fragments 

func (v *View) Fragments() []*Fragment

Fragments returns a list of all fragments in the view.

func (*View) Frame 

func (v *View) Frame() string

Frame returns the frame name the view was initialized with.

func (*View) Index 

func (v *View) Index() string

Index returns the index name the view was initialized with.

func (*View) MaxSlice 

func (v *View) MaxSlice() uint64

MaxSlice returns the max slice in the view.

func (*View) Name 

func (v *View) Name() string

Name returns the name the view was initialized with.

func (*View) Open 

func (v *View) Open() error

Open opens and initializes the view.

func (*View) Path 

func (v *View) Path() string

Path returns the path the view was initialized with.

func (*View) RecalculateCaches added in v0.8.0 

func (v *View) RecalculateCaches()

RecalculateCaches recalculates the cache on every fragment in the view.

func (*View) SetBit 

func (v *View) SetBit(rowID, columnID uint64) (changed bool, err error)

SetBit sets a bit within the view.

func (*View) SetFieldValue added in v0.5.0 

func (v *View) SetFieldValue(columnID uint64, bitDepth uint, value uint64) (changed bool, err error)

SetFieldValue uses a column of bits to set a multi-bit value.

type ViewInfo 

type ViewInfo struct {
	Name string `json:"name"`
}

ViewInfo represents schema information for a view.

Source Files

View all Source files

Directories

Path Synopsis
cmd
Package cmd contains all the pilosa subcommand definitions (1 per file).
 Package cmd contains all the pilosa subcommand definitions (1 per file).
pilosa
This is the entrypoint for the Pilosa binary.
 This is the entrypoint for the Pilosa binary.
package ctl contains all pilosa subcommands other than 'server'.
 package ctl contains all pilosa subcommands other than 'server'.
Package internal is a generated protocol buffer package.
 Package internal is a generated protocol buffer package.
Package lru implements an LRU cache.
 Package lru implements an LRU cache.
Package pql defines the Pilosa Query Language.
 Package pql defines the Pilosa Query Language.
Package roaring implements roaring bitmaps with support for incremental changes.
 Package roaring implements roaring bitmaps with support for incremental changes.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.