pilosa

package module
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 8, 2017 License: Apache-2.0 Imports: 46 Imported by: 0

README

Build Status GoDoc Go Report Card license CLA Assistant GitHub release

An open source, distributed bitmap index.

Docs

See our Documentation for information about installing and working with Pilosa.

Getting Started

  1. Install Pilosa.

  2. Start Pilosa with the default configuration:

    pilosa server

    and verify that it's running:

    curl localhost:10101/nodes

  3. Follow along with the Sample Project to get a better understanding of Pilosa's capabilities.

Data Model

Check out how the Pilosa Data Model works.

Query Language

You can interact with Pilosa directly in the console using the Pilosa Query Language (PQL).

Client Libraries

There are supported libraries for the following languages:

Get Support

There are several channels available for you to reach out to us for support.

Contributing

Pilosa is an open source project. Please see our Contributing Guide for information about how to get involved.

Documentation

Overview

Package pilosa implements the core of the Pilosa distributed bitmap index. It contains all the domain objects, interfaces, and logic that defines pilosa.

Index

Constants

View Source 
const (
	AttrTypeString = 1
	AttrTypeInt    = 2
	AttrTypeBool   = 3
	AttrTypeFloat  = 4
)

Attribute data type enum.

View Source 
const (
	MessageTypeCreateSlice = 1
	MessageTypeCreateIndex = 2
	MessageTypeDeleteIndex = 3
	MessageTypeCreateFrame = 4
	MessageTypeDeleteFrame = 5
)

Broadcast message types.

View Source 
const (
	// DefaultPartitionN is the default number of partitions in a cluster.
	DefaultPartitionN = 256

	// DefaultReplicaN is the default number of replicas per partition.
	DefaultReplicaN = 1
)
View Source 
const (
	NodeStateUp   = "UP"
	NodeStateDown = "DOWN"
)

NodeState represents node state returned in /status endpoint for a node in the cluster.

View Source 
const (
	// DefaultHost is the default hostname to use.
	DefaultHost = "localhost"

	// DefaultPort is the default port use with the hostname.
	DefaultPort = "10101"

	// DefaultClusterType sets the node intercommunication method.
	DefaultClusterType = "static"

	// DefaultInternalPort the port the nodes intercommunicate on.
	DefaultInternalPort = "14000"

	// DefaultMetrics sets the internal metrics to no op
	DefaultMetrics = "nop"

	// DefaultMaxWritesPerRequest is the default number of writes per request.
	DefaultMaxWritesPerRequest = 5000
)
View Source 
const (
	DefaultFrame = "general"

	// MinThreshold is the lowest count to use in a Top-N operation when
	// looking for additional id/count pairs.
	MinThreshold = 1
)

DefaultFrame is the frame used if one is not specified.

View Source 
const (
	// SliceWidth is the number of column IDs in a slice.
	SliceWidth = 1048576

	// SnapshotExt is the file extension used for an in-process snapshot.
	SnapshotExt = ".snapshotting"

	// CopyExt is the file extension used for the temp file used while copying.
	CopyExt = ".copying"

	// CacheExt is the file extension for persisted cache ids.
	CacheExt = ".cache"

	// HashBlockSize is the number of rows in a merkle hash block.
	HashBlockSize = 100
)
View Source 
const (
	DefaultRowLabel       = "rowID"
	DefaultCacheType      = CacheTypeRanked
	DefaultInverseEnabled = false

	// Default ranked frame cache
	DefaultCacheSize = 50000
)

Default frame settings.

View Source 
const (
	CacheTypeLRU    = "lru"
	CacheTypeRanked = "ranked"
)

Cache types.

View Source 
const (
	DefaultAntiEntropyInterval = 10 * time.Minute
	DefaultPollingInterval     = 60 * time.Second
)

Default server settings.

View Source 
const (
	ViewStandard = "standard"
	ViewInverse  = "inverse"
)

View layout modes.

View Source 
const AttrBlockSize = 100

AttrBlockSize is the size of attribute blocks for anti-entropy.

View Source 
const DefaultCacheFlushInterval = 1 * time.Minute

DefaultCacheFlushInterval is the default value for Fragment.CacheFlushInterval.

View Source 
const (
	DefaultColumnLabel = "columnID"
)

Default index settings.

View Source 
const (
	// DefaultFragmentMaxOpN is the default value for Fragment.MaxOpN.
	DefaultFragmentMaxOpN = 2000
)
View Source 
const (
	// ThresholdFactor is used to calculate the threshold for new items entering the cache
	ThresholdFactor = 1.1
)
View Source 
const TimeFormat = "2006-01-02T15:04"

TimeFormat is the go-style time format used to parse string dates.

Variables

View Source 
var (
	ErrHostRequired = errors.New("host required")

	ErrIndexRequired = errors.New("index required")
	ErrIndexExists   = errors.New("index already exists")
	ErrIndexNotFound = errors.New("index not found")

	// ErrFrameRequired is returned when no frame is specified.
	ErrFrameRequired        = errors.New("frame required")
	ErrFrameExists          = errors.New("frame already exists")
	ErrFrameNotFound        = errors.New("frame not found")
	ErrFrameInverseDisabled = errors.New("frame inverse disabled")
	ErrColumnRowLabelEqual  = errors.New("column and row labels cannot be equal")

	ErrInvalidView      = errors.New("invalid view")
	ErrInvalidCacheType = errors.New("invalid cache type")

	ErrName  = errors.New("invalid index or frame's name, must match [a-z0-9_-]")
	ErrLabel = errors.New("invalid row or column label, must match [A-Za-z0-9_-]")

	// ErrFragmentNotFound is returned when a fragment does not exist.
	ErrFragmentNotFound = errors.New("fragment not found")
	ErrQueryRequired    = errors.New("query required")
	ErrTooManyWrites    = errors.New("too many write commands")
)

System errors.

View Source 
var BuildTime = "not recorded"
View Source 
var ErrInvalidTimeQuantum = errors.New("invalid time quantum")

ErrInvalidTimeQuantum is returned when parsing a time quantum.

View Source 
var Expvar = expvar.NewMap("index")

Expvar global expvar map.

View Source 
var NopBroadcastReceiver = &nopBroadcastReceiver{}

NopBroadcastReceiver is a no-op implementation of the BroadcastReceiver.

View Source 
var Version = "v0.0.0"

Functions

func EncodeIndexes added in v0.4.0 

func EncodeIndexes(a []*Index) []*internal.Index

EncodeIndexes converts a into its internal representation.

func IsInverseView 

func IsInverseView(name string) bool

IsInverseView returns true if the view is used for storing an inverted representation.

func IsValidCacheType 

func IsValidCacheType(v string) bool

IsValidCacheType returns true if v is a valid cache type.

func IsValidView 

func IsValidView(name string) bool

IsValidView returns true if name is valid.

func MarshalImportPayload 

func MarshalImportPayload(index, frame string, slice uint64, bits []Bit) ([]byte, error)

MarshalImportPayload marshalls the import parameters into a protobuf byte slice.

func MarshalMessage 

func MarshalMessage(m proto.Message) ([]byte, error)

MarshalMessage encodes the protobuf message into a byte slice.

func NewRouter 

func NewRouter(handler *Handler) *mux.Router

NewRouter creates a Gorilla Mux http router.

func Pos 

func Pos(rowID, columnID uint64) uint64

Pos returns the row position of a row/column pair.

func UnionStringSlice 

func UnionStringSlice(a, b []string) []string

UnionStringSlice returns a sorted set of tags which combine a & b.

func UnmarshalMessage 

func UnmarshalMessage(buf []byte) (proto.Message, error)

UnmarshalMessage decodes the byte slice into a protobuf message.

func ValidateLabel 

func ValidateLabel(label string) error

ValidateLabel ensures that the label is a valid format.

func ValidateName 

func ValidateName(name string) error

ValidateName ensures that the name is a valid format.

func ViewByTimeUnit 

func ViewByTimeUnit(name string, t time.Time, unit rune) string

ViewByTimeUnit returns the view name for time with a given quantum unit.

func ViewsByTime 

func ViewsByTime(name string, t time.Time, q TimeQuantum) []string

ViewsByTime returns a list of views for a given timestamp.

func ViewsByTimeRange 

func ViewsByTimeRange(name string, start, end time.Time, q TimeQuantum) []string

ViewsByTimeRange returns a list of views to traverse to query a time range.

Types

type AttrBlock 

type AttrBlock struct {
	ID       uint64 `json:"id"`
	Checksum []byte `json:"checksum"`
}

AttrBlock represents a checksummed block of the attribute store.

type AttrBlocks 

type AttrBlocks []AttrBlock

AttrBlocks represents a list of blocks.

func (AttrBlocks) Diff 

func (a AttrBlocks) Diff(other []AttrBlock) []uint64

Diff returns a list of block ids that are different or are new in other. Block lists must be in sorted order.

type AttrStore 

type AttrStore struct {
	// contains filtered or unexported fields
}

AttrStore represents a storage layer for attributes.

func NewAttrStore 

func NewAttrStore(path string) *AttrStore

NewAttrStore returns a new instance of AttrStore.

func (*AttrStore) Attrs 

func (s *AttrStore) Attrs(id uint64) (m map[string]interface{}, err error)

Attrs returns a set of attributes by ID.

func (*AttrStore) BlockData 

func (s *AttrStore) BlockData(i uint64) (map[uint64]map[string]interface{}, error)

BlockData returns all data for a single block.

func (*AttrStore) Blocks 

func (s *AttrStore) Blocks() ([]AttrBlock, error)

Blocks returns a list of all blocks in the store.

func (*AttrStore) Close 

func (s *AttrStore) Close() error

Close closes the store.

func (*AttrStore) Open 

func (s *AttrStore) Open() error

Open opens and initializes the store.

func (*AttrStore) Path 

func (s *AttrStore) Path() string

Path returns path to the store's data file.

func (*AttrStore) SetAttrs 

func (s *AttrStore) SetAttrs(id uint64, m map[string]interface{}) error

SetAttrs sets attribute values for a given ID.

func (*AttrStore) SetBulkAttrs 

func (s *AttrStore) SetBulkAttrs(m map[uint64]map[string]interface{}) error

SetBulkAttrs sets attribute values for a set of ids.

type Bit 

type Bit struct {
	RowID     uint64
	ColumnID  uint64
	Timestamp int64
}

Bit represents the location of a single bit.

type Bitmap 

type Bitmap struct {

	// Attributes associated with the bitmap.
	Attrs map[string]interface{}
	// contains filtered or unexported fields
}

Bitmap represents a set of bits.

func NewBitmap 

func NewBitmap(bits ...uint64) *Bitmap

NewBitmap returns a new instance of Bitmap.

func Union 

func Union(bitmaps []*Bitmap) *Bitmap

Union performs a union on a slice of bitmaps.

func (*Bitmap) Bits 

func (b *Bitmap) Bits() []uint64

Bits returns the bits in b as a slice of ints.

func (*Bitmap) ClearBit 

func (b *Bitmap) ClearBit(i uint64) (changed bool)

ClearBit clears the i-th bit of the bitmap.

func (*Bitmap) Count 

func (b *Bitmap) Count() uint64

Count returns the number of set bits in the bitmap.

func (*Bitmap) DecrementCount 

func (b *Bitmap) DecrementCount(i uint64)

DecrementCount decrements the bitmap cached counter.

func (*Bitmap) Difference 

func (b *Bitmap) Difference(other *Bitmap) *Bitmap

Difference returns the diff of b and other.

func (*Bitmap) IncrementCount 

func (b *Bitmap) IncrementCount(i uint64)

IncrementCount increments the bitmap cached counter, note this is an optimization that assumes that the caller is aware the size increased.

func (*Bitmap) Intersect 

func (b *Bitmap) Intersect(other *Bitmap) *Bitmap

Intersect returns the itersection of b and other.

func (*Bitmap) IntersectionCount 

func (b *Bitmap) IntersectionCount(other *Bitmap) uint64

IntersectionCount returns the number of intersections between b and other.

func (*Bitmap) InvalidateCount 

func (b *Bitmap) InvalidateCount()

InvalidateCount updates the cached count in the bitmap.

func (*Bitmap) MarshalJSON 

func (b *Bitmap) MarshalJSON() ([]byte, error)

MarshalJSON returns a JSON-encoded byte slice of b.

func (*Bitmap) Merge 

func (b *Bitmap) Merge(other *Bitmap)

Merge merges data from other into b.

func (*Bitmap) SetBit 

func (b *Bitmap) SetBit(i uint64) (changed bool)

SetBit sets the i-th bit of the bitmap.

func (*Bitmap) Union 

func (b *Bitmap) Union(other *Bitmap) *Bitmap

Union returns the bitwise union of b and other.

type BitmapCache 

type BitmapCache interface {
	Fetch(id uint64) (*Bitmap, bool)
	Add(id uint64, b *Bitmap)
}

BitmapCache provides an interface for caching full bitmaps.

type BitmapPair 

type BitmapPair struct {
	ID    uint64
	Count uint64
}

BitmapPair represents a id/count pair with an associated identifier.

type BitmapPairs 

type BitmapPairs []BitmapPair

BitmapPairs is a sortable list of BitmapPair objects.

func (BitmapPairs) Len 

func (p BitmapPairs) Len() int

func (BitmapPairs) Less 

func (p BitmapPairs) Less(i, j int) bool

func (BitmapPairs) Swap 

func (p BitmapPairs) Swap(i, j int)

type BitmapSegment 

type BitmapSegment struct {
	// contains filtered or unexported fields
}

BitmapSegment holds a subset of a bitmap. This could point to a mmapped roaring bitmap or an in-memory bitmap. The width of the segment will always match the slice width.

func (*BitmapSegment) Bits 

func (s *BitmapSegment) Bits() []uint64

Bits returns a list of all bits set in the segment.

func (*BitmapSegment) ClearBit 

func (s *BitmapSegment) ClearBit(i uint64) (changed bool)

ClearBit clears the i-th bit of the bitmap.

func (*BitmapSegment) Count 

func (s *BitmapSegment) Count() uint64

Count returns the number of set bits in the bitmap.

func (*BitmapSegment) Difference 

func (s *BitmapSegment) Difference(other *BitmapSegment) *BitmapSegment

Difference returns the diff of s and other.

func (*BitmapSegment) Intersect 

func (s *BitmapSegment) Intersect(other *BitmapSegment) *BitmapSegment

Intersect returns the itersection of s and other.

func (*BitmapSegment) IntersectionCount 

func (s *BitmapSegment) IntersectionCount(other *BitmapSegment) uint64

IntersectionCount returns the number of intersections between s and other.

func (*BitmapSegment) InvalidateCount 

func (s *BitmapSegment) InvalidateCount()

InvalidateCount updates the cached count in the bitmap.

func (*BitmapSegment) Merge 

func (s *BitmapSegment) Merge(other *BitmapSegment)

Merge adds chunks from other to s. Chunks in s are overwritten if they exist in other.

func (*BitmapSegment) SetBit 

func (s *BitmapSegment) SetBit(i uint64) (changed bool)

SetBit sets the i-th bit of the bitmap.

func (*BitmapSegment) Union 

func (s *BitmapSegment) Union(other *BitmapSegment) *BitmapSegment

Union returns the bitwise union of s and other.

type Bits 

type Bits []Bit

Bits represents a slice of bits.

func (Bits) ColumnIDs 

func (p Bits) ColumnIDs() []uint64

ColumnIDs returns a slice of all the column IDs.

func (Bits) GroupBySlice 

func (p Bits) GroupBySlice() map[uint64][]Bit

GroupBySlice returns a map of bits by slice.

func (Bits) Len 

func (p Bits) Len() int

func (Bits) Less 

func (p Bits) Less(i, j int) bool

func (Bits) RowIDs 

func (p Bits) RowIDs() []uint64

RowIDs returns a slice of all the row IDs.

func (Bits) Swap 

func (p Bits) Swap(i, j int)

func (Bits) Timestamps 

func (p Bits) Timestamps() []int64

Timestamps returns a slice of all the timestamps.

type BitsByPos 

type BitsByPos []Bit

BitsByPos represents a slice of bits sorted by internal position.

func (BitsByPos) Len 

func (p BitsByPos) Len() int

func (BitsByPos) Less 

func (p BitsByPos) Less(i, j int) bool

func (BitsByPos) Swap 

func (p BitsByPos) Swap(i, j int)

type BroadcastHandler 

type BroadcastHandler interface {
	ReceiveMessage(pb proto.Message) error
}

BroadcastHandler is the interface for the pilosa object which knows how to handle broadcast messages. (Hint: this is implemented by pilosa.Server)

type BroadcastReceiver 

type BroadcastReceiver interface {
	// Start starts listening for broadcast messages - it should return
	// immediately, spawning a goroutine if necessary.
	Start(BroadcastHandler) error
}

BroadcastReceiver is the interface for the object which will listen for and decode broadcast messages before passing them to pilosa to handle. The implementation of this could be an http server which listens for messages, gets the protobuf payload, and then passes it to BroadcastHandler.ReceiveMessage.

type Broadcaster 

type Broadcaster interface {
	SendSync(pb proto.Message) error
	SendAsync(pb proto.Message) error
}

Broadcaster is an interface for broadcasting messages. 

var NopBroadcaster Broadcaster

NopBroadcaster represents a Broadcaster that doesn't do anything.

type BufIterator 

type BufIterator struct {
	// contains filtered or unexported fields
}

BufIterator wraps an iterator to provide the ability to unread values.

func NewBufIterator 

func NewBufIterator(itr Iterator) *BufIterator

NewBufIterator returns a buffered iterator that wraps itr.

func (*BufIterator) Next 

func (itr *BufIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next pair in the row. If a value has been buffered then it is returned and the buffer is cleared.

func (*BufIterator) Peek 

func (itr *BufIterator) Peek() (rowID, columnID uint64, eof bool)

Peek reads the next value but leaves it on the buffer.

func (*BufIterator) Seek 

func (itr *BufIterator) Seek(rowID, columnID uint64)

Seek moves to the first pair equal to or greater than pseek/bseek.

func (*BufIterator) Unread 

func (itr *BufIterator) Unread()

Unread pushes previous pair on to the buffer. Panics if the buffer is already full.

type Cache 

type Cache interface {
	Add(id uint64, n uint64)
	BulkAdd(id uint64, n uint64)
	Get(id uint64) uint64
	Len() int

	// Returns a list of all IDs.
	IDs() []uint64

	// Updates the cache, if necessary.
	Invalidate()

	// Rebuilds the cache
	Recalculate()

	// Returns an ordered list of the top ranked bitmaps.
	Top() []BitmapPair

	// SetStats defines the stats client used in the cache.
	SetStats(s StatsClient)
}

Cache represents a cache of counts.

type Client 

type Client struct {

	// The client to use for HTTP communication.
	// Defaults to the http.DefaultClient.
	HTTPClient *http.Client
	// contains filtered or unexported fields
}

Client represents a client to the Pilosa cluster.

func NewClient 

func NewClient(host string) (*Client, error)

NewClient returns a new instance of Client to connect to host.

func (*Client) BackupSlice 

func (c *Client) BackupSlice(ctx context.Context, index, frame, view string, slice uint64) (io.ReadCloser, error)

BackupSlice retrieves a streaming backup from a single slice. This function tries slice owners until one succeeds.

func (*Client) BackupTo 

func (c *Client) BackupTo(ctx context.Context, w io.Writer, index, frame, view string) error

BackupTo backs up an entire frame from a cluster to w.

func (*Client) BlockData 

func (c *Client) BlockData(ctx context.Context, index, frame, view string, slice uint64, block int) ([]uint64, []uint64, error)

BlockData returns row/column id pairs for a block.

func (*Client) ColumnAttrDiff 

func (c *Client) ColumnAttrDiff(ctx context.Context, index string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)

ColumnAttrDiff returns data from differing blocks on a remote host.

func (*Client) CreateFrame 

func (c *Client) CreateFrame(ctx context.Context, index, frame string, opt FrameOptions) error

CreateFrame creates a new frame on the server.

func (*Client) CreateIndex 

func (c *Client) CreateIndex(ctx context.Context, index string, opt IndexOptions) error

CreateIndex creates a new index on the server.

func (*Client) ExecutePQL 

func (c *Client) ExecutePQL(ctx context.Context, index, query string) (interface{}, error)

ExecutePQL executes query string against index on the server.

func (*Client) ExecuteQuery 

func (c *Client) ExecuteQuery(ctx context.Context, index, query string, allowRedirect bool) (result interface{}, err error)

ExecuteQuery executes query against index on the server.

func (*Client) ExportCSV 

func (c *Client) ExportCSV(ctx context.Context, index, frame, view string, slice uint64, w io.Writer) error

ExportCSV bulk exports data for a single slice from a host to CSV format.

func (*Client) FragmentBlocks 

func (c *Client) FragmentBlocks(ctx context.Context, index, frame, view string, slice uint64) ([]FragmentBlock, error)

FragmentBlocks returns a list of block checksums for a fragment on a host. Only returns blocks which contain data.

func (*Client) FragmentNodes 

func (c *Client) FragmentNodes(ctx context.Context, index string, slice uint64) ([]*Node, error)

FragmentNodes returns a list of nodes that own a slice.

func (*Client) FrameViews 

func (c *Client) FrameViews(ctx context.Context, index, frame string) ([]string, error)

FrameViews returns a list of view names for a frame.

func (*Client) Host 

func (c *Client) Host() string

Host returns the host the client was initialized with.

func (*Client) Import 

func (c *Client) Import(ctx context.Context, index, frame string, slice uint64, bits []Bit) error

Import bulk imports bits for a single slice to a host.

func (*Client) MaxInverseSliceByIndex 

func (c *Client) MaxInverseSliceByIndex(ctx context.Context) (map[string]uint64, error)

MaxInverseSliceByIndex returns the number of inverse slices on a server by index.

func (*Client) MaxSliceByIndex 

func (c *Client) MaxSliceByIndex(ctx context.Context) (map[string]uint64, error)

MaxSliceByIndex returns the number of slices on a server by index.

func (*Client) RestoreFrame 

func (c *Client) RestoreFrame(ctx context.Context, host, index, frame string) error

RestoreFrame restores an entire frame from a host in another cluster.

func (*Client) RestoreFrom 

func (c *Client) RestoreFrom(ctx context.Context, r io.Reader, index, frame, view string) error

RestoreFrom restores a frame from a backup file to an entire cluster.

func (*Client) RowAttrDiff 

func (c *Client) RowAttrDiff(ctx context.Context, index, frame string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)

RowAttrDiff returns data from differing blocks on a remote host.

func (*Client) Schema 

func (c *Client) Schema(ctx context.Context) ([]*IndexInfo, error)

Schema returns all index and frame schema information.

type Cluster 

type Cluster struct {
	Nodes   []*Node
	NodeSet NodeSet

	// Hashing algorithm used to assign partitions to nodes.
	Hasher Hasher

	// The number of partitions in the cluster.
	PartitionN int

	// The number of replicas a partition has.
	ReplicaN int

	// Threshold for logging long-running queries
	LongQueryTime time.Duration
}

Cluster represents a collection of nodes.

func NewCluster 

func NewCluster() *Cluster

NewCluster returns a new instance of Cluster with defaults.

func (*Cluster) FragmentNodes 

func (c *Cluster) FragmentNodes(index string, slice uint64) []*Node

FragmentNodes returns a list of nodes that own a fragment.

func (*Cluster) NodeByHost 

func (c *Cluster) NodeByHost(host string) *Node

NodeByHost returns a node reference by host.

func (*Cluster) NodeSetHosts 

func (c *Cluster) NodeSetHosts() []string

NodeSetHosts returns the list of host strings for NodeSet members.

func (*Cluster) NodeStates 

func (c *Cluster) NodeStates() map[string]string

NodeStates returns a map of nodes in the cluster with each node's state (UP/DOWN) as the value.

func (*Cluster) OwnsFragment 

func (c *Cluster) OwnsFragment(host string, index string, slice uint64) bool

OwnsFragment returns true if a host owns a fragment.

func (*Cluster) OwnsSlices 

func (c *Cluster) OwnsSlices(index string, maxSlice uint64, host string) []uint64

OwnsSlices find the set of slices owned by the node per Index

func (*Cluster) Partition 

func (c *Cluster) Partition(index string, slice uint64) int

Partition returns the partition that a slice belongs to.

func (*Cluster) PartitionNodes 

func (c *Cluster) PartitionNodes(partitionID int) []*Node

PartitionNodes returns a list of nodes that own a partition.

func (*Cluster) Status 

func (c *Cluster) Status() *internal.ClusterStatus

Status returns the internal ClusterStatus representation.

type CmdIO 

type CmdIO struct {
	Stdin  io.Reader
	Stdout io.Writer
	Stderr io.Writer
}

CmdIO holds standard unix inputs and outputs.

func NewCmdIO 

func NewCmdIO(stdin io.Reader, stdout, stderr io.Writer) *CmdIO

NewCmdIO returns a new instance of CmdIO with inputs and outputs set to the arguments.

type ColumnAttrSet 

type ColumnAttrSet struct {
	ID    uint64                 `json:"id"`
	Attrs map[string]interface{} `json:"attrs,omitempty"`
}

ColumnAttrSet represents a set of attributes for a vertical column in an index. Can have a set of attributes attached to it.

type Config 

type Config struct {
	DataDir string `toml:"data-dir"`
	Host    string `toml:"host"`

	Cluster struct {
		ReplicaN        int      `toml:"replicas"`
		Type            string   `toml:"type"`
		Hosts           []string `toml:"hosts"`
		InternalHosts   []string `toml:"internal-hosts"`
		PollingInterval Duration `toml:"polling-interval"`
		InternalPort    string   `toml:"internal-port"`
		GossipSeed      string   `toml:"gossip-seed"`
		LongQueryTime   Duration `toml:"long-query-time"`
	} `toml:"cluster"`

	Plugins struct {
		Path string `toml:"path"`
	} `toml:"plugins"`

	AntiEntropy struct {
		Interval Duration `toml:"interval"`
	} `toml:"anti-entropy"`

	// Limits the number of mutating commands that can be in a single request to
	// the server. This includes SetBit, ClearBit, SetRowAttrs & SetColumnAttrs.
	MaxWritesPerRequest int `toml:"max-writes-per-request"`

	LogPath string `toml:"log-path"`

	Metric struct {
		Service         string   `toml:"service"`
		Host            string   `toml:"host"`
		PollingInterval Duration `toml:"interval"`
	} `toml:"metrics"`
}

Config represents the configuration for the command.

func NewConfig 

func NewConfig() *Config

NewConfig returns an instance of Config with default options.

type Duration 

type Duration time.Duration

Duration is a TOML wrapper type for time.Duration.

func (Duration) MarshalTOML added in v0.4.0 

func (d Duration) MarshalTOML() ([]byte, error)

func (Duration) MarshalText 

func (d Duration) MarshalText() (text []byte, err error)

MarshalText writes duration value in text format.

func (Duration) String 

func (d Duration) String() string

String returns the string representation of the duration.

func (*Duration) UnmarshalText 

func (d *Duration) UnmarshalText(text []byte) error

UnmarshalText parses a TOML value into a duration value.

type ExecOptions 

type ExecOptions struct {
	Remote bool
}

ExecOptions represents an execution context for a single Execute() call.

type Executor 

type Executor struct {
	Holder *Holder

	// Local hostname & cluster configuration.
	Host    string
	Cluster *Cluster

	// Client used for remote HTTP requests.
	HTTPClient *http.Client

	// Maximum number of SetBit() or ClearBit() commands per request.
	MaxWritesPerRequest int
}

Executor recursively executes calls in a PQL query across all slices.

func NewExecutor 

func NewExecutor() *Executor

NewExecutor returns a new instance of Executor.

func (*Executor) Execute 

func (e *Executor) Execute(ctx context.Context, index string, q *pql.Query, slices []uint64, opt *ExecOptions) ([]interface{}, error)

Execute executes a PQL query.

type ExpvarStatsClient 

type ExpvarStatsClient struct {
	// contains filtered or unexported fields
}

ExpvarStatsClient writes stats out to expvars.

func NewExpvarStatsClient 

func NewExpvarStatsClient() *ExpvarStatsClient

NewExpvarStatsClient returns a new instance of ExpvarStatsClient. This client points at the root of the expvar index map.

func (*ExpvarStatsClient) Count 

func (c *ExpvarStatsClient) Count(name string, value int64, rate float64)

Count tracks the number of times something occurs.

func (*ExpvarStatsClient) CountWithCustomTags added in v0.4.0 

func (c *ExpvarStatsClient) CountWithCustomTags(name string, value int64, rate float64, tags []string)

CountWithCustomTags Tracks the number of times something occurs per second with custom tags

func (*ExpvarStatsClient) Gauge 

func (c *ExpvarStatsClient) Gauge(name string, value float64, rate float64)

Gauge sets the value of a metric.

func (*ExpvarStatsClient) Histogram 

func (c *ExpvarStatsClient) Histogram(name string, value float64, rate float64)

Histogram tracks statistical distribution of a metric. This works the same as gauge for this client.

func (*ExpvarStatsClient) Set 

func (c *ExpvarStatsClient) Set(name string, value string, rate float64)

Set tracks number of unique elements.

func (*ExpvarStatsClient) SetLogger added in v0.4.0 

func (c *ExpvarStatsClient) SetLogger(logger io.Writer)

SetLogger has no logger

func (*ExpvarStatsClient) Tags 

func (c *ExpvarStatsClient) Tags() []string

Tags returns a sorted list of tags on the client.

func (*ExpvarStatsClient) Timing 

func (c *ExpvarStatsClient) Timing(name string, value time.Duration, rate float64)

Timing tracks timing information for a metric.

func (*ExpvarStatsClient) WithTags 

func (c *ExpvarStatsClient) WithTags(tags ...string) StatsClient

WithTags returns a new client with additional tags appended.

type Fragment 

type Fragment struct {

	// Cache for row counts.
	CacheType string // passed in by frame

	CacheSize uint32

	// Number of operations performed before performing a snapshot.
	// This limits the size of fragments on the heap and flushes them to disk
	// so that they can be mmapped and heap utilization can be kept low.
	MaxOpN int

	// Writer used for out-of-band log entries.
	LogOutput io.Writer

	// Row attribute storage.
	// This is set by the parent frame unless overridden for testing.
	RowAttrStore *AttrStore
	// contains filtered or unexported fields
}

Fragment represents the intersection of a frame and slice in an index.

func NewFragment 

func NewFragment(path, index, frame, view string, slice uint64) *Fragment

NewFragment returns a new instance of Fragment.

func (*Fragment) BlockData 

func (f *Fragment) BlockData(id int) (rowIDs, columnIDs []uint64)

BlockData returns bits in a block as row & column ID pairs.

func (*Fragment) BlockN 

func (f *Fragment) BlockN() int

BlockN returns the number of blocks in the fragment.

func (*Fragment) Blocks 

func (f *Fragment) Blocks() []FragmentBlock

Blocks returns info for all blocks containing data.

func (*Fragment) Cache 

func (f *Fragment) Cache() Cache

Cache returns the fragment's cache. This is not safe for concurrent use.

func (*Fragment) CachePath 

func (f *Fragment) CachePath() string

CachePath returns the path to the fragment's cache data.

func (*Fragment) Checksum 

func (f *Fragment) Checksum() []byte

Checksum returns a checksum for the entire fragment. If two fragments have the same checksum then they have the same data.

func (*Fragment) ClearBit 

func (f *Fragment) ClearBit(rowID, columnID uint64) (bool, error)

ClearBit clears a bit for a given column & row within the fragment. This updates both the on-disk storage and the in-cache bitmap.

func (*Fragment) Close 

func (f *Fragment) Close() error

Close flushes the underlying storage, closes the file and unlocks it.

func (*Fragment) FlushCache 

func (f *Fragment) FlushCache() error

FlushCache writes the cache data to disk.

func (*Fragment) ForEachBit 

func (f *Fragment) ForEachBit(fn func(rowID, columnID uint64) error) error

ForEachBit executes fn for every bit set in the fragment. Errors returned from fn are passed through.

func (*Fragment) Frame 

func (f *Fragment) Frame() string

Frame returns the frame the fragment was initialized with.

func (*Fragment) Import 

func (f *Fragment) Import(rowIDs, columnIDs []uint64) error

Import bulk imports a set of bits and then snapshots the storage. This does not affect the fragment's cache.

func (*Fragment) Index 

func (f *Fragment) Index() string

Index returns the index that the fragment was initialized with.

func (*Fragment) InvalidateChecksums 

func (f *Fragment) InvalidateChecksums()

InvalidateChecksums clears all cached block checksums.

func (*Fragment) MergeBlock 

func (f *Fragment) MergeBlock(id int, data []PairSet) (sets, clears []PairSet, err error)

MergeBlock compares the block's bits and computes a diff with another set of block bits. The state of a bit is determined by consensus from all blocks being considered.

For example, if 3 blocks are compared and two have a set bit and one has a cleared bit then the bit is considered cleared. The function returns the diff per incoming block so that all can be in sync.

func (*Fragment) Open 

func (f *Fragment) Open() error

Open opens the underlying storage.

func (*Fragment) Path 

func (f *Fragment) Path() string

Path returns the path the fragment was initialized with.

func (*Fragment) ReadFrom 

func (f *Fragment) ReadFrom(r io.Reader) (n int64, err error)

ReadFrom reads a data file from r and loads it into the fragment.

func (*Fragment) RecalculateCache 

func (f *Fragment) RecalculateCache()

RecalculateCache rebuilds the cache regardless of invalidate time delay.

func (*Fragment) Row 

func (f *Fragment) Row(rowID uint64) *Bitmap

Row returns a row by ID.

func (*Fragment) SetBit 

func (f *Fragment) SetBit(rowID, columnID uint64) (changed bool, err error)

SetBit sets a bit for a given column & row within the fragment. This updates both the on-disk storage and the in-cache bitmap.

func (*Fragment) Slice 

func (f *Fragment) Slice() uint64

Slice returns the slice the fragment was initialized with.

func (*Fragment) Snapshot 

func (f *Fragment) Snapshot() error

Snapshot writes the storage bitmap to disk and reopens it.

func (*Fragment) Top 

func (f *Fragment) Top(opt TopOptions) ([]Pair, error)

Top returns the top rows from the fragment. If opt.Src is specified then only rows which intersect src are returned. If opt.FilterValues exist then the row attribute specified by field is matched.

func (*Fragment) View 

func (f *Fragment) View() string

View returns the view the fragment was initialized with.

func (*Fragment) WriteTo 

func (f *Fragment) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the fragment's data to w.

type FragmentBlock 

type FragmentBlock struct {
	ID       int    `json:"id"`
	Checksum []byte `json:"checksum"`
}

FragmentBlock represents info about a subsection of the rows in a block. This is used for comparing data in remote blocks for active anti-entropy.

type FragmentSyncer 

type FragmentSyncer struct {
	Fragment *Fragment

	Host    string
	Cluster *Cluster

	Closing <-chan struct{}
}

FragmentSyncer syncs a local fragment to one on a remote host.

func (*FragmentSyncer) SyncFragment 

func (s *FragmentSyncer) SyncFragment() error

SyncFragment compares checksums for the local and remote fragments and then merges any blocks which have differences.

type Frame 

type Frame struct {
	Stats StatsClient

	LogOutput io.Writer
	// contains filtered or unexported fields
}

Frame represents a container for views.

func NewFrame 

func NewFrame(path, index, name string) (*Frame, error)

NewFrame returns a new instance of frame.

func (*Frame) CacheSize 

func (f *Frame) CacheSize() uint32

CacheSize returns the ranked frame cache size.

func (*Frame) CacheType 

func (f *Frame) CacheType() string

CacheType returns the caching mode for the frame.

func (*Frame) ClearBit 

func (f *Frame) ClearBit(name string, rowID, colID uint64, t *time.Time) (changed bool, err error)

ClearBit clears a bit within the frame.

func (*Frame) Close 

func (f *Frame) Close() error

Close closes the frame and its views.

func (*Frame) CreateViewIfNotExists 

func (f *Frame) CreateViewIfNotExists(name string) (*View, error)

CreateViewIfNotExists returns the named view, creating it if necessary.

func (*Frame) Import 

func (f *Frame) Import(rowIDs, columnIDs []uint64, timestamps []*time.Time) error

Import bulk imports data.

func (*Frame) Index 

func (f *Frame) Index() string

Index returns the index name the frame was initialized with.

func (*Frame) InverseEnabled 

func (f *Frame) InverseEnabled() bool

InverseEnabled returns true if an inverse view is available.

func (*Frame) MaxInverseSlice 

func (f *Frame) MaxInverseSlice() uint64

MaxInverseSlice returns the max inverse slice in the frame.

func (*Frame) MaxSlice 

func (f *Frame) MaxSlice() uint64

MaxSlice returns the max slice in the frame.

func (*Frame) Name 

func (f *Frame) Name() string

Name returns the name the frame was initialized with.

func (*Frame) Open 

func (f *Frame) Open() error

Open opens and initializes the frame.

func (*Frame) Options 

func (f *Frame) Options() FrameOptions

Options returns all options for this frame.

func (*Frame) Path 

func (f *Frame) Path() string

Path returns the path the frame was initialized with.

func (*Frame) RowAttrStore 

func (f *Frame) RowAttrStore() *AttrStore

RowAttrStore returns the attribute storage.

func (*Frame) RowLabel 

func (f *Frame) RowLabel() string

RowLabel returns the row label.

func (*Frame) SetBit 

func (f *Frame) SetBit(name string, rowID, colID uint64, t *time.Time) (changed bool, err error)

SetBit sets a bit on a view within the frame.

func (*Frame) SetCacheSize 

func (f *Frame) SetCacheSize(v uint32) error

SetCacheSize sets the cache size for ranked fames. Persists to meta file on update. defaults to DefaultCacheSize 50000

func (*Frame) SetRowLabel 

func (f *Frame) SetRowLabel(v string) error

SetRowLabel sets the row labels. Persists to meta file on update.

func (*Frame) SetTimeQuantum 

func (f *Frame) SetTimeQuantum(q TimeQuantum) error

SetTimeQuantum sets the time quantum for the frame.

func (*Frame) TimeQuantum 

func (f *Frame) TimeQuantum() TimeQuantum

TimeQuantum returns the time quantum for the frame.

func (*Frame) View 

func (f *Frame) View(name string) *View

View returns a view in the frame by name.

func (*Frame) ViewPath 

func (f *Frame) ViewPath(name string) string

ViewPath returns the path to a view in the frame.

func (*Frame) Views 

func (f *Frame) Views() []*View

Views returns a list of all views in the frame.

type FrameInfo 

type FrameInfo struct {
	Name  string      `json:"name"`
	Views []*ViewInfo `json:"views,omitempty"`
}

FrameInfo represents schema information for a frame.

type FrameOptions 

type FrameOptions struct {
	RowLabel       string      `json:"rowLabel,omitempty"`
	InverseEnabled bool        `json:"inverseEnabled,omitempty"`
	CacheType      string      `json:"cacheType,omitempty"`
	CacheSize      uint32      `json:"cacheSize,omitempty"`
	TimeQuantum    TimeQuantum `json:"timeQuantum,omitempty"`
}

FrameOptions represents options to set when initializing a frame.

func (*FrameOptions) Encode 

func (o *FrameOptions) Encode() *internal.FrameMeta

Encode converts o into its internal representation.

type Handler 

type Handler struct {
	Holder        *Holder
	Broadcaster   Broadcaster
	StatusHandler StatusHandler

	// Local hostname & cluster configuration.
	Host    string
	Cluster *Cluster

	Router *mux.Router

	// The execution engine for running queries.
	Executor interface {
		Execute(context context.Context, index string, query *pql.Query, slices []uint64, opt *ExecOptions) ([]interface{}, error)
	}

	// The writer for any logging.
	LogOutput io.Writer
}

Handler represents an HTTP handler.

func NewHandler 

func NewHandler() *Handler

NewHandler returns a new instance of Handler with a default logger.

func (*Handler) ServeHTTP 

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP handles an HTTP request.

type Hasher 

type Hasher interface {
	// Hashes the key into a number between [0,N).
	Hash(key uint64, n int) int
}

Hasher represents an interface to hash integers into buckets.

func NewHasher 

func NewHasher() Hasher

NewHasher returns a new instance of the default hasher.

type Holder 

type Holder struct {
	Broadcaster Broadcaster

	// Stats
	Stats StatsClient

	// Data directory path.
	Path string

	// The interval at which the cached row ids are persisted to disk.
	CacheFlushInterval time.Duration

	LogOutput io.Writer
	// contains filtered or unexported fields
}

Holder represents a container for indexes.

func NewHolder 

func NewHolder() *Holder

NewHolder returns a new instance of Holder.

func (*Holder) Close 

func (h *Holder) Close() error

Close closes all open fragments.

func (*Holder) CreateIndex 

func (h *Holder) CreateIndex(name string, opt IndexOptions) (*Index, error)

CreateIndex creates an index. An error is returned if the index already exists.

func (*Holder) CreateIndexIfNotExists 

func (h *Holder) CreateIndexIfNotExists(name string, opt IndexOptions) (*Index, error)

CreateIndexIfNotExists returns an index by name. The index is created if it does not already exist.

func (*Holder) DeleteIndex 

func (h *Holder) DeleteIndex(name string) error

DeleteIndex removes an index from the holder.

func (*Holder) Fragment 

func (h *Holder) Fragment(index, frame, view string, slice uint64) *Fragment

Fragment returns the fragment for an index, frame & slice.

func (*Holder) Frame 

func (h *Holder) Frame(index, name string) *Frame

Frame returns the frame for an index and name.

func (*Holder) Index 

func (h *Holder) Index(name string) *Index

Index returns the index by name.

func (*Holder) IndexPath 

func (h *Holder) IndexPath(name string) string

IndexPath returns the path where a given index is stored.

func (*Holder) Indexes 

func (h *Holder) Indexes() []*Index

Indexes returns a list of all indexes in the holder.

func (*Holder) MaxInverseSlices 

func (h *Holder) MaxInverseSlices() map[string]uint64

MaxInverseSlices returns MaxInverseSlice map for all indexes.

func (*Holder) MaxSlices 

func (h *Holder) MaxSlices() map[string]uint64

MaxSlices returns MaxSlice map for all indexes.

func (*Holder) Open 

func (h *Holder) Open() error

Open initializes the root data directory for the holder.

func (*Holder) Schema 

func (h *Holder) Schema() []*IndexInfo

Schema returns schema data for all indexes and frames.

func (*Holder) View 

func (h *Holder) View(index, frame, name string) *View

View returns the view for an index, frame, and name.

type HolderSyncer 

type HolderSyncer struct {
	Holder *Holder

	Host    string
	Cluster *Cluster

	// Signals that the sync should stop.
	Closing <-chan struct{}
}

HolderSyncer is an active anti-entropy tool that compares the local holder with a remote holder based on block checksums and resolves differences.

func (*HolderSyncer) IsClosing 

func (s *HolderSyncer) IsClosing() bool

IsClosing returns true if the syncer has been marked to close.

func (*HolderSyncer) SyncHolder 

func (s *HolderSyncer) SyncHolder() error

SyncHolder compares the holder on host with the local holder and resolves differences.

type Index 

type Index struct {
	Stats StatsClient

	LogOutput io.Writer
	// contains filtered or unexported fields
}

Index represents a container for frames.

func NewIndex 

func NewIndex(path, name string) (*Index, error)

NewIndex returns a new instance of Index.

func (*Index) Close 

func (i *Index) Close() error

Close closes the index and its frames.

func (*Index) ColumnAttrStore 

func (i *Index) ColumnAttrStore() *AttrStore

ColumnAttrStore returns the storage for column attributes.

func (*Index) ColumnLabel 

func (i *Index) ColumnLabel() string

ColumnLabel returns the column label.

func (*Index) CreateFrame 

func (i *Index) CreateFrame(name string, opt FrameOptions) (*Frame, error)

CreateFrame creates a frame.

func (*Index) CreateFrameIfNotExists 

func (i *Index) CreateFrameIfNotExists(name string, opt FrameOptions) (*Frame, error)

CreateFrameIfNotExists creates a frame with the given options if it doesn't exist.

func (*Index) DeleteFrame 

func (i *Index) DeleteFrame(name string) error

DeleteFrame removes a frame from the index.

func (*Index) Frame 

func (i *Index) Frame(name string) *Frame

Frame returns a frame in the index by name.

func (*Index) FramePath 

func (i *Index) FramePath(name string) string

FramePath returns the path to a frame in the index.

func (*Index) Frames 

func (i *Index) Frames() []*Frame

Frames returns a list of all frames in the index.

func (*Index) MaxInverseSlice 

func (i *Index) MaxInverseSlice() uint64

MaxInverseSlice returns the max inverse slice in the index according to this node.

func (*Index) MaxSlice 

func (i *Index) MaxSlice() uint64

MaxSlice returns the max slice in the index according to this node.

func (*Index) Name 

func (i *Index) Name() string

Name returns name of the index.

func (*Index) Open 

func (i *Index) Open() error

Open opens and initializes the index.

func (*Index) Path 

func (i *Index) Path() string

Path returns the path the index was initialized with.

func (*Index) SetColumnLabel 

func (i *Index) SetColumnLabel(v string) error

SetColumnLabel sets the column label. Persists to meta file on update.

func (*Index) SetRemoteMaxInverseSlice 

func (i *Index) SetRemoteMaxInverseSlice(v uint64)

SetRemoteMaxInverseSlice sets the remote max inverse slice value received from another node.

func (*Index) SetRemoteMaxSlice 

func (i *Index) SetRemoteMaxSlice(newmax uint64)

SetRemoteMaxSlice sets the remote max slice value received from another node.

func (*Index) SetTimeQuantum 

func (i *Index) SetTimeQuantum(q TimeQuantum) error

SetTimeQuantum sets the default time quantum for the index.

func (*Index) TimeQuantum 

func (i *Index) TimeQuantum() TimeQuantum

TimeQuantum returns the default time quantum for the index.

type IndexInfo 

type IndexInfo struct {
	Name   string       `json:"name"`
	Frames []*FrameInfo `json:"frames"`
}

IndexInfo represents schema information for an index.

func MergeSchemas 

func MergeSchemas(a, b []*IndexInfo) []*IndexInfo

MergeSchemas combines indexes and frames from a and b into one schema.

type IndexOptions 

type IndexOptions struct {
	ColumnLabel string      `json:"columnLabel,omitempty"`
	TimeQuantum TimeQuantum `json:"timeQuantum,omitempty"`
}

IndexOptions represents options to set when initializing an index.

func (*IndexOptions) Encode 

func (o *IndexOptions) Encode() *internal.IndexMeta

Encode converts o into its internal representation.

type Iterator 

type Iterator interface {
	Seek(rowID, columnID uint64)
	Next() (rowID, columnID uint64, eof bool)
}

Iterator is an interface for looping over row/column pairs.

type LRUCache 

type LRUCache struct {
	// contains filtered or unexported fields
}

LRUCache represents a least recently used Cache implemenation.

func NewLRUCache 

func NewLRUCache(maxEntries uint32) *LRUCache

NewLRUCache returns a new instance of LRUCache.

func (*LRUCache) Add 

func (c *LRUCache) Add(id, n uint64)

Add adds a count to the cache.

func (*LRUCache) BulkAdd 

func (c *LRUCache) BulkAdd(id, n uint64)

BulkAdd adds a count to the cache unsorted. You should Invalidate after completion.

func (*LRUCache) Get 

func (c *LRUCache) Get(id uint64) uint64

Get returns a count for a given id.

func (*LRUCache) IDs 

func (c *LRUCache) IDs() []uint64

IDs returns a list of all IDs in the cache.

func (*LRUCache) Invalidate 

func (c *LRUCache) Invalidate()

Invalidate is a no-op.

func (*LRUCache) Len 

func (c *LRUCache) Len() int

Len returns the number of items in the cache.

func (*LRUCache) Recalculate 

func (c *LRUCache) Recalculate()

Recalculate is a no-op.

func (*LRUCache) SetStats added in v0.4.0 

func (c *LRUCache) SetStats(s StatsClient)

SetStats defines the stats client used in the cache.

func (*LRUCache) Top 

func (c *LRUCache) Top() []BitmapPair

Top returns all counts in the cache.

type LimitIterator 

type LimitIterator struct {
	// contains filtered or unexported fields
}

LimitIterator wraps an Iterator and limits it to a max column/row pair.

func NewLimitIterator 

func NewLimitIterator(itr Iterator, maxRowID, maxColumnID uint64) *LimitIterator

NewLimitIterator returns a new LimitIterator.

func (*LimitIterator) Next 

func (itr *LimitIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next row/column ID pair. If the underlying iterator returns a pair higher than the max then EOF is returned.

func (*LimitIterator) Seek 

func (itr *LimitIterator) Seek(rowID, columnID uint64)

Seek moves the underlying iterator to a column/row pair.

type MultiStatsClient 

type MultiStatsClient []StatsClient

MultiStatsClient joins multiple stats clients together.

func (MultiStatsClient) Count 

func (a MultiStatsClient) Count(name string, value int64, rate float64)

Count tracks the number of times something occurs per second on all clients.

func (MultiStatsClient) CountWithCustomTags added in v0.4.0 

func (a MultiStatsClient) CountWithCustomTags(name string, value int64, rate float64, tags []string)

CountWithCustomTags Tracks the number of times something occurs per second with custom tags

func (MultiStatsClient) Gauge 

func (a MultiStatsClient) Gauge(name string, value float64, rate float64)

Gauge sets the value of a metric on all clients.

func (MultiStatsClient) Histogram 

func (a MultiStatsClient) Histogram(name string, value float64, rate float64)

Histogram tracks statistical distribution of a metric on all clients.

func (MultiStatsClient) Set 

func (a MultiStatsClient) Set(name string, value string, rate float64)

Set tracks number of unique elements on all clients.

func (MultiStatsClient) SetLogger added in v0.4.0 

func (a MultiStatsClient) SetLogger(logger io.Writer)

SetLogger Sets the StatsD logger output type

func (MultiStatsClient) Tags 

func (a MultiStatsClient) Tags() []string

Tags returns tags from the first client.

func (MultiStatsClient) Timing 

func (a MultiStatsClient) Timing(name string, value time.Duration, rate float64)

Timing tracks timing information for a metric on all clients.

func (MultiStatsClient) WithTags 

func (a MultiStatsClient) WithTags(tags ...string) StatsClient

WithTags returns a new set of clients with the additional tags.

type Node 

type Node struct {
	Host         string `json:"host"`
	InternalHost string `json:"internalHost"`
	// contains filtered or unexported fields
}

Node represents a node in the cluster.

func (*Node) SetState 

func (n *Node) SetState(s string)

SetState sets the Node.status.state.

func (*Node) SetStatus 

func (n *Node) SetStatus(s *internal.NodeStatus)

SetStatus sets the NodeStatus.

type NodeSet 

type NodeSet interface {
	// Returns a list of all Nodes in the cluster
	Nodes() []*Node

	// Open starts any network activity implemented by the NodeSet
	Open() error
}

NodeSet represents an interface for Node membership and inter-node communication.

type Nodes 

type Nodes []*Node

Nodes represents a list of nodes.

func (Nodes) Clone 

func (a Nodes) Clone() []*Node

Clone returns a shallow copy of nodes.

func (Nodes) Contains 

func (a Nodes) Contains(n *Node) bool

Contains returns true if a node exists in the list.

func (Nodes) ContainsHost 

func (a Nodes) ContainsHost(host string) bool

ContainsHost returns true if host matches one of the node's host.

func (Nodes) Filter 

func (a Nodes) Filter(n *Node) []*Node

Filter returns a new list of nodes with node removed.

func (Nodes) FilterHost 

func (a Nodes) FilterHost(host string) []*Node

FilterHost returns a new list of nodes with host removed.

func (Nodes) Hosts 

func (a Nodes) Hosts() []string

Hosts returns a list of all hostnames.

type Pair 

type Pair struct {
	ID    uint64 `json:"id"`
	Count uint64 `json:"count"`
}

Pair holds an id/count pair.

type PairHeap 

type PairHeap struct {
	Pairs
}

PairHeap is a heap implementation over a group of Pairs.

func (PairHeap) Less 

func (p PairHeap) Less(i, j int) bool

Less implemets the Sort interface. reports whether the element with index i should sort before the element with index j.

type PairSet 

type PairSet struct {
	RowIDs    []uint64
	ColumnIDs []uint64
}

PairSet is a list of equal length row and column id lists.

type Pairs 

type Pairs []Pair

Pairs is a sortable slice of Pair objects.

func (Pairs) Add 

func (p Pairs) Add(other []Pair) []Pair

Add merges other into p and returns a new slice.

func (Pairs) Keys 

func (p Pairs) Keys() []uint64

Keys returns a slice of all keys in p.

func (Pairs) Len 

func (p Pairs) Len() int

func (Pairs) Less 

func (p Pairs) Less(i, j int) bool

func (*Pairs) Pop 

func (p *Pairs) Pop() interface{}

Pop removes the minimum element from the Pair slice.

func (*Pairs) Push 

func (p *Pairs) Push(x interface{})

Push appends the element onto the Pair slice.

func (Pairs) String 

func (p Pairs) String() string

func (Pairs) Swap 

func (p Pairs) Swap(i, j int)

type QueryRequest 

type QueryRequest struct {
	// Index to execute query against.
	Index string

	// The query string to parse and execute.
	Query string

	// The slices to include in the query execution.
	// If empty, all slices are included.
	Slices []uint64

	// Return column attributes, if true.
	ColumnAttrs bool

	// Time granularity to use with the timestamp.
	Quantum TimeQuantum

	// If true, indicates that query is part of a larger distributed query.
	// If false, this request is on the originating node.
	Remote bool
}

QueryRequest represent a request to process a query.

type QueryResponse 

type QueryResponse struct {
	// Result for each top-level query call.
	// Can be a Bitmap, Pairs, or uint64.
	Results []interface{}

	// Set of column attribute objects matching IDs returned in Result.
	ColumnAttrSets []*ColumnAttrSet

	// Error during parsing or execution.
	Err error
}

QueryResponse represent a response from a processed query.

func (*QueryResponse) MarshalJSON 

func (resp *QueryResponse) MarshalJSON() ([]byte, error)

MarshalJSON marshals QueryResponse into a JSON-encoded byte slice

type RankCache 

type RankCache struct {
	// contains filtered or unexported fields
}

RankCache represents a cache with sorted entries.

func NewRankCache 

func NewRankCache(maxEntries uint32) *RankCache

NewRankCache returns a new instance of RankCache.

func (*RankCache) Add 

func (c *RankCache) Add(id uint64, n uint64)

Add adds a count to the cache.

func (*RankCache) BulkAdd 

func (c *RankCache) BulkAdd(id uint64, n uint64)

BulkAdd adds a count to the cache unsorted. You should Invalidate after completion.

func (*RankCache) Get 

func (c *RankCache) Get(id uint64) uint64

Get returns a count for a given id.

func (*RankCache) IDs 

func (c *RankCache) IDs() []uint64

IDs returns a list of all IDs in the cache.

func (*RankCache) Invalidate 

func (c *RankCache) Invalidate()

Invalidate recalculates the the entries by rank.

func (*RankCache) Len 

func (c *RankCache) Len() int

Len returns the number of items in the cache.

func (*RankCache) ReadFrom 

func (c *RankCache) ReadFrom(r io.Reader) (n int64, err error)

ReadFrom read from r into the cache.

func (*RankCache) Recalculate 

func (c *RankCache) Recalculate()

Recalculate rebuilds the cache.

func (*RankCache) SetStats added in v0.4.0 

func (c *RankCache) SetStats(s StatsClient)

SetStats defines the stats client used in the cache.

func (*RankCache) Top 

func (c *RankCache) Top() []BitmapPair

Top returns an ordered list of pairs.

func (*RankCache) WriteTo 

func (c *RankCache) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the cache to w.

type RoaringIterator 

type RoaringIterator struct {
	// contains filtered or unexported fields
}

RoaringIterator converts a roaring.Iterator to output column/row pairs.

func NewRoaringIterator 

func NewRoaringIterator(itr *roaring.Iterator) *RoaringIterator

NewRoaringIterator returns a new iterator wrapping itr.

func (*RoaringIterator) Next 

func (itr *RoaringIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next column/row ID pair.

func (*RoaringIterator) Seek 

func (itr *RoaringIterator) Seek(bseek, pseek uint64)

Seek moves the cursor to a pair matching bseek/pseek. If the pair is not found then it moves to the next pair.

type Server 

type Server struct {

	// Data storage and HTTP interface.
	Holder            *Holder
	Handler           *Handler
	Broadcaster       Broadcaster
	BroadcastReceiver BroadcastReceiver

	// Cluster configuration.
	// Host is replaced with actual host after opening if port is ":0".
	Network string
	Host    string
	Cluster *Cluster

	// Background monitoring intervals.
	AntiEntropyInterval time.Duration
	PollingInterval     time.Duration
	MetricInterval      time.Duration

	// Misc options.
	MaxWritesPerRequest int

	LogOutput io.Writer
	// contains filtered or unexported fields
}

Server represents a holder wrapped by a running HTTP server.

func NewServer 

func NewServer() *Server

NewServer returns a new instance of Server.

func (*Server) Addr 

func (s *Server) Addr() net.Addr

Addr returns the address of the listener.

func (*Server) Close 

func (s *Server) Close() error

Close closes the server and waits for it to shutdown.

func (*Server) ClusterStatus 

func (s *Server) ClusterStatus() (proto.Message, error)

ClusterStatus returns the NodeState for all nodes in the cluster.

func (*Server) HandleRemoteStatus 

func (s *Server) HandleRemoteStatus(pb proto.Message) error

HandleRemoteStatus receives incoming NodeState from remote nodes.

func (*Server) LocalStatus 

func (s *Server) LocalStatus() (proto.Message, error)

LocalStatus returns the state of the local node as well as the holder (indexes/frames) according to the local node. In a gossip implementation, memberlist.Delegate.LocalState() uses this. Server implements StatusHandler.

func (*Server) Open 

func (s *Server) Open() error

Open opens and initializes the server.

func (*Server) ReceiveMessage 

func (s *Server) ReceiveMessage(pb proto.Message) error

ReceiveMessage represents an implementation of BroadcastHandler.

type SimpleCache 

type SimpleCache struct {
	// contains filtered or unexported fields
}

SimpleCache implements BitmapCache it is meant to be a short-lived cache for cases where writes are continuing to access the same bit within a short time frame (i.e. good for write-heavy loads) A read-heavy use case would cause the cache to get bigger, potentially causing the node to run out of memory.

func (*SimpleCache) Add 

func (s *SimpleCache) Add(id uint64, b *Bitmap)

Add adds the bitmap to the cache, keyed on the id.

func (*SimpleCache) Fetch 

func (s *SimpleCache) Fetch(id uint64) (*Bitmap, bool)

Fetch retrieves the bitmap at the id in the cache.

type SliceIterator 

type SliceIterator struct {
	// contains filtered or unexported fields
}

SliceIterator iterates over a pair of row/column ID slices.

func NewSliceIterator 

func NewSliceIterator(rowIDs, columnIDs []uint64) *SliceIterator

NewSliceIterator returns an iterator to iterate over a set of row/column ID pairs. Both slices MUST have an equal length. Otherwise the function will panic.

func (*SliceIterator) Next 

func (itr *SliceIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next row/column ID pair.

func (*SliceIterator) Seek 

func (itr *SliceIterator) Seek(bseek, pseek uint64)

Seek moves the cursor to a given pair. If the pair is not found, the iterator seeks to the next pair.

type StaticNodeSet 

type StaticNodeSet struct {
	// contains filtered or unexported fields
}

StaticNodeSet represents a basic NodeSet for testing.

func NewStaticNodeSet 

func NewStaticNodeSet() *StaticNodeSet

NewStaticNodeSet creates a statically defined NodeSet.

func (*StaticNodeSet) Join 

func (s *StaticNodeSet) Join(nodes []*Node) error

Join sets the NodeSet nodes to the slice of Nodes passed in.

func (*StaticNodeSet) Nodes 

func (s *StaticNodeSet) Nodes() []*Node

Nodes implements the NodeSet interface and returns a list of nodes in the cluster.

func (*StaticNodeSet) Open 

func (s *StaticNodeSet) Open() error

Open implements the NodeSet interface to start network activity, but for a static NodeSet it does nothing.

type StatsClient 

type StatsClient interface {
	// Returns a sorted list of tags on the client.
	Tags() []string

	// Returns a new client with additional tags appended.
	WithTags(tags ...string) StatsClient

	// Tracks the number of times something occurs per second.
	Count(name string, value int64, rate float64)

	// Tracks the number of times something occurs per second with custom tags
	CountWithCustomTags(name string, value int64, rate float64, tags []string)

	// Sets the value of a metric.
	Gauge(name string, value float64, rate float64)

	// Tracks statistical distribution of a metric.
	Histogram(name string, value float64, rate float64)

	// Tracks number of unique elements.
	Set(name string, value string, rate float64)

	// Tracks timing information for a metric.
	Timing(name string, value time.Duration, rate float64)

	// SetLogger Set the logger output type
	SetLogger(logger io.Writer)
}

StatsClient represents a client to a stats server. 

var NopStatsClient StatsClient

NopStatsClient represents a client that doesn't do anything.

type StatusHandler 

type StatusHandler interface {
	LocalStatus() (proto.Message, error)
	ClusterStatus() (proto.Message, error)
	HandleRemoteStatus(proto.Message) error
}

StatusHandler specifies two methods which an object must implement to share state in the cluster. These are used by the GossipNodeSet to implement the LocalState and MergeRemoteState methods of memberlist.Delegate

type TimeQuantum 

type TimeQuantum string

TimeQuantum represents a time granularity for time-based bitmaps.

func ParseTimeQuantum 

func ParseTimeQuantum(v string) (TimeQuantum, error)

ParseTimeQuantum parses v into a time quantum.

func (TimeQuantum) HasDay 

func (q TimeQuantum) HasDay() bool

HasDay returns true if the quantum contains a 'D' unit.

func (TimeQuantum) HasHour 

func (q TimeQuantum) HasHour() bool

HasHour returns true if the quantum contains a 'H' unit.

func (TimeQuantum) HasMonth 

func (q TimeQuantum) HasMonth() bool

HasMonth returns true if the quantum contains a 'M' unit.

func (TimeQuantum) HasYear 

func (q TimeQuantum) HasYear() bool

HasYear returns true if the quantum contains a 'Y' unit.

func (TimeQuantum) Valid 

func (q TimeQuantum) Valid() bool

Valid returns true if q is a valid time quantum value.

type TopOptions 

type TopOptions struct {
	// Number of rows to return.
	N int

	// Bitmap to intersect with.
	Src *Bitmap

	// Specific rows to filter against.
	RowIDs       []uint64
	MinThreshold uint64

	// Filter field name & values.
	FilterField       string
	FilterValues      []interface{}
	TanimotoThreshold uint64
}

TopOptions represents options passed into the Top() function.

type View 

type View struct {
	RowAttrStore *AttrStore
	LogOutput    io.Writer
	// contains filtered or unexported fields
}

View represents a container for frame data.

func NewView 

func NewView(path, index, frame, name string, cacheSize uint32) *View

NewView returns a new instance of View.

func (*View) ClearBit 

func (v *View) ClearBit(rowID, columnID uint64) (changed bool, err error)

ClearBit clears a bit within the view.

func (*View) Close 

func (v *View) Close() error

Close closes the view and its fragments.

func (*View) CreateFragmentIfNotExists 

func (v *View) CreateFragmentIfNotExists(slice uint64) (*Fragment, error)

CreateFragmentIfNotExists returns a fragment in the view by slice.

func (*View) Fragment 

func (v *View) Fragment(slice uint64) *Fragment

Fragment returns a fragment in the view by slice.

func (*View) FragmentPath 

func (v *View) FragmentPath(slice uint64) string

FragmentPath returns the path to a fragment in the view.

func (*View) Fragments 

func (v *View) Fragments() []*Fragment

Fragments returns a list of all fragments in the view.

func (*View) Frame 

func (v *View) Frame() string

Frame returns the frame name the view was initialized with.

func (*View) Index 

func (v *View) Index() string

Index returns the index name the view was initialized with.

func (*View) MaxSlice 

func (v *View) MaxSlice() uint64

MaxSlice returns the max slice in the view.

func (*View) Name 

func (v *View) Name() string

Name returns the name the view was initialized with.

func (*View) Open 

func (v *View) Open() error

Open opens and initializes the view.

func (*View) Path 

func (v *View) Path() string

Path returns the path the view was initialized with.

func (*View) SetBit 

func (v *View) SetBit(rowID, columnID uint64) (changed bool, err error)

SetBit sets a bit within the view.

type ViewInfo 

type ViewInfo struct {
	Name string `json:"name"`
}

ViewInfo represents schema information for a view.

Source Files

View all Source files

Directories

Path Synopsis
cmd
Package cmd contains all the pilosa subcommand definitions (1 per file).
 Package cmd contains all the pilosa subcommand definitions (1 per file).
pilosa
This is the entrypoint for the Pilosa binary.
 This is the entrypoint for the Pilosa binary.
package ctl contains all pilosa subcommands other than 'server'.
 package ctl contains all pilosa subcommands other than 'server'.
Package internal is a generated protocol buffer package.
 Package internal is a generated protocol buffer package.
Package pql defines the Pilosa Query Language.
 Package pql defines the Pilosa Query Language.
package roaring implements roaring bitmaps with support for incremental changes.
 package roaring implements roaring bitmaps with support for incremental changes.
Package server contains the `pilosa server` subcommand which runs Pilosa itself.
 Package server contains the `pilosa server` subcommand which runs Pilosa itself.
Package statik contains static assets for the Web UI.
 Package statik contains static assets for the Web UI.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.