stats

Documentation

Overview

Package stats calculates statistical metadata for a given dataset

Index

• Variables
• func ToMap(s Statser) []map[string]interface{}
• type Accumulator
  • func NewAccumulator(r dsio.EntryReader) *Accumulator
  • func (r *Accumulator) Close() error
  • func (r *Accumulator) ReadEntry() (dsio.Entry, error)
  • func (r *Accumulator) Stats() []Stat
  • func (r *Accumulator) Structure() *dataset.Structure
• type Cache
  • func NewOSCache(rootDir string, maxSize uint64) Cache
• type Stat
• type Stats
  • func New(cache Cache) *Stats
  • func (s *Stats) JSON(ctx context.Context, ds *dataset.Dataset) (r io.Reader, err error)
• type Statser

Variables

var (
	// ErrCacheMiss indicates a requested path isn't in the cache
	ErrCacheMiss = fmt.Errorf("stats: cache miss")
	// ErrNoCache indicates there is no cache
	ErrNoCache = fmt.Errorf("stats: no cache")
)

var (
	// StopFreqCountThreshold is the number of unique values past which we will
	// stop keeping frequencies. This is a simplistic line of defense against
	// unweildly memory consumption
	StopFreqCountThreshold = 10000
)

Functions

func ToMap

func ToMap(s Statser) []map[string]interface{}

ToMap converts stats to a Plain Old Data object

Types

type Accumulator

type Accumulator struct {
	// contains filtered or unexported fields
}

Accumulator wraps a dsio.EntryReader, on each call to read stats will update it's internal statistics Consumers can only assume the return value of Accumulator.Stats is final after a call to Close

func NewAccumulator

func NewAccumulator(r dsio.EntryReader) *Accumulator

NewAccumulator wraps an entry reader to create a stat accumulator

func (*Accumulator) Close

func (r *Accumulator) Close() error

Close finalizes the Reader

func (*Accumulator) ReadEntry

func (r *Accumulator) ReadEntry() (dsio.Entry, error)

ReadEntry reads one row of structured data from the reader

func (*Accumulator) Stats

func (r *Accumulator) Stats() []Stat

Stats gets the statistics created by the accumulator

func (*Accumulator) Structure

func (r *Accumulator) Structure() *dataset.Structure

Structure gives the structure being read

type Cache

type Cache interface {
	// Put places stats in the cache, keyed by path
	PutJSON(ctx context.Context, path string, r io.Reader) error
	// JSON gets cached byte data for a path
	JSON(ctx context.Context, path string) (r io.Reader, err error)
}

Cache is a store of JSON-formated stats data, keyed by path Consumers of a cache must not rely on the cache for persistence Implementations are expected to maintain their own size bounding semantics internally Cache implementations must be safe for concurrent use, and must be nil-callable

func NewOSCache

func NewOSCache(rootDir string, maxSize uint64) Cache

NewOSCache creates a cache in a local direcory

type Stat

type Stat interface {
	// Type returns a string identifier for the kind of statistic being reported
	Type() string
	// Map reports statistical details as a map, map must not return nil
	Map() map[string]interface{}
}

Stat describes common features of all statistical types

type Stats

type Stats struct {
	// contains filtered or unexported fields
}

Stats can generate an array of statistical info for a dataset

func New

func New(cache Cache) *Stats

New allocates a Stats service

func (*Stats) JSON

func (s *Stats) JSON(ctx context.Context, ds *dataset.Dataset) (r io.Reader, err error)

JSON gets stats data as reader of JSON-formatted bytes

type Statser

type Statser interface {
	Stats() []Stat
}

Statser produces a slice of Stat objects