Documentation
¶
Index ¶
- Constants
- Variables
- func GetFinishedBlockDates(blocksDirectory string) (map[time.Time]bool, error)
- func LabelsFromUntaggedName(name string, builder *labels.Builder) labels.Labels
- func PathsForWorker(paths []string, workerCount, workerID int) []string
- type ErrSeek
- type MimirSeries
- type Progress
- type ProtoConstructor
- type ProtoUnmarshaler
- type USTable
- func NewUSTableForAppend(fname string, overwrite bool, constructor ProtoConstructor, logger log.Logger) (*USTable, error)
- func NewUSTableForAppendWithIndex(fname string, overwrite bool, constructor ProtoConstructor, logger log.Logger) (*USTable, map[string]int64, error)
- func NewUSTableForRead(fname string, constructor ProtoConstructor, logger log.Logger) (*USTable, error)
- func (t *USTable) Append(key string, value proto.Marshaler) error
- func (t *USTable) Close() error
- func (t *USTable) Index() (map[string]int64, error)
- func (t *USTable) Next() (key string, value ProtoUnmarshaler, err error)
- func (t *USTable) ReadAt(seekPos int64) (key string, value ProtoUnmarshaler, err error)
- func (t *USTable) SeekLastValid() (err error)
Constants ¶
const ( TaggedMetricName = "graphite_tagged" UntaggedMetricName = "graphite_untagged" )
const ( READ = iota APPEND )
Variables ¶
var ( ErrInvalidForMode = errors.New("the operation is not allowed under the mode") ErrBadSentinal = errors.New("did not find end-of-file sentinel") ErrAtSentinel = errors.New("file is complete, reads have been exhausted. File must only be Appended to") ErrBadData = errors.New("invalid data in file") )
Functions ¶
func GetFinishedBlockDates ¶
GetFinishedBlockDates walks the blocks directory and builds a list of dates that have already been processed.
func LabelsFromUntaggedName ¶
func PathsForWorker ¶
PathsForWorker returns a subset of the paths slice for the worker with the given index. If there are more workers than there are paths, and the worker ID is greater than the number of paths, the returned slice will be empty. If the worker ID is greater than the workerCount, returns empty.
Types ¶
type MimirSeries ¶
type MimirSeries struct {
// contains filtered or unexported fields
}
MimirSeries satisfies the storage.Series interface for writing out to Blocks.
func NewMimirSeries ¶
func NewMimirSeries(l labels.Labels, samples []mimirpb.Sample) *MimirSeries
func (*MimirSeries) Iterator ¶
func (s *MimirSeries) Iterator(_ chunkenc.Iterator) chunkenc.Iterator
func (*MimirSeries) Labels ¶
func (s *MimirSeries) Labels() labels.Labels
type Progress ¶
type Progress struct {
// contains filtered or unexported fields
}
Progress is a concurrent-safe type for tracking the progress of the file conversion. It can keep track of the number of processed and skipped records. Every 100 processed files it emits an info log line.
func NewProgress ¶
func (*Progress) GetProcessedCount ¶
GetProcessedCount atomically loads and returns the current processed count.
func (*Progress) GetSkippedCount ¶
GetSkippedCount atomically loads and returns the current skipped count.
func (*Progress) IncProcessed ¶
func (p *Progress) IncProcessed()
IncProcessed atomically increases the processed count and prints a message if processing is complete.
func (*Progress) IncSkipped ¶
func (p *Progress) IncSkipped()
IncSkipped atomically increases the skipped file count.
type ProtoConstructor ¶
type ProtoConstructor func() ProtoUnmarshaler
ProtoConstructor is a function which returns a new proto for this table's values. The return value from implementations should ideally be a pointer to a proto.
type ProtoUnmarshaler ¶
ProtoUnmarshaler is similar to proto.Marshaler, except for the opposite direction.
func NewMimirSeriesProto ¶
func NewMimirSeriesProto() ProtoUnmarshaler
NewMimirSeriesProto creates a new proto for use in unmarshalling data from USTable files.
type USTable ¶
type USTable struct {
// contains filtered or unexported fields
}
USTable is an Unsorted String-keyed Table file containing unsorted proto items. An USTable file can either be opened for reading or appending.
The file layout is:
repeated: int64 key length OR metadata code if <= 0 key in bytes int64 proto length marshaled proto
int64 0 value SENTINEL string indicating EOF
Files without a SENTINEL are partial and would indicate incomplete processing. Futureproofing: negative values in the key length field can be used as an enum to indicate other metadata, such as error correction or an index.
func NewUSTableForAppend ¶
func NewUSTableForAppend(fname string, overwrite bool, constructor ProtoConstructor, logger log.Logger) (*USTable, error)
NewUSTableForAppend creates a new USTable file of the given name, creating one if it doesn't exist, and opening it if it does. If overwrite is true, the existing contents will be overwritten. If the file doesn't exist, it will be created regardless. If the file already existed, the write head will seek such as it will preserve existing valid contents.
func NewUSTableForAppendWithIndex ¶
func NewUSTableForAppendWithIndex(fname string, overwrite bool, constructor ProtoConstructor, logger log.Logger) (*USTable, map[string]int64, error)
NewUSTableForAppendWithIndex opens the file for appending, and also does a pre-read of the protos that are already written to the file.
func NewUSTableForRead ¶
func NewUSTableForRead(fname string, constructor ProtoConstructor, logger log.Logger) (*USTable, error)
NewUSTableForRead opens an USTable file for reading. Attempts to append will result in bad file descriptor errors.
func (*USTable) Append ¶
Append writes a new proto to the USTable file. This function assumes the file descriptor is at the end of the file. This function should not be used in conjunction with Next. Append() may be called concurrently.
func (*USTable) Index ¶
Index returns a map of key to position in the file. Performing a ReadAt for the given positions will read the value of the associated key. When done, the read head will be at the next valid append point.
func (*USTable) Next ¶
func (t *USTable) Next() (key string, value ProtoUnmarshaler, err error)
Next calls ReadAt for the current position.
func (*USTable) ReadAt ¶
func (t *USTable) ReadAt(seekPos int64) (key string, value ProtoUnmarshaler, err error)
ReadAt reads the record at the given file position and returns it. If the seek position is -1, uses the current position. In case of error, seeks back to the previous position before the call and no other calls to Next should be attempted (they will just fail again). If the file is complete, the last call to Next() will return nil and ErrAtSentinel. ReadAt() may be called concurrently, but if using seekPos -1 this may result in unpredictable behavior.
func (*USTable) SeekLastValid ¶
SeekLastValid seeks to the end of the valid portion of the file, before any existing sentinel. Can be called in either Read or Append modes. Unmarshals values to confirm validity. Only returns error if there is an unrecoverable issue finding a valid position in the file.
Source Files
Directories