whisperconverter

Documentation

Index

• func CreateWhisperFile(path string, timestamps []*time.Time) error
• func ListFilesInDir(dirPath string) ([]string, error)
• func ReadPoints(w Archive, name string) ([]whisper.Point, error)
• func SplitSamplesByDays(samples []mimirpb.Sample) [][]mimirpb.Sample
• func ToMimirSamples(points []whisper.Point) ([]mimirpb.Sample, error)
• func ToTime(in string) (time.Time, error)
• func ToTimes(in []string) ([]*time.Time, error)
• func WhisperToMimirSamples(whisperFile, name string) ([]mimirpb.Sample, error)
• type Archive
• type WhisperConverter
  • func NewWhisperConverter(namePrefix, whisperDirectory string, fileFilter *regexp.Regexp, ...) *WhisperConverter
  • func (c *WhisperConverter) CommandDateRange(targetWhisperFiles string)
  • func (c *WhisperConverter) CommandFileList(targetWhisperFiles string) error
  • func (c *WhisperConverter) CommandPass1(targetWhisperFiles, intermediateDir string, resumeIntermediate bool) error
  • func (c *WhisperConverter) CommandPass2(intermediateDir, blocksDir string, overwriteBlocks bool) error
  • func (c *WhisperConverter) GetProcessedCount() uint64
  • func (c *WhisperConverter) GetSkippedCount() uint64

Functions

func CreateWhisperFile

func CreateWhisperFile(path string, timestamps []*time.Time) error

func ListFilesInDir

func ListFilesInDir(dirPath string) ([]string, error)

func ReadPoints

func ReadPoints(w Archive, name string) ([]whisper.Point, error)

ReadPoints reads and concatenates all of the points in a whisper Archive.

func SplitSamplesByDays

func SplitSamplesByDays(samples []mimirpb.Sample) [][]mimirpb.Sample

SplitSamplesByDays separates a slice of samples over the midnight UTC boundary.

func ToMimirSamples

func ToMimirSamples(points []whisper.Point) ([]mimirpb.Sample, error)

ToMimirSamples converts a Whisper metric with the given name to a slice of labels and series of mimir samples. Returns error if no points.

func ToTime

func ToTime(in string) (time.Time, error)

func ToTimes

func ToTimes(in []string) ([]*time.Time, error)

func WhisperToMimirSamples

func WhisperToMimirSamples(whisperFile, name string) ([]mimirpb.Sample, error)

WhisperToMimirSamples opens the given whisper file, applying the given metric name, and writes it to the given block directory with blocks covering the given duration.

Types

type Archive

type Archive interface {
	GetArchives() []whisper.ArchiveInfo
	DumpArchive(int) ([]whisper.Point, error)
}

Archive provides a testable interface for converting whisper databases.

type WhisperConverter

type WhisperConverter struct {
	// contains filtered or unexported fields
}

WhisperConverter is an object for performing various steps of the conversion from Whisper files to Mimir blocks.

func NewWhisperConverter

func NewWhisperConverter(
	namePrefix,
	whisperDirectory string,
	fileFilter *regexp.Regexp,
	threads,
	workerCount,
	workerID int,
	customLabels labels.Labels,
	dates []time.Time,
	logger log.Logger) *WhisperConverter

func (*WhisperConverter) CommandDateRange

func (c *WhisperConverter) CommandDateRange(targetWhisperFiles string)

CommandDateRange prints the minimum and maximum timestamps for the dataset in a format suitable for use as arguments to pass1 and pass2. targetWhisperFiles is a filename containing the list of files to process, or if blank, files will be walked using c.whisperDirectory.

func (*WhisperConverter) CommandFileList

func (c *WhisperConverter) CommandFileList(targetWhisperFiles string) error

CommandFileList traverses the file structure starting from whisperDirectory, discovers files that match fileFilter, and outputs those files to the destination specified by targetWhisperFiles.

func (*WhisperConverter) CommandPass1

func (c *WhisperConverter) CommandPass1(targetWhisperFiles, intermediateDir string, resumeIntermediate bool) error

CommandPass1 performs the first pass conversion to intermediate files. Each metric is dumped, sorted, and split into days, and the individual days are written to the intermediate files. If this stage crashes, rerunning the stage will automatically resume. targetWhisperFiles is a filename containing the list of files to process, or if blank, files will be walked using c.whisperDirectory.

func (*WhisperConverter) CommandPass2

func (c *WhisperConverter) CommandPass2(intermediateDir, blocksDir string, overwriteBlocks bool) error

CommandPass2 performs the second pass conversion to Mimir blocks. It reads each intermediate file, sorts the metrics by labels, and outputs the block.

func (*WhisperConverter) GetProcessedCount

func (c *WhisperConverter) GetProcessedCount() uint64

func (*WhisperConverter) GetSkippedCount

func (c *WhisperConverter) GetSkippedCount() uint64