Documentation
¶
Overview ¶
Package dataobj holds utilities for working with data objects.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrBufferFull = errors.New("buffer full")
ErrBufferFull is returned by Builder.Append when the buffer is full and needs to flush; call Builder.Flush to flush it.
Functions ¶
This section is empty.
Types ¶
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
A Builder builds data objects from a set of incoming log data. Log data is appended to a builder by calling Builder.Append. Buffered log data is flushed manually by calling Builder.Flush.
Methods on Builder are not goroutine-safe; callers are responsible for synchronizing calls.
func NewBuilder ¶
NewBuilder creates a new Builder which stores data objects for the specified tenant in a bucket.
NewBuilder returns an error if BuilderConfig is invalid.
func (*Builder) Append ¶
Append buffers a stream to be written to a data object. Append returns an error if the stream labels cannot be parsed or ErrBufferFull if the builder is full.
Once a Builder is full, call Builder.Flush to flush the buffered data, then call Append again with the same entry.
func (*Builder) Flush ¶
Flush flushes all buffered data to object storage. Calling Flush can result in a no-op if there is no buffered data to flush.
If Flush builds an object but fails to upload it to object storage, the built object is cached and can be retried. Builder.Reset can be called to discard any pending data and allow new data to be appended.
func (*Builder) RegisterMetrics ¶
func (b *Builder) RegisterMetrics(reg prometheus.Registerer) error
RegisterMetrics registers metrics about builder to report to reg. All metrics will have a tenant label set to the tenant ID of the Builder.
If multiple Builders for the same tenant are running in the same process, reg must contain additional labels to differentiate between them.
func (*Builder) Reset ¶
func (b *Builder) Reset()
Reset discards pending data and resets the builder to an empty state.
func (*Builder) UnregisterMetrics ¶
func (b *Builder) UnregisterMetrics(reg prometheus.Registerer)
UnregisterMetrics unregisters metrics about builder from reg.
type BuilderConfig ¶
type BuilderConfig struct {
// SHAPrefixSize sets the number of bytes of the SHA filename to use as a
// folder path.
SHAPrefixSize int `yaml:"sha_prefix_size"`
// TargetPageSize configures a target size for encoded pages within the data
// object. TargetPageSize accounts for encoding, but not for compression.
TargetPageSize flagext.Bytes `yaml:"target_page_size"`
// TargetObjectSize configures a target size for data objects.
TargetObjectSize flagext.Bytes `yaml:"target_object_size"`
// TargetSectionSize configures the maximum size of data in a section. Sections
// which support this parameter will place overflow data into new sections of
// the same type.
TargetSectionSize flagext.Bytes `yaml:"target_section_size"`
// BufferSize configures the size of the buffer used to accumulate
// uncompressed logs in memory prior to sorting.
BufferSize flagext.Bytes `yaml:"buffer_size"`
}
BuilderConfig configures a data object Builder.
func (*BuilderConfig) RegisterFlagsWithPrefix ¶
func (cfg *BuilderConfig) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet)
RegisterFlagsWithPrefix registers flags with the given prefix.
func (*BuilderConfig) Validate ¶
func (cfg *BuilderConfig) Validate() error
Validate validates the BuilderConfig.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
internal
|
|
|
dataset
Package dataset contains utilities for working with datasets.
|
Package dataset contains utilities for working with datasets. |
|
encoding
Package encoding provides utilities for encoding and decoding data objects.
|
Package encoding provides utilities for encoding and decoding data objects. |
|
result
Package result provides utilities for dealing with iterators that can fail during iteration.
|
Package result provides utilities for dealing with iterators that can fail during iteration. |
|
sections/logs
Package logs defines types used for the data object logs section.
|
Package logs defines types used for the data object logs section. |
|
sections/streams
Package streams defines types used for the data object streams section.
|
Package streams defines types used for the data object streams section. |
|
streamio
Package streamio defines interfaces shared by other packages for streaming binary data.
|
Package streamio defines interfaces shared by other packages for streaming binary data. |
|
util/bufpool
Package bufpool offers a pool of *bytes.Buffer objects that are placed into exponentially sized buckets.
|
Package bufpool offers a pool of *bytes.Buffer objects that are placed into exponentially sized buckets. |
|
util/sliceclear
Package sliceclear provides a way to clear and truncate the length of a slice.
|
Package sliceclear provides a way to clear and truncate the length of a slice. |