Documentation ¶
Index ¶
- func CreateUUIDBoundaries(nRanges uint32) ([]uuid.UUID, error)
- func MakeUUIDWithTopBits(topBits uint32) (uuid.UUID, error)
- type Config
- type MetabaseRangeSplitter
- type MetabaseSegmentProvider
- type Observer
- type ObserverDuration
- type Partial
- type RangeSplitter
- type SegmentProvider
- type Service
- type UUIDRange
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CreateUUIDBoundaries ¶
CreateUUIDBoundaries splits up the entire 128-bit UUID range into equal parts.
Types ¶
type Config ¶ added in v1.68.1
type Config struct { Parallelism int `help:"how many chunks of segments to process in parallel" default:"2"` BatchSize int `help:"how many items to query in a batch" default:"2500"` AsOfSystemInterval time.Duration `help:"as of system interval" releaseDefault:"-5m" devDefault:"-1us" testDefault:"-1us"` }
Config contains configurable values for the shared loop.
type MetabaseRangeSplitter ¶ added in v1.70.1
type MetabaseRangeSplitter struct {
// contains filtered or unexported fields
}
MetabaseRangeSplitter implements RangeSplitter.
func NewMetabaseRangeSplitter ¶ added in v1.70.1
func NewMetabaseRangeSplitter(db *metabase.DB, batchSize int) MetabaseRangeSplitter
NewMetabaseRangeSplitter creates the segment provider.
func (*MetabaseRangeSplitter) CreateRanges ¶ added in v1.70.1
func (provider *MetabaseRangeSplitter) CreateRanges(nRanges int, batchSize int) ([]SegmentProvider, error)
CreateRanges splits the segment table into chunks.
type MetabaseSegmentProvider ¶ added in v1.70.1
type MetabaseSegmentProvider struct {
// contains filtered or unexported fields
}
MetabaseSegmentProvider implements SegmentProvider.
func (*MetabaseSegmentProvider) Iterate ¶ added in v1.70.1
func (provider *MetabaseSegmentProvider) Iterate(ctx context.Context, fn func([]segmentloop.Segment) error) error
Iterate loops over a part of the segment table.
type Observer ¶ added in v1.68.1
type Observer interface { // Start is called at the beginning of each segment loop. Start(context.Context, time.Time) error // Fork creates a Partial to process a chunk of all the segments. It is // called after Start. It is not called concurrently. Fork(context.Context) (Partial, error) // Join is called for each partial returned by Fork. This gives the // opportunity to merge the output like in a reduce step. It will be called // before Finish. It is not called concurrently. Join(context.Context, Partial) error // Finish is called after all segments are processed by all observers. Finish(context.Context) error }
Observer subscribes to the parallel segment loop. It is intended that a naïve implementation is threadsafe.
type ObserverDuration ¶ added in v1.70.1
ObserverDuration reports back on how long it took the observer to process all the segments.
type Partial ¶ added in v1.68.1
type Partial interface { // Process is called repeatedly with batches of segments. // It is not called concurrently on the same instance. Process(context.Context, []segmentloop.Segment) error }
Partial processes a part of the total range of segments.
type RangeSplitter ¶ added in v1.70.1
type RangeSplitter interface {
CreateRanges(nRanges int, batchSize int) ([]SegmentProvider, error)
}
RangeSplitter splits a source of segments into ranges, so that multiple segments can be processed concurrently. It usually abstracts over a database. It is a subcomponent of the ranged segment loop.
type SegmentProvider ¶ added in v1.70.1
type SegmentProvider interface {
Iterate(ctx context.Context, fn func([]segmentloop.Segment) error) error
}
SegmentProvider iterates through a range of segments.
type Service ¶ added in v1.68.1
type Service struct {
// contains filtered or unexported fields
}
Service iterates through all segments and calls the attached observers for every segment
architecture: Service
func NewService ¶ added in v1.68.1
func NewService(log *zap.Logger, config Config, provider RangeSplitter, observers []Observer) *Service
NewService creates a new instance of the ranged loop service.
type UUIDRange ¶
UUIDRange describes a range of UUID values. Start and End can be open-ended.
func CreateUUIDRanges ¶
CreateUUIDRanges splits up the entire 128-bit UUID range into equal parts.