indexshipper

Documentation

Index

• Constants
• func NewTableClient(objectClient client.ObjectClient, storageKeyPrefix string) index.TableClient
• type Config
  • func (cfg *Config) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet)
  • func (cfg *Config) Validate() error
• type Index
• type IndexShipper
  • func NewIndexShipper(cfg Config, storageClient client.ObjectClient, limits downloads.Limits, ...) (IndexShipper, error)
• type Mode

Constants

const (
	// ModeReadWrite is to allow both read and write
	ModeReadWrite = Mode("RW")
	// ModeReadOnly is to allow only read operations
	ModeReadOnly = Mode("RO")
	// ModeWriteOnly is to allow only write operations
	ModeWriteOnly = Mode("WO")

	// FilesystemObjectStoreType holds the periodic config type for the filesystem store
	FilesystemObjectStoreType = "filesystem"

	// UploadInterval defines interval for when we check if there are new index files to upload.
	// It's also used to snapshot the currently written index tables so the snapshots can be used for reads.
	UploadInterval = 1 * time.Minute
)

Functions

func NewTableClient

func NewTableClient(objectClient client.ObjectClient, storageKeyPrefix string) index.TableClient

NewTableClient creates a client for managing tables in object storage based index store. It is typically used when running a table manager.

Types

type Config

type Config struct {
	ActiveIndexDirectory     string                                 `yaml:"active_index_directory"`
	SharedStoreType          string                                 `yaml:"shared_store"`
	SharedStoreKeyPrefix     string                                 `yaml:"shared_store_key_prefix"`
	CacheLocation            string                                 `yaml:"cache_location"`
	CacheTTL                 time.Duration                          `yaml:"cache_ttl"`
	ResyncInterval           time.Duration                          `yaml:"resync_interval"`
	QueryReadyNumDays        int                                    `yaml:"query_ready_num_days"`
	IndexGatewayClientConfig gatewayclient.IndexGatewayClientConfig `yaml:"index_gateway_client"`

	IngesterName           string
	Mode                   Mode
	IngesterDBRetainPeriod time.Duration
}

func (*Config) RegisterFlagsWithPrefix

func (cfg *Config) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet)

RegisterFlagsWithPrefix registers flags.

func (*Config) Validate

func (cfg *Config) Validate() error

type Index

type Index interface {
	Close() error
}

type IndexShipper

type IndexShipper interface {
	// AddIndex adds an immutable index to a logical table which would eventually get uploaded to the object store.
	AddIndex(tableName, userID string, index index.Index) error
	// ForEach lets us iterates through each index file in a table for a specific user.
	// On the write path, it would iterate on the files given to the shipper for uploading, until they eventually get dropped from local disk.
	// On the read path, it would iterate through the files if already downloaded else it would download and iterate through them.
	ForEach(ctx context.Context, tableName, userID string, callback index.ForEachIndexCallback) error
	Stop()
}

func NewIndexShipper

func NewIndexShipper(cfg Config, storageClient client.ObjectClient, limits downloads.Limits,
	ownsTenantFn downloads.IndexGatewayOwnsTenant, open index.OpenIndexFileFunc, tableRangesToHandle config.TableRanges, reg prometheus.Registerer) (IndexShipper, error)

NewIndexShipper creates a shipper for providing index store functionality using index files and object storage. It manages the whole life cycle of uploading the index and downloading the index at query time.

Since IndexShipper is generic, which means it can be used to manage various index types under the same object storage and/or local disk path, it accepts ranges of table numbers(config.TableRanges) to be managed by the shipper. This is mostly useful on the read path to sync and manage specific index tables within the given table number ranges.

type Mode

type Mode string