audit

README

Loki Index Auditing

Usage

To audit your index data:

1. Make sure you're authenticated to the cloud where your bucket lives in. In this example I'll be using GCP.
2. Create a new YAML configuration file that defines your storage configuration. lokitool will use it to communicate with your data. Only TSDB is supported. Make sure you give all three fields: schema_config, storage_config and tenant. In this example I'm naming my file configfile.yaml:

schema_config:
  configs:
    - from: "2023-08-21"
      index:
        period: 24h
        prefix: loki_env_tsdb_index_
      object_store: gcs
      schema: v13
      store: tsdb

storage_config:
  gcs:
    bucket_name: loki-bucket

tenant: 12345

1. Build a new lokitool binary:

go build ./cmd/lokitool

1. Finally, invoke the audit index command the following way:

./lokitool audit index --period=19856 --config.file=configfile.yaml --index.file=index/loki_env_tsdb_index_19856/12345/1715707992714992001-compactor-1715199977885-1815707796275-g8003361.tsdb.gz

The --period is the period of the index being audited. You can find it by checking the 5-digits number appended as a suffix of the Loki environment name in the index file. Example: For index/loki_env_tsdb_index_19856/12345/..., the period is 19856. The --config.file is the YAML configuration described in the first step. The --index.file is the path to the index file you want to audit. Take a look at your bucket to see its exactly path and substitute it accordingly.

Documentation

Index

• Constants
• func CheckChunkExistance(key string, objClient client.ObjectClient) (bool, error)
• func DownloadIndexFile(ctx context.Context, cfg Config, cloudIndexPath string, ...) (string, error)
• func GetObjectClient(cfg Config) (client.ObjectClient, error)
• func ParseCompactexIndex(ctx context.Context, localFilePath, table string, cfg Config) (compactor.CompactedIndex, error)
• func Run(ctx context.Context, cloudIndexPath, table string, cfg Config, ...) (int, int, error)
• func ValidateCompactedIndex(ctx context.Context, objClient client.ObjectClient, ...) (int, int, error)
• type Config
  • func (c *Config) Clone() flagext.Registerer
  • func (c *Config) RegisterFlags(f *flag.FlagSet)
  • func (c *Config) Validate() error
• type FileConfig
  • func (c *FileConfig) RegisterFlags(f *flag.FlagSet)

Constants

const (
	TsFormat = time.RFC3339Nano
)

Functions

func CheckChunkExistance

func CheckChunkExistance(key string, objClient client.ObjectClient) (bool, error)

func DownloadIndexFile

func DownloadIndexFile(ctx context.Context, cfg Config, cloudIndexPath string, objClient client.ObjectClient, logger log.Logger) (string, error)

func GetObjectClient

func GetObjectClient(cfg Config) (client.ObjectClient, error)

func ParseCompactexIndex

func ParseCompactexIndex(ctx context.Context, localFilePath, table string, cfg Config) (compactor.CompactedIndex, error)

func Run

func Run(ctx context.Context, cloudIndexPath, table string, cfg Config, logger log.Logger) (int, int, error)

func ValidateCompactedIndex

func ValidateCompactedIndex(ctx context.Context, objClient client.ObjectClient, compactedIdx compactor.CompactedIndex, parallelism int, logger log.Logger) (int, int, error)

Types

type Config

type Config struct {
	FileConfig    `yaml:",inline"`
	Tenant        string                   `yaml:"tenant,omitempty"`
	SchemaConfig  lokiStorage.SchemaConfig `yaml:"schema_config,omitempty"`
	StorageConfig storage.Config           `yaml:"storage_config,omitempty"`
	LogLevel      dskitlog.Level           `yaml:"log_level"`
	Concurrency   int                      `yaml:"concurrency"`
	WorkingDir    string                   `yaml:"working_dir"`
	Period        string                   `yaml:"period,omitempty"`
}

Config Loki related storage and schema configs

func (*Config) Clone

func (c *Config) Clone() flagext.Registerer

Clone takes advantage of pass-by-value semantics to return a distinct *Config. This is primarily used to parse a different flag set without mutating the original *Config.

func (*Config) RegisterFlags

func (c *Config) RegisterFlags(f *flag.FlagSet)

func (*Config) Validate

func (c *Config) Validate() error

type FileConfig

type FileConfig struct {
	ConfigFile string
}

func (*FileConfig) RegisterFlags

func (c *FileConfig) RegisterFlags(f *flag.FlagSet)