globalstats

Documentation

Index

• Constants
• func MergeGlobalStatsTopNByConcurrency(gp *gp.Pool, mergeConcurrency, mergeBatchSize int, wrapper *StatsWrapper, ...) (*statistics.TopN, []statistics.TopNMeta, []*statistics.Histogram, error)
• func MergePartTopN2GlobalTopN(loc *time.Location, version int, topNs []*statistics.TopN, n uint32, ...) (*statistics.TopN, []statistics.TopNMeta, []*statistics.Histogram, error)
• func NewStatsGlobal(statsHandler statstypes.StatsHandle) statstypes.StatsGlobal
• func NewTopnStatsMergeWorker(taskCh <-chan *TopnStatsMergeTask, respCh chan<- *TopnStatsMergeResponse, ...) *topnStatsMergeWorker
• func WriteGlobalStatsToStorage(statsHandle statstypes.StatsHandle, globalStats *GlobalStats, ...) (err error)
• type AsyncMergePartitionStats2GlobalStats
  • func NewAsyncMergePartitionStats2GlobalStats(statsHandle statstypes.StatsHandle, globalTableInfo *model.TableInfo, ...) (*AsyncMergePartitionStats2GlobalStats, error)
  • func (a *AsyncMergePartitionStats2GlobalStats) MergePartitionStats2GlobalStats(sctx sessionctx.Context, opts map[ast.AnalyzeOptionType]uint64, isIndex bool) error
  • func (a *AsyncMergePartitionStats2GlobalStats) Result() *GlobalStats
• type GlobalStats
  • func MergePartitionStats2GlobalStats(sc sessionctx.Context, statsHandle statstypes.StatsHandle, ...) (globalStats *GlobalStats, err error)
  • func MergePartitionStats2GlobalStatsByTableID(sc sessionctx.Context, statsHandle statstypes.StatsHandle, ...) (globalStats *GlobalStats, err error)
• type StatsWrapper
  • func NewStatsWrapper(hg []*statistics.Histogram, topN []*statistics.TopN) *StatsWrapper
• type TopnStatsMergeResponse
• type TopnStatsMergeTask
  • func NewTopnStatsMergeTask(start, end int) *TopnStatsMergeTask

Constants

const (
	// MaxPartitionMergeBatchSize indicates the max batch size for a worker to merge partition stats
	MaxPartitionMergeBatchSize = 256
)

Functions

func MergeGlobalStatsTopNByConcurrency

func MergeGlobalStatsTopNByConcurrency(
	gp *gp.Pool,
	mergeConcurrency, mergeBatchSize int,
	wrapper *StatsWrapper,
	timeZone *time.Location,
	version int,
	n uint32,
	isIndex bool,
	killer *sqlkiller.SQLKiller,
) (*statistics.TopN,
	[]statistics.TopNMeta, []*statistics.Histogram, error)

MergeGlobalStatsTopNByConcurrency merge partition topN by concurrency. To merge global stats topN by concurrency, we will separate the partition topN in concurrency part and deal it with different worker. mergeConcurrency is used to control the total concurrency of the running worker, and mergeBatchSize is sued to control the partition size for each worker to solve it

func MergePartTopN2GlobalTopN

func MergePartTopN2GlobalTopN(
	loc *time.Location,
	version int,
	topNs []*statistics.TopN,
	n uint32,
	hists []*statistics.Histogram,
	isIndex bool,
	killer *sqlkiller.SQLKiller,
) (*statistics.TopN, []statistics.TopNMeta, []*statistics.Histogram, error)

MergePartTopN2GlobalTopN is used to merge the partition-level topN to global-level topN. The input parameters:

1. `topNs` are the partition-level topNs to be merged.
2. `n` is the size of the global-level topN. Notice: This value can be 0 and has no default value, we must explicitly specify this value.
3. `hists` are the partition-level histograms. Some values not in topN may be placed in the histogram. We need it here to make the value in the global-level TopN more accurate.

The output parameters:

1. `*TopN` is the final global-level topN.
2. `[]TopNMeta` is the left topN value from the partition-level TopNs, but is not placed to global-level TopN. We should put them back to histogram latter.
3. `[]*Histogram` are the partition-level histograms which just delete some values when we merge the global-level topN.

func NewStatsGlobal

func NewStatsGlobal(statsHandler statstypes.StatsHandle) statstypes.StatsGlobal

NewStatsGlobal creates a new StatsGlobal.

func NewTopnStatsMergeWorker

func NewTopnStatsMergeWorker(
	taskCh <-chan *TopnStatsMergeTask,
	respCh chan<- *TopnStatsMergeResponse,
	wrapper *StatsWrapper,
	killer *sqlkiller.SQLKiller) *topnStatsMergeWorker

NewTopnStatsMergeWorker returns topn merge worker

func WriteGlobalStatsToStorage

func WriteGlobalStatsToStorage(statsHandle statstypes.StatsHandle, globalStats *GlobalStats, info *statstypes.GlobalStatsInfo, gid int64) (err error)

WriteGlobalStatsToStorage is to write global stats to storage

Types

type AsyncMergePartitionStats2GlobalStats

type AsyncMergePartitionStats2GlobalStats struct {
	PartitionDefinition map[int64]model.PartitionDefinition
	// contains filtered or unexported fields
}

AsyncMergePartitionStats2GlobalStats is used to merge partition stats to global stats. it divides the merge task into two parts. - IOWorker: load stats from storage. it will load fmsketch, cmsketch, histogram and topn. and send them to cpuWorker. - CPUWorker: merge the stats from IOWorker and generate global stats.

┌────────────────────────┐ ┌───────────────────────┐ │ │ │ │ │ │ │ │ │ │ │ │ │ IOWorker │ │ CPUWorker │ │ │ ────► │ │ │ │ │ │ │ │ │ │ │ │ │ │ └────────────────────────┘ └───────────────────────┘

func NewAsyncMergePartitionStats2GlobalStats

func NewAsyncMergePartitionStats2GlobalStats(
	statsHandle statstypes.StatsHandle,
	globalTableInfo *model.TableInfo,
	histIDs []int64,
	is infoschema.InfoSchema) (*AsyncMergePartitionStats2GlobalStats, error)

NewAsyncMergePartitionStats2GlobalStats creates a new AsyncMergePartitionStats2GlobalStats.

func (*AsyncMergePartitionStats2GlobalStats) MergePartitionStats2GlobalStats

func (a *AsyncMergePartitionStats2GlobalStats) MergePartitionStats2GlobalStats(
	sctx sessionctx.Context,
	opts map[ast.AnalyzeOptionType]uint64,
	isIndex bool,
) error

MergePartitionStats2GlobalStats merges partition stats to global stats.

func (*AsyncMergePartitionStats2GlobalStats) Result

func (a *AsyncMergePartitionStats2GlobalStats) Result() *GlobalStats

Result returns the global stats.

type GlobalStats

type GlobalStats struct {
	Hg                    []*statistics.Histogram
	Cms                   []*statistics.CMSketch
	TopN                  []*statistics.TopN
	Fms                   []*statistics.FMSketch
	MissingPartitionStats []string
	Num                   int
	Count                 int64
	ModifyCount           int64
}

GlobalStats is used to store the statistics contained in the global-level stats which is generated by the merge of partition-level stats. It will both store the column stats and index stats. In the column statistics, the variable `num` is equal to the number of columns in the partition table. In the index statistics, the variable `num` is always equal to one.

func MergePartitionStats2GlobalStats

func MergePartitionStats2GlobalStats(
	sc sessionctx.Context,
	statsHandle statstypes.StatsHandle,
	opts map[ast.AnalyzeOptionType]uint64,
	is infoschema.InfoSchema,
	globalTableInfo *model.TableInfo,
	isIndex bool,
	histIDs []int64,
) (globalStats *GlobalStats, err error)

MergePartitionStats2GlobalStats merge the partition-level stats to global-level stats based on the tableInfo.

func MergePartitionStats2GlobalStatsByTableID

func MergePartitionStats2GlobalStatsByTableID(
	sc sessionctx.Context,
	statsHandle statstypes.StatsHandle,
	opts map[ast.AnalyzeOptionType]uint64,
	is infoschema.InfoSchema,
	tableID int64,
	isIndex bool,
	histIDs []int64,
) (globalStats *GlobalStats, err error)

MergePartitionStats2GlobalStatsByTableID merge the partition-level stats to global-level stats based on the tableID.

type StatsWrapper

type StatsWrapper struct {
	AllHg   []*statistics.Histogram
	AllTopN []*statistics.TopN
}

StatsWrapper wrapper stats

func NewStatsWrapper

func NewStatsWrapper(hg []*statistics.Histogram, topN []*statistics.TopN) *StatsWrapper

NewStatsWrapper returns wrapper

type TopnStatsMergeResponse

type TopnStatsMergeResponse struct {
	Err error
}

TopnStatsMergeResponse indicates topn merge worker response

type TopnStatsMergeTask

type TopnStatsMergeTask struct {
	// contains filtered or unexported fields
}

TopnStatsMergeTask indicates a task for merge topn stats

func NewTopnStatsMergeTask

func NewTopnStatsMergeTask(start, end int) *TopnStatsMergeTask

NewTopnStatsMergeTask returns task