housekeeping

package
v16.11.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 25, 2024 License: MIT Imports: 23 Imported by: 0

Documentation

Index

Constants

View Source 
const (

	// ReferenceLockfileGracePeriod is the grace period when cleaning up lock files of individual references in the
	// repository. Lock files existing less than this period are ignored.
	ReferenceLockfileGracePeriod = 1 * time.Hour
	// ReftableLockfileGracePeriod is the grace period when cleaning up lock files reftable.
	ReftableLockfileGracePeriod = 1 * time.Hour
)
View Source 
const (
	// WorktreesPrefix is a subdirectory in the repository where Gitaly creates worktrees.
	WorktreesPrefix = "worktrees"
	// GitlabWorktreePrefix is a subdirectory in the repository where Gitaly creates worktrees.
	GitlabWorktreePrefix = "gitlab-worktree"
)
View Source 
const (
	// FullRepackCooldownPeriod is the cooldown period that needs to pass since the last full
	// repack before we consider doing another full repack.
	FullRepackCooldownPeriod = 5 * 24 * time.Hour
)
View Source 
const (
	// LooseObjectLimit is the limit of loose objects we accept both when doing incremental
	// repacks and when pruning objects.
	LooseObjectLimit = 1024
)

Variables

This section is empty.

Functions

func CleanupWorktrees 

func CleanupWorktrees(ctx context.Context, repo *localrepo.Repo) error

CleanupWorktrees cleans up stale and disconnected worktrees for the given repository.

func FindBrokenLooseReferences added in v16.10.0 

func FindBrokenLooseReferences(ctx context.Context, repoPath string) ([]string, error)

FindBrokenLooseReferences return the list of broken refs. A ref is considered to be broken when the loose ref file is empty. We don't validate the existence of referred objects.

func FindPackedRefsLock added in v16.10.0 

func FindPackedRefsLock(ctx context.Context, repoPath string) ([]string, error)

FindPackedRefsLock returns stale lockfiles for the packed-refs file.

func FindPackedRefsNew added in v16.10.0 

func FindPackedRefsNew(ctx context.Context, repoPath string) ([]string, error)

FindPackedRefsNew returns stale temporary packed-refs files.

func FindServerInfo added in v16.10.0 

func FindServerInfo(ctx context.Context, repoPath string) ([]string, error)

FindServerInfo returns files generated by git-update-server-info(1). These files are only required to serve Git fetches via the dumb HTTP protocol, which we don't serve at all. It's thus safe to remove all of those files without a grace period.

func FindStaleLockfiles added in v16.10.0 

func FindStaleLockfiles(ctx context.Context, repoPath string) ([]string, error)

FindStaleLockfiles finds a subset of lockfiles which may be created by git commands. We're quite conservative with what we're removing, we certainly don't just scan the repo for `*.lock` files. Instead, we only remove a known set of lockfiles which have caused problems in the past.

func FindStalePackFileLocks added in v16.10.0 

func FindStalePackFileLocks(ctx context.Context, repoPath string) ([]string, error)

FindStalePackFileLocks finds packfile locks (`.keep` files) that git-receive-pack(1) and git-fetch-pack(1) write in order to not have the newly written packfile be deleted while refs have not yet been updated to point to their objects. Normally, these locks would get removed by both Git commands once the refs have been updated. But when the command gets killed meanwhile, then it can happen that the locks are left behind. As Git will never delete packfiles with an associated `.keep` file, the end result is that we may accumulate more and more of these locked packfiles over time.

func FindStaleReftableLock added in v16.10.0 

func FindStaleReftableLock(_ context.Context, repoPath string) ([]string, error)

FindStaleReftableLock provides a function which scans the reftable folder for stale a reftable lock and deletes it.

func FindTemporaryObjects added in v16.10.0 

func FindTemporaryObjects(ctx context.Context, repoPath string) ([]string, error)

FindTemporaryObjects find temporary objects of a repository. Those objects are created by some git processes and are sure to be removed when the processes are done.

func GetRepackGitConfig 

func GetRepackGitConfig(ctx context.Context, repo storage.Repository, bitmap bool) []git.ConfigPair

GetRepackGitConfig returns configuration suitable for Git commands which write new packfiles.

func PerformFullRepackingWithUnreachable added in v16.9.0 

func PerformFullRepackingWithUnreachable(ctx context.Context, repo *localrepo.Repo, cfg config.RepackObjectsConfig) error

PerformFullRepackingWithUnreachable performs a full repacking task using git-repack(1) command. This will omit packing objects part of alternates.

func PerformGeometricRepacking added in v16.9.0 

func PerformGeometricRepacking(ctx context.Context, repo *localrepo.Repo, cfg config.RepackObjectsConfig) error

PerformGeometricRepacking performs geometric repacking task using git-repack(1) command. It allows us to merge multiple packfiles without having to rewrite all packfiles into one. This new "geometric" strategy tries to ensure that existing packfiles in the repository form a geometric sequence where each successive packfile contains at least n times as many objects as the preceding packfile. If the sequence isn't maintained, Git will determine a slice of packfiles that it must repack to maintain the sequence again. With this process, we can limit the number of packfiles that exist in the repository without having to repack all objects into a single packfile regularly. This repacking does not take reachability into account. For more information, https://about.gitlab.com/blog/2023/11/02/rearchitecting-git-object-database-mainentance-for-scale/#geometric-repacking

func PerformRepack added in v16.9.0 

func PerformRepack(ctx context.Context, repo *localrepo.Repo, cfg config.RepackObjectsConfig, opts ...git.Option) error

PerformRepack performs `git-repack(1)` command on a repository with some pre-built configs.

func PruneEmptyConfigSections added in v16.10.0 

func PruneEmptyConfigSections(ctx context.Context, repo *localrepo.Repo) (_ int, returnedErr error)

PruneEmptyConfigSections prunes all empty sections from the repo's config.

func PruneObjects 

func PruneObjects(ctx context.Context, repo *localrepo.Repo, cfg PruneObjectsConfig) error

PruneObjects prunes loose objects from the repository that are already packed or which are unreachable and older than the configured expiry date.

func RemoveRefEmptyDirs added in v16.10.0 

func RemoveRefEmptyDirs(ctx context.Context, repository *localrepo.Repo) (int, error)

RemoveRefEmptyDirs removes empty directories inside refs directory. They are left-over of ref deletion operations.

func RepackObjects 

func RepackObjects(ctx context.Context, repo *localrepo.Repo, cfg config.RepackObjectsConfig) error

RepackObjects repacks objects in the given repository and updates the commit-graph. The way objects are repacked is determined via the config.RepackObjectsConfig.

func ValidateRepacking added in v16.9.0 

func ValidateRepacking(cfg config.RepackObjectsConfig) (bool, error)

ValidateRepacking validates the input repacking config. This function any validating error and if the configuration is for full repack.

func WriteCommitGraph 

func WriteCommitGraph(ctx context.Context, repo *localrepo.Repo, cfg config.WriteCommitGraphConfig) error

WriteCommitGraph updates the commit-graph in the given repository. The commit-graph is updated incrementally, except in the case where it doesn't exist yet or in case it is detected that the commit-graph is missing bloom filters.

func WriteCommitGraphConfigForRepository 

func WriteCommitGraphConfigForRepository(ctx context.Context, repo *localrepo.Repo) (config.WriteCommitGraphConfig, error)

WriteCommitGraphConfigForRepository returns the optimal default-configuration for the given repo. By default, the configuration will ask for an incremental commit-graph update. If the preexisting commit-graph is missing bloom filters though then the whole commit-graph chain will be rewritten.

Types

type CleanStaleDataConfig 

type CleanStaleDataConfig struct {
	// StaleFileFinders contains the list of finder functions to find stale files in the repository.
	StaleFileFinders map[string]FindStaleFileFunc
	// RepoCleanups contains the list of clean-up functions for the repository.
	RepoCleanups map[string]CleanupRepoFunc
	// RepoCleanupWithTxManagers contains the list of clean-up functions with transaction manager support.
	RepoCleanupWithTxManagers map[string]cleanupRepoWithTxManagerFunc
}

CleanStaleDataConfig is the configuration for running CleanStaleData. It is used to define the different types of cleanups we want to run.

func DefaultStaleDataCleanup 

func DefaultStaleDataCleanup() CleanStaleDataConfig

DefaultStaleDataCleanup is the default configuration for CleanStaleData which contains all the cleanup functions.

func OnlyStaleReferenceLockCleanup 

func OnlyStaleReferenceLockCleanup(gracePeriod time.Duration) CleanStaleDataConfig

OnlyStaleReferenceLockCleanup returns a config which only contains a stale reference lock cleaner with the provided grace period.

type CleanupRepoFunc added in v16.10.0 

type CleanupRepoFunc func(context.Context, *localrepo.Repo) (int, error)

CleanupRepoFunc is a type of function that performs a clean up task in the repository.

type EagerOptimizationStrategy 

type EagerOptimizationStrategy struct {
	// contains filtered or unexported fields
}

EagerOptimizationStrategy is a strategy that will eagerly perform optimizations. All of the data structures will be optimized regardless of whether they already are in an optimal state or not.

func NewEagerOptimizationStrategy 

func NewEagerOptimizationStrategy(info stats.RepositoryInfo) EagerOptimizationStrategy

NewEagerOptimizationStrategy creates a new EagerOptimizationStrategy.

func (EagerOptimizationStrategy) ShouldPruneObjects 

func (s EagerOptimizationStrategy) ShouldPruneObjects(context.Context) (bool, PruneObjectsConfig)

ShouldPruneObjects always instructs the caller to prune objects, unless the repository is an object pool.

func (EagerOptimizationStrategy) ShouldRepackObjects 

func (s EagerOptimizationStrategy) ShouldRepackObjects(ctx context.Context) (bool, config.RepackObjectsConfig)

ShouldRepackObjects always instructs the caller to repack objects. The strategy will always be to repack all objects into a single packfile. The bitmap will be written in case the repository does not have any alternates.

func (EagerOptimizationStrategy) ShouldRepackReferences 

func (s EagerOptimizationStrategy) ShouldRepackReferences(context.Context) bool

ShouldRepackReferences always instructs the caller to repack references.

func (EagerOptimizationStrategy) ShouldWriteCommitGraph 

func (s EagerOptimizationStrategy) ShouldWriteCommitGraph(context.Context) (bool, config.WriteCommitGraphConfig, error)

ShouldWriteCommitGraph always instructs the caller to write the commit-graph. The strategy will always be to completely rewrite the commit-graph chain.

type FindStaleFileFunc added in v16.10.0 

type FindStaleFileFunc func(context.Context, string) ([]string, error)

FindStaleFileFunc is a type of function that returns the list of stale files in the repository.

func FindStaleReferenceLocks added in v16.10.0 

func FindStaleReferenceLocks(gracePeriod time.Duration) FindStaleFileFunc

FindStaleReferenceLocks provides a function which scans the refdb for stale locks and loose references against the provided grace period.

type HeuristicalOptimizationStrategy 

type HeuristicalOptimizationStrategy struct {
	// contains filtered or unexported fields
}

HeuristicalOptimizationStrategy is an optimization strategy that is based on a set of heuristics.

func NewHeuristicalOptimizationStrategy 

func NewHeuristicalOptimizationStrategy(gitVersion git.Version, info stats.RepositoryInfo) HeuristicalOptimizationStrategy

NewHeuristicalOptimizationStrategy constructs a heuristicalOptimizationStrategy for the given repository info. It derives all data from the repository so that the heuristics used by this repository can be decided without further disk reads.

func (HeuristicalOptimizationStrategy) ShouldPruneObjects 

func (s HeuristicalOptimizationStrategy) ShouldPruneObjects(context.Context) (bool, PruneObjectsConfig)

ShouldPruneObjects determines whether the repository has stale objects that should be pruned. Object pools are never pruned to not lose data in them, but otherwise we prune when we've found enough stale objects that might in fact get pruned.

func (HeuristicalOptimizationStrategy) ShouldRepackObjects 

func (s HeuristicalOptimizationStrategy) ShouldRepackObjects(ctx context.Context) (bool, config.RepackObjectsConfig)

ShouldRepackObjects checks whether the repository's objects need to be repacked. This uses a set of heuristics that scales with the size of the object database: the larger the repository, the less frequent does it get a full repack.

func (HeuristicalOptimizationStrategy) ShouldRepackReferences 

func (s HeuristicalOptimizationStrategy) ShouldRepackReferences(context.Context) bool

ShouldRepackReferences determines whether the repository's references need to be repacked based on heuristics. The more references there are, the more loose referencos may exist until they are packed again.

func (HeuristicalOptimizationStrategy) ShouldWriteCommitGraph 

func (s HeuristicalOptimizationStrategy) ShouldWriteCommitGraph(ctx context.Context) (bool, config.WriteCommitGraphConfig, error)

ShouldWriteCommitGraph determines whether we need to write the commit-graph and how it should be written.

type Metrics added in v16.11.0 

type Metrics struct {
	DataStructureCount                     *prometheus.HistogramVec
	DataStructureExistence                 *prometheus.CounterVec
	DataStructureSize                      *prometheus.HistogramVec
	DataStructureTimeSinceLastOptimization *prometheus.HistogramVec
	PrunedFilesTotal                       *prometheus.CounterVec
	TasksLatency                           *prometheus.HistogramVec
	TasksTotal                             *prometheus.CounterVec
}

Metrics stores prometheus metrics of housekeeping tasks.

func NewMetrics added in v16.11.0 

func NewMetrics(promCfg gitalycfgprom.Config) *Metrics

NewMetrics returns a new metric wrapper object.

func (*Metrics) Collect added in v16.11.0 

func (m *Metrics) Collect(metrics chan<- prometheus.Metric)

Collect is used to collect Prometheus metrics.

func (*Metrics) Describe added in v16.11.0 

func (m *Metrics) Describe(descs chan<- *prometheus.Desc)

Describe is used to describe Prometheus metrics.

func (*Metrics) ReportRepositoryInfo added in v16.11.0 

func (m *Metrics) ReportRepositoryInfo(info stats.RepositoryInfo)

ReportRepositoryInfo reports the repository info in the form of prometheus metrics.

func (*Metrics) ReportTaskLatency added in v16.11.0 

func (m *Metrics) ReportTaskLatency(task, stage string) func() time.Duration

ReportTaskLatency reports the latency of a housekeeping task at a specific stage. The stage of a task is meaningful in the context of WAL transaction where a task is split into "prepare", "verify", and "apply". When the task runs in non-WAL context, it has "apply" stage only. The caller calls this function to mark the starting point of them. The returned function is triggered when the task finishes.

type OptimizationStrategy 

type OptimizationStrategy interface {
	// ShouldRepackObjects determines whether the repository needs to be repacked and, if so,
	// how it should be done.
	ShouldRepackObjects(context.Context) (bool, config.RepackObjectsConfig)
	// ShouldPruneObjects determines whether the repository has stale objects that should be
	// pruned and, if so, how it should be done.
	ShouldPruneObjects(context.Context) (bool, PruneObjectsConfig)
	// ShouldRepackReferences determines whether the repository's references need to be
	// repacked.
	ShouldRepackReferences(context.Context) bool
	// ShouldWriteCommitGraph determines whether we need to write the commit-graph and how it
	// should be written.
	ShouldWriteCommitGraph(context.Context) (bool, config.WriteCommitGraphConfig, error)
}

OptimizationStrategy is an interface to determine which parts of a repository should be optimized.

type PruneObjectsConfig 

type PruneObjectsConfig struct {
	// ExpireBefore controls the grace period after which unreachable objects shall be pruned.
	// An unreachable object must be older than the given date in order to be considered for
	// deletion.
	ExpireBefore time.Time
}

PruneObjectsConfig determines which objects should be pruned in PruneObjects.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.