backup

package
v16.5.6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 11, 2024 License: MIT Imports: 38 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrSkipped means the repository was skipped because there was nothing to backup
	ErrSkipped = errors.New("repository skipped")
	// ErrDoesntExist means that the data was not found.
	ErrDoesntExist = errors.New("doesn't exist")
)

Functions

This section is empty.

Types

type Backup 

type Backup struct {
	// ID is the identifier that uniquely identifies the backup for this repository.
	ID string `toml:"-"`
	// Repository is the repository being backed up.
	Repository storage.Repository `toml:"-"`
	// Steps are the ordered list of steps required to restore this backup
	Steps []Step `toml:"steps"`
	// ObjectFormat is the name of the object hash used by the repository.
	ObjectFormat string `toml:"object_format"`
}

Backup represents all the information needed to restore a backup for a repository

type Command 

type Command interface {
	Repository() *gitalypb.Repository
	Name() string
	Execute(context.Context) error
}

Command handles a specific backup operation

type CreateCommand 

type CreateCommand struct {
	// contains filtered or unexported fields
}

CreateCommand creates a backup for a repository

func NewCreateCommand 

func NewCreateCommand(strategy Strategy, request CreateRequest) *CreateCommand

NewCreateCommand builds a CreateCommand

func (CreateCommand) Execute 

func (cmd CreateCommand) Execute(ctx context.Context) error

Execute performs the backup

func (CreateCommand) Name 

func (cmd CreateCommand) Name() string

Name is the name of the command

func (CreateCommand) Repository 

func (cmd CreateCommand) Repository() *gitalypb.Repository

Repository is the repository that will be acted on

type CreateRequest 

type CreateRequest struct {
	// Server contains gitaly server connection information required to call
	// RPCs in the non-local backup.Manager configuration.
	Server storage.ServerInfo
	// Repository is the repository to be backed up.
	Repository *gitalypb.Repository
	// VanityRepository is used to determine the backup path.
	VanityRepository *gitalypb.Repository
	// Incremental when true will create an increment on the specified full backup.
	Incremental bool
	// BackupID is used to determine a unique path for the backup when a full
	// backup is created.
	BackupID string
}

CreateRequest is the request to create a backup

type FilesystemSink 

type FilesystemSink struct {
	// contains filtered or unexported fields
}

FilesystemSink is a sink for creating and restoring backups from the local filesystem.

func NewFilesystemSink 

func NewFilesystemSink(path string) *FilesystemSink

NewFilesystemSink returns a sink that uses a local filesystem to work with data.

func (*FilesystemSink) Close added in v16.2.0 

func (fs *FilesystemSink) Close() error

Close is a no-op to implement the Sink interface

func (*FilesystemSink) GetReader 

func (fs *FilesystemSink) GetReader(ctx context.Context, relativePath string) (io.ReadCloser, error)

GetReader returns a reader of the requested file path. It's the caller's responsibility to Close returned reader once it is not needed anymore. If relativePath doesn't exist the ErrDoesntExist is returned.

func (*FilesystemSink) GetWriter 

func (fs *FilesystemSink) GetWriter(ctx context.Context, relativePath string) (io.WriteCloser, error)

GetWriter opens a io.WriteCloser that can be used to write data into a relativePath path on the filesystem. It is the callers responsibility to Close the writer after usage.

type LazyWriter 

type LazyWriter struct {
	// contains filtered or unexported fields
}

LazyWriter is a WriteCloser that will call Create when on the first call to Write. This means it will only create a file if there will be data written to it.

func NewLazyWriter 

func NewLazyWriter(create func() (io.WriteCloser, error)) *LazyWriter

NewLazyWriter initializes a new LazyWriter. create is called on the first call of Write, any errors will be returned by this call.

func (*LazyWriter) Close 

func (w *LazyWriter) Close() error

Close calls Close on the WriteCloser returned by Create, passing on any returned error. Close must be called to properly clean up resources.

func (*LazyWriter) Write 

func (w *LazyWriter) Write(p []byte) (int, error)

type LegacyLocator 

type LegacyLocator struct{}

LegacyLocator locates backup paths for historic backups. This is the structure that gitlab used before incremental backups were introduced.

Existing backup files are expected to be overwritten by the latest backup files.

Structure: 

<repo relative path>.bundle
<repo relative path>.refs
<repo relative path>/custom_hooks.tar

func (LegacyLocator) BeginFull 

func (l LegacyLocator) BeginFull(ctx context.Context, repo storage.Repository, backupID string) *Backup

BeginFull returns the static paths for a legacy repository backup

func (LegacyLocator) BeginIncremental 

func (l LegacyLocator) BeginIncremental(ctx context.Context, repo storage.Repository, backupID string) (*Backup, error)

BeginIncremental is not supported for legacy backups

func (LegacyLocator) Commit 

func (l LegacyLocator) Commit(ctx context.Context, full *Backup) error

Commit is unused as the locations are static

func (LegacyLocator) Find added in v16.2.0 

func (l LegacyLocator) Find(ctx context.Context, repo storage.Repository, backupID string) (*Backup, error)

Find is not supported for legacy backups.

func (LegacyLocator) FindLatest 

func (l LegacyLocator) FindLatest(ctx context.Context, repo storage.Repository) (*Backup, error)

FindLatest returns the static paths for a legacy repository backup

type Locator 

type Locator interface {
	// BeginFull returns the tentative backup paths needed to create a full backup.
	BeginFull(ctx context.Context, repo storage.Repository, backupID string) *Backup

	// BeginIncremental returns the backup with the last element of Steps being
	// the tentative step needed to create an incremental backup.
	BeginIncremental(ctx context.Context, repo storage.Repository, backupID string) (*Backup, error)

	// Commit persists the backup so that it can be looked up by FindLatest. It
	// is expected that the last element of Steps will be the newly created
	// backup.
	Commit(ctx context.Context, backup *Backup) error

	// FindLatest returns the latest backup that was written by Commit
	FindLatest(ctx context.Context, repo storage.Repository) (*Backup, error)

	// Find returns the repository backup at the given backupID. If the backup does
	// not exist then the error ErrDoesntExist is returned.
	Find(ctx context.Context, repo storage.Repository, backupID string) (*Backup, error)
}

Locator finds sink backup paths for repositories

func ResolveLocator 

func ResolveLocator(layout string, sink Sink) (Locator, error)

ResolveLocator returns a locator implementation based on a locator identifier.

type LoggingPipeline 

type LoggingPipeline struct {
	// contains filtered or unexported fields
}

LoggingPipeline outputs logging for each command executed

func NewLoggingPipeline 

func NewLoggingPipeline(log log.Logger) *LoggingPipeline

NewLoggingPipeline creates a new logging pipeline

func (*LoggingPipeline) Done 

func (p *LoggingPipeline) Done() error

Done indicates that the pipeline is complete and returns any accumulated errors

func (*LoggingPipeline) Handle 

func (p *LoggingPipeline) Handle(ctx context.Context, cmd Command)

Handle takes a command to process. Commands are logged and executed immediately.

type Manager 

type Manager struct {
	// contains filtered or unexported fields
}

Manager manages process of the creating/restoring backups.

func NewManager 

func NewManager(sink Sink, locator Locator, pool *client.Pool) *Manager

NewManager creates and returns initialized *Manager instance.

func NewManagerLocal 

func NewManagerLocal(
	sink Sink,
	logger log.Logger,
	locator Locator,
	storageLocator storage.Locator,
	gitCmdFactory git.CommandFactory,
	catfileCache catfile.Cache,
	txManager transaction.Manager,
	repoCounter *counter.RepositoryCounter,
) *Manager

NewManagerLocal creates and returns a *Manager instance for operating on local repositories.

func (*Manager) Create 

func (mgr *Manager) Create(ctx context.Context, req *CreateRequest) error

Create creates a repository backup.

func (*Manager) RemoveAllRepositories 

func (mgr *Manager) RemoveAllRepositories(ctx context.Context, req *RemoveAllRepositoriesRequest) error

RemoveAllRepositories removes all repositories in the specified storage name.

func (*Manager) Restore 

func (mgr *Manager) Restore(ctx context.Context, req *RestoreRequest) error

Restore restores a repository from a backup. If req.BackupID is empty, the latest backup will be used.

type ManifestLocator added in v16.5.0 

type ManifestLocator struct {
	Sink     Sink
	Fallback Locator
}

ManifestLocator locates backup paths based on manifest files that are written to a predetermined path: 

manifests/<repo_storage_name>/<repo_relative_path>/<backup_id>.toml

It relies on Fallback to determine paths of new backups.

func (ManifestLocator) BeginFull added in v16.5.0 

func (l ManifestLocator) BeginFull(ctx context.Context, repo storage.Repository, backupID string) *Backup

BeginFull passes through to Fallback

func (ManifestLocator) BeginIncremental added in v16.5.0 

func (l ManifestLocator) BeginIncremental(ctx context.Context, repo storage.Repository, backupID string) (*Backup, error)

BeginIncremental passes through to Fallback

func (ManifestLocator) Commit added in v16.5.0 

func (l ManifestLocator) Commit(ctx context.Context, backup *Backup) (returnErr error)

Commit passes through to Fallback, then writes a manifest file for the backup.

func (ManifestLocator) Find added in v16.5.0 

func (l ManifestLocator) Find(ctx context.Context, repo storage.Repository, backupID string) (*Backup, error)

Find loads the manifest for the provided repo and backupID. If this manifest does not exist, the fallback is used.

func (ManifestLocator) FindLatest added in v16.5.0 

func (l ManifestLocator) FindLatest(ctx context.Context, repo storage.Repository) (*Backup, error)

FindLatest passes through to Fallback

type ParallelPipeline 

type ParallelPipeline struct {
	// contains filtered or unexported fields
}

ParallelPipeline is a pipeline that executes commands in parallel

func NewParallelPipeline 

func NewParallelPipeline(next Pipeline, parallel, parallelStorage int) *ParallelPipeline

NewParallelPipeline creates a new ParallelPipeline where all commands are passed onto `next` to be processed, `parallel` is the maximum number of parallel backups that will run and `parallelStorage` is the maximum number of parallel backups that will run per storage. Since the number of storages is unknown at initialisation, workers are created lazily as new storage names are encountered.

Note: When both `parallel` and `parallelStorage` are zero or less no workers are created and the pipeline will block forever.

func (*ParallelPipeline) Done 

func (p *ParallelPipeline) Done() error

Done waits for any in progress calls to `next` to complete then reports any accumulated errors

func (*ParallelPipeline) Handle 

func (p *ParallelPipeline) Handle(ctx context.Context, cmd Command)

Handle queues a request to create a backup. Commands are processed by n-workers per storage.

type Pipeline 

type Pipeline interface {
	Handle(context.Context, Command)
	Done() error
}

Pipeline executes a series of commands and encapsulates error handling for the caller.

type PipelineErrors 

type PipelineErrors []error

PipelineErrors represents a summary of errors by repository

func (*PipelineErrors) AddError 

func (e *PipelineErrors) AddError(repo *gitalypb.Repository, err error)

AddError adds an error associated with a repository to the summary.

func (PipelineErrors) Error 

func (e PipelineErrors) Error() string

type PointerLocator 

type PointerLocator struct {
	Sink     Sink
	Fallback Locator
}

PointerLocator locates backup paths where each full backup is put into a unique timestamp directory and the latest backup taken is pointed to by a file named LATEST.

Structure: 

<repo relative path>/LATEST
<repo relative path>/<backup id>/LATEST
<repo relative path>/<backup id>/<nnn>.bundle
<repo relative path>/<backup id>/<nnn>.refs
<repo relative path>/<backup id>/<nnn>.custom_hooks.tar

func (PointerLocator) BeginFull 

func (l PointerLocator) BeginFull(ctx context.Context, repo storage.Repository, backupID string) *Backup

BeginFull returns a tentative first step needed to create a new full backup.

func (PointerLocator) BeginIncremental 

func (l PointerLocator) BeginIncremental(ctx context.Context, repo storage.Repository, fallbackBackupID string) (*Backup, error)

BeginIncremental returns a tentative step needed to create a new incremental backup. The incremental backup is always based off of the latest full backup. If there is no latest backup, a new full backup step is returned using fallbackBackupID

func (PointerLocator) Commit 

func (l PointerLocator) Commit(ctx context.Context, backup *Backup) error

Commit persists the step so that it can be looked up by FindLatest

func (PointerLocator) Find added in v16.2.0 

func (l PointerLocator) Find(ctx context.Context, repo storage.Repository, backupID string) (*Backup, error)

Find returns the repository backup at the given backupID. If the backup does not exist then the error ErrDoesntExist is returned.

func (PointerLocator) FindLatest 

func (l PointerLocator) FindLatest(ctx context.Context, repo storage.Repository) (*Backup, error)

FindLatest returns the paths committed by the latest call to CommitFull.

If there is no `LATEST` file, the result of the `Fallback` is used.

type RemoveAllRepositoriesRequest 

type RemoveAllRepositoriesRequest struct {
	Server      storage.ServerInfo
	StorageName string
}

RemoveAllRepositoriesRequest is the request to remove all repositories in the specified storage name.

type Repository 

type Repository interface {
	// ListRefs fetches the full set of refs and targets for the repository.
	ListRefs(ctx context.Context) ([]git.Reference, error)
	// GetCustomHooks fetches the custom hooks archive.
	GetCustomHooks(ctx context.Context, out io.Writer) error
	// CreateBundle fetches a bundle that contains refs matching patterns. When
	// patterns is nil all refs are bundled.
	CreateBundle(ctx context.Context, out io.Writer, patterns io.Reader) error
	// Remove removes the repository. Does not return an error if the
	// repository cannot be found.
	Remove(ctx context.Context) error
	// Create creates the repository.
	Create(ctx context.Context, hash git.ObjectHash) error
	// FetchBundle fetches references from a bundle. Refs will be mirrored to
	// the repository.
	FetchBundle(ctx context.Context, reader io.Reader) error
	// SetCustomHooks updates the custom hooks for the repository.
	SetCustomHooks(ctx context.Context, reader io.Reader) error
}

Repository abstracts git access required to make a repository backup

type RestoreCommand 

type RestoreCommand struct {
	// contains filtered or unexported fields
}

RestoreCommand restores a backup for a repository

func NewRestoreCommand 

func NewRestoreCommand(strategy Strategy, request RestoreRequest) *RestoreCommand

NewRestoreCommand builds a RestoreCommand

func (RestoreCommand) Execute 

func (cmd RestoreCommand) Execute(ctx context.Context) error

Execute performs the restore

func (RestoreCommand) Name 

func (cmd RestoreCommand) Name() string

Name is the name of the command

func (RestoreCommand) Repository 

func (cmd RestoreCommand) Repository() *gitalypb.Repository

Repository is the repository that will be acted on

type RestoreRequest 

type RestoreRequest struct {
	// Server contains gitaly server connection information required to call
	// RPCs in the non-local backup.Manager configuration.
	Server storage.ServerInfo
	// Repository is the repository to be restored.
	Repository *gitalypb.Repository
	// VanityRepository is used to determine the backup path.
	VanityRepository *gitalypb.Repository
	// AlwaysCreate forces the repository to be created even if no bundle for
	// it exists. See https://gitlab.com/gitlab-org/gitlab/-/issues/357044
	AlwaysCreate bool
	// BackupID is the ID of the full backup to restore. If not specified, the
	// latest backup is restored..
	BackupID string
}

RestoreRequest is the request to restore from a backup

type ServerSideAdapter added in v16.2.0 

type ServerSideAdapter struct {
	// contains filtered or unexported fields
}

ServerSideAdapter allows calling the server-side backup RPCs `BackupRepository` and `RestoreRepository` through `backup.Strategy` such that server-side backups can be used with `backup.Pipeline`.

func NewServerSideAdapter added in v16.2.0 

func NewServerSideAdapter(pool *client.Pool) *ServerSideAdapter

NewServerSideAdapter creates and returns initialized *ServerSideAdapter instance.

func (ServerSideAdapter) Create added in v16.2.0 

func (ss ServerSideAdapter) Create(ctx context.Context, req *CreateRequest) error

Create calls the BackupRepository RPC.

func (ServerSideAdapter) RemoveAllRepositories added in v16.2.0 

func (ss ServerSideAdapter) RemoveAllRepositories(ctx context.Context, req *RemoveAllRepositoriesRequest) error

RemoveAllRepositories removes all repositories in the specified storage name.

func (ServerSideAdapter) Restore added in v16.2.0 

func (ss ServerSideAdapter) Restore(ctx context.Context, req *RestoreRequest) error

Restore calls the RestoreRepository RPC.

type Sink 

type Sink interface {
	io.Closer
	// GetWriter saves the written data to relativePath. It is the callers
	// responsibility to call Close and check any subsequent errors.
	GetWriter(ctx context.Context, relativePath string) (io.WriteCloser, error)
	// GetReader returns a reader that servers the data stored by relativePath.
	// If relativePath doesn't exists the ErrDoesntExist will be returned.
	GetReader(ctx context.Context, relativePath string) (io.ReadCloser, error)
}

Sink is an abstraction over the real storage used for storing/restoring backups.

func ResolveSink 

func ResolveSink(ctx context.Context, path string) (Sink, error)

ResolveSink returns a sink implementation based on the provided path.

type Step 

type Step struct {
	// BundlePath is the path of the bundle
	BundlePath string `toml:"bundle_path,omitempty"`
	// RefPath is the path of the ref file
	RefPath string `toml:"ref_path,omitempty"`
	// PreviousRefPath is the path of the previous ref file
	PreviousRefPath string `toml:"previous_ref_path,omitempty"`
	// CustomHooksPath is the path of the custom hooks archive
	CustomHooksPath string `toml:"custom_hooks_path,omitempty"`
}

Step represents an incremental step that makes up a complete backup for a repository

type StorageServiceSink 

type StorageServiceSink struct {
	// contains filtered or unexported fields
}

StorageServiceSink uses a storage engine that can be defined by the construction url on creation.

func NewStorageServiceSink 

func NewStorageServiceSink(ctx context.Context, url string) (*StorageServiceSink, error)

NewStorageServiceSink returns initialized instance of StorageServiceSink instance. The storage engine is chosen based on the provided url value and a set of pre-registered blank imports in that file. It is the caller's responsibility to provide all required environment variables in order to get properly initialized storage engine driver.

func (*StorageServiceSink) Close 

func (s *StorageServiceSink) Close() error

Close releases resources associated with the bucket communication.

func (*StorageServiceSink) GetReader 

func (s *StorageServiceSink) GetReader(ctx context.Context, relativePath string) (io.ReadCloser, error)

GetReader returns a reader to consume the data from the configured bucket. It is the caller's responsibility to Close the reader after usage.

func (*StorageServiceSink) GetWriter 

func (s *StorageServiceSink) GetWriter(ctx context.Context, relativePath string) (io.WriteCloser, error)

GetWriter stores the written data into a relativePath path on the configured bucket. It is the callers responsibility to Close the reader after usage.

type Strategy 

type Strategy interface {
	Create(context.Context, *CreateRequest) error
	Restore(context.Context, *RestoreRequest) error
	RemoveAllRepositories(context.Context, *RemoveAllRepositoriesRequest) error
}

Strategy used to create/restore backups

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.