storagemgr

package
v16.3.8-rc42 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 17, 2024 License: MIT Imports: 31 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrRepositoryNotFound is returned when the repository doesn't exist.
	ErrRepositoryNotFound = structerr.NewNotFound("repository not found")
	// ErrTransactionProcessingStopped is returned when the TransactionManager stops processing transactions.
	ErrTransactionProcessingStopped = errors.New("transaction processing stopped")
	// ErrTransactionAlreadyCommitted is returned when attempting to rollback or commit a transaction that
	// already had commit called on it.
	ErrTransactionAlreadyCommitted = errors.New("transaction already committed")
	// ErrTransactionAlreadyRollbacked is returned when attempting to rollback or commit a transaction that
	// already had rollback called on it.
	ErrTransactionAlreadyRollbacked = errors.New("transaction already rollbacked")
)
View Source 
var ErrPartitionManagerClosed = errors.New("partition manager closed")

ErrPartitionManagerClosed is returned when the PartitionManager stops processing transactions.

Functions

func OpenDatabase 

func OpenDatabase(databasePath string) (*badger.DB, error)

OpenDatabase opens a new database handle to a database at the given path.

Types

type CustomHooksUpdate 

type CustomHooksUpdate struct {
	// CustomHooksTAR contains the custom hooks as a TAR. The TAR contains a `custom_hooks`
	// directory which contains the hooks. Setting the update with nil `custom_hooks_tar` clears
	// the hooks from the repository.
	CustomHooksTAR []byte
}

CustomHooksUpdate models an update to the custom hooks.

type DefaultBranchUpdate 

type DefaultBranchUpdate struct {
	// Reference is the reference to update the default branch to.
	Reference git.ReferenceName
}

DefaultBranchUpdate provides the information to update the default branch of the repo.

type InvalidReferenceFormatError 

type InvalidReferenceFormatError struct {
	// ReferenceName is the reference with invalid format.
	ReferenceName git.ReferenceName
}

InvalidReferenceFormatError is returned when a reference name was invalid.

func (InvalidReferenceFormatError) Error 

func (err InvalidReferenceFormatError) Error() string

Error returns the formatted error string.

type LogIndex 

type LogIndex uint64

LogIndex points to a specific position in a repository's write-ahead log.

func (LogIndex) String 

func (index LogIndex) String() string

String returns a string representation of the LogIndex.

type PartitionManager 

type PartitionManager struct {
	// contains filtered or unexported fields
}

PartitionManager is responsible for managing the lifecycle of each TransactionManager.

func NewPartitionManager 

func NewPartitionManager(
	configuredStorages []config.Storage,
	cmdFactory git.CommandFactory,
	housekeepingManager housekeeping.Manager,
	localRepoFactory localrepo.Factory,
	logger logrus.FieldLogger,
) (*PartitionManager, error)

NewPartitionManager returns a new PartitionManager.

func (*PartitionManager) Begin 

func (pm *PartitionManager) Begin(ctx context.Context, repo storage.Repository, opts TransactionOptions) (*finalizableTransaction, error)

Begin gets the TransactionManager for the specified repository and starts a transaction. If a TransactionManager is not already running, a new one is created and used. The partition tracks the number of pending transactions and this counter gets incremented when Begin is invoked.

func (*PartitionManager) Close added in v16.3.0 

func (pm *PartitionManager) Close()

Close closes transaction processing for all storages and waits for closing completion.

type ReferenceUpdate 

type ReferenceUpdate struct {
	// Force indicates this is a forced reference update. If set, the reference is pointed
	// to the new value regardless of the old value.
	Force bool
	// OldOID is the old OID the reference is expected to point to prior to updating it.
	// If the reference does not point to the old value, the reference verification fails.
	OldOID git.ObjectID
	// NewOID is the new desired OID to point the reference to.
	NewOID git.ObjectID
}

ReferenceUpdate describes the state of a reference's old and new tip in an update.

type ReferenceUpdates 

type ReferenceUpdates map[git.ReferenceName]ReferenceUpdate

ReferenceUpdates contains references to update. Reference name is used as the key and the value is the expected old tip and the desired new tip.

type ReferenceVerificationError 

type ReferenceVerificationError struct {
	// ReferenceName is the name of the reference that failed verification.
	ReferenceName git.ReferenceName
	// ExpectedOID is the OID the reference was expected to point to.
	ExpectedOID git.ObjectID
	// ActualOID is the OID the reference actually pointed to.
	ActualOID git.ObjectID
}

ReferenceVerificationError is returned when a reference's old OID did not match the expected.

func (ReferenceVerificationError) Error 

func (err ReferenceVerificationError) Error() string

Error returns the formatted error string.

type Snapshot 

type Snapshot struct {
	// ReadIndex is the index of the log entry this Transaction is reading the data at.
	ReadIndex LogIndex
	// CustomHookIndex is index of the custom hooks on the disk that are included in this Transactions's
	// snapshot and were the latest on the read index.
	CustomHookIndex LogIndex
	// CustomHookPath is an absolute filesystem path to the custom hooks in this snapshot.
	CustomHookPath string
}

Snapshot contains the read snapshot details of a Transaction.

type Transaction 

type Transaction struct {
	// contains filtered or unexported fields
}

Transaction is a unit-of-work that contains reference changes to perform on the repository.

func (*Transaction) Commit 

func (txn *Transaction) Commit(ctx context.Context) (returnedErr error)

Commit performs the changes. If no error is returned, the transaction was successful and the changes have been performed. If an error was returned, the transaction may or may not be persisted.

func (*Transaction) DeleteRepository 

func (txn *Transaction) DeleteRepository()

DeleteRepository deletes the repository when the transaction is committed.

func (*Transaction) IncludeObject added in v16.3.0 

func (txn *Transaction) IncludeObject(oid git.ObjectID)

IncludeObject includes the given object and its dependencies in the transaction's logged pack file even if the object is unreachable from the references.

func (*Transaction) RewriteRepository added in v16.3.0 

func (txn *Transaction) RewriteRepository(repo *gitalypb.Repository) *gitalypb.Repository

RewriteRepository returns a copy of the repository that has been set up to correctly access the transaction's repository.

func (*Transaction) Rollback 

func (txn *Transaction) Rollback() error

Rollback releases resources associated with the transaction without performing any changes.

func (*Transaction) SetCustomHooks 

func (txn *Transaction) SetCustomHooks(customHooksTAR []byte)

SetCustomHooks sets the custom hooks as part of the transaction. If SetCustomHooks is called multiple times, only the changes from the latest invocation take place. The custom hooks are extracted as is and are not validated. Setting a nil hooksTAR removes the hooks from the repository.

func (*Transaction) SetDefaultBranch 

func (txn *Transaction) SetDefaultBranch(new git.ReferenceName)

SetDefaultBranch sets the default branch as part of the transaction. If SetDefaultBranch is called multiple times, only the changes from the latest invocation take place. The reference is validated to exist.

func (*Transaction) SkipVerificationFailures 

func (txn *Transaction) SkipVerificationFailures()

SkipVerificationFailures configures the transaction to skip reference updates that fail verification. If a reference update fails verification with this set, the update is dropped from the transaction but other successful reference updates will be made. By default, the entire transaction is aborted if a reference fails verification.

The default behavior models `git push --atomic`. Toggling this option models the behavior without the `--atomic` flag.

func (*Transaction) Snapshot 

func (txn *Transaction) Snapshot() Snapshot

Snapshot returns the details of the Transaction's read snapshot.

func (*Transaction) UpdateReferences 

func (txn *Transaction) UpdateReferences(updates ReferenceUpdates)

UpdateReferences updates the given references as part of the transaction. If UpdateReferences is called multiple times, only the changes from the latest invocation take place.

type TransactionManager 

type TransactionManager struct {
	// contains filtered or unexported fields
}

TransactionManager is responsible for transaction management of a single repository. Each repository has a single TransactionManager; it is the repository's single-writer. It accepts writes one at a time from the admissionQueue. Each admitted write is processed in three steps:

  1. The references being updated are verified by ensuring the expected old tips match what the references actually point to prior to update. The entire transaction is by default aborted if a single reference fails the verification step. The reference verification behavior can be controlled on a per-transaction level by setting: - The reference verification failures can be ignored instead of aborting the entire transaction. If done, the references that failed verification are dropped from the transaction but the updates that passed verification are still performed. - The reference verification may also be skipped if the write is force updating references. If done, the current state of the references is ignored and they are directly updated to point to the new tips.
  2. The transaction is appended to the write-ahead log. Once the write has been logged, it is effectively committed and will be applied to the repository even after restarting.
  3. The transaction is applied from the write-ahead log to the repository by actually performing the reference changes.

The goroutine that issued the transaction is waiting for the result while these steps are being performed. As there is no transaction control for readers yet, the issuer is only notified of a successful write after the write has been applied to the repository.

TransactionManager recovers transactions after interruptions by applying the write-ahead logged transactions to the repository on start up.

TransactionManager maintains the write-ahead log in a key-value store. It maintains the following key spaces: - `repository/<repository_id:string>/log/index/applied`

  • This key stores the index of the log entry that has been applied to the repository. This allows for determining how far a repository is in processing the log and which log entries need to be applied after starting up. Repository starts from log index 0 if there are no log entries recorded to have been applied.

- `repository/<repository_id:string>/log/entry/<log_index:uint64>`

  • These keys hold the actual write-ahead log entries. A repository's first log entry starts at index 1 and the log index keeps monotonically increasing from there on without gaps. The write-ahead log entries are processed in ascending order.

The values in the database are marshaled protocol buffer messages. Numbers in the keys are encoded as big endian to preserve the sort order of the numbers also in lexicographical order.

func NewTransactionManager 

func NewTransactionManager(
	db *badger.DB,
	storagePath,
	relativePath,
	stagingDir string,
	cmdFactory git.CommandFactory,
	housekeepingManager housekeeping.Manager,
	repositoryFactory localrepo.StorageScopedFactory,
) *TransactionManager

NewTransactionManager returns a new TransactionManager for the given repository.

func (*TransactionManager) Begin 

func (mgr *TransactionManager) Begin(ctx context.Context, opts TransactionOptions) (_ *Transaction, returnedErr error)

Begin opens a new transaction. The caller must call either Commit or Rollback to release the resources tied to the transaction. The returned Transaction is not safe for concurrent use.

The returned Transaction's read snapshot includes all writes that were committed prior to the Begin call. Begin blocks until the committed writes have been applied to the repository.

func (*TransactionManager) Close added in v16.3.0 

func (mgr *TransactionManager) Close()

Close stops the transaction processing causing Run to return.

func (*TransactionManager) Run 

func (mgr *TransactionManager) Run() (returnedErr error)

Run starts the transaction processing. On start up Run loads the indexes of the last appended and applied log entries from the database. It will then apply any transactions that have been logged but not applied to the repository. Once the recovery is completed, Run starts processing new transactions by verifying the references, logging the transaction and finally applying it to the repository. The transactions are acknowledged once they've been applied to the repository.

Run keeps running until Stop is called or it encounters a fatal error. All transactions will error with ErrTransactionProcessingStopped when Run returns.

type TransactionOptions added in v16.3.0 

type TransactionOptions struct {
	// ReadOnly indicates whether this is a read-only transaction. Read-only transactions are not
	// configured with a quarantine directory and do not commit a log entry.
	ReadOnly bool
}

TransactionOptions configures transaction options when beginning a transaction.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.