praefect

package
v1.59.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 22, 2019 License: MIT Imports: 32 Imported by: 0

Documentation

Overview

Package praefect provides data models and datastore persistence abstractions for tracking the state of repository replicas.

See original design discussion: https://gitlab.com/gitlab-org/gitaly/issues/1495

Package praefect is a Gitaly reverse proxy for transparently routing gRPC calls to a set of Gitaly services.

Index

Constants

This section is empty.

Variables

View Source 
var ErrInvalidReplTarget = errors.New("targetNode repository fails preconditions for replication")

ErrInvalidReplTarget indicates a targetNode repository cannot be chosen because it fails preconditions for being replicatable

View Source 
var (
	// ErrPrimaryNotSet indicates the primary has not been set in the datastore
	ErrPrimaryNotSet = errors.New("primary is not set")
)
View Source 
var ErrSecondariesMissing = errors.New("repository missing secondary replicas")

ErrSecondariesMissing indicates the repository does not have any backup replicas

Functions

func GetBuildTime added in v1.53.0 

func GetBuildTime() string

GetBuildTime returns the time at which the build took place

func GetVersion added in v1.53.0 

func GetVersion() string

GetVersion returns the semver compatible version number

func GetVersionString added in v1.53.0 

func GetVersionString() string

GetVersionString returns a standard version header

Types

type Coordinator 

type Coordinator struct {
	// contains filtered or unexported fields
}

Coordinator takes care of directing client requests to the appropriate downstream server. The coordinator is thread safe; concurrent calls to register nodes are safe.

func NewCoordinator added in v1.34.0 

func NewCoordinator(l *logrus.Logger, datastore Datastore, fileDescriptors ...*descriptor.FileDescriptorProto) *Coordinator

NewCoordinator returns a new Coordinator that utilizes the provided logger

func (*Coordinator) FailoverRotation added in v1.56.0 

func (c *Coordinator) FailoverRotation()

FailoverRotation waits for the SIGUSR1 signal, then promotes the next secondary to be primary

func (*Coordinator) GetStorageNode added in v1.34.0 

func (c *Coordinator) GetStorageNode(storage string) (Node, error)

GetStorageNode returns the registered node for the given storage location

func (*Coordinator) RegisterNode 

func (c *Coordinator) RegisterNode(storageName, listenAddr string) error

RegisterNode will direct traffic to the supplied downstream connection when the storage location is encountered.

func (*Coordinator) RegisterProtos added in v1.35.1 

func (c *Coordinator) RegisterProtos(protos ...*descriptor.FileDescriptorProto) error

RegisterProtos allows coordinator to register new protos on the fly

type Datastore added in v1.34.0 

type Datastore interface {
	ReplJobsDatastore
	ReplicasDatastore
	TemporaryDatastore
}

Datastore is a data persistence abstraction for all of Praefect's persistence needs

type JobState added in v1.36.0 

type JobState uint8

JobState is an enum that indicates the state of a job 

const (
	// JobStatePending is the initial job state when it is not yet ready to run
	// and may indicate recovery from a failure prior to the ready-state
	JobStatePending JobState = 1 << iota
	// JobStateReady indicates the job is now ready to proceed
	JobStateReady
	// JobStateInProgress indicates the job is being processed by a worker
	JobStateInProgress
	// JobStateComplete indicates the job is now complete
	JobStateComplete
	// JobStateCancelled indicates the job was cancelled. This can occur if the
	// job is no longer relevant (e.g. a node is moved out of a shard)
	JobStateCancelled
)

type MemoryDatastore added in v1.34.0 

type MemoryDatastore struct {
	// contains filtered or unexported fields
}

MemoryDatastore is a simple datastore that isn't persisted to disk. It is only intended for early beta requirements and as a reference implementation for the eventual SQL implementation

func NewMemoryDatastore added in v1.34.0 

func NewMemoryDatastore(cfg config.Config) *MemoryDatastore

NewMemoryDatastore returns an initialized in-memory datastore

func (*MemoryDatastore) CreateSecondaryReplJobs added in v1.36.0 

func (md *MemoryDatastore) CreateSecondaryReplJobs(source models.Repository) ([]uint64, error)

CreateSecondaryReplJobs creates a replication job for each secondary that backs the specified repository. Upon success, the job IDs will be returned.

func (*MemoryDatastore) GetDefaultPrimary added in v1.56.0 

func (md *MemoryDatastore) GetDefaultPrimary() (models.GitalyServer, error)

GetDefaultPrimary gets the primary datastore location

func (*MemoryDatastore) GetJobs added in v1.44.0 

func (md *MemoryDatastore) GetJobs(state JobState, storage string, count int) ([]ReplJob, error)

GetJobs is a more general method to retrieve jobs of a certain state from the datastore

func (*MemoryDatastore) GetRepositoriesForPrimary added in v1.56.0 

func (md *MemoryDatastore) GetRepositoriesForPrimary(primary models.GitalyServer) ([]string, error)

GetRepositoriesForPrimary gets all repositories

func (*MemoryDatastore) GetShardPrimary added in v1.56.0 

func (md *MemoryDatastore) GetShardPrimary(repo models.Repository) (models.GitalyServer, error)

GetShardPrimary gets the primary for a repository

func (*MemoryDatastore) GetShardSecondaries added in v1.56.0 

func (md *MemoryDatastore) GetShardSecondaries(primary models.Repository) ([]models.GitalyServer, error)

GetShardSecondaries will return the set of secondary storage locations for a given repository if they exist

func (*MemoryDatastore) SetDefaultPrimary added in v1.56.0 

func (md *MemoryDatastore) SetDefaultPrimary(primary models.GitalyServer) error

SetDefaultPrimary gets the primary datastore location

func (*MemoryDatastore) SetPrimary added in v1.50.0 

func (md *MemoryDatastore) SetPrimary(primary models.GitalyServer) error

SetPrimary sets the primary datastore location

func (*MemoryDatastore) SetShardPrimary added in v1.56.0 

func (md *MemoryDatastore) SetShardPrimary(repo models.Repository, primary models.GitalyServer) error

SetShardPrimary sets the primary for a repository

func (*MemoryDatastore) SetShardSecondaries added in v1.56.0 

func (md *MemoryDatastore) SetShardSecondaries(repo models.Repository, secondaries []models.GitalyServer) error

SetShardSecondaries will replace the set of replicas for a repository

func (*MemoryDatastore) UpdateReplJob added in v1.36.0 

func (md *MemoryDatastore) UpdateReplJob(jobID uint64, newState JobState) error

UpdateReplJob updates an existing replication job's state

type Node added in v1.34.0 

type Node struct {
	Storage string
	// contains filtered or unexported fields
}

Node is a wrapper around the grpc client connection for a backend Gitaly node

type ReplJob added in v1.34.0 

type ReplJob struct {
	ID     uint64            // autoincrement ID
	Target string            // which storage location to replicate to?
	Source models.Repository // source for replication
	State  JobState
}

ReplJob is an instance of a queued replication job. A replication job is meant for updating the repository so that it is synced with the primary copy. Scheduled indicates when a replication job should be performed.

type ReplJobsDatastore added in v1.34.0 

type ReplJobsDatastore interface {
	// GetJobs fetches a list of chronologically ordered replication
	// jobs for the given storage replica. The returned list will be at most
	// count-length.
	GetJobs(flag JobState, node string, count int) ([]ReplJob, error)

	// CreateSecondaryJobs will create replication jobs for each secondary
	// replica of a repository known to the datastore. A set of replication job
	// ID's for the created jobs will be returned upon success.
	CreateSecondaryReplJobs(source models.Repository) ([]uint64, error)

	// UpdateReplJob updates the state of an existing replication job
	UpdateReplJob(jobID uint64, newState JobState) error
}

ReplJobsDatastore represents the behavior needed for fetching and updating replication jobs from the datastore

type ReplMgr added in v1.34.0 

type ReplMgr struct {
	// contains filtered or unexported fields
}

ReplMgr is a replication manager for handling replication jobs

func NewReplMgr added in v1.34.0 

func NewReplMgr(targetNode string, log *logrus.Logger, ds Datastore, c *Coordinator, opts ...ReplMgrOpt) ReplMgr

NewReplMgr initializes a replication manager with the provided dependencies and options

func (ReplMgr) ProcessBacklog added in v1.34.0 

func (r ReplMgr) ProcessBacklog(ctx context.Context) error

ProcessBacklog will process queued jobs. It will block while processing jobs.

func (ReplMgr) ScheduleReplication added in v1.34.0 

func (r ReplMgr) ScheduleReplication(ctx context.Context, repo models.Repository) error

ScheduleReplication will store a replication job in the datastore for later execution. It filters out projects that are not whitelisted. TODO: add a parameter to delay replication

type ReplMgrOpt added in v1.34.0 

type ReplMgrOpt func(*ReplMgr)

ReplMgrOpt allows a replicator to be configured with additional options

func WithReplicator added in v1.34.0 

func WithReplicator(r Replicator) ReplMgrOpt

WithReplicator overrides the default replicator

func WithWhitelist added in v1.34.0 

func WithWhitelist(whitelistedRepos []string) ReplMgrOpt

WithWhitelist will configure a whitelist for repos to allow replication

type ReplicasDatastore added in v1.34.0 

type ReplicasDatastore interface {
	// GetSecondaries will retrieve all secondary replica storage locations for
	// a primary replica
	GetShardSecondaries(repo models.Repository) ([]models.GitalyServer, error)

	GetShardPrimary(repo models.Repository) (models.GitalyServer, error)

	// SetSecondaries will set the secondary storage locations for a repository
	// in a primary replica.
	SetShardSecondaries(repo models.Repository, secondaries []models.GitalyServer) error

	SetShardPrimary(repo models.Repository, primary models.GitalyServer) error

	// GetRepositoriesForPrimary returns a map of all of the active shards for a given primary
	GetRepositoriesForPrimary(primary models.GitalyServer) ([]string, error)
}

ReplicasDatastore manages accessing and setting which secondary replicas backup a repository

type Replicator added in v1.34.0 

type Replicator interface {
	Replicate(ctx context.Context, source models.Repository, target Node) error
}

Replicator performs the actual replication logic between two nodes

type Server 

type Server struct {
	// contains filtered or unexported fields
}

Server is a praefect server

func NewServer 

func NewServer(c *Coordinator, repl ReplMgr, grpcOpts []grpc.ServerOption, l *logrus.Logger) *Server

NewServer returns an initialized praefect gPRC proxy server configured with the provided gRPC server options

func (*Server) Shutdown 

func (srv *Server) Shutdown(ctx context.Context) error

Shutdown will attempt a graceful shutdown of the grpc server. If unable to gracefully shutdown within the context deadline, it will then forcefully shutdown the server and return a context cancellation error.

func (*Server) Start 

func (srv *Server) Start(lis net.Listener) error

Start will start the praefect gRPC proxy server listening at the provided listener. Function will block until the server is stopped or an unrecoverable error occurs.

type TemporaryDatastore added in v1.56.0 

type TemporaryDatastore interface {
	GetDefaultPrimary() (models.GitalyServer, error)
	SetDefaultPrimary(primary models.GitalyServer) error
}

TemporaryDatastore contains methods that will go away once we move to a SQL datastore

Source Files

View all Source files

Directories

Path Synopsis
grpc-proxy
proxy
Package proxy provides a reverse proxy handler for gRPC.
 Package proxy provides a reverse proxy handler for gRPC.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.