praefect

package
v1.72.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 18, 2019 License: MIT Imports: 38 Imported by: 0

Documentation

Overview

Package praefect provides data models and datastore persistence abstractions for tracking the state of repository replicas.

See original design discussion: https://gitlab.com/gitlab-org/gitaly/issues/1495

Package praefect is a Gitaly reverse proxy for transparently routing gRPC calls to a set of Gitaly services.

Index

Constants

This section is empty.

Variables

View Source 
var ErrInvalidReplTarget = errors.New("targetStorage repository fails preconditions for replication")

ErrInvalidReplTarget indicates a targetStorage repository cannot be chosen because it fails preconditions for being replicatable

View Source 
var (
	// ErrPrimaryNotSet indicates the primary has not been set in the datastore
	ErrPrimaryNotSet = errors.New("primary is not set")
)
View Source 
var ErrReplicasMissing = errors.New("repository missing secondary replicas")

ErrReplicasMissing indicates the repository does not have any backup replicas

Functions

func GetBuildTime added in v1.53.0 

func GetBuildTime() string

GetBuildTime returns the time at which the build took place

func GetVersion added in v1.53.0 

func GetVersion() string

GetVersion returns the semver compatible version number

func GetVersionString added in v1.53.0 

func GetVersionString() string

GetVersionString returns a standard version header

Types

type ChangeType added in v1.72.0 

type ChangeType int

ChangeType indicates what kind of change the replication is propagating 

const (
	// UpdateRepo is when a replication updates a repository in place
	UpdateRepo ChangeType = iota + 1
	// DeleteRepo is when a replication deletes a repo
	DeleteRepo
)

type Coordinator 

type Coordinator struct {
	// contains filtered or unexported fields
}

Coordinator takes care of directing client requests to the appropriate downstream server. The coordinator is thread safe; concurrent calls to register nodes are safe.

func NewCoordinator added in v1.34.0 

func NewCoordinator(l *logrus.Entry, datastore Datastore, clientConnections *conn.ClientConnections, conf config.Config, fileDescriptors ...*descriptor.FileDescriptorProto) *Coordinator

NewCoordinator returns a new Coordinator that utilizes the provided logger

func (*Coordinator) FailoverRotation added in v1.56.0 

func (c *Coordinator) FailoverRotation()

FailoverRotation waits for the SIGUSR1 signal, then promotes the next secondary to be primary

func (*Coordinator) RegisterProtos added in v1.35.1 

func (c *Coordinator) RegisterProtos(protos ...*descriptor.FileDescriptorProto) error

RegisterProtos allows coordinator to register new protos on the fly

type Datastore added in v1.34.0 

type Datastore interface {
	ReplJobsDatastore
	ReplicasDatastore
}

Datastore is a data persistence abstraction for all of Praefect's persistence needs

type JobState added in v1.36.0 

type JobState uint8

JobState is an enum that indicates the state of a job 

const (
	// JobStatePending is the initial job state when it is not yet ready to run
	// and may indicate recovery from a failure prior to the ready-state
	JobStatePending JobState = 1 << iota
	// JobStateReady indicates the job is now ready to proceed
	JobStateReady
	// JobStateInProgress indicates the job is being processed by a worker
	JobStateInProgress
	// JobStateComplete indicates the job is now complete
	JobStateComplete
	// JobStateCancelled indicates the job was cancelled. This can occur if the
	// job is no longer relevant (e.g. a node is moved out of a repository)
	JobStateCancelled
)

type MemoryDatastore added in v1.34.0 

type MemoryDatastore struct {
	// contains filtered or unexported fields
}

MemoryDatastore is a simple datastore that isn't persisted to disk. It is only intended for early beta requirements and as a reference implementation for the eventual SQL implementation

func NewMemoryDatastore added in v1.34.0 

func NewMemoryDatastore(cfg config.Config) *MemoryDatastore

NewMemoryDatastore returns an initialized in-memory datastore

func (*MemoryDatastore) AddReplica added in v1.60.0 

func (md *MemoryDatastore) AddReplica(relativePath string, storageNodeID int) error

AddReplica adds a secondary to a repository of a repository relative path

func (*MemoryDatastore) CreateReplicaReplJobs added in v1.60.0 

func (md *MemoryDatastore) CreateReplicaReplJobs(relativePath string, change ChangeType) ([]uint64, error)

CreateReplicaReplJobs creates a replication job for each secondary that backs the specified repository. Upon success, the job IDs will be returned.

func (*MemoryDatastore) GetJobs added in v1.44.0 

func (md *MemoryDatastore) GetJobs(state JobState, targetNodeID int, count int) ([]ReplJob, error)

GetJobs is a more general method to retrieve jobs of a certain state from the datastore

func (*MemoryDatastore) GetPrimary added in v1.50.0 

func (md *MemoryDatastore) GetPrimary(relativePath string) (*models.Node, error)

GetPrimary gets the primary storage node for a repository of a repository relative path

func (*MemoryDatastore) GetReplicas added in v1.60.0 

func (md *MemoryDatastore) GetReplicas(relativePath string) ([]models.Node, error)

GetReplicas gets the secondaries for a repository based on the relative path

func (*MemoryDatastore) GetRepository added in v1.60.0 

func (md *MemoryDatastore) GetRepository(relativePath string) (*models.Repository, error)

GetRepository gets the repository for a repository relative path

func (*MemoryDatastore) GetStorageNode added in v1.60.0 

func (md *MemoryDatastore) GetStorageNode(nodeID int) (models.Node, error)

GetStorageNode gets all storage nodes

func (*MemoryDatastore) GetStorageNodes added in v1.60.0 

func (md *MemoryDatastore) GetStorageNodes() ([]models.Node, error)

GetStorageNodes gets all storage nodes

func (*MemoryDatastore) PickAPrimary added in v1.66.0 

func (md *MemoryDatastore) PickAPrimary() (*models.Node, error)

PickAPrimary returns the primary configured in the config file

func (*MemoryDatastore) RemoveReplica added in v1.60.0 

func (md *MemoryDatastore) RemoveReplica(relativePath string, storageNodeID int) error

RemoveReplica removes a secondary from a repository of a repository relative path

func (*MemoryDatastore) SetPrimary added in v1.50.0 

func (md *MemoryDatastore) SetPrimary(relativePath string, storageNodeID int) error

SetPrimary sets the primary storagee node for a repository of a repository relative path

func (*MemoryDatastore) UpdateReplJob added in v1.36.0 

func (md *MemoryDatastore) UpdateReplJob(jobID uint64, newState JobState) error

UpdateReplJob updates an existing replication job's state

type ReplJob added in v1.34.0 

type ReplJob struct {
	Change                 ChangeType
	ID                     uint64            // autoincrement ID
	TargetNode, SourceNode models.Node       // which node to replicate to?
	Repository             models.Repository // source for replication
	State                  JobState
}

ReplJob is an instance of a queued replication job. A replication job is meant for updating the repository so that it is synced with the primary copy. Scheduled indicates when a replication job should be performed.

type ReplJobsDatastore added in v1.34.0 

type ReplJobsDatastore interface {
	// GetJobs fetches a list of chronologically ordered replication
	// jobs for the given storage replica. The returned list will be at most
	// count-length.
	GetJobs(flag JobState, nodeID int, count int) ([]ReplJob, error)

	// CreateReplicaJobs will create replication jobs for each secondary
	// replica of a repository known to the datastore. A set of replication job
	// ID's for the created jobs will be returned upon success.
	CreateReplicaReplJobs(relativePath string, change ChangeType) ([]uint64, error)

	// UpdateReplJob updates the state of an existing replication job
	UpdateReplJob(jobID uint64, newState JobState) error
}

ReplJobsDatastore represents the behavior needed for fetching and updating replication jobs from the datastore

type ReplMgr added in v1.34.0 

type ReplMgr struct {
	// contains filtered or unexported fields
}

ReplMgr is a replication manager for handling replication jobs

func NewReplMgr added in v1.34.0 

func NewReplMgr(targetNode string, log *logrus.Entry, datastore Datastore, c *conn.ClientConnections, opts ...ReplMgrOpt) ReplMgr

NewReplMgr initializes a replication manager with the provided dependencies and options

func (ReplMgr) ProcessBacklog added in v1.34.0 

func (r ReplMgr) ProcessBacklog(ctx context.Context) error

ProcessBacklog will process queued jobs. It will block while processing jobs.

func (ReplMgr) ScheduleReplication added in v1.34.0 

func (r ReplMgr) ScheduleReplication(ctx context.Context, repo models.Repository) error

ScheduleReplication will store a replication job in the datastore for later execution. It filters out projects that are not whitelisted. TODO: add a parameter to delay replication

type ReplMgrOpt added in v1.34.0 

type ReplMgrOpt func(*ReplMgr)

ReplMgrOpt allows a replicator to be configured with additional options

func WithReplicator added in v1.34.0 

func WithReplicator(r Replicator) ReplMgrOpt

WithReplicator overrides the default replicator

func WithWhitelist added in v1.34.0 

func WithWhitelist(whitelistedRepos []string) ReplMgrOpt

WithWhitelist will configure a whitelist for repos to allow replication

type ReplicasDatastore added in v1.34.0 

type ReplicasDatastore interface {
	PickAPrimary() (*models.Node, error)

	GetReplicas(relativePath string) ([]models.Node, error)

	GetStorageNode(nodeID int) (models.Node, error)

	GetStorageNodes() ([]models.Node, error)

	GetPrimary(relativePath string) (*models.Node, error)

	SetPrimary(relativePath string, storageNodeID int) error

	AddReplica(relativePath string, storageNodeID int) error

	RemoveReplica(relativePath string, storageNodeID int) error

	GetRepository(relativePath string) (*models.Repository, error)
}

ReplicasDatastore manages accessing and setting which secondary replicas backup a repository

type Replicator added in v1.34.0 

type Replicator interface {
	// Replicate propagates changes from the source to the target
	Replicate(ctx context.Context, job ReplJob, source, target *grpc.ClientConn) error
	// Destroy will remove the target repo on the specified target connection
	Destroy(ctx context.Context, job ReplJob, target *grpc.ClientConn) error
}

Replicator performs the actual replication logic between two nodes

type Server 

type Server struct {
	// contains filtered or unexported fields
}

Server is a praefect server

func NewServer 

func NewServer(c *Coordinator, repl ReplMgr, grpcOpts []grpc.ServerOption, l *logrus.Entry, clientConnections *conn.ClientConnections, conf config.Config) *Server

NewServer returns an initialized praefect gPRC proxy server configured with the provided gRPC server options

func (*Server) Shutdown 

func (srv *Server) Shutdown(ctx context.Context) error

Shutdown will attempt a graceful shutdown of the grpc server. If unable to gracefully shutdown within the context deadline, it will then forcefully shutdown the server and return a context cancellation error.

func (*Server) Start 

func (srv *Server) Start(lis net.Listener) error

Start will start the praefect gRPC proxy server listening at the provided listener. Function will block until the server is stopped or an unrecoverable error occurs.

Source Files

View all Source files

Directories

Path Synopsis
grpc-proxy
proxy
Package proxy provides a reverse proxy handler for gRPC.
 Package proxy provides a reverse proxy handler for gRPC.
service
server

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.