dbspgraph

Documentation

Index

• Variables
• type Master
  • func NewMaster(cfg MasterConfig) (*Master, error)
  • func (m *Master) Close() error
  • func (m *Master) RunJob(ctx context.Context, minWorkers int, workerAcquireTimeout time.Duration) error
  • func (m *Master) Start() error
• type MasterConfig
  • func (cfg *MasterConfig) Validate() error
• type Serializer
• type Worker
  • func NewWorker(cfg WorkerConfig) (*Worker, error)
  • func (w *Worker) Close() error
  • func (w *Worker) Dial(masterEndpoint string, dialTimeout time.Duration) error
  • func (w *Worker) RunJob(ctx context.Context) error
• type WorkerConfig
  • func (cfg *WorkerConfig) Validate() error

Variables

var (

	// ErrUnableToReserveWorkers is returned by the master to indicate that
	// the required number of workers for running a job is not available.
	ErrUnableToReserveWorkers = xerrors.Errorf("unable to reserve required number of workers")
)

Types

type Master

type Master struct {
	// contains filtered or unexported fields
}

Master orchestrates the execution of a distributed graph-based algorithm built on top of the bspgraph framework across multiple workers.

func NewMaster

func NewMaster(cfg MasterConfig) (*Master, error)

NewMaster creates a new Master instance with the specified configuration.

func (*Master) Close

func (m *Master) Close() error

Close disconnects any connected workers and shuts down the gRPC server.

func (*Master) RunJob

func (m *Master) RunJob(ctx context.Context, minWorkers int, workerAcquireTimeout time.Duration) error

RunJob creates a new job and coordinates its execution until the job completes, the context expires or some error occurs. The minWorkers parameter defines the minimum number of connected workers required for the job. It may be set to 0 to reserve all workers currently available. If the required number of workers is not available, RunJob blocks until either enough workers connect, or the workerAcquireTimeout (if non-zero) expires or if the provided context expires.

func (*Master) Start

func (m *Master) Start() error

Start listening on the configured address for incoming worker connections. Calls to Start are non-blocking. The caller must invoke the Close method to shutdown the server and clean up any reserved resources.

type MasterConfig

type MasterConfig struct {
	// The address where the master will listen for incoming gRPC
	// connections from workers.
	ListenAddress string

	// JobRunner
	JobRunner job.Runner

	// A helper for serializing and unserializing aggregator values.
	Serializer Serializer

	// A logger instance to use. If not specified, a null logger will be
	// used instead.
	Logger *logrus.Entry
}

MasterConfig encapsulates the configuration options for a master node.

func (*MasterConfig) Validate

func (cfg *MasterConfig) Validate() error

Validate the config options.

type Serializer

type Serializer interface {
	// Serialize encodes the given value into an any.Any protobuf message.
	Serialize(interface{}) (*any.Any, error)

	// Unserialize decodes the given any.Any protobuf value.
	Unserialize(*any.Any) (interface{}, error)
}

Serializer is implemented by types that can serialize aggregator and graph messages from and to an any.Any value.

type Worker

type Worker struct {
	// contains filtered or unexported fields
}

Worker coordinates the execution of a distributed graph-based algorithm built on top of the bspgraph framework with a remote master node.

func NewWorker

func NewWorker(cfg WorkerConfig) (*Worker, error)

NewWorker creates a new Worker instance with the specified configuration.

func (*Worker) Close

func (w *Worker) Close() error

Close shuts down the worker.

func (*Worker) Dial

func (w *Worker) Dial(masterEndpoint string, dialTimeout time.Duration) error

Dial establishes a connection to the master node.

func (*Worker) RunJob

func (w *Worker) RunJob(ctx context.Context) error

RunJob waits for a new job announcement from the master and coordinates its execution with the master until it either completes or is aborted due to a context expiration or a local/remote error.

type WorkerConfig

type WorkerConfig struct {
	// JobRunner
	JobRunner job.Runner

	// A helper for serializing and unserializing aggregator values and
	// vertex messages to/from protocol buffer messages.
	Serializer Serializer

	// A logger instance to use. If not specified, a null logger will be
	// used instead.
	Logger *logrus.Entry
}

WorkerConfig encapsulates the configuration options for a worker node.

func (*WorkerConfig) Validate

func (cfg *WorkerConfig) Validate() error

Validate the config options.