pachd

Documentation

Overview

Package pachd implements the Pachyderm dæmon and its various modes.

Callers need only provide a context and a configuration.

Adding a new mode

The idea is that there is a builder for each mode; each builder is responsible for building and running an instance of the single type daemon representing a pachd instance.

To add a new mode one will at least add a new builder; one may also need to add new members to daemon. Daemon should contain only those members needed at run time for any mode; other, transient, values should be members of the pertinent builder.

Index

• func AdminEnv(senv serviceenv.ServiceEnv, paused bool) admin_server.Env
• func AuthEnv(senv serviceenv.ServiceEnv, txnEnv *txnenv.TransactionEnv) auth_server.Env
• func DebugEnv(env serviceenv.ServiceEnv) debug_server.Env
• func EnterpriseEnv(senv serviceenv.ServiceEnv, etcdPrefix string, txEnv *txnenv.TransactionEnv) *enterprise_server.Env
• func EnterpriseMode(ctx context.Context, config *pachconfig.EnterpriseServerConfiguration) error
• func FullMode(ctx context.Context, config *pachconfig.PachdFullConfiguration) error
• func LicenseEnv(senv serviceenv.ServiceEnv) *license_server.Env
• func NewTestPachd(t testing.TB, opts ...TestPachdOption) *client.APIClient
• func PFSEnv(env serviceenv.ServiceEnv, txnEnv *txnenv.TransactionEnv) (*pfs_server.Env, error)
• func PFSWorkerEnv(env serviceenv.ServiceEnv) (*pfs_server.WorkerEnv, error)
• func PPSEnv(senv serviceenv.ServiceEnv, txnEnv *txnenv.TransactionEnv, ...) pps_server.Env
• func PachwEnv(env serviceenv.ServiceEnv) (*pachw_server.Env, error)
• func PachwMode(ctx context.Context, config *pachconfig.PachdFullConfiguration) error
• func PausedMode(ctx context.Context, config *pachconfig.PachdFullConfiguration) error
• func SidecarMode(ctx context.Context, config *pachconfig.PachdFullConfiguration) error
• func StorageEnv(env serviceenv.ServiceEnv) (*storage.Env, error)
• type Env
• type Full
  • func BuildTestPachd(ctx context.Context, opts ...TestPachdOption) (*Full, *cleanup.Cleaner, error)
  • func NewFull(env Env, config pachconfig.PachdFullConfiguration, opt *FullOption) *Full
  • func (pd *Full) AwaitAuth(ctx context.Context)
  • func (pd *Full) PachClient(ctx context.Context) (*client.APIClient, error)
  • func (b *Full) Run(ctx context.Context) error
• type FullOption
• type PreFlight
  • func NewPreflight(env PreFlightEnv, config pachconfig.PachdPreflightConfiguration) *PreFlight
  • func (b *PreFlight) Run(ctx context.Context) error
• type PreFlightEnv
• type TestPachdOption
  • func ActivateAuthOption(rootToken string) TestPachdOption
  • func NoLogToFileOption() TestPachdOption

Functions

func AdminEnv

func AdminEnv(senv serviceenv.ServiceEnv, paused bool) admin_server.Env

func AuthEnv

func AuthEnv(senv serviceenv.ServiceEnv, txnEnv *txnenv.TransactionEnv) auth_server.Env

func DebugEnv

func DebugEnv(env serviceenv.ServiceEnv) debug_server.Env

func EnterpriseEnv

func EnterpriseEnv(senv serviceenv.ServiceEnv, etcdPrefix string, txEnv *txnenv.TransactionEnv) *enterprise_server.Env

func EnterpriseMode

func EnterpriseMode(ctx context.Context, config *pachconfig.EnterpriseServerConfiguration) error

EnterpriseMode runs an enterprise-mode pachd.

Enterprise mode is the enterprise server which is used to manage multiple Pachyderm installations.

func FullMode

func FullMode(ctx context.Context, config *pachconfig.PachdFullConfiguration) error

FullMode runs a full-mode pachd.

Full mode is that standard pachd which users interact with using pachctl and which manages pipelines, files and so forth.

func LicenseEnv

func LicenseEnv(senv serviceenv.ServiceEnv) *license_server.Env

func NewTestPachd

func NewTestPachd(t testing.TB, opts ...TestPachdOption) *client.APIClient

NewTestPachd creates an environment suitable for non-k8s tests and then calls pachd.NewFull with that environment.

func PFSEnv

func PFSEnv(env serviceenv.ServiceEnv, txnEnv *txnenv.TransactionEnv) (*pfs_server.Env, error)

func PFSWorkerEnv

func PFSWorkerEnv(env serviceenv.ServiceEnv) (*pfs_server.WorkerEnv, error)

func PPSEnv

func PPSEnv(senv serviceenv.ServiceEnv, txnEnv *txnenv.TransactionEnv, reporter *metrics.Reporter) pps_server.Env

func PachwEnv

func PachwEnv(env serviceenv.ServiceEnv) (*pachw_server.Env, error)

func PachwMode

func PachwMode(ctx context.Context, config *pachconfig.PachdFullConfiguration) error

PachwMode runs a pachw-mode pachd. When in pachw mode, the pachd instance processes storage and url tasks.

func PausedMode

func PausedMode(ctx context.Context, config *pachconfig.PachdFullConfiguration) error

PausedMode runs a paused-mode pachd.

Paused mode is a restricted mode which runs Pachyderm read-only in order to take offline backups.

func SidecarMode

func SidecarMode(ctx context.Context, config *pachconfig.PachdFullConfiguration) error

SidecarMode runs a sidecar-mode pachd.

Sidecar mode is run as a sidecar in a pipeline pod; it provides services to the pipeline worker code running in that pod.

func StorageEnv

func StorageEnv(env serviceenv.ServiceEnv) (*storage.Env, error)

Types

type Env

type Env struct {
	DB               *pachsql.DB
	DirectDB         *pachsql.DB
	DBListenerConfig string
	ObjClient        obj.Client
	Bucket           *obj.Bucket
	EtcdClient       *clientv3.Client
	Listener         net.Listener
	K8sObjects       []runtime.Object
	GetLokiClient    func() (*lokiclient.Client, error)
}

type Full

type Full struct {
	// contains filtered or unexported fields
}

func BuildTestPachd

func BuildTestPachd(ctx context.Context, opts ...TestPachdOption) (*Full, *cleanup.Cleaner, error)

BuildTestPachd returns a test pachd that can be run outside of tests. The returned cleanup handler frees all ephemeral resources associated with the instance.

func NewFull

func NewFull(env Env, config pachconfig.PachdFullConfiguration, opt *FullOption) *Full

NewFull sets up a new Full pachd and returns it.

func (*Full) AwaitAuth

func (pd *Full) AwaitAuth(ctx context.Context)

AwaitAuth returns when auth is ready. It must be called after Run() starts.

func (*Full) PachClient

func (pd *Full) PachClient(ctx context.Context) (*client.APIClient, error)

PachClient returns a pach client that can talk to the server with root privileges.

func (*Full) Run

func (b *Full) Run(ctx context.Context) error

Run runs all the setup steps serially, returning the first error encountered. Then all the background tasks are run concurrently, if any of them error then the others are cancelled, and Run returns the error.

type FullOption

type FullOption struct {
	DesiredState *migrations.State
}

type PreFlight

type PreFlight struct {
	// contains filtered or unexported fields
}

PreFlight is a minimal pachd for running preflight checks.

func NewPreflight

func NewPreflight(env PreFlightEnv, config pachconfig.PachdPreflightConfiguration) *PreFlight

func (*PreFlight) Run

func (b *PreFlight) Run(ctx context.Context) error

Run runs all the setup steps serially, returning the first error encountered. Then all the background tasks are run concurrently, if any of them error then the others are cancelled, and Run returns the error.

type PreFlightEnv

type PreFlightEnv struct {
	// DB should be a direct database connection.
	DB *sqlx.DB
}

type TestPachdOption

type TestPachdOption struct {
	MutateContext    func(ctx context.Context) context.Context
	MutateEnv        func(env *Env)
	MutateConfig     func(config *pachconfig.PachdFullConfiguration)
	MutatePachd      func(full *Full)
	MutateFullOption func(fullOption *FullOption)
	// contains filtered or unexported fields
}

TestPachdOptions allow a testpachd to be customized.

func ActivateAuthOption

func ActivateAuthOption(rootToken string) TestPachdOption

ActivateAuthOption is an option that activates auth inside the created pachd. Outside of tests, you must manually call pachd.AwaitAuth(ctx).

func NoLogToFileOption

func NoLogToFileOption() TestPachdOption

NoLogToFileOption is an option that disable's NewTestPachd's default behavior of logging pachd logs to a file.