pool

Documentation

Overview

Package pool implements resource pools for reflow. Reflow manages resources in units of "allocs" -- an a resource allocation that exists on a single machine, and to which is attached a shared repository with the results of all execs within that Alloc. Allocs are leased-- they must be kept alive to guarantee continuity; they are collected as a unit.

Index

• func Keepalive(ctx context.Context, log *log.Logger, alloc Alloc) error
• type Alloc
  • func Allocate(ctx context.Context, pool Pool, req reflow.Requirements, labels Labels) (Alloc, error)
  • func Allocs(ctx context.Context, pool Pool, log *log.Logger) []Alloc
• type AllocInspect
• type AllocMeta
• type Labels
  • func (l Labels) Add(k, v string) Labels
  • func (l Labels) Copy() Labels
• type Mux
  • func (m *Mux) Alloc(ctx context.Context, uri string) (Alloc, error)
  • func (m *Mux) Allocs(ctx context.Context) ([]Alloc, error)
  • func (m *Mux) ID() string
  • func (m *Mux) Offer(ctx context.Context, uri string) (Offer, error)
  • func (m *Mux) Offers(ctx context.Context) ([]Offer, error)
  • func (m *Mux) Pools() []Pool
  • func (m *Mux) SetPools(pools []Pool)
  • func (m *Mux) Size() int
• type Offer
  • func Pick(offers []Offer, min, max reflow.Resources) Offer
• type OfferJSON
• type Pool

Functions

func Keepalive

func Keepalive(ctx context.Context, log *log.Logger, alloc Alloc) error

Keepalive maintains the lease on alloc until it expires (e.g., by calling Free), or until the passed-in context is cancelled. Keepalive retries errors by exponential backoffs with a fixed configuration.

Types

type Alloc

type Alloc interface {
	reflow.Executor

	// Pool returns the pool from which the alloc is reserved.
	Pool() Pool

	// ID returns the ID of alloc in the pool. The format of the ID is opaque.
	ID() string

	// Keepalive maintains the lease of this Alloc. It must be called again
	// before the expiration of the returned duration. The user may also
	// request a maintenance interval. This is just a hint and may not be
	// respected by the Alloc.
	Keepalive(ctx context.Context, interval time.Duration) (time.Duration, error)

	// Inspect returns Alloc metadata.
	Inspect(ctx context.Context) (AllocInspect, error)

	// Free frees the alloc. Pending tasks are killed but its Repository
	// is not collected. Some implementations may implement "zombie"
	// allocs so that they can be inspected after Free is called.
	Free(ctx context.Context) error
}

Alloc represent a resource allocation attached to a single executor, a reservation of resources on a single node.

func Allocate

func Allocate(ctx context.Context, pool Pool, req reflow.Requirements, labels Labels) (Alloc, error)

Allocate attempts to place an Alloc on a pool with the given resource requirements.

func Allocs

func Allocs(ctx context.Context, pool Pool, log *log.Logger) []Alloc

Allocs fetches all of the allocs from the provided pool. If it encounters any failure (e.g., due to a context timeout), they are logged, but ignored. The returned slice contains all the successfuly fetched allocs.

type AllocInspect

type AllocInspect struct {
	ID            string
	Resources     reflow.Resources
	Meta          AllocMeta
	Created       time.Time
	LastKeepalive time.Time
	Expires       time.Time
}

AllocInspect contains Alloc metadata.

type AllocMeta

type AllocMeta struct {
	Want   reflow.Resources
	Owner  string
	Labels Labels
}

AllocMeta contains Alloc requester metadata.

type Labels

type Labels map[string]string

Labels represents a set of metadata labels for a run.

func (Labels) Add

func (l Labels) Add(k, v string) Labels

Add returns a copy of Labels l with an added key and value.

func (Labels) Copy

func (l Labels) Copy() Labels

Copy returns a copy of l.

type Mux

type Mux struct {
	// contains filtered or unexported fields
}

Mux is a Pool implementation that multiplexes and aggregates multiple underlying pools. Mux uses a URI naming scheme to address allocs and offers. Namely, the ID the underlying pool, followed by '/' and then the ID of the alloc or offer. For example, the URI

1.worker.us-west-2a.reflowy.eng.aws.grail.com:9000/4640204a5fd6ce42

Names the alloc with ID "4640204a5fd6ce42" of the pool named 1.worker.us-west-2a.reflowy.eng.aws.grail.com:9000.

func (*Mux) Alloc

func (m *Mux) Alloc(ctx context.Context, uri string) (Alloc, error)

Alloc returns an alloc named by a URI.

func (*Mux) Allocs

func (m *Mux) Allocs(ctx context.Context) ([]Alloc, error)

Allocs returns the current set of allocs over all underlying pools.

func (*Mux) ID

func (m *Mux) ID() string

ID returns the ID of this pool. It is always empty.

func (*Mux) Offer

func (m *Mux) Offer(ctx context.Context, uri string) (Offer, error)

Offer looks up the offer named by the given URI.

func (*Mux) Offers

func (m *Mux) Offers(ctx context.Context) ([]Offer, error)

Offers enumerates all the offers available from the underlying pools. Offers applies a timeout to the underlying requests; requests that do not meet the deadline are simply dropped.

func (*Mux) Pools

func (m *Mux) Pools() []Pool

Pools retrieves the Mux's underlying pools.

func (*Mux) SetPools

func (m *Mux) SetPools(pools []Pool)

SetPools sets the Mux's underlying pools.

func (*Mux) Size

func (m *Mux) Size() int

Size tells how many pools the Mux comprises.

type Offer

type Offer interface {
	// ID returns the ID of the offer. It is an opaque string.
	ID() string

	// Pool returns the pool from which this Offer is extended.
	Pool() Pool

	// Available returns the amount of total available resources
	// that can be accepted.
	Available() reflow.Resources

	// Accept accepts this Offer with the given Alloc metadata. The
	// metadata includes how many resources are requested. Accept may
	// return ErrOfferExpired if another client accepted the offer
	// first.
	Accept(ctx context.Context, meta AllocMeta) (Alloc, error)
}

Offer represents an offer of resources, from which an Alloc can be created.

func Pick

func Pick(offers []Offer, min, max reflow.Resources) Offer

Pick selects an offer best matching a minimum and maximum resource requirements. It picks the offer which has at least the minimum amount of resources but as close to maximum as possible.

type OfferJSON

type OfferJSON struct {
	// The ID of the offer.
	ID string
	// The amount of available resources the offer represents.
	Available reflow.Resources
}

OfferJSON is the JSON structure used to describe offers.

type Pool

type Pool interface {
	// ID returns the ID of the pool. It is an opaque string.
	ID() string

	// Alloc returns the Alloc named by an ID.
	Alloc(ctx context.Context, id string) (Alloc, error)

	// Allocs enumerates the available Allocs in this Pool.
	Allocs(ctx context.Context) ([]Alloc, error)

	// Offer returns the Offer identified by an id.
	Offer(ctx context.Context, id string) (Offer, error)

	// Offers returns the set of current Offers from this Pool.
	// TODO(marius): it would be good to have a scanning/long-poll
	// version of this so that clients do not have to do their own polling.
	Offers(ctx context.Context) ([]Offer, error)
}

Pool is a resource pool which manages a set of allocs.