pool

package
v0.0.0-...-2f2bd00 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 19, 2024 License: BSD-3-Clause Imports: 42 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var TestPoolHook func(*dashboard.HostConfig) Buildlet

TestPoolHook is used to override the buildlet returned by ForConf. It should only be used for testing purposes.

Functions

func GCENumCPU 

func GCENumCPU(machineType string) int

GCENumCPU returns the number of GCE CPUs used by the specified machine type.

func HandleReverse 

func HandleReverse(w http.ResponseWriter, r *http.Request)

HandleReverse handles reverse buildlet connections.

func HasScope 

func HasScope(want string) bool

HasScope returns true if the GCE metadata contains the default scopes.

func InitGCE 

func InitGCE(sc *secret.Client, basePin *atomic.Value, fn IsRemoteBuildletFunc, buildEnvName, mode string) error

InitGCE initializes the GCE buildlet pool.

func SetBuilderMasterKey 

func SetBuilderMasterKey(masterKey []byte)

SetBuilderMasterKey sets the builder master key used to generate keys used by the builders.

func StorageClient 

func StorageClient(ctx context.Context) (*storage.Client, error)

StorageClient retrieves the GCE storage client.

Types

type Buildlet 

type Buildlet interface {
	// GetBuildlet returns a new buildlet client.
	//
	// The hostType is the key into the dashboard.Hosts
	// map (such as "host-linux-bullseye"), NOT the buidler type
	// ("linux-386").
	//
	// Users of GetBuildlet must both call Client.Close when done
	// with the client as well as cancel the provided Context.
	GetBuildlet(ctx context.Context, hostType string, lg Logger, item *queue.SchedItem) (buildlet.Client, error)

	String() string // TODO(bradfitz): more status stuff
}

Buildlet defines an interface for a pool of buildlets.

func ForHost 

func ForHost(conf *dashboard.HostConfig) Buildlet

ForHost returns the appropriate buildlet depending on the host configuration that is passed it. The returned buildlet can be overridden for testing purposes by registering a test hook.

type ByCreationTime 

type ByCreationTime []ResourceTime

ByCreationTime provides the functionality to sort resource times by the time of creation.

func (ByCreationTime) Len 

func (s ByCreationTime) Len() int

func (ByCreationTime) Less 

func (s ByCreationTime) Less(i, j int) bool

func (ByCreationTime) Swap 

func (s ByCreationTime) Swap(i, j int)

type EC2Buildlet 

type EC2Buildlet struct {
	// contains filtered or unexported fields
}

EC2Buildlet manages a pool of AWS EC2 buildlets.

func EC2BuildetPool 

func EC2BuildetPool() *EC2Buildlet

EC2BuildetPool retrieves the package level EC2Buildlet pool set by the constructor.

TODO(golang.org/issues/38337) remove once a package level variable is no longer required by the main package.

func NewEC2Buildlet 

func NewEC2Buildlet(client *cloud.AWSClient, buildEnv *buildenv.Environment, hosts map[string]*dashboard.HostConfig, fn IsRemoteBuildletFunc, opts ...EC2Opt) (*EC2Buildlet, error)

NewEC2Buildlet creates a new EC2 buildlet pool used to create and manage the lifecycle of EC2 buildlets. Information about ARM64 instance types is retrieved before starting the pool. EC2 quota types are also retrieved before starting the pool. The pool will continuously poll for quotas which limit the resources that can be consumed by the pool. It will also periodically search for VMs which are no longer in use or are untracked by the pool in order to delete them.

func (*EC2Buildlet) Close 

func (eb *EC2Buildlet) Close()

Close stops the pollers used by the EC2Buildlet pool from running.

func (*EC2Buildlet) GetBuildlet 

func (eb *EC2Buildlet) GetBuildlet(ctx context.Context, hostType string, lg Logger, si *queue.SchedItem) (buildlet.Client, error)

GetBuildlet retrieves a buildlet client for a newly created buildlet.

func (*EC2Buildlet) QuotaStats 

func (eb *EC2Buildlet) QuotaStats() map[string]*queue.QuotaStats

func (*EC2Buildlet) String 

func (eb *EC2Buildlet) String() string

String gives a report of capacity usage for the EC2 buildlet pool.

func (*EC2Buildlet) WriteHTMLStatus 

func (eb *EC2Buildlet) WriteHTMLStatus(w io.Writer)

WriteHTMLStatus writes the status of the EC2 buildlet pool to an io.Writer.

type EC2Opt 

type EC2Opt func(*EC2Buildlet)

EC2Opt is optional configuration for the buildlet.

type EventTimeLogger 

type EventTimeLogger interface {
	LogEventTime(event string, optText ...string)
}

EventTimeLogger is the logging interface used to log an event at a point in time.

type GCEBuildlet 

type GCEBuildlet struct {
	// contains filtered or unexported fields
}

GCEBuildlet manages a pool of GCE buildlets.

func (*GCEBuildlet) CleanUpOldVMs 

func (p *GCEBuildlet) CleanUpOldVMs()

CleanUpOldVMs loops forever and periodically enumerates virtual machines and deletes those which have expired.

A VM is considered expired if it has a "delete-at" metadata attribute having a unix timestamp before the current time.

This is the safety mechanism to delete VMs which stray from the normal deleting process. VMs are created to run a single build and should be shut down by a controlling process. Due to various types of failures, they might get stranded. To prevent them from getting stranded and wasting resources forever, we instead set the "delete-at" metadata attribute on them when created to some time that's well beyond their expected lifetime.

func (*GCEBuildlet) GetBuildlet 

func (p *GCEBuildlet) GetBuildlet(ctx context.Context, hostType string, lg Logger, si *queue.SchedItem) (bc buildlet.Client, err error)

GetBuildlet retrieves a buildlet client for an available buildlet.

func (*GCEBuildlet) QuotaStats 

func (p *GCEBuildlet) QuotaStats() map[string]*queue.QuotaStats

func (*GCEBuildlet) SetEnabled 

func (p *GCEBuildlet) SetEnabled(enabled bool)

SetEnabled marks the buildlet pool as enabled.

func (*GCEBuildlet) String 

func (p *GCEBuildlet) String() string

func (*GCEBuildlet) WriteHTMLStatus 

func (p *GCEBuildlet) WriteHTMLStatus(w io.Writer)

WriteHTMLStatus writes the status of the buildlet pool to an io.Writer.

type GCEConfiguration 

type GCEConfiguration struct{}

GCEConfiguration manages and contains all of the GCE configuration.

func NewGCEConfiguration 

func NewGCEConfiguration() *GCEConfiguration

NewGCEConfiguration creates a new GCEConfiguration.

func (*GCEConfiguration) BuildEnv 

func (c *GCEConfiguration) BuildEnv() *buildenv.Environment

BuildEnv retrieves the GCE build env.

func (*GCEConfiguration) BuildletPool 

func (c *GCEConfiguration) BuildletPool() *GCEBuildlet

BuildletPool retrieves the GCE buildlet pool.

func (*GCEConfiguration) DSClient 

func (c *GCEConfiguration) DSClient() *datastore.Client

DSClient retrieves the datastore client.

func (*GCEConfiguration) ErrorsClient 

func (c *GCEConfiguration) ErrorsClient() *errorreporting.Client

ErrorsClient retrieves the stackdriver errors client.

func (*GCEConfiguration) GCPCredentials 

func (c *GCEConfiguration) GCPCredentials() *google.Credentials

GCPCredentials retrieves the GCP credentials.

func (*GCEConfiguration) GKENodeHostname 

func (c *GCEConfiguration) GKENodeHostname() string

GKENodeHostname retrieves the GKE node hostname.

func (*GCEConfiguration) GerritClient 

func (c *GCEConfiguration) GerritClient() *gerrit.Client

GerritClient retrieves a gerrit client.

func (*GCEConfiguration) GoDSClient 

func (c *GCEConfiguration) GoDSClient() *datastore.Client

GoDSClient retrieves the datastore client for golang.org project.

func (*GCEConfiguration) InStaging 

func (c *GCEConfiguration) InStaging() bool

InStaging returns a boolean denoting if the environment is staging.

func (*GCEConfiguration) OAuthHTTPClient 

func (c *GCEConfiguration) OAuthHTTPClient() *http.Client

OAuthHTTPClient retrieves an OAuth2 HTTP client used to make API calls to GCP.

func (*GCEConfiguration) SetBuildEnv 

func (c *GCEConfiguration) SetBuildEnv(b *buildenv.Environment)

SetBuildEnv sets the GCE build env. This is primarily reserved for testing purposes.

func (*GCEConfiguration) StorageClient 

func (c *GCEConfiguration) StorageClient() *storage.Client

StorageClient retrieves the GCE storage client.

func (*GCEConfiguration) TryDepsErr 

func (c *GCEConfiguration) TryDepsErr() error

TryDepsErr retrieves any Trybot dependency error.

type IsRemoteBuildletFunc 

type IsRemoteBuildletFunc func(instanceName string) bool

IsRemoteBuildletFunc should report whether the buildlet instance name is is a remote buildlet. This is applicable to GCE and EC2 instances.

TODO(golang.org/issue/38337): should be removed once remote buildlet management functions are moved into a package.

type Logger 

type Logger interface {
	EventTimeLogger // point in time
	spanlog.Logger  // action spanning time
}

Logger is the logging interface used within the coordinator. It can both log a message at a point in time, as well as log a span (something having a start and end time, as well as a final success status).

type Process 

type Process struct {
	// contains filtered or unexported fields
}

func CoordinatorProcess 

func CoordinatorProcess() *Process

CoordinatorProcess returns the package level process logger.

func SetProcessMetadata 

func SetProcessMetadata(id string, startTime time.Time) *Process

SetProcessMetadata sets metadata about the process. This should be called before any event logging operations.

func (*Process) PutBuildRecord 

func (p *Process) PutBuildRecord(br *types.BuildRecord)

func (*Process) PutSpanRecord 

func (p *Process) PutSpanRecord(sr *types.SpanRecord)

func (*Process) UpdateInstanceRecord 

func (p *Process) UpdateInstanceRecord()

type ProcessRecord 

type ProcessRecord struct {
	ID            string
	Start         time.Time
	LastHeartbeat time.Time
}

ProcessRecord is a datastore record about the lifetime of a coordinator process.

Example GQL query: SELECT * From Process where LastHeartbeat > datetime("2016-01-01T00:00:00Z")

type ResourceTime 

type ResourceTime struct {
	Name     string
	Creation time.Time
}

ResourceTime is a GCE instance or Kube pod name and its creation time.

type ReverseBuildletPool 

type ReverseBuildletPool struct {
	// contains filtered or unexported fields
}

ReverseBuildletPool manages the pool of reverse buildlet pools.

func ReversePool 

func ReversePool() *ReverseBuildletPool

ReversePool retrieves the reverse buildlet pool.

func (*ReverseBuildletPool) BuildletHostnames 

func (p *ReverseBuildletPool) BuildletHostnames() []string

BuildletHostnames returns a slice of reverse buildlet hostnames.

func (*ReverseBuildletPool) BuildletLastSeen 

func (p *ReverseBuildletPool) BuildletLastSeen(host string) (time.Time, bool)

BuildletLastSeen gives the last time a buildlet was connected to the pool. If the buildlet has not been seen a false is returned by the boolean.

func (*ReverseBuildletPool) CanBuild 

func (p *ReverseBuildletPool) CanBuild(hostType string) bool

CanBuild reports whether the pool has a machine capable of building mode, even if said machine isn't currently idle.

func (*ReverseBuildletPool) GetBuildlet 

func (p *ReverseBuildletPool) GetBuildlet(ctx context.Context, hostType string, lg Logger, si *queue.SchedItem) (buildlet.Client, error)

GetBuildlet builds a buildlet client for the passed in host.

func (*ReverseBuildletPool) HostTypeCount 

func (p *ReverseBuildletPool) HostTypeCount() map[string]int

HostTypeCount iterates through the running reverse buildlets, and constructs a count of running buildlets per hostType.

func (*ReverseBuildletPool) HostTypes 

func (p *ReverseBuildletPool) HostTypes() (types []string)

HostTypes returns a sorted, deduplicated list of buildlet types currently supported by the pool.

func (*ReverseBuildletPool) QuotaStats 

func (p *ReverseBuildletPool) QuotaStats() map[string]*queue.QuotaStats

func (*ReverseBuildletPool) SingleHostTypeCount 

func (p *ReverseBuildletPool) SingleHostTypeCount(hostType string) int

SingleHostTypeCount iterates through the running reverse buildlets, and constructs a count of the running buildlet hostType requested.

func (*ReverseBuildletPool) String 

func (p *ReverseBuildletPool) String() string

func (*ReverseBuildletPool) WriteHTMLStatus 

func (p *ReverseBuildletPool) WriteHTMLStatus(w io.Writer)

WriteHTMLStatus writes a status of the reverse buildlet pool, in HTML format, to the passed in io.Writer.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.