awsmgr

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 5, 2022 License: MIT Imports: 27 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type AwsInstance 

type AwsInstance struct {
	Region   string
	Instance *types.Instance
}

AwsInstance describes a running AWS EC2 instance

type AwsLaunchTemplate 

type AwsLaunchTemplate struct {
	Region       string `json:"region"`
	TemplateID   string `json:"id"`
	Description  string `json:"description"`
	InstanceType string `json:"instanceType"`
	VCPU         string `json:"vCPU"`
	VCPUCount    int32
	RAM          string `json:"ram"`
	Bandwidth    string `json:"bandwidth"`
}

AwsLaunchTemplate describes an AWS EC2 launch template

type AwsManager 

type AwsManager struct {
	Enabled bool
	// contains filtered or unexported fields
}

AwsManager is the main type for managing AWS resources from the coordinator

func NewAwsManager 

func NewAwsManager() *AwsManager

NewAwsManager creates a new AwsManager instance

func (*AwsManager) CancelSpotRequests 

func (awsm *AwsManager) CancelSpotRequests(
	clt *ec2.Client,
	spotRequestTag string,
) error

TerminateSpotRequests will search for spot request(s) with the given tag and cancel them

func (*AwsManager) DownloadFromS3 

func (am *AwsManager) DownloadFromS3(d common.S3Download) error

DownloadFromS3 downloads an object from an S3 bucket

func (*AwsManager) DownloadMultipleFromS3 

func (am *AwsManager) DownloadMultipleFromS3(
	downloads []common.S3Download,
) error

DownloadMultipleFromS3 will run multiple downloads in parallel to speed up the process when downloading many files. The number of CPUs in the system is used as a parallelism limit. If an error occurred with one of the downloads, the function will return the first error that occurred.

func (*AwsManager) FileExistsOnS3 

func (am *AwsManager) FileExistsOnS3(
	region, bucket, path string,
) (bool, error)

FileExistsOnS3 checks if a file in the given region, bucket and path exists in S3. The return value indicates existence (true/false). In case an error occurs during checking the file's existence, it will be returned

func (*AwsManager) ForceRefreshLaunchTemplates 

func (am *AwsManager) ForceRefreshLaunchTemplates()

ForceRefreshLaunchTemplates is a method to force refreshing the launch templates. This is currently unused but could be hooked up to a REST API to allow refreshing this by force from the UI

func (*AwsManager) ForceRefreshSubnets 

func (am *AwsManager) ForceRefreshSubnets()

ForceRefreshSubnets is a method to force refreshing the subnets This is currently unused but could be hooked up to a REST API to allow refreshing this by force from the UI

func (*AwsManager) GenerateSeed 

func (am *AwsManager) GenerateSeed(seed ShardSeed) error

func (*AwsManager) GetAllRegions 

func (am *AwsManager) GetAllRegions() []string

GetAllRegions returns a list of available regions. This now returns the hard coded array, but in the future the array could be periodically re-read using whatever discovery method we find for the regions we want enabled.

func (*AwsManager) GetAvailableSeeds 

func (am *AwsManager) GetAvailableSeeds() []*ShardSeed

func (*AwsManager) GetLaunchTemplate 

func (am *AwsManager) GetLaunchTemplate(
	templateID string,
) (AwsLaunchTemplate, error)

GetLaunchTemplate returns a single launch template identified by its ID or an error if it was not found in the set of available launch templates

func (*AwsManager) GetLaunchTemplateRegion 

func (am *AwsManager) GetLaunchTemplateRegion(launchTemplateID string) string

GetLaunchTemplateRegion will return the AWS region for a given launch template

func (*AwsManager) GetSubnetsForRegion 

func (am *AwsManager) GetSubnetsForRegion(region string) []AwsSubnet

GetSubnetsForRegion returns the subnets available in the given region

func (*AwsManager) GetVCPULimit 

func (am *AwsManager) GetVCPULimit(key string) int32

GetVCPULimit will return the limit of number of VCPUs for a certain `key` where key is `<region>-[spot|ondem]`, for instance 'us-east-1-spot' describes how many vcpus we can launch in spot markets in region us-east-1

func (*AwsManager) HasSeed 

func (am *AwsManager) HasSeed(seed ShardSeed, pending bool) (bool, error)

func (*AwsManager) KillAllInstances 

func (am *AwsManager) KillAllInstances() error

KillAllInstances will terminate all known running instances in EC2

func (*AwsManager) LaunchTemplates 

func (am *AwsManager) LaunchTemplates() []AwsLaunchTemplate

LaunchTemplates will return the available launch templates or an empty array if AWS was not enabled

func (*AwsManager) ListObjectsInS3 

func (am *AwsManager) ListObjectsInS3(
	region, bucket, prefix string,
) ([]string, error)

ListObjectsInS3 will scan a bucket with a particular prefix and return all the objects that match the prefix

func (*AwsManager) ReadFromS3 

func (am *AwsManager) ReadFromS3(
	d common.S3Download,
	tail int,
) ([]byte, error)

ReadFromS3 fetches an object as byte slice from an S3 bucket. If tail is -1 it iwll read the entire object - if it is higher than -1 it will only read the last tail bytes of the object

func (*AwsManager) RunEC2ForAllRegions 

func (am *AwsManager) RunEC2ForAllRegions(
	f func(e *ec2.Client, r string) error,
) error

RunEC2ForAllRegions will execute a function for each enabled region, passing in an EC2 client configured for that region and wait for the function to complete for all regions. If an error occurs for any region, it is returned to the caller

func (*AwsManager) RunSQForAllRegions 

func (am *AwsManager) RunSQForAllRegions(
	f func(e *servicequotas.Client, r string) error,
) error

RunSQForAllRegions will execute a function for each enabled region, passing in an SQ client configured for that region and wait for the function to complete for all regions. If an error occurs for any region, it is returned to the caller

func (*AwsManager) RunningInstances 

func (am *AwsManager) RunningInstances() []*AwsInstance

RunningInstances returns the currently running instances in EC2

func (*AwsManager) StartNewAgents 

func (am *AwsManager) StartNewAgents(
	templateIDs []string,
	testRunID string,
) ([]*AwsInstance, []error)

StartNewAgents is the main logic that spawns our new instances. We pass it an array of templateIDs to launch. For each instance we want to launch there is one entry in the array. So there can be many entries of the same launch template in this array - each entry in the list will result in an instance launched. The return array will contain the instances that we launched in the exact same sequence as the templateIDs that were passed.

func (*AwsManager) StopAgents 

func (am *AwsManager) StopAgents(a []*AwsInstance) error

StopAgents will terminate the EC2 instances by the instance objects passed

func (*AwsManager) StopAgentsByInstanceIds 

func (am *AwsManager) StopAgentsByInstanceIds(ids []string) error

StopAgentsByInstanceIds will terminate the EC2 instances specified in the ids array. It will enumerate the runningInstances and match their ID to the IDs in the passed array, and then call StopAgents()

func (*AwsManager) UploadToS3 

func (am *AwsManager) UploadToS3(d common.S3Upload) error

UploadToS3 will upload a local file to a bucket and key in S3

func (*AwsManager) UploadToS3IfNotExists 

func (am *AwsManager) UploadToS3IfNotExists(d common.S3Upload) error

UploadToS3IfNotExists will check a file's existence and skip uploading if the file already exists

type AwsSubnet 

type AwsSubnet struct {
	Region   string `json:"region"`
	SubnetID string `json:"id"`
	AZ       string `json:"az"`
	Name     string `json:"name"`
}

AwsSubnet describes an AWS EC2 subnet

type ShardSeed 

type ShardSeed struct {
	CommitHash string `json:"commitHash"`
	SeedMode   int    `json:"mode"`
	Shards     int    `json:"shards"`
	Outputs    int    `json:"outputs"`
	TestRunID  string
	// contains filtered or unexported fields
}

func (*ShardSeed) Equals 

func (s *ShardSeed) Equals(s2 *ShardSeed) bool

type StartAgent 

type StartAgent struct {
	TemplateID  string
	Count       int
	InstanceIDs []string
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.