realis

package module
v1.3.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 8, 2019 License: Apache-2.0 Imports: 23 Imported by: 6

README

gorealis GoDoc Build Status codecov

Go library for interacting with Apache Aurora.

Aurora version compatibility

Please see .auroraversion to see the latest Aurora version against which this library has been tested.

Usage

Projects using gorealis

Contributions

Contributions are always welcome. Please raise an issue to discuss a contribution before it is made.

Documentation

Overview

Collection of monitors to create synchronicity

Package realis provides the ability to use Thrift API to communicate with Apache Aurora.

Index

Constants

View Source 
const (
	UpdateFailed = "update failed"
	RolledBack   = "update rolled back"
	Timeout      = "timeout"
)
View Source 
const VERSION = "1.3.1"

Variables

This section is empty.

Functions

func ExponentialBackoff added in v1.1.0 

func ExponentialBackoff(backoff Backoff, logger Logger, condition ConditionFunc) error

func GetCerts added in v1.2.0 

func GetCerts(certpath string) (*x509.CertPool, error)

func IsTemporary added in v1.2.0 

func IsTemporary(err error) bool

func IsTimeout added in v1.2.0 

func IsTimeout(err error) bool

func Jitter added in v1.1.0 

func Jitter(duration time.Duration, maxFactor float64) time.Duration

Jitter returns a time.Duration between duration and duration + maxFactor * duration.

This allows clients to avoid converging on periodic behavior. If maxFactor is 0.0, a suggested default value will be chosen.

func LeaderFromZK 

func LeaderFromZK(cluster Cluster) (string, error)

Retrieves current Aurora leader from ZK.

func LeaderFromZKOpts added in v1.3.0 

func LeaderFromZKOpts(options ...ZKOpt) (string, error)

Retrieves current Aurora leader from ZK with a custom configuration.

func LoadClusters 

func LoadClusters(config string) (map[string]Cluster, error)

Loads clusters.json file traditionally located at /etc/aurora/clusters.json

func NewUpdateSettings added in v1.1.0 

func NewUpdateSettings() *aurora.JobUpdateSettings

func ToRetryCount added in v1.3.0 

func ToRetryCount(err error) *retryErr

Helper function for testing verification to avoid whitebox testing as well as keeping retryErr as a private. Should NOT be used under any other context.

Types

type AuroraJob 

type AuroraJob struct {
	// contains filtered or unexported fields
}

Structure to collect all information pertaining to an Aurora job.

func (*AuroraJob) AddDedicatedConstraint added in v1.3.1 

func (j *AuroraJob) AddDedicatedConstraint(role, name string) Job

func (*AuroraJob) AddLabel 

func (j *AuroraJob) AddLabel(key string, value string) Job

Adds a Mesos label to the job. Note that Aurora will add the prefix "org.apache.aurora.metadata." to the beginning of each key.

func (*AuroraJob) AddLimitConstraint 

func (j *AuroraJob) AddLimitConstraint(name string, limit int32) Job

From Aurora Docs: A constraint that specifies the maximum number of active tasks on a host with a matching attribute that may be scheduled simultaneously.

func (*AuroraJob) AddNamedPorts 

func (j *AuroraJob) AddNamedPorts(names ...string) Job

Add a named port to the job configuration These are random ports as it's not currently possible to request specific ports using Aurora.

func (*AuroraJob) AddPorts 

func (j *AuroraJob) AddPorts(num int) Job

Adds a request for a number of ports to the job configuration. The names chosen for these ports will be org.apache.aurora.port.X, where X is the current port count for the job configuration starting at 0. These are random ports as it's not currently possible to request specific ports using Aurora.

func (*AuroraJob) AddURIs 

func (j *AuroraJob) AddURIs(extract bool, cache bool, values ...string) Job

Add a list of URIs with the same extract and cache configuration. Scheduler must have --enable_mesos_fetcher flag enabled. Currently there is no duplicate detection.

func (*AuroraJob) AddValueConstraint 

func (j *AuroraJob) AddValueConstraint(name string, negated bool, values ...string) Job

From Aurora Docs: Add a Value constraint name - Mesos slave attribute that the constraint is matched against. If negated = true , treat this as a 'not' - to avoid specific values. Values - list of values we look for in attribute name

func (*AuroraJob) CPU 

func (j *AuroraJob) CPU(cpus float64) Job

func (*AuroraJob) Container added in v1.0.2 

func (j *AuroraJob) Container(container Container) Job

Set a container to run for the job configuration to run.

func (*AuroraJob) CronCollisionPolicy 

func (j *AuroraJob) CronCollisionPolicy(policy aurora.CronCollisionPolicy) Job

func (*AuroraJob) CronSchedule 

func (j *AuroraJob) CronSchedule(cron string) Job

func (*AuroraJob) Disk 

func (j *AuroraJob) Disk(disk int64) Job

func (*AuroraJob) Environment 

func (j *AuroraJob) Environment(env string) Job

Set Job Key environment.

func (*AuroraJob) ExecutorData 

func (j *AuroraJob) ExecutorData(data string) Job

Will be included as part of entire task inside the scheduler that will be serialized.

func (*AuroraJob) ExecutorName 

func (j *AuroraJob) ExecutorName(name string) Job

Set name of the executor that will the task will be configured to.

func (*AuroraJob) GPU added in v1.3.1 

func (j *AuroraJob) GPU(gpus int64) Job

func (*AuroraJob) GetInstanceCount 

func (j *AuroraJob) GetInstanceCount() int32

How many instances of the job to run

func (*AuroraJob) InstanceCount 

func (j *AuroraJob) InstanceCount(instCount int32) Job

How many instances of the job to run

func (*AuroraJob) IsService 

func (j *AuroraJob) IsService(isService bool) Job

Restart the job's tasks if they fail

func (*AuroraJob) JobConfig 

func (j *AuroraJob) JobConfig() *aurora.JobConfiguration

Get the current job configurations key to use for some realis calls.

func (*AuroraJob) JobKey 

func (j *AuroraJob) JobKey() *aurora.JobKey

Get the current job configurations key to use for some realis calls.

func (*AuroraJob) MaxFailure 

func (j *AuroraJob) MaxFailure(maxFail int32) Job

How many failures to tolerate before giving up.

func (*AuroraJob) Name 

func (j *AuroraJob) Name(name string) Job

Set Job Key Name.

func (*AuroraJob) PartitionPolicy added in v1.3.1 

func (j *AuroraJob) PartitionPolicy(policy *aurora.PartitionPolicy) Job

Set a partition policy for the job configuration to implement.

func (*AuroraJob) RAM 

func (j *AuroraJob) RAM(ram int64) Job

func (*AuroraJob) Role 

func (j *AuroraJob) Role(role string) Job

Set Job Key Role.

func (*AuroraJob) TaskConfig 

func (j *AuroraJob) TaskConfig() *aurora.TaskConfig

type Backoff added in v1.1.0 

type Backoff struct {
	Duration time.Duration // the base duration
	Factor   float64       // Duration is multipled by factor each iteration
	Jitter   float64       // The amount of jitter applied each iteration
	Steps    int           // Exit with error after this many steps
}

type ClientOption added in v1.1.0 

type ClientOption func(*RealisConfig)

func BackOff added in v1.1.0 

func BackOff(b Backoff) ClientOption

func BasicAuth added in v1.1.0 

func BasicAuth(username, password string) ClientOption

Config sets for options in RealisConfig.

func Certspath added in v1.1.0 

func Certspath(certspath string) ClientOption

func ClientCerts added in v1.1.0 

func ClientCerts(clientKey, clientCert string) ClientOption

func Debug added in v1.3.0 

func Debug() ClientOption

Enable debug statements.

func InsecureSkipVerify added in v1.1.0 

func InsecureSkipVerify(InsecureSkipVerify bool) ClientOption

func Retries added in v1.1.0 

func Retries(backoff Backoff) ClientOption

func SchedulerUrl added in v1.1.0 

func SchedulerUrl(url string) ClientOption

func SetLogger added in v1.1.0 

func SetLogger(l Logger) ClientOption

Using the word set to avoid name collision with Interface.

func ThriftBinary added in v1.1.0 

func ThriftBinary() ClientOption

func ThriftJSON added in v1.1.0 

func ThriftJSON() ClientOption

func TimeoutMS added in v1.1.0 

func TimeoutMS(timeout int) ClientOption

func ZKCluster added in v1.1.0 

func ZKCluster(cluster *Cluster) ClientOption

func ZKUrl added in v1.1.0 

func ZKUrl(url string) ClientOption

func ZookeeperOptions added in v1.3.0 

func ZookeeperOptions(opts ...ZKOpt) ClientOption

Use this option if you'd like to override default settings for connecting to Zookeeper. See zk.go for what is possible to set as an option.

type Cluster 

type Cluster struct {
	Name          string `json:"name"`
	AgentRoot     string `json:"slave_root"`
	AgentRunDir   string `json:"slave_run_directory"`
	ZK            string `json:"zk"`
	ZKPort        int    `json:"zk_port"`
	SchedZKPath   string `json:"scheduler_zk_path"`
	SchedURI      string `json:"scheduler_uri"`
	ProxyURL      string `json:"proxy_url"`
	AuthMechanism string `json:"auth_mechanism"`
}

func GetDefaultClusterFromZKUrl added in v1.1.0 

func GetDefaultClusterFromZKUrl(zkurl string) *Cluster

type ConditionFunc added in v1.1.0 

type ConditionFunc func() (done bool, err error)

ConditionFunc returns true if the condition is satisfied, or an error if the loop should be aborted.

type Container added in v1.0.2 

type Container interface {
	Build() *aurora.Container
}

type DockerContainer added in v1.0.2 

type DockerContainer struct {
	// contains filtered or unexported fields
}

func NewDockerContainer added in v1.0.2 

func NewDockerContainer() DockerContainer

func (DockerContainer) AddParameter added in v1.0.2 

func (c DockerContainer) AddParameter(name, value string) DockerContainer

func (DockerContainer) Build added in v1.0.2 

func (c DockerContainer) Build() *aurora.Container

func (DockerContainer) Image added in v1.0.2 

func (c DockerContainer) Image(image string) DockerContainer

type Endpoint 

type Endpoint struct {
	Host string `json:"host"`
	Port int    `json:"port"`
}

type Job 

type Job interface {
	// Set Job Key environment.
	Environment(env string) Job
	Role(role string) Job
	Name(name string) Job
	CPU(cpus float64) Job
	CronSchedule(cron string) Job
	CronCollisionPolicy(policy aurora.CronCollisionPolicy) Job
	Disk(disk int64) Job
	RAM(ram int64) Job
	ExecutorName(name string) Job
	ExecutorData(data string) Job
	AddPorts(num int) Job
	AddLabel(key string, value string) Job
	AddNamedPorts(names ...string) Job
	AddLimitConstraint(name string, limit int32) Job
	AddValueConstraint(name string, negated bool, values ...string) Job

	// From Aurora Docs:
	// dedicated attribute. Aurora treats this specially, and only allows matching jobs
	// to run on these machines, and will only schedule matching jobs on these machines.
	// When a job is created, the scheduler requires that the $role component matches
	// the role field in the job configuration, and will reject the job creation otherwise.
	// A wildcard (*) may be used for the role portion of the dedicated attribute, which
	// will allow any owner to elect for a job to run on the host(s)
	AddDedicatedConstraint(role, name string) Job
	AddURIs(extract bool, cache bool, values ...string) Job
	JobKey() *aurora.JobKey
	JobConfig() *aurora.JobConfiguration
	TaskConfig() *aurora.TaskConfig
	IsService(isService bool) Job
	InstanceCount(instCount int32) Job
	GetInstanceCount() int32
	MaxFailure(maxFail int32) Job
	Container(container Container) Job
	PartitionPolicy(policy *aurora.PartitionPolicy) Job
}

func NewJob 

func NewJob() Job

Create a Job object with everything initialized.

type LevelLogger added in v1.3.0 

type LevelLogger struct {
	Logger
	// contains filtered or unexported fields
}

func (LevelLogger) DebugPrint added in v1.3.0 

func (l LevelLogger) DebugPrint(a ...interface{})

func (LevelLogger) DebugPrintf added in v1.3.0 

func (l LevelLogger) DebugPrintf(format string, a ...interface{})

func (LevelLogger) DebugPrintln added in v1.3.0 

func (l LevelLogger) DebugPrintln(a ...interface{})

func (*LevelLogger) EnableDebug added in v1.3.0 

func (l *LevelLogger) EnableDebug(enable bool)

type Logger added in v1.1.0 

type Logger interface {
	Println(v ...interface{})
	Printf(format string, v ...interface{})
	Print(v ...interface{})
}

type MesosContainer added in v1.0.2 

type MesosContainer struct {
	// contains filtered or unexported fields
}

func NewMesosContainer added in v1.1.0 

func NewMesosContainer() MesosContainer

func (MesosContainer) AppcImage added in v1.1.0 

func (c MesosContainer) AppcImage(name, imageId string) MesosContainer

func (MesosContainer) Build added in v1.1.0 

func (c MesosContainer) Build() *aurora.Container

func (MesosContainer) DockerImage added in v1.1.0 

func (c MesosContainer) DockerImage(name, tag string) MesosContainer

type Monitor 

type Monitor struct {
	Client Realis
}

func (*Monitor) HostMaintenance added in v1.1.0 

func (m *Monitor) HostMaintenance(hosts []string, modes []aurora.MaintenanceMode, interval, timeout int) (map[string]bool, error)

Monitor host status until all hosts match the status provided. Returns a map where the value is true if the host is in one of the desired mode(s) or false if it is not as of the time when the monitor exited.

func (*Monitor) Instances 

func (m *Monitor) Instances(key *aurora.JobKey, instances int32, interval, timeout int) (bool, error)

Monitor a Job until all instances enter one of the LIVE_STATES

func (*Monitor) JobUpdate 

func (m *Monitor) JobUpdate(updateKey aurora.JobUpdateKey, interval int, timeout int) (bool, error)

Polls the scheduler every certain amount of time to see if the update has succeeded

func (*Monitor) ScheduleStatus added in v1.2.0 

func (m *Monitor) ScheduleStatus(key *aurora.JobKey, instanceCount int32, desiredStatuses map[aurora.ScheduleStatus]bool, interval, timeout int) (bool, error)

Monitor a Job until all instances enter a desired status. Defaults sets of desired statuses provided by the thrift API include: ACTIVE_STATES, SLAVE_ASSIGNED_STATES, LIVE_STATES, and TERMINAL_STATES

type NoopLogger added in v1.0.4 

type NoopLogger struct{}

func (NoopLogger) Print added in v1.1.0 

func (NoopLogger) Print(a ...interface{})

func (NoopLogger) Printf added in v1.0.4 

func (NoopLogger) Printf(format string, a ...interface{})

func (NoopLogger) Println added in v1.1.0 

func (NoopLogger) Println(a ...interface{})

type Realis 

type Realis interface {
	AbortJobUpdate(updateKey aurora.JobUpdateKey, message string) (*aurora.Response, error)
	AddInstances(instKey aurora.InstanceKey, count int32) (*aurora.Response, error)
	CreateJob(auroraJob Job) (*aurora.Response, error)
	CreateService(auroraJob Job, settings *aurora.JobUpdateSettings) (*aurora.Response, *aurora.StartJobUpdateResult_, error)
	DescheduleCronJob(key *aurora.JobKey) (*aurora.Response, error)
	FetchTaskConfig(instKey aurora.InstanceKey) (*aurora.TaskConfig, error)
	GetInstanceIds(key *aurora.JobKey, states map[aurora.ScheduleStatus]bool) (map[int32]bool, error)
	GetJobUpdateSummaries(jobUpdateQuery *aurora.JobUpdateQuery) (*aurora.Response, error)
	GetTaskStatus(query *aurora.TaskQuery) ([]*aurora.ScheduledTask, error)
	GetTasksWithoutConfigs(query *aurora.TaskQuery) ([]*aurora.ScheduledTask, error)
	GetJobs(role string) (*aurora.Response, *aurora.GetJobsResult_, error)
	GetPendingReason(query *aurora.TaskQuery) (pendingReasons []*aurora.PendingReason, e error)
	JobUpdateDetails(updateQuery aurora.JobUpdateQuery) (*aurora.Response, error)
	KillJob(key *aurora.JobKey) (*aurora.Response, error)
	KillInstances(key *aurora.JobKey, instances ...int32) (*aurora.Response, error)
	RemoveInstances(key *aurora.JobKey, count int32) (*aurora.Response, error)
	RestartInstances(key *aurora.JobKey, instances ...int32) (*aurora.Response, error)
	RestartJob(key *aurora.JobKey) (*aurora.Response, error)
	RollbackJobUpdate(key aurora.JobUpdateKey, message string) (*aurora.Response, error)
	ScheduleCronJob(auroraJob Job) (*aurora.Response, error)
	StartJobUpdate(updateJob *UpdateJob, message string) (*aurora.Response, error)

	PauseJobUpdate(key *aurora.JobUpdateKey, message string) (*aurora.Response, error)
	ResumeJobUpdate(key *aurora.JobUpdateKey, message string) (*aurora.Response, error)
	PulseJobUpdate(key *aurora.JobUpdateKey) (*aurora.Response, error)
	StartCronJob(key *aurora.JobKey) (*aurora.Response, error)
	// TODO: Remove this method and make it private to avoid race conditions
	ReestablishConn() error
	RealisConfig() *RealisConfig
	Close()

	// Admin functions
	DrainHosts(hosts ...string) (*aurora.Response, *aurora.DrainHostsResult_, error)
	SLADrainHosts(policy *aurora.SlaPolicy, timeout int64, hosts ...string) (*aurora.DrainHostsResult_, error)
	StartMaintenance(hosts ...string) (*aurora.Response, *aurora.StartMaintenanceResult_, error)
	EndMaintenance(hosts ...string) (*aurora.Response, *aurora.EndMaintenanceResult_, error)
	MaintenanceStatus(hosts ...string) (*aurora.Response, *aurora.MaintenanceStatusResult_, error)
	SetQuota(role string, cpu *float64, ram *int64, disk *int64) (*aurora.Response, error)
	GetQuota(role string) (*aurora.Response, error)
	Snapshot() error
	PerformBackup() error
	// Force an Implicit reconciliation between Mesos and Aurora
	ForceImplicitTaskReconciliation() error
	// Force an Explicit reconciliation between Mesos and Aurora
	ForceExplicitTaskReconciliation(batchSize *int32) error
}

TODO(rdelvalle): Move documentation to interface in order to make godoc look better/more accessible

func NewRealisClient added in v1.1.0 

func NewRealisClient(options ...ClientOption) (Realis, error)

This client implementation of the realis interface uses a retry mechanism for all Thrift Calls. It will retry all calls which result in a temporary failure as well as calls that fail due to an EOF being returned by the http client. Most permanent failures are now being caught by the thriftCallWithRetries function and not being retried but there may be corner cases not yet handled.

type RealisConfig 

type RealisConfig struct {
	InsecureSkipVerify bool
	// contains filtered or unexported fields
}

type ResourceType added in v1.3.1 

type ResourceType int
const (
	CPU ResourceType = iota
	RAM
	DISK
	GPU
)

type ServiceInstance 

type ServiceInstance struct {
	Service             Endpoint            `json:"serviceEndpoint"`
	AdditionalEndpoints map[string]Endpoint `json:"additionalEndpoints"`
	Status              string              `json:"status"`
}

type TemporaryErr added in v1.2.0 

type TemporaryErr struct {
	// contains filtered or unexported fields
}

func NewTemporaryError added in v1.2.0 

func NewTemporaryError(err error) *TemporaryErr

Retrying after receiving this error is advised

func (*TemporaryErr) Temporary added in v1.2.0 

func (t *TemporaryErr) Temporary() bool

type UpdateJob 

type UpdateJob struct {
	Job // SetInstanceCount for job is hidden, access via full qualifier
	// contains filtered or unexported fields
}

Structure to collect all information required to create job update

func NewDefaultUpdateJob added in v1.1.0 

func NewDefaultUpdateJob(config *aurora.TaskConfig) *UpdateJob

Create a default UpdateJob object.

func NewUpdateJob 

func NewUpdateJob(config *aurora.TaskConfig, settings *aurora.JobUpdateSettings) *UpdateJob

func (*UpdateJob) BatchSize 

func (u *UpdateJob) BatchSize(size int32) *UpdateJob

Max number of instances being updated at any given moment.

func (*UpdateJob) InstanceCount 

func (u *UpdateJob) InstanceCount(inst int32) *UpdateJob

Set instance count the job will have after the update.

func (*UpdateJob) MaxFailedInstances 

func (u *UpdateJob) MaxFailedInstances(inst int32) *UpdateJob

Max number of FAILED instances to tolerate before terminating the update.

func (*UpdateJob) MaxPerInstanceFailures 

func (u *UpdateJob) MaxPerInstanceFailures(inst int32) *UpdateJob

Max number of instance failures to tolerate before marking instance as FAILED.

func (*UpdateJob) RollbackOnFail 

func (u *UpdateJob) RollbackOnFail(rollback bool) *UpdateJob

When False, prevents auto rollback of a failed update.

func (*UpdateJob) WaitForBatchCompletion 

func (u *UpdateJob) WaitForBatchCompletion(batchWait bool) *UpdateJob

Wait for all instances in a group to be done before moving on.

func (*UpdateJob) WatchTime 

func (u *UpdateJob) WatchTime(ms int32) *UpdateJob

Minimum number of seconds a shard must remain in RUNNING state before considered a success.

type ZKOpt added in v1.3.0 

type ZKOpt func(z *zkConfig)

func ZKBackoff added in v1.3.0 

func ZKBackoff(b Backoff) ZKOpt

func ZKEndpoints added in v1.3.0 

func ZKEndpoints(endpoints ...string) ZKOpt

func ZKLogger added in v1.3.0 

func ZKLogger(l Logger) ZKOpt

func ZKPath added in v1.3.0 

func ZKPath(path string) ZKOpt

func ZKTimeout added in v1.3.0 

func ZKTimeout(d time.Duration) ZKOpt

Source Files

View all Source files

Directories

Path Synopsis
gen-go
apache/aurora
apache/aurora/aurora_admin-remote
apache/aurora/aurora_scheduler_manager-remote
apache/aurora/read_only_scheduler-remote
Helper functions to process aurora.Response
 Helper functions to process aurora.Response

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.