Documentation
¶
Index ¶
- Constants
- Variables
- type BootstrapImage
- type BucketNameFlagsTrait
- type CacheMode
- type CacheProvider
- type CacheProviderOff
- type CacheProviderRead
- type CacheProviderReadWrite
- type CacheProviderWrite
- type DockerConfig
- type KV
- type Logger
- type PredictorConfig
- type PredictorTestConfig
- type ReflowVersion
- type ReflowletConfig
- type Ssh
- type SshKey
- type TableNameFlagsTrait
- type User
- type VolumeWatcher
Constants ¶
const ( AWSCreds = "awscreds" AWSRegion = "awsregion" Assoc = "assoc" AWSTool = "awstool" Cache = "cache" Cluster = "cluster" Labels = "labels" Log = "logger" Repository = "repository" Reflow = "reflow" Reflowlet = "reflowlet" Bootstrap = "bootstrap" Session = "session" SSHKey = "sshkey" Username = "user" TLS = "tls" Tracer = "tracer" Metrics = "metrics" TaskDB = "taskdb" Docker = "docker" Predictor = "predictor" RunID = "runid" )
Reflow infra schema key names.
Variables ¶
var DefaultPredictorConfig = PredictorConfig{ MinData: 5, MaxInspect: 50, MemPercentile: 95, NonEC2Ok: false, }
var DefaultPredictorTestConfig = PredictorTestConfig{ &PredictorConfig{ MinData: 1, MaxInspect: 50, MemPercentile: 95, NonEC2Ok: true, }, }
var DefaultReflowletConfig = ReflowletConfig{ MaxIdleDuration: 10 * time.Minute, LogStatsDuration: 1 * time.Minute, VolumeWatcher: DefaultVolumeWatcher, }
var DefaultVolumeWatcher = VolumeWatcher{ LowThresholdPct: 55.0, HighThresholdPct: 75.0, WatcherSleepDuration: 20 * time.Second, ResizeSleepDuration: 5 * time.Second, FastThresholdDuration: 24 * time.Hour, FastIncreaseFactor: 20, SlowIncreaseFactor: 5, }
DefaultVolumeWatcher are a default set of volume watcher parameters which will double the disk size if the disk usage >75%, or quadruple if the usage went from <55% to >75% within 24 hours.
Functions ¶
This section is empty.
Types ¶
type BootstrapImage ¶
type BootstrapImage string
BootstrapImage is the URL of the image used for instance bootstrap.
func (*BootstrapImage) Flags ¶
func (r *BootstrapImage) Flags(flags *flag.FlagSet)
Flags implements infra.Provider.
func (*BootstrapImage) Set ¶
func (r *BootstrapImage) Set(image string) bool
Set sets this BootstrapImage's value with the given one (if different) but only if its current value is "bootstrap". Returns true if the value was set.
func (*BootstrapImage) Value ¶
func (r *BootstrapImage) Value() string
Value returns the bootstrap image uri.
type BucketNameFlagsTrait ¶
type BucketNameFlagsTrait struct {
BucketName string
}
func (*BucketNameFlagsTrait) Flags ¶
func (b *BucketNameFlagsTrait) Flags(flags *flag.FlagSet)
type CacheMode ¶
type CacheMode int
CacheMode is a bitmask that determines how caching is to be used in the evaluator.
const ( // CacheOff is CacheMode's default value and indicates // no caching (read or write) is to be performed. CacheOff CacheMode = 0 // CacheRead indicates that cache lookups should be performed // during evaluation. CacheRead CacheMode = 1 << iota // CacheWrite indicates that the evaluator should write evaluation // results to the cache. CacheWrite )
type CacheProvider ¶
type CacheProvider struct {
// CacheMode is the configured cache mode of this CacheMode infra provider.
CacheMode
}
CacheProvider is an infra provider for cache mode.
type CacheProviderOff ¶
type CacheProviderOff struct {
*CacheProvider
}
CacheProviderOff is the provider to turn off cache.
func (*CacheProviderOff) Init ¶
func (c *CacheProviderOff) Init() error
Init implements infra.Provider
type CacheProviderRead ¶
type CacheProviderRead struct {
*CacheProvider
}
CacheProviderRead is the provider to only read from cache.
func (CacheProviderRead) Help ¶
func (CacheProviderRead) Help() string
Help implements infra.Provider
func (*CacheProviderRead) Init ¶
func (c *CacheProviderRead) Init() error
Init implements infra.Provider
type CacheProviderReadWrite ¶
type CacheProviderReadWrite struct {
*CacheProvider
}
CacheProviderReadWrite is the provider to read and write to cache.
func (CacheProviderReadWrite) Help ¶
func (CacheProviderReadWrite) Help() string
Help implements infra.Provider
func (*CacheProviderReadWrite) Init ¶
func (c *CacheProviderReadWrite) Init() error
Init implements infra.Provider
type CacheProviderWrite ¶
type CacheProviderWrite struct {
*CacheProvider
}
CacheProviderWrite is the provider to only write to cache.
func (CacheProviderWrite) Help ¶
func (CacheProviderWrite) Help() string
Help implements infra.Provider
func (*CacheProviderWrite) Init ¶
func (c *CacheProviderWrite) Init() error
Init implements infra.Provider
type DockerConfig ¶
type DockerConfig string
DockerConfig sets the docker memory limit to either be hard or soft.
func (*DockerConfig) Flags ¶
func (m *DockerConfig) Flags(flags *flag.FlagSet)
Flags implements infra.Provider.
func (*DockerConfig) Value ¶
func (m *DockerConfig) Value() string
Value returns the docker memory limit mode.
type PredictorConfig ¶
type PredictorConfig struct {
// MinData is the minimum number of datapoints
// required to make a resource prediction.
MinData int `yaml:"mindata"`
// MaxInspect is the maximum number of ExecInspects
// that can be used to make a prediction.
MaxInspect int `yaml:"maxinspect"`
// MemPercentile is the percentile that will
// be used to predict memory usage for all tasks.
MemPercentile float64 `yaml:"mempercentile"`
// NonEC2Ok determines if the predictor should
// run on any machine. If set to false, the predictor
// will only work when 'reflow run' is invoked from
// an EC2 instance
NonEC2Ok bool `yaml:"alwaysrun"`
}
PredictorConfig represents the set of parameters that govern the behavior of the resource prediction system. The resource prediction system predicts the resource usage of an exec based on previous runs. A prediction requires at least MinData datapoints--profiles containing the resource to be predicted. No prediction will look at more than MaxInspect ExecInspects to obtain theses profiles. An exec's predicted memory usage is the MemPercentile'th percentile of the maximum memory usage of the exec.
func (PredictorConfig) Help ¶
func (p PredictorConfig) Help() string
Help implements infra.Provider.
func (*PredictorConfig) Init ¶
func (p *PredictorConfig) Init() error
Init implements infra.Provider.
func (*PredictorConfig) InstanceConfig ¶
func (p *PredictorConfig) InstanceConfig() interface{}
InstanceConfig implements infra.Provider.
type PredictorTestConfig ¶
type PredictorTestConfig struct {
*PredictorConfig
}
func (PredictorTestConfig) Help ¶
func (p PredictorTestConfig) Help() string
Help implements infra.Provider.
func (*PredictorTestConfig) Init ¶
func (p *PredictorTestConfig) Init() error
Init implements infra.Provider.
func (*PredictorTestConfig) InstanceConfig ¶
func (p *PredictorTestConfig) InstanceConfig() interface{}
InstanceConfig implements infra.Provider.
type ReflowVersion ¶
type ReflowVersion string
ReflowVersion is the infrastructure provider for the reflow version.
func (*ReflowVersion) Flags ¶
func (r *ReflowVersion) Flags(flags *flag.FlagSet)
Flags implements infra.Provider.
func (*ReflowVersion) Value ¶
func (r *ReflowVersion) Value() string
Value returns the reflow version name.
type ReflowletConfig ¶
type ReflowletConfig struct {
// MaxIdleDuration is the maximum duration the reflowlet will be idle waiting to receive work after which it dies.
MaxIdleDuration time.Duration `yaml:"maxidleduration,omitempty"`
// LogStatsDuration is the periodicity with which the reflowlet will log statistics.
// Note: The json field name is "logmemstatsduration" due to legacy reasons, and not renamed
// to avoid having to deal with migrating marshaled configs.
LogStatsDuration time.Duration `yaml:"logmemstatsduration,omitempty"`
// VolumeWatcher defines a set of parameters for the volume watcher on the reflowlet.
VolumeWatcher VolumeWatcher `yaml:"volumewatcher,omitempty"`
}
ReflowletConfig is a provider for reflowlet configuration parameters which control its behavior.
func MergeReflowletConfig ¶
func MergeReflowletConfig(base, other ReflowletConfig) ReflowletConfig
MergeReflowletConfig merges/overrides field values into `base` from `other`.
func (*ReflowletConfig) Flags ¶
func (rp *ReflowletConfig) Flags(flags *flag.FlagSet)
Flags implements infra.Provider.
func (*ReflowletConfig) Init ¶
func (rp *ReflowletConfig) Init() error
Init implements infra.Provider.
func (*ReflowletConfig) InstanceConfig ¶
func (rp *ReflowletConfig) InstanceConfig() interface{}
Instance implements infra.Provider.
type SshKey ¶
type SshKey struct {
Key string `yaml:"key,omitempty"`
}
SshKey is the infrastructure provider for ssh key
type TableNameFlagsTrait ¶
type TableNameFlagsTrait struct {
TableName string
}
func (*TableNameFlagsTrait) Flags ¶
func (t *TableNameFlagsTrait) Flags(flags *flag.FlagSet)
type VolumeWatcher ¶
type VolumeWatcher struct {
// LowThresholdPct defines how full the filesystem needs to be to trigger the low threshold.
LowThresholdPct float64 `yaml:"lowthresholdpct,omitempty"`
// HighThresholdPct defines how full the filesystem needs to be to trigger the high threshold.
// The time it takes for the filesystem to fill from low threshold to high threshold
// selects between a lower (time is longer) or higher (time is shorter)
// amount of extra space to get added to the volume.
HighThresholdPct float64 `yaml:"highthresholdpct,omitempty"`
// WatcherSleepDuration is the frequency with which to check
// whether disk are full over threshold and need resizing
WatcherSleepDuration time.Duration `yaml:"watchersleepduration,omitempty"`
// ResizeSleepDuration is the frequency with which to attempt
// resizing the volume and filesystem once we've hit above HighThresholdPct
ResizeSleepDuration time.Duration `yaml:"resizesleepduration,omitempty"`
// FastThresholdDuration is the time duration within which if the disk usage
// went from below LowThresholdPct to above HighThresholdPct, then
// we increase the disk size by FastIncreaseFactor (otherwise, by SlowIncreaseFactor)
FastThresholdDuration time.Duration `yaml:"fastthresholdduration,omitempty"`
// FastIncreaseFactor is factor by which the disk size is increased,
// if disk usage went from below LowThresholdPct to above HighThresholdPct,
// within FastThresholdDuration.
FastIncreaseFactor uint `yaml:"fastincreasefactor,omitempty"`
// SlowIncreaseFactor is factor by which the disk size is increased,
// if disk usage went from below LowThresholdPct to above HighThresholdPct,
// in more than FastThresholdDuration duration.
SlowIncreaseFactor uint `yaml:"slowincreasefactor,omitempty"`
}
VolumeWatcher represents the set of parameters that govern the behavior of a volume watcher. Every WatcherSleepDuration, the watcher will check the disk usage and keep track of the last time at which the usage was below the LowThresholdPct. If the disk usage goes above HighThresholdPct, then a resize is triggered. The volume size will be increased to 2x the current size unless the time taken to go from below LowThresholdPct to above HighThresholdPct was within FastThresholdDuration, in which case the size will be increased to 4x the current size. Once the underlying volume is resized, a filesystem resize will be attempted every ResizeSleepDuration until success.
Source Files