evergreen

README

Evergreen

Version 0.9.0 Alpha

Evergreen is a distributed continuous integration system built by MongoDB. It dynamically allocates hosts (via AWS, digitalocean, etc) to run tasks in parallel across many machines at once to decrease the total amount of time needed to complete a test workload.

Using Evergreen, we've significantly enhanced the productivity of our engineers.

Features

Elastic Host Allocation

Use only the computing resources you need.

Clean UI

Easily navigate the state of your tests, logs, and commit history.

Multiplatform Support

Run jobs on Linux (including PowerPC and ZSeries), Windows, OSX, Solaris, and BSD.

Spawn Hosts

Spin up a copy of any machine in your test infrastructure for debugging.

Patch Builds

See test results for your code changes before committing.

Stepback on Failure

Automatically run past commits to pinpoint the origin of a test failure

See the documentation for a full feature list!

Usage

Evergreen requires the configuration of a main server along with a cloud provider or static servers. Please refer to our tutorial for full installation instructions.

System Requirements

The Evergreen Agent and Command Line Tool are supported on Linux, OSX, Windows, and Solaris operating systems. However, the Evergreen API Server, UI Server, and Runner program are currently only supported and tested on Linux and OSX.

Go Requirements

• Install Go 1.7 or later.

Vendoring Dependencies

Our dependencies live in the vendor subdirectory of the repo, and are managed using glide. To add a new dependency, add the package to glide.yaml and the package name and revision to glide.lock, and run glide install -s. To refresh the entire vendor tree, run make vendor-sync

Building the Binaries

Setup:

• ensure that your GOPATH environment variable is set.
• check out a copy of the repo into your gopath. You can use: go get github.com/evergreen-ci/evergreen. If you have an existing checkout of the evergreen repository that is not in $GOPATH/src/github.com/evergreen-ci/ move or create a symlink.
• run make vendor to set up the vendoring environment.

Possible Targets:

• run make build to compile all server binaries for your local system.
• run make agent cli to compile the agent and cli for your local platform.
• run make agents clis to compile and cross compile all agent and command line interface binaries.
• run make dist to compile all server, commandline, and agent binaries and create a dist tarball with all artifacts.

Terminology

• distro: A platform type (e.g. Windows or OSX) plus large-scale architectural details. One of the targets that we produce executables for.
• host: One of the machines used to run tasks on; typically an instance of a distro that does the actual work (see task). Equivalently, any machine in our system other than MOTU.
• revision: A revision SHA; e.g. ad8a1d4678122bada9d4479b114cf68d39c27724.
• project: A branch in a given repository. e.g. master in the mongodb/mongo repository
• version: (project + revision). A version of a project.
• buildvariant: build specific information.
• build: (version + buildvariant) = (project + revision + buildvariant).
• task: “compile”, “test”, or “push”. The kinds of things that we want to do on a host.

Running Evergreen Processes

A single configuration file drives most of how Evergreen works. Each Evergreen process must be supplied this settings file to run properly. For development, tweak the sample configuration file here.

All Evergreen programs accept a configuration file with the -conf flag.

How It Works

• A MongoDB server must be already running on the port specified in the configuration file.
• Both the API and UI server processes are started manually and listen for connections.
• All other processes are run by the runner process.
• For detailed instructions, please see the wiki.

Documentation

Index

• Constants
• Variables
• func FindConfig(configName string) (string, error)
• func FindEvergreenHome() string
• func GetAppendingFile(path string) (*os.File, error)
• func IsSystemActivator(caller string) bool
• func SetLogger(logPath string)
• type APIConfig
• type AWSConfig
• type AlertsConfig
• type AuthConfig
• type AuthUser
• type ClientBinary
• type ClientConfig
• type CloudProviders
• type ConfigValidator
• type CrowdConfig
• type DBSettings
• type DigitalOceanConfig
• type GithubAuthConfig
• type HostInitConfig
• type JiraConfig
• type LoggingWriter
  • func NewErrorLoggingWriter(logger *slogger.Logger) *LoggingWriter
  • func NewInfoLoggingWriter(logger *slogger.Logger) *LoggingWriter
  • func (self *LoggingWriter) Write(p []byte) (n int, err error)
• type MonitorConfig
• type NaiveAuthConfig
• type NotifyConfig
• type PluginConfig
• type RepoTrackerConfig
• type RunnerConfig
• type SMTPConfig
• type SchedulerConfig
• type Settings
  • func GetSettings() (*Settings, error)
  • func GetSettingsOrExit() *Settings
  • func NewSettings(filename string) (*Settings, error)
  • func TestConfig() *Settings
  • func (settings *Settings) Validate(validators []ConfigValidator) error
• type SliceAppender
  • func (self *SliceAppender) Append(log *slogger.Log) error
• type TaskRunnerConfig
• type UIConfig
• type WriteConcern

Constants

const (
	User = "mci"

	TestDir      = "config_test"
	TestSettings = "evg_settings.yml"

	HostRunning         = "running"
	HostTerminated      = "terminated"
	HostUninitialized   = "starting"
	HostInitializing    = "provisioning"
	HostProvisionFailed = "provision failed"
	HostUnreachable     = "unreachable"
	HostQuarantined     = "quarantined"
	HostDecommissioned  = "decommissioned"

	HostStatusSuccess = "success"
	HostStatusFailed  = "failed"

	TaskStarted      = "started"
	TaskUndispatched = "undispatched"
	TaskDispatched   = "dispatched"
	TaskFailed       = "failed"
	TaskSucceeded    = "success"
	TaskInactive     = "inactive"

	TestFailedStatus    = "fail"
	TestSkippedStatus   = "skip"
	TestSucceededStatus = "pass"

	BuildStarted   = "started"
	BuildCreated   = "created"
	BuildFailed    = "failed"
	BuildSucceeded = "success"

	VersionStarted   = "started"
	VersionCreated   = "created"
	VersionFailed    = "failed"
	VersionSucceeded = "success"

	PatchCreated   = "created"
	PatchStarted   = "started"
	PatchSucceeded = "succeeded"
	PatchFailed    = "failed"

	PushLogPushing = "pushing"
	PushLogSuccess = "success"

	HostTypeStatic = "static"

	CompileStage = "compile"
	TestStage    = "single_test"
	SanityStage  = "smokeCppUnitTests"
	PushStage    = "push"

	// maximum task (zero based) execution number
	MaxTaskExecution = 3

	// maximum task priority
	MaxTaskPriority = 100

	// LogMessage struct versions
	LogmessageFormatTimestamp = 1
	LogmessageCurrentVersion  = LogmessageFormatTimestamp

	EvergreenHome = "EVGHOME"

	DefaultTaskActivator   = ""
	APIServerTaskActivator = "apiserver"

	AgentAPIVersion = 2
)

const (
	AuthTokenCookie  = "mci-token"
	TaskSecretHeader = "Task-Secret"
)

const (
	// DefaultConfFile is the default config file path for Evergreen.
	DefaultConfFile = "/etc/mci_settings.yml"
)

const (
	UIPackage = "EVERGREEN_UI"
)

evergreen package names

Variables

var (
	// Should be specified with -ldflags at build time
	BuildRevision = ""

	// Commandline Version String; used to control auto-updating.
	ClientVersion = "e4185327b58797d5ebcc88f092b33e319d8c8521" // in the future this should probably be a date, but this is the legacy value
)

var (
	// UphostStatus is a list of all host statuses that are considered "up."
	// This is used for query building.
	UphostStatus = []string{
		HostRunning,
		HostUninitialized,
		HostInitializing,
		HostProvisionFailed,
	}

	// Logger is our global logger. It can be changed for testing.
	Logger = slogger.Logger{
		Prefix:    "",
		Appenders: []slogger.Appender{slogger.StdOutAppender()},
	}

	// database and config directory, set to the testing version by default for safety
	NotificationsFile = "mci-notifications.yml"
	ClientDirectory   = "clients"

	// version requester types
	PatchVersionRequester       = "patch_request"
	RepotrackerVersionRequester = "gitter_request"

	// constant arrays for db update logic
	AbortableStatuses = []string{TaskStarted, TaskDispatched}
	CompletedStatuses = []string{TaskSucceeded, TaskFailed}
)

var ConfigValidationRules = []ConfigValidator{
	func(settings *Settings) error {
		if settings.Database.Url == "" || settings.Database.DB == "" {
			return fmt.Errorf("DBUrl and DB must not be empty")
		}
		return nil
	},

	func(settings *Settings) error {
		if settings.ApiUrl == "" {
			return fmt.Errorf("API hostname must not be empty")
		}
		return nil
	},

	func(settings *Settings) error {
		if settings.Ui.Secret == "" {
			return fmt.Errorf("UI Secret must not be empty")
		}
		return nil
	},

	func(settings *Settings) error {
		if settings.ConfigDir == "" {
			return fmt.Errorf("Config directory must not be empty")
		}
		return nil
	},

	func(settings *Settings) error {
		if settings.Ui.DefaultProject == "" {
			return fmt.Errorf("You must specify a default project in UI")
		}
		return nil
	},

	func(settings *Settings) error {
		if settings.Ui.Url == "" {
			return fmt.Errorf("You must specify a default UI url")
		}
		return nil
	},

	func(settings *Settings) error {
		notifyConfig := settings.Notify.SMTP

		if notifyConfig == nil {
			return nil
		}

		if notifyConfig.Server == "" || notifyConfig.Port == 0 {
			return fmt.Errorf("You must specify a SMTP server and port")
		}

		if len(notifyConfig.AdminEmail) == 0 {
			return fmt.Errorf("You must specify at least one admin_email")
		}

		if notifyConfig.From == "" {
			return fmt.Errorf("You must specify a from address")
		}

		return nil
	},

	func(settings *Settings) error {
		if settings.AuthConfig.Crowd == nil && settings.AuthConfig.Naive == nil && settings.AuthConfig.Github == nil {
			return fmt.Errorf("You must specify one form of authentication")
		}
		if settings.AuthConfig.Naive != nil {
			used := map[string]bool{}
			for _, x := range settings.AuthConfig.Naive.Users {
				if used[x.Username] {
					return fmt.Errorf("Duplicate user in list")
				}
				used[x.Username] = true
			}
		}
		if settings.AuthConfig.Github != nil {
			if settings.AuthConfig.Github.Users == nil && settings.AuthConfig.Github.Organization == "" {
				return fmt.Errorf("Must specify either a set of users or an organization for Github Authentication")
			}
		}
		return nil
	},
}

ConfigValidationRules is the set of all ConfigValidator functions.

Functions

func FindConfig

func FindConfig(configName string) (string, error)

FindConfig finds the config root in the home directory. Returns an error if the root cannot be found or EVGHOME is unset.

func FindEvergreenHome

func FindEvergreenHome() string

FindEvergreenHome finds the directory of the EVGHOME environment variable.

func GetAppendingFile

func GetAppendingFile(path string) (*os.File, error)

GetAppendingFile opens a file for appending. The file will be created if it does not already exist.

func IsSystemActivator

func IsSystemActivator(caller string) bool

IsSystemActivator returns true when the task activator is Evergreen.

func SetLogger

func SetLogger(logPath string)

SetLogger sets the global logger to write to the given path.

Types

type APIConfig

type APIConfig struct {
	LogFile         string
	HttpListenAddr  string
	HttpsListenAddr string
	HttpsKey        string
	HttpsCert       string
}

APIConfig holds relevant encryption and log settings for the API server.

type AWSConfig

type AWSConfig struct {
	Secret string `yaml:"aws_secret"`
	Id     string `yaml:"aws_id"`
}

AWSConfig stores auth info for Amazon Web Services.

type AlertsConfig

type AlertsConfig struct {
	LogFile string
	SMTP    *SMTPConfig `yaml:"smtp"`
}

type AuthConfig

type AuthConfig struct {
	Crowd  *CrowdConfig      `yaml:"crowd"`
	Naive  *NaiveAuthConfig  `yaml:"naive"`
	Github *GithubAuthConfig `yaml:"github"`
}

AuthConfig has a pointer to either a CrowConfig or a NaiveAuthConfig.

type AuthUser

type AuthUser struct {
	Username    string `yaml:"username"`
	DisplayName string `yaml:"display_name"`
	Password    string `yaml:"password"`
	Email       string `yaml:"email"`
}

AuthUser configures a user for our Naive authentication setup.

type ClientBinary

type ClientBinary struct {
	Arch string `yaml:"arch" json:"arch"`
	OS   string `yaml:"os" json:"os"`
	URL  string `yaml:"url" json:"url"`
}

type ClientConfig

type ClientConfig struct {
	ClientBinaries []ClientBinary `yaml:"client_binaries"`
	LatestRevision string         `yaml:"latest_revision"`
}

type CloudProviders

type CloudProviders struct {
	AWS          AWSConfig          `yaml:"aws"`
	DigitalOcean DigitalOceanConfig `yaml:"digitalocean"`
}

CloudProviders stores configuration settings for the supported cloud host providers.

type ConfigValidator

type ConfigValidator func(settings *Settings) error

ConfigValidator is a type of function that checks the settings struct for any errors or missing required fields.

type CrowdConfig

type CrowdConfig struct {
	Username string
	Password string
	Urlroot  string
}

CrowdConfig holds settings for interacting with Atlassian Crowd.

type DBSettings

type DBSettings struct {
	Url                  string       `yaml:"url"`
	SSL                  bool         `yaml:"ssl"`
	DB                   string       `yaml:"db"`
	WriteConcernSettings WriteConcern `yaml:"write_concern"`
}

type DigitalOceanConfig

type DigitalOceanConfig struct {
	ClientId string `yaml:"client_id"`
	Key      string `yaml:"key"`
}

DigitalOceanConfig stores auth info for Digital Ocean.

type GithubAuthConfig

type GithubAuthConfig struct {
	ClientId     string   `yaml:"client_id"`
	ClientSecret string   `yaml:"client_secret"`
	Users        []string `yaml:"users"`
	Organization string   `yaml:"organization"`
}

GithubAuthConfig holds settings for interacting with Github Authentication including the ClientID, ClientSecret and CallbackUri which are given when registering the application Furthermore,

type HostInitConfig

type HostInitConfig struct {
	LogFile           string
	SSHTimeoutSeconds int64
}

HostInitConfig holds logging settings for the hostinit process.

type JiraConfig

type JiraConfig struct {
	Host     string
	Username string
	Password string
}

JiraConfig stores auth info for interacting with Atlassian Jira.

type LoggingWriter

type LoggingWriter struct {
	Logger   *slogger.Logger
	Severity slogger.Level
}

LoggingWriter is a struct - with an associated log level severity - that implements io.Writer

func NewErrorLoggingWriter

func NewErrorLoggingWriter(logger *slogger.Logger) *LoggingWriter

NewErrorLoggingWriter is a helper function that returns a LoggingWriter for errors

func NewInfoLoggingWriter

func NewInfoLoggingWriter(logger *slogger.Logger) *LoggingWriter

NewInfoLoggingWriter is a helper function that returns a LoggingWriter for information logs

func (*LoggingWriter) Write

func (self *LoggingWriter) Write(p []byte) (n int, err error)

Since LoggingWriter is an io.Writer, it must implement the Write function

type MonitorConfig

type MonitorConfig struct {
	LogFile string
}

MonitorConfig holds logging settings for the monitor process.

type NaiveAuthConfig

type NaiveAuthConfig struct {
	Users []*AuthUser
}

NaiveAuthConfig contains a list of AuthUsers from the settings file.

type NotifyConfig

type NotifyConfig struct {
	LogFile string
	SMTP    *SMTPConfig `yaml:"smtp"`
}

NotifyConfig hold logging and email settings for the notify package.

type PluginConfig

type PluginConfig map[string]map[string]interface{}

PluginConfig holds plugin-specific settings, which are handled. manually by their respective plugins

type RepoTrackerConfig

type RepoTrackerConfig struct {
	NumNewRepoRevisionsToFetch int
	MaxRepoRevisionsToSearch   int
	LogFile                    string
}

RepoTrackerConfig holds settings for polling project repositories.

type RunnerConfig

type RunnerConfig struct {
	LogFile         string
	IntervalSeconds int64
}

RunnerConfig holds logging and timing settings for the runner process.

type SMTPConfig

type SMTPConfig struct {
	Server     string   `yaml:"server"`
	Port       int      `yaml:"port"`
	UseSSL     bool     `yaml:"use_ssl"`
	Username   string   `yaml:"username"`
	Password   string   `yaml:"password"`
	From       string   `yaml:"from"`
	AdminEmail []string `yaml:"admin_email"`
}

SMTPConfig holds SMTP email settings.

type SchedulerConfig

type SchedulerConfig struct {
	LogFile     string
	MergeToggle int
}

SchedulerConfig holds relevant settings for the scheduler process.

type Settings

type Settings struct {
	Database            DBSettings        `yaml:"database"`
	WriteConcern        WriteConcern      `yaml:"write_concern"`
	ConfigDir           string            `yaml:"configdir"`
	ApiUrl              string            `yaml:"api_url"`
	AgentExecutablesDir string            `yaml:"agentexecutablesdir"`
	ClientBinariesDir   string            `yaml:"client_binaries_dir"`
	SuperUsers          []string          `yaml:"superusers"`
	Jira                JiraConfig        `yaml:"jira"`
	Providers           CloudProviders    `yaml:"providers"`
	Keys                map[string]string `yaml:"keys"`
	Credentials         map[string]string `yaml:"credentials"`
	AuthConfig          AuthConfig        `yaml:"auth"`
	RepoTracker         RepoTrackerConfig `yaml:"repotracker"`
	Monitor             MonitorConfig     `yaml:"monitor"`
	Api                 APIConfig         `yaml:"api"`
	Alerts              AlertsConfig      `yaml:"alerts"`
	Ui                  UIConfig          `yaml:"ui"`
	HostInit            HostInitConfig    `yaml:"hostinit"`
	Notify              NotifyConfig      `yaml:"notify"`
	Runner              RunnerConfig      `yaml:"runner"`
	Scheduler           SchedulerConfig   `yaml:"scheduler"`
	TaskRunner          TaskRunnerConfig  `yaml:"taskrunner"`
	Expansions          map[string]string `yaml:"expansions"`
	Plugins             PluginConfig      `yaml:"plugins"`
	IsProd              bool              `yaml:"isprod"`
}

Settings contains all configuration settings for running Evergreen.

func GetSettings

func GetSettings() (*Settings, error)

GetSettings returns Evergreen Settings or an error.

func GetSettingsOrExit

func GetSettingsOrExit() *Settings

GetSettingsOrExit loads the evergreen settings file, or exits with a non-0 code if any errors occur.

func NewSettings

func NewSettings(filename string) (*Settings, error)

NewSettings builds an in-memory representation of the given settings file.

func TestConfig

func TestConfig() *Settings

TestConfig creates test settings from a test config.

func (*Settings) Validate

func (settings *Settings) Validate(validators []ConfigValidator) error

Validate checks the settings and returns nil if the config is valid, or an error with a message explaining why otherwise.

type SliceAppender

type SliceAppender struct {
	Messages []*slogger.Log
}

SliceAppender is a slogger.Appender implemenation that just adds every log message to an internal slice. Useful for testing when a test needs to capture data sent to a slogger.Logger and verify what data was written.

func (*SliceAppender) Append

func (self *SliceAppender) Append(log *slogger.Log) error

type TaskRunnerConfig

type TaskRunnerConfig struct {
	LogFile string
}

TaskRunnerConfig holds logging settings for the scheduler process.

type UIConfig

type UIConfig struct {
	Url            string
	HelpUrl        string
	HttpListenAddr string
	// Secret to encrypt session storage
	Secret  string
	LogFile string
	// Default project to assume when none specified, e.g. when using
	// the /waterfall route use this project, while /waterfall/other-project
	// then use `other-project`
	DefaultProject string
	// Cache results of template compilation, so you don't have to re-read files
	// on every request. Note that if this is true, changes to HTML templates
	// won't take effect until server restart.
	CacheTemplates bool
	// SecureCookies sets the "secure" flag on user tokens. Evergreen
	// does not yet natively support SSL UI connections, but this option
	// is available, for example, for deployments behind HTTPS load balancers.
	SecureCookies bool
}

UIConfig holds relevant settings for the UI server.

type WriteConcern

type WriteConcern struct {
	W        int    `yaml:"w"`
	WMode    string `yaml:"wmode"`
	WTimeout int    `yaml:"wtimeout"`
	FSync    bool   `yaml:"fsync"`
	J        bool   `yaml:"j"`
}