evergreen

README

Evergreen

Version 0.9.0 Alpha

Evergreen is a distributed continuous integration system built by MongoDB. It utilizes multi-platform cloud computing to run tasks as quickly as possible.

Using Evergreen, we've significantly enhanced the productivity of our engineers. We hope you can get something out of it as well!

Features

Elastic Host Allocation

Use only the computing resources you need

Clean UI

Easily navigate and explore the state of your tests

Multiplatform Support

Run jobs on any operating system supported by the Go compiler

Patch Builds

Test your changes before your commit

Stepback on Failure

Automatically run past commits to pinpoint the origin of a test failure

See the documentation for a full feature list!

Usage

Evergreen requires the configuration of a main server along with a cloud provider or static servers. Please refer to our tutorial for full installation instructions.

System Requirements

The Evergreen Agent and Command Line Tool are supported on Linux, OSX, Windows, and Solaris operating systems. However, the Evergreen API Server, UI Server, and Runner program are currently only supported and tested on Linux and OSX.

Go Requirements

• Install Go from source. This is needed to build the binaries using goxc. This requirement will be simplified by the release of Go 1.5.

Vendoring Dependencies

Our dependencies live in the vendor subdirectory of the repo. The specifications for what version of each dependency is used resides in the Godeps file in the root of the repo. If you add a new dependency, or change the version of an existing one, run the vendor.sh script to refresh the downloaded versions in vendor.

Building the Binaries

• To build the binaries for the cron tasks and servers, run the build.sh script in the root of the repo. This will install the binaries into the bin subdirectory.
• To build the agent binary, run the build_agent.sh script in the root of the repo. This will use goxc (located in vendor/src/github.com/laher/goxc) to cross-compile the agent binary into the executables subdirectory. Before doing this, you have to run go run vendor/src/github.com/laher/goxc/goxc.go -t once on the machine where you are building the binaries.
• To run tests or build manually, you must update the GOPATH variable by running . setgopath.sh in your shell.

Terminology

• distro: A platform type (e.g. Windows or OSX) plus large-scale architectural details. One of the targets that we produce executables for.
• host: One of the machines used to run tasks on; typically an instance of a distro that does the actual work (see task). Equivalently, any machine in our system other than MOTU.
• revision: A revision SHA; e.g. ad8a1d4678122bada9d4479b114cf68d39c27724.
• project: A branch in a given repository. e.g. master in the mongodb/mongo repository
• version: (project + revision). A version of a project.
• buildvariant: build specific information.
• build: (version + buildvariant) = (project + revision + buildvariant).
• task: “compile”, “test”, or “push”. The kinds of things that we want to do on a host.

Running Evergreen Processes

A single configuration (settings) file drives most of how Evergreen works. Each Evergreen process must be supplied this settings file to run properly. For development, tweak the sample configuration file here.

All Evergreen programs accept a configuration file with the -conf flag.

How It Works

• A MongoDB server must be already running on the port specified in the configuration file.
• Both the API and UI server processes are started manually and listen for connections.
• All other processes are run by the runner process.
• For detailed instructions, please see the wiki.

Documentation

Index

• Constants
• Variables
• func FindConfig(configName string) (string, error)
• func FindEvergreenHome() string
• func SetLogger(logPath string)
• type APIConfig
• type AWSConfig
• type AlertsConfig
• type AuthConfig
• type AuthUser
• type ClientBinary
• type ClientConfig
• type CloudProviders
• type ConfigValidator
• type CrowdConfig
• type DigitalOceanConfig
• type GithubAuthConfig
• type HostInitConfig
• type JiraConfig
• type LoggingWriter
  • func NewErrorLoggingWriter(logger *slogger.Logger) *LoggingWriter
  • func NewInfoLoggingWriter(logger *slogger.Logger) *LoggingWriter
  • func (self *LoggingWriter) Write(p []byte) (n int, err error)
• type MonitorConfig
• type NaiveAuthConfig
• type NotifyConfig
• type PluginConfig
• type RepoTrackerConfig
• type RunnerConfig
• type SMTPConfig
• type SchedulerConfig
• type Settings
  • func GetSettings() (*Settings, error)
  • func GetSettingsOrExit() *Settings
  • func NewSettings(filename string) (*Settings, error)
  • func TestConfig() *Settings
  • func (settings *Settings) Validate(validators []ConfigValidator) error
• type SliceAppender
  • func (self *SliceAppender) Append(log *slogger.Log) error
• type TaskRunnerConfig
• type UIConfig
• type WriteConcern

Constants

const (
	User = "mci"

	TestDir      = "config_test"
	TestSettings = "mci_settings.yml"

	HostRunning         = "running"
	HostTerminated      = "terminated"
	HostUninitialized   = "starting"
	HostInitializing    = "provisioning"
	HostProvisionFailed = "provision failed"
	HostUnreachable     = "unreachable"
	HostQuarantined     = "quarantined"
	HostDecommissioned  = "decommissioned"

	HostStatusSuccess = "success"
	HostStatusFailed  = "failed"

	SpawnRequestInit       = "initializing"
	SpawnRequestReady      = "ready"
	SpawnRequestUnusable   = "unusable"
	SpawnRequestTerminated = "terminated"

	TaskStarted      = "started"
	TaskUndispatched = "undispatched"
	TaskDispatched   = "dispatched"
	TaskFailed       = "failed"
	TaskCancelled    = "cancelled"
	TaskSucceeded    = "success"

	TestFailedStatus    = "fail"
	TestSkippedStatus   = "skip"
	TestSucceededStatus = "pass"

	BuildStarted   = "started"
	BuildCreated   = "created"
	BuildFailed    = "failed"
	BuildSucceeded = "success"
	BuildCancelled = "cancelled"

	VersionStarted   = "started"
	VersionCreated   = "created"
	VersionFailed    = "failed"
	VersionSucceeded = "success"
	VersionCancelled = "cancelled"

	PatchCreated   = "created"
	PatchStarted   = "started"
	PatchSucceeded = "succeeded"
	PatchFailed    = "failed"
	PatchCancelled = "cancelled"

	PushLogPushing = "pushing"
	PushLogSuccess = "success"

	HostTypeStatic = "static"
	HostTypeEC2    = "ec2"

	CompileStage = "compile"
	TestStage    = "single_test"
	SanityStage  = "smokeCppUnitTests"
	PushStage    = "push"

	// maximum task (zero based) execution number
	MaxTaskExecution = 3

	// LogMessage struct versions
	LogmessageFormatTimestamp = 1
	LogmessageCurrentVersion  = LogmessageFormatTimestamp

	EvergreenHome = "EVGHOME"
)

const (
	AuthTokenCookie  = "mci-token"
	TaskSecretHeader = "Task-Secret"
)

const (
	// DefaultConfFile is the default config file path for Evergreen.
	DefaultConfFile = "/etc/mci_settings.yml"
)

const (
	UIPackage = "EVERGREEN_UI"
)

evergreen package names

Variables

var (
	// UphostStatus is a list of all host statuses that are considered "up."
	// This is used for query building.
	UphostStatus = []string{
		HostRunning,
		HostUninitialized,
		HostInitializing,
		HostProvisionFailed,
		HostUnreachable,
	}

	// Logger is our global logger. It can be changed for testing.
	Logger = slogger.Logger{
		Prefix:    "",
		Appenders: []slogger.Appender{slogger.StdOutAppender()},
	}

	// database and config directory, set to the testing version by default for safety
	NotificationsFile = "mci-notifications.yml"

	// version requester types
	PatchVersionRequester       = "patch_request"
	RepotrackerVersionRequester = "gitter_request"

	// constant arrays for db update logic
	AbortableStatuses = []string{TaskStarted, TaskDispatched}
	CompletedStatuses = []string{TaskSucceeded, TaskFailed}
)

var BuildRevision = ""

Should be specified with -ldflags at build time

var ConfigValidationRules = []ConfigValidator{
	func(settings *Settings) error {
		if settings.DbUrl == "" || settings.Db == "" {
			return fmt.Errorf("DBUrl and DB must not be empty")
		}
		return nil
	},

	func(settings *Settings) error {
		if settings.Providers.AWS.Secret == "" || settings.Providers.AWS.Id == "" {
			return fmt.Errorf("AWS Secret and ID must not be empty")
		}
		return nil
	},

	func(settings *Settings) error {
		if settings.ApiUrl == "" {
			return fmt.Errorf("API hostname must not be empty")
		}
		return nil
	},

	func(settings *Settings) error {
		if settings.Ui.Secret == "" {
			return fmt.Errorf("UI Secret must not be empty")
		}
		return nil
	},

	func(settings *Settings) error {
		if settings.ConfigDir == "" {
			return fmt.Errorf("Config directory must not be empty")
		}
		return nil
	},

	func(settings *Settings) error {
		if settings.Ui.DefaultProject == "" {
			return fmt.Errorf("You must specify a default project in UI")
		}
		return nil
	},

	func(settings *Settings) error {
		if settings.Ui.Url == "" {
			return fmt.Errorf("You must specify a default UI url")
		}
		return nil
	},

	func(settings *Settings) error {
		notifyConfig := settings.Notify.SMTP

		if notifyConfig == nil {
			return nil
		}

		if notifyConfig.Server == "" || notifyConfig.Port == 0 {
			return fmt.Errorf("You must specify a SMTP server and port")
		}

		if len(notifyConfig.AdminEmail) == 0 {
			return fmt.Errorf("You must specify at least one admin_email")
		}

		if notifyConfig.From == "" {
			return fmt.Errorf("You must specify a from address")
		}

		return nil
	},

	func(settings *Settings) error {
		if settings.AuthConfig.Crowd == nil && settings.AuthConfig.Naive == nil && settings.AuthConfig.Github == nil {
			return fmt.Errorf("You must specify one form of authentication")
		}
		if settings.AuthConfig.Naive != nil {
			used := map[string]bool{}
			for _, x := range settings.AuthConfig.Naive.Users {
				if used[x.Username] {
					return fmt.Errorf("Duplicate user in list")
				}
				used[x.Username] = true
			}
		}
		if settings.AuthConfig.Github != nil {
			if settings.AuthConfig.Github.Users == nil && settings.AuthConfig.Github.Organization == "" {
				return fmt.Errorf("Must specify either a set of users or an organization for Github Authentication")
			}
		}
		return nil
	},
}

ConfigValidationRules is the set of all ConfigValidator functions.

Functions

func FindConfig

func FindConfig(configName string) (string, error)

FindConfig finds the config root in the home directory. Returns an error if the root cannot be found or EVGHOME is unset.

func FindEvergreenHome

func FindEvergreenHome() string

FindEvergreenHome finds the directory of the EVGHOME environment variable.

func SetLogger

func SetLogger(logPath string)

SetLogger sets the global logger to write to the given path.

Types

type APIConfig

type APIConfig struct {
	Clients         ClientConfig `yaml:"clients"`
	LogFile         string
	HttpListenAddr  string
	HttpsListenAddr string
	HttpsKey        string
}

APIConfig holds relevant encryption and log settings for the API server.

type AWSConfig

type AWSConfig struct {
	Secret string `yaml:"aws_secret"`
	Id     string `yaml:"aws_id"`
}

AWSConfig stores auth info for Amazon Web Services.

type AlertsConfig

type AlertsConfig struct {
	LogFile string
	SMTP    *SMTPConfig `yaml:"smtp"`
}

type AuthConfig

type AuthConfig struct {
	Crowd  *CrowdConfig      `yaml:"crowd"`
	Naive  *NaiveAuthConfig  `yaml:"naive"`
	Github *GithubAuthConfig `yaml:"github"`
}

AuthConfig has a pointer to either a CrowConfig or a NaiveAuthConfig.

type AuthUser

type AuthUser struct {
	Username    string `yaml:"username"`
	DisplayName string `yaml:"display_name"`
	Password    string `yaml:"password"`
	Email       string `yaml:"email"`
}

AuthUser configures a user for our Naive authentication setup.

type ClientBinary

type ClientBinary struct {
	Arch string `yaml:"arch"`
	OS   string `yaml:"os"`
	URL  string `yaml:"url"`
}

type ClientConfig

type ClientConfig struct {
	ClientBinaries []ClientBinary `yaml:"client_binaries"`
	LatestRevision string         `yaml:"latest_revision"`
}

type CloudProviders

type CloudProviders struct {
	AWS          AWSConfig          `yaml:"aws"`
	DigitalOcean DigitalOceanConfig `yaml:"digitalocean"`
}

CloudProviders stores configuration settings for the supported cloud host providers.

type ConfigValidator

type ConfigValidator func(settings *Settings) error

ConfigValidator is a type of function that checks the settings struct for any errors or missing required fields.

type CrowdConfig

type CrowdConfig struct {
	Username string
	Password string
	Urlroot  string
}

CrowdConfig holds settings for interacting with Atlassian Crowd.

type DigitalOceanConfig

type DigitalOceanConfig struct {
	ClientId string `yaml:"client_id"`
	Key      string `yaml:"key"`
}

DigitalOceanConfig stores auth info for Digital Ocean.

type GithubAuthConfig

type GithubAuthConfig struct {
	ClientId     string   `yaml:"client_id"`
	ClientSecret string   `yaml:"client_secret"`
	Users        []string `yaml:"users"`
	Organization string   `yaml:"organization"`
}

GithubAuthConfig holds settings for interacting with Github Authentication including the ClientID, ClientSecret and CallbackUri which are given when registering the application Furthermore,

type HostInitConfig

type HostInitConfig struct {
	LogFile           string
	SSHTimeoutSeconds int64
}

HostInitConfig holds logging settings for the hostinit process.

type JiraConfig

type JiraConfig struct {
	Host     string
	Username string
	Password string
}

JiraConfig stores auth info for interacting with Atlassian Jira.

type LoggingWriter

type LoggingWriter struct {
	Logger   *slogger.Logger
	Severity slogger.Level
}

LoggingWriter is a struct - with an associated log level severity - that implements io.Writer

func NewErrorLoggingWriter

func NewErrorLoggingWriter(logger *slogger.Logger) *LoggingWriter

NewErrorLoggingWriter is a helper function that returns a LoggingWriter for errors

func NewInfoLoggingWriter

func NewInfoLoggingWriter(logger *slogger.Logger) *LoggingWriter

NewInfoLoggingWriter is a helper function that returns a LoggingWriter for information logs

func (*LoggingWriter) Write

func (self *LoggingWriter) Write(p []byte) (n int, err error)

Since LoggingWriter is an io.Writer, it must implement the Write function

type MonitorConfig

type MonitorConfig struct {
	LogFile string
}

MonitorConfig holds logging settings for the monitor process.

type NaiveAuthConfig

type NaiveAuthConfig struct {
	Users []*AuthUser
}

NaiveAuthConfig contains a list of AuthUsers from the settings file.

type NotifyConfig

type NotifyConfig struct {
	LogFile string
	SMTP    *SMTPConfig `yaml:"smtp"`
}

NotifyConfig hold logging and email settings for the notify package.

type PluginConfig

type PluginConfig map[string]map[string]interface{}

PluginConfig holds plugin-specific settings, which are handled. manually by their respective plugins

type RepoTrackerConfig

type RepoTrackerConfig struct {
	NumNewRepoRevisionsToFetch int
	MaxRepoRevisionsToSearch   int
	LogFile                    string
}

RepoTrackerConfig holds settings for polling project repositories.

type RunnerConfig

type RunnerConfig struct {
	LogFile         string
	IntervalSeconds int64
}

RunnerConfig holds logging and timing settings for the runner process.

type SMTPConfig

type SMTPConfig struct {
	Server     string   `yaml:"server"`
	Port       int      `yaml:"port"`
	UseSSL     bool     `yaml:"use_ssl"`
	Username   string   `yaml:"username"`
	Password   string   `yaml:"password"`
	From       string   `yaml:"from"`
	AdminEmail []string `yaml:"admin_email"`
}

SMTPConfig holds SMTP email settings.

type SchedulerConfig

type SchedulerConfig struct {
	LogFile     string
	MergeToggle int
}

SchedulerConfig holds relevant settings for the scheduler process.

type Settings

type Settings struct {
	DbUrl               string            `yaml:"dburl"`
	Db                  string            `yaml:"db"`
	WriteConcern        WriteConcern      `yaml:"write_concern"`
	ConfigDir           string            `yaml:"configdir"`
	ApiUrl              string            `yaml:"api_url"`
	AgentExecutablesDir string            `yaml:"agentexecutablesdir"`
	SuperUsers          []string          `yaml:"superusers"`
	Jira                JiraConfig        `yaml:"jira"`
	Providers           CloudProviders    `yaml:"providers"`
	Keys                map[string]string `yaml:"keys"`
	Credentials         map[string]string `yaml:"credentials"`
	AuthConfig          AuthConfig        `yaml:"auth"`
	RepoTracker         RepoTrackerConfig `yaml:"repotracker"`
	Monitor             MonitorConfig     `yaml:"monitor"`
	Api                 APIConfig         `yaml:"api"`
	Alerts              AlertsConfig      `yaml:"alerts"`
	Ui                  UIConfig          `yaml:"ui"`
	HostInit            HostInitConfig    `yaml:"hostinit"`
	Notify              NotifyConfig      `yaml:"notify"`
	Runner              RunnerConfig      `yaml:"runner"`
	Scheduler           SchedulerConfig   `yaml:"scheduler"`
	TaskRunner          TaskRunnerConfig  `yaml:"taskrunner"`
	Expansions          map[string]string `yaml:"expansions"`
	Plugins             PluginConfig      `yaml:"plugins"`
	IsProd              bool              `yaml:"isprod"`
}

Settings contains all configuration settings for running Evergreen.

func GetSettings

func GetSettings() (*Settings, error)

GetSettings returns Evergreen Settings or an error.

func GetSettingsOrExit

func GetSettingsOrExit() *Settings

GetSettingsOrExit loads the evergreen settings file, or exits with a non-0 code if any errors occur.

func NewSettings

func NewSettings(filename string) (*Settings, error)

NewSettings builds an in-memory representation of the given settings file.

func TestConfig

func TestConfig() *Settings

TestConfig creates test settings from a test config.

func (*Settings) Validate

func (settings *Settings) Validate(validators []ConfigValidator) error

Validate checks the settings and returns nil if the config is valid, or an error with a message explaining why otherwise.

type SliceAppender

type SliceAppender struct {
	Messages []*slogger.Log
}

SliceAppender is a slogger.Appender implemenation that just adds every log message to an internal slice. Useful for testing when a test needs to capture data sent to a slogger.Logger and verify what data was written.

func (*SliceAppender) Append

func (self *SliceAppender) Append(log *slogger.Log) error

type TaskRunnerConfig

type TaskRunnerConfig struct {
	LogFile string
}

TaskRunnerConfig holds logging settings for the scheduler process.

type UIConfig

type UIConfig struct {
	Url            string
	HelpUrl        string
	HttpListenAddr string
	// Secret to encrypt session storage
	Secret  string
	LogFile string
	// Default project to assume when none specified, e.g. when using
	// the /waterfall route use this project, while /waterfall/other-project
	// then use `other-project`
	DefaultProject string
	// Cache results of template compilation, so you don't have to re-read files
	// on every request. Note that if this is true, changes to HTML templates
	// won't take effect until server restart.
	CacheTemplates bool
}

UIConfig holds relevant settings for the UI server.

type WriteConcern

type WriteConcern struct {
	W        int    `yaml:"w"`
	WMode    string `yaml:"wmode"`
	WTimeout int    `yaml:"wtimeout"`
	FSync    bool   `yaml:"fsync"`
	J        bool   `yaml:"j"`
}