inspeqtor

package module
v0.7.0-2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 24, 2014 License: GPL-3.0 Imports: 30 Imported by: 0

README

Inspeqtor

GoDoc

Inspeqtor monitors your application infrastructure. It gathers and verifies key metrics from all the moving parts in your application and alerts you when something looks wrong. It understands the application deployment workflow so it won't bother you during a deploy.

What it does:

  • Monitor init.d-, systemd-, upstart-, runit- or launchd-managed services
  • Monitor process memory and CPU usage
  • Monitor daemon-specific metrics (e.g. redis, memcached, mysql, nginx...)
  • Monitor and alert based on host CPU, load, swap and disk usage
  • Alert or restart a process if a rule threshold is breached
  • Alert if a process disappears or changes PID
  • Signal deploy start/stop to silence alerts during deploy

What it doesn't:

  • monitor or control arbitrary processes, services must be init-managed
  • have any runtime dependencies at all, not even libc.

Installation

See the Inspeqtor wiki for complete documentation.

Requirements

Linux 3.0+. It will run on OS X. FreeBSD is untested. It uses about 5-10MB of RAM at runtime.

Upgrade

Inspeqtor Pro is the commercial upgrade for Inspeqtor and offers more features, dedicated support and a non-GPL license:

  • Monitor cron job execution
  • Export metrics to Statsd
  • Route alerts to different teams or individuals
  • Send alerts to Slack or other team chat rooms
  • More to come...

To buy, go to the homepage and select the right plan for your organization.

See the wiki documentation for in-depth documentation around each Pro feature.

License

Inspeqtor is licensed under GPLv3. Inspeqtor Pro is licensed under the custom commercial license in COMM-LICENSE.

Want to Help?

See the Development wiki page for details on how to get the source code and build Inspeqtor locally.

Author

Inspeqtor is written by Mike Perham of Contributed Systems. We build awesome open source-based infrastructure to help you build awesome apps.

We also develop Sidekiq and sell Sidekiq Pro, the best Ruby background job processing system.

Documentation

Index

Constants

View Source 
const (
	VERSION = "0.7.0"
)

Variables

View Source 
var (
	Actions = map[string]ActionBuilder{
		"alert":   buildAlerter,
		"restart": buildRestarter,
		"reload":  buildReloader,
	}
	Notifiers = map[string]NotifierBuilder{
		"email": buildEmailNotifier,
		"gmail": buildGmailNotifier,
		"null":  buildNullNotifier,
	}
)
View Source 
var (
	BuildHost    = convertHost
	BuildService = convertService
	BuildRule    = convertRule
	BuildAction  = convertAction
)
View Source 
var (
	Term os.Signal = syscall.SIGTERM
	Hup  os.Signal = syscall.SIGHUP

	SignalHandlers = map[os.Signal]func(*Inspeqtor){
		Term:         exit,
		os.Interrupt: exit,
		Hup:          reload,
	}
	Name      string                               = "Inspeqtor"
	Licensing string                               = "Licensed under the GNU Public License 3.0"
	Singleton *Inspeqtor                           = nil
	Reloaders []func(*Inspeqtor, *Inspeqtor) error = []func(*Inspeqtor, *Inspeqtor) error{basicReloader}
)
View Source 
var (
	CommandHandlers = map[string]commandFunc{
		"start":  startDeploy,
		"finish": finishDeploy,
		"status": currentStatus,
		"export": export,
		"show":   sparkline,
		"♡":      heart,
	}
)
View Source 
var Defaults = GlobalConfig{15, 300, map[string]string{}}
View Source 
var (
	Events = []EventType{ProcessDoesNotExist, ProcessExists, RuleFailed, RuleRecovered}
)

Functions

func HandleSignal 

func HandleSignal(sig os.Signal, handler func(*Inspeqtor))

func HandleSignals 

func HandleSignals()

Types

type Action 

type Action interface {
	Trigger(event *Event) error
}

type ActionBuilder 

type ActionBuilder func(Eventable, *AlertRoute) (Action, error)

An Action is something which is triggered when a rule is broken. This is typically either a Notification, Restart or a Reload to the the service.

type AlertRoute 

type AlertRoute struct {
	Name    string
	Channel string
	Config  map[string]string
}

An alert route is a way to send an alert to a recipient.

Channel is the notification mechanism: email, campfire, etc. Config is an undefined set of kv pairs for configuring the channel.

The configuration looks like this: 

send alerts
  via CHANNEL with K V, K V, K V

You'd then write a rule like: 

if foo > 10 then alert

func ValidateChannel 

func ValidateChannel(name string, channel string, config map[string]string) (*AlertRoute, error)

type Checkable 

type Checkable interface {
	Name() string
	Parameter(string) string
	Metrics() metrics.Store
	Resolve([]services.InitSystem) error
	Rules() []*Rule
	Verify() []*Event
	Collect(bool, func(Checkable))
}

func ParseServices 

func ParseServices(global *ConfigFile, confDir string) ([]Checkable, error)

Parses the service-specific rules in /etc/inspeqtor/services.d/*.inq

type ConfigFile 

type ConfigFile struct {
	GlobalConfig
	AlertRoutes map[string]*AlertRoute
}

func ParseGlobal 

func ParseGlobal(rootDir string) (*ConfigFile, error)

type Controllable 

type Controllable interface {
	Restart() error
	Reload() error
}

A Service is Controllable, a Host is not.

type EmailEvent 

type EmailEvent struct {
	*Event
	Config *EmailNotifier
}

type EmailNotifier 

type EmailNotifier struct {
	Username string
	Password string
	Host     string
	From     string
	To       string
}

func (EmailNotifier) Trigger 

func (e EmailNotifier) Trigger(event *Event) error

func (*EmailNotifier) TriggerEmail 

func (e *EmailNotifier) TriggerEmail(event *Event, sender EmailSender) error

type EmailSender 

type EmailSender func(e *EmailNotifier, doc bytes.Buffer) error

type Entity 

type Entity struct {
	// contains filtered or unexported fields
}

A named thing which can checked by Inspeqtor

func (*Entity) CycleTime 

func (e *Entity) CycleTime() uint

func (*Entity) Metrics 

func (e *Entity) Metrics() metrics.Store

func (*Entity) Name 

func (e *Entity) Name() string

func (*Entity) Parameter 

func (e *Entity) Parameter(key string) string

func (*Entity) Parameters 

func (e *Entity) Parameters() map[string]string

func (*Entity) Rules 

func (e *Entity) Rules() []*Rule

type Event 

type Event struct {
	Type EventType
	Eventable
	*Rule
}

func (*Event) Hostname 

func (e *Event) Hostname() string

func (*Event) Service 

func (e *Event) Service() *Service

func (*Event) Target 

func (e *Event) Target() string

func (*Event) Thing 

func (e *Event) Thing() Eventable

type EventType 

type EventType string

There are several different types of Events:

* Process disappeared (or did not exist when we started up) * Process appeared * Rule failed check * Rule has recovered 

const (
	ProcessDoesNotExist EventType = "ProcessDoesNotExist"
	ProcessExists       EventType = "ProcessExists"
	RuleFailed          EventType = "RuleFailed"
	RuleRecovered       EventType = "RuleRecovered"
)

func (EventType) String 

func (s EventType) String() string

type Eventable 

type Eventable interface {
	Name() string
	Parameter(string) string
}

type GlobalConfig 

type GlobalConfig struct {
	CycleTime    uint
	DeployLength uint
	Variables    map[string]string
}

Parses the global inspeqtor configuration in /etc/inspeqtor/inspeqtor.conf.

type Host 

type Host struct {
	*Entity
}

Host is the local machine.

func NewHost 

func NewHost() *Host

func ParseHost 

func ParseHost(global *ConfigFile, hostInq string) (*Host, error)

Parses the host-specific rules in /etc/inspeqtor/host.inq

func (*Host) Collect 

func (h *Host) Collect(silenced bool, completeCallback func(Checkable))

func (*Host) Resolve 

func (h *Host) Resolve(_ []services.InitSystem) error

func (*Host) Verify 

func (s *Host) Verify() []*Event

type Inspeqtor 

type Inspeqtor struct {
	RootDir    string
	SocketPath string
	StartedAt  time.Time

	ServiceManagers []services.InitSystem
	Host            *Host
	Services        []Checkable
	GlobalConfig    *ConfigFile
	Socket          net.Listener
	SilenceUntil    time.Time
	Stopping        chan struct{}
	Handlers        map[string][]func(*Inspeqtor)
}

func New 

func New(dir string, socketpath string) (*Inspeqtor, error)

func (*Inspeqtor) Fire 

func (i *Inspeqtor) Fire(eventName string)

func (*Inspeqtor) Listen 

func (i *Inspeqtor) Listen(eventName string, handler func(*Inspeqtor))

func (*Inspeqtor) Parse 

func (i *Inspeqtor) Parse() error

func (*Inspeqtor) Shutdown 

func (i *Inspeqtor) Shutdown()

func (*Inspeqtor) Start 

func (i *Inspeqtor) Start()

func (*Inspeqtor) TestAlertRoutes 

func (i *Inspeqtor) TestAlertRoutes() int

type NotifierBuilder 

type NotifierBuilder func(Eventable, map[string]string) (Action, error)

A Notifier is a route to send an alert somewhere else. The global conf sets up the necessary params for the notification to work.

type NullNotifier 

type NullNotifier struct {
	LastEvent *Event
}

Useful for testing

func (*NullNotifier) Trigger 

func (e *NullNotifier) Trigger(event *Event) error

type Operator 

type Operator uint8
const (
	LT Operator = iota
	GT
)

func (Operator) String 

func (o Operator) String() string

type Reloader 

type Reloader struct {
	*Service
}

func (Reloader) Trigger 

func (r Reloader) Trigger(event *Event) error

type Restarter 

type Restarter struct {
	*Service
}

func (Restarter) Trigger 

func (r Restarter) Trigger(event *Event) error

type Rule 

type Rule struct {
	Entity           Checkable
	MetricFamily     string
	MetricName       string
	Op               Operator
	DisplayThreshold string
	Threshold        float64
	CurrentValue     float64
	PerSec           bool
	CycleCount       int
	TrippedCount     int
	State            RuleState
	Actions          []Action
}

func (*Rule) Check 

func (rule *Rule) Check(cycleTime uint) *Event

Run through all Rules and check if we need to trigger actions.

"tripped" means the Rule threshold was breached **this cycle**. "triggered" means the Rule threshold was breached enough cycles in a row to fire the alerts associated with the Rule.

There are three possible states for Rules: Ok - rule was fine last time and passed this time too. Triggered - threshold breached enough times, action should be taken Recovered - rule is currently Triggered but threshold was not breached this time

func (*Rule) Consequence 

func (r *Rule) Consequence() string

func (*Rule) DisplayState 

func (r *Rule) DisplayState() string

func (*Rule) EntityName 

func (r *Rule) EntityName() string

func (*Rule) FetchDisplayCurrentValue 

func (r *Rule) FetchDisplayCurrentValue() string

func (*Rule) FetchLatestMetricValue 

func (r *Rule) FetchLatestMetricValue() float64

func (*Rule) Metric 

func (r *Rule) Metric() string

func (*Rule) Reset 

func (r *Rule) Reset()

type RuleState 

type RuleState string
const (
	Ok        RuleState = "Ok"
	Triggered RuleState = "Triggered"
	Recovered RuleState = "Recovered"
)

func (RuleState) String 

func (o RuleState) String() string

type Service 

type Service struct {
	*Entity
	// Handles process events: exists, doesn't exist
	EventHandler Action
	Process      *services.ProcessStatus
	Manager      services.InitSystem
}

A service is an Entity which resolves to a Process we can monitor.

func NewService 

func NewService(name string) *Service

func (*Service) Collect 

func (svc *Service) Collect(silenced bool, completeCallback func(Checkable))

Called for each service each cycle, in parallel. This method must be thread-safe. Since this method executes in a goroutine, errors must be handled/logged here and not just returned.

Each cycle we need to: 1. verify service is Up and running. 2. capture process metrics 3. run rules 4. trigger any necessary actions

func (*Service) Reload 

func (s *Service) Reload() error

func (*Service) Resolve 

func (svc *Service) Resolve(mgrs []services.InitSystem) error

Resolve each defined service to its managing init system. Called only at startup, this is what maps services to init and fires ProcessDoesNotExist events.

func (*Service) Restart 

func (s *Service) Restart() error

func (*Service) SetMetrics 

func (s *Service) SetMetrics(newStore metrics.Store)

func (*Service) String 

func (s *Service) String() string

func (*Service) Transition 

func (s *Service) Transition(ps *services.ProcessStatus, emitter func(EventType))

func (*Service) Verify 

func (s *Service) Verify() []*Event

Source Files

View all Source files

Directories

Path Synopsis
conf
global/ast
global/errors
global/lexer
global/parser
global/token
global/util
inq/ast
inq/errors
inq/lexer
inq/parser
inq/token
inq/util
daemon
runit manages services usually found in /etc/service or /service, which are soft links to the actual service directories in /etc/sv: <service_name>/ run log/ run supervise/ pid # => 4994 stat # => run / down
 runit manages services usually found in /etc/service or /service, which are soft links to the actual service directories in /etc/sv: <service_name>/ run log/ run supervise/ pid # => 4994 stat # => run / down

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.