repbak

README

Repbak

Repbak is a simple database backup tool made specifically to backup replicated databases. Repbak will send notifications based on database backup failure. Repbak also optionally support HTTP healthchecks for liveness.

Supported Dumpers

• mysqldump

Supported Notifications

• Email

How does it work?

1. Download the latest release.
2. Create a YAML configuration file
3. Run it repbak -conf repbak.yaml

Configuration file

The YAML file defines repbak's operation.

Full config example

log_path: /var/log/repbak.log
log_level: error
lib_path: /var/lib/repbak
time_format: Mon Jan 02 03:04:05 PM MST
retention: 7
http:
  addr: 0.0.0.0
  port: 4060
mysqldump:
  retention: 30
  output_path: /mnt/backups/mysql.dump
  schedule: "0 0 * * *"
  executable_path: mysqldump
  executable_args: --add-drop-database --all-databases -u user -ppass -h 127.0.0.1
  time_limit: 8h
email:
  host: mail.me.com
  port: 587
  user: me
  pass: pass
  starttls: true
  ssl: false
  subject: Database Replication Failure
  from: me@me.com
  to:
    - you@me.com
  history_subject: Database Backup History
  history_schedule: "0 0 * * *"
  history_template: /home/repbak/email.template
  on_failure: true

Global Options

log_path - File on disk where repbak logs will be stored. Defaults to /var/log/repbak.log.

log_level - Sets the log level. Valid levels are: panic, fatal, trace, debug, warn, info, and error. Defaults to error.

lib_path - The directory on disk where repbak lib files are stored. Defaults to /var/lib/repbak.

time_format - The format used when displaying backup stats. See formatting options in the go time.Time package. Defaults to Mon Jan 02 03:04:05 PM MST.

retention - The number of stats that are stored for each backup. If set to less than 0 no stats are saved. Defaults to 7.

HTTP

addr - The listening address for the HTTP server. Default to 127.0.0.1

port - The listening port for the HTTP server. Default to 4040

mysqldump

retention - The number of backups to keep before rotating old backups out. Defaults to 7.

output_path - The path where backups will be stored.

schedule - The cron expression that defines when backups are created.

executable_path - The path to the mysqldump binary. Defaults to mysqldump.

executable_args - The arguments passed to the executable used to create the mysql backup. Defaults to --add-drop-database --all-databases.

time_limit - Optional limit to the time it takes to run the backup.

Email

host - The hostname or IP of the SMTP server.

port - The port of the SMTP server.

user - The username used to authenticate.

pass - The password used to authenticate.

start_tls - StartTLS enables TLS security. If both StartTLS and SSL are true then StartTLS will be used.

insecure_skip_verify - When using TLS skip verifying the server's certificate chain and host name.

ssl - SSL enables SSL security. If both StartTLS and SSL are true then StartTLS will be used.

from - The email address the email will be sent from.

to - An array of email addresses for which emails will be sent.

history_subject - An optional subject to use when sending sync history emails. Defaults to Database Backup History.

history_schedule - An optional cron expression. If set then an email with sync history will be sent based on the schedule.

history_template - An optional path to an email template to use when sending history emails. If not set uses the default template.

on_failure - An optional value that will send an email for each backup failure if true.

Flags

-conf - Path to the repbak configuration file

-debug - Log to STDOUT

HTTP Health Checks

The optional HTTP server creates two endpoints.

/live - A liveness check that always returns 200.

/health - A health check that returns 200 if the latest run for each backup was successful and 503 otherwise.

Road Map

• Docker Image
• Systemd service file
• Create rpm
• Create deb
• Support for more dumpers
• Support for more notifiers

Documentation

Index

• type BoltDB
  • func NewBoltDB(config *Config) (*BoltDB, error)
  • func (s *BoltDB) Close() error
  • func (s *BoltDB) Insert(stat Stat) error
  • func (s *BoltDB) List() (map[string][]Stat, error)
  • func (s *BoltDB) Prune() error
• type Config
  • func OpenConfig(path string) (*Config, error)
• type DB
• type Dumper
• type Email
• type EmailNotifier
  • func NewEmailNotifier(config *Config) *EmailNotifier
  • func (n *EmailNotifier) Notify(stat Stat) error
  • func (n *EmailNotifier) NotifyHistory(statMap map[string][]Stat) error
• type HTTP
• type MySQLDump
• type MySQLDumpDumper
  • func NewMySQLDumpDumper(config *Config) *MySQLDumpDumper
  • func (d *MySQLDumpDumper) Dump() Stat
  • func (d *MySQLDumpDumper) Stop()
• type Notifier
• type RepBak
  • func New(config *Config, db DB, dumper Dumper, notifier Notifier) *RepBak
  • func (r *RepBak) Start() error
  • func (r *RepBak) Stop()
• type Stat
  • func NewStat(name string, format string) Stat
  • func (s Stat) Finish(err error) Stat

Types

type BoltDB

type BoltDB struct {
	// contains filtered or unexported fields
}

BoltDB is the default and only database for storing stats. In the future other databases could be added.

func NewBoltDB

func NewBoltDB(config *Config) (*BoltDB, error)

NewBoltDB creates the underlying boltdb database. If retention is less than 1 than the database isn't created and all calls are no-ops.

func (*BoltDB) Close

func (s *BoltDB) Close() error

Close closes the bolt database file.

func (*BoltDB) Insert

func (s *BoltDB) Insert(stat Stat) error

Insert adds one Stat to bolt.

func (*BoltDB) List

func (s *BoltDB) List() (map[string][]Stat, error)

List returns all stats stored as a map. The map keys are sync names and the values are a list of all stored stats for. that sysnc. Stats are returned storted by Start in descending order.

func (*BoltDB) Prune

func (s *BoltDB) Prune() error

Prune removes all entries for a Sync job that exceed the rentention config.

type Config

type Config struct {
	// LogPath is the oath on disk where repbak log file. Defaults to /var/log/repbak.log.
	LogPath string `yaml:"log_path"`

	// LogLevel sets the level of logging. Valid levels are: panic, fatal, trace, debug, warn, info, and error. Defaults to error
	LogLevel string `yaml:"log_level"`

	// LibPath is the directory on disk where repbak lib files are stored. Defaults to /var/lib/repbak.
	LibPath string `yaml:"lib_path"`

	// The time format used when displaying backup stats. See formatting options in the go time.Time package.
	// Defaults to Mon Jan 02 03:04:05 PM MST
	TimeFormat string `yaml:"time_format"`

	// Retention is the number of logs and stats that are stored for each backup. If set to less than 0 no
	// logs or stats are saved. Defaults to 7.
	Retention int `yaml:"retention"`

	HTTP      *HTTP      `yaml:"http"`
	MySQLDump *MySQLDump `yaml:"mysqldump"`
	Email     *Email     `yaml:"email"`
}

Config is an object representation of the YAML configuration file.

func OpenConfig

func OpenConfig(path string) (*Config, error)

OpenConfig returns a new Config option by reading the YAML file at path. If the file doesn't exist, can't be read, is invalid YAML, or doesn't match the repbak spec then an error is returned.

type DB

type DB interface {
	// Prune removes old stats based on the configured retention value.
	Prune() error

	// List returns all stats stored as a map. The map keys are sync names and the values are a list of all stored stats for.
	//that sysnc. Stats should be returned storted by Start in descending order.
	List() (map[string][]Stat, error)

	// Insert adds a stat to the database.
	Insert(Stat) error

	// Closes the connection the database
	Close() error
}

DB defines an interface for persisting Stats. Stats are simple metrics for each job.

type Dumper

type Dumper interface {
	// Dump does a backup of the database
	Dump() Stat

	// Stop stops the database backup if one is running
	Stop()
}

Dumper defines an interface for backing up a database.

type Email

type Email struct {
	// Host is the hostname or IP of the SMTP server.
	Host string `yaml:"host"`

	// Port is the port of the SMTP server.
	Port int `yaml:"port"`

	// User is the username used to authenticate.
	User string `yaml:"user"`

	// Pass is the password used to authenticate.
	Pass string `yaml:"pass"`

	// StartTLS enables TLS security. If both StartTLS and SSL are true then StartTLS will be used.
	StartTLS bool `yaml:"starttls"`

	// Skip verifying the server's certificate chain and host name.
	InsecureSkipVerify bool `yaml:"insecure_skip_verify"`

	// SSL enables SSL security. If both StartTLS and SSL are true then StartTLS will be used.
	SSL bool `yaml:"ssl"`

	// Optional subject field for notification emails
	Subject string `yaml:"subject"`

	// From is the email address the email will be sent from.
	From string `yaml:"from"`

	// To is an array of email addresses for which emails will be sent.
	To []string `yaml:"to"`

	// HistorySubject is an optional subject to use when sending sync history emails. Defaults to Database Backup History.
	HistorySubject string `yaml:"history_subject"`

	// HistorySchedule is a cron expression. If set then an email with sync history will be sent based on the schedule.
	HistorySchedule string `yaml:"history_schedule"`

	// HistoryTemplate is an optional path to an email template to use when sending history emails. If not set uses the default template.
	HistoryTemplate string `yaml:"history_template"`

	// OnFailure will send an email for each backup failure if true.
	OnFailure bool `yaml:"on_failure"`
}

type EmailNotifier

type EmailNotifier struct {
	// contains filtered or unexported fields
}

EmailNotifier sends emails based on repliaction failure

func NewEmailNotifier

func NewEmailNotifier(config *Config) *EmailNotifier

NewEmailNotifier creates a EmailNotifier using the config

func (*EmailNotifier) Notify

func (n *EmailNotifier) Notify(stat Stat) error

Notify sends a failure notification

func (*EmailNotifier) NotifyHistory

func (n *EmailNotifier) NotifyHistory(statMap map[string][]Stat) error

type HTTP

type HTTP struct {
	// The address the http server will listen on.
	Addr string `yaml:"addr"`

	// The port the http server will listen on.
	Port int `yaml:"port"`
}

HTTP defines the configuration for http health checks.

type MySQLDump

type MySQLDump struct {
	// Retention is the number of backups to keep before rotating old backups out. Defaults to 7.
	Retention int `yaml:"retention"`

	// OutputPath is the path where backups will be stored.
	OutputPath string `yaml:"output_path"`

	// Schedule is the cron expression that defines when backups are created.
	Schedule string `yaml:"schedule"`

	// ExecutablePath is the path to the tool used to create the mysql backup. Defaults to mysqldump.
	ExecutablePath string `yaml:"executable_path"`

	// ExecutableArgs are the arguments passed to the executable used to create the mysql backup. Defaults to --add-drop-database --all-databases.
	ExecutableArgs string `yaml:"executable_args"`

	// TimeLimit is an optional limit to the time it takes to run the backup.
	TimeLimit string `yaml:"time_limit"`
	// contains filtered or unexported fields
}

MySQL defines how a mysql backup will be created.

type MySQLDumpDumper

type MySQLDumpDumper struct {
	// contains filtered or unexported fields
}

MySQLDumpDumper dumps a mysql backup to a file.

func NewMySQLDumpDumper

func NewMySQLDumpDumper(config *Config) *MySQLDumpDumper

NewMySQLDumpDumper creates a NewMySQLDumpDumper.

func (*MySQLDumpDumper) Dump

func (d *MySQLDumpDumper) Dump() Stat

Dump dumps the mysql data to a file based on the settings in config.

func (*MySQLDumpDumper) Stop

func (d *MySQLDumpDumper) Stop()

Stop stops the current dump if one is running.

type Notifier

type Notifier interface {
	// Notify sends a notification
	Notify(stat Stat) error

	// Notify History sends a notification with the backup history.
	NotifyHistory(map[string][]Stat) error
}

Notifier defines a notification method.

type RepBak

type RepBak struct {
	// contains filtered or unexported fields
}

RepBak reforms scheduled database backups.

func New

func New(config *Config, db DB, dumper Dumper, notifier Notifier) *RepBak

New returns a new RepBak instance.

func (*RepBak) Start

func (r *RepBak) Start() error

Start runs until stopped and does scheduled database backups.

func (*RepBak) Stop

func (r *RepBak) Stop()

Stop stops repmon from running for schedule database backups.

type Stat

type Stat struct {
	Name     string
	Success  bool
	Start    string
	End      string
	Duration time.Duration
	Error    error `json:"-"`
	Skip     bool
	// contains filtered or unexported fields
}

Stat defines basic statistics for a single sync. Stats are stored so that historical data from past syncs can be viewed.

func NewStat

func NewStat(name string, format string) Stat

NewStat creates a new Stat with Name set to name, Success set to false, and Start set to the current time.

func (Stat) Finish

func (s Stat) Finish(err error) Stat

Finish sets the Success based on err, End based on the current time, and Duration based on Start and End.