backup

README

a8n-backup

a8n-backup is a wrapper for Restic to back up files.

Note: Do NOT use in production. This is alpha quality code with very few features.

Introduction

Here is the general process to run a8n-backup. The detailed walk through is further down.

1. git clone the repo
2. Create the backup config files.
3. go build the backup CLI tool to run the management commands.
4. Run the backup tool to do the following.
   1. Generate the encryption keys.
   2. Encrypt the backup config file.
   3. Build the a8n-backup agent.
   4. Upload the a8n-backup agent to a file server.
5. Upload the a8n-backup-linux.sh script to TRMM.
6. Add the encrypted payload as a client/site/agent custom field to TRMM.
7. Run the script.
   1. Run the script with restic init to initialize the repository. Do this if this is your first time running Restic.
   2. Run the script with backup to run the backup.

TODO

• Cleanup and improve logging. Escape sequences are output which makes reading the logs much harder. The logs are only needed when problems happen. Is there a way to capture the logs and output them only if there was an error?
• Support Windows (PowerShell) and Mac (TRMM agent is needed).
• Add TRMM integration.
• Add support for pipes.
• Check for newer versions and update automatically.
• Support other file servers. Only GoHttpServer is supported.

Wish list

• Use koanf for configurations. koanf supports HCL, etcd, and Vault.
• Kopia may be added in the future.
• wal-g is an archival restoration tool for PostgreSQL, MySQL/MariaDB, and MS SQL Server (beta for MongoDB and Redis).
• wal-e - Continuous Archiving for Postgres
• Add a GUI

Limitations

The encryption keys are per file generated, and that file has "a8n-backup_${os}_${arch}" format. This means the same OS/Arch combination will have the same name and same encryption key. Separating desktop from servers is not possible. While the agent can do this for Windows, it's not so easy for MacOS and Linux.

The client configuration files contain the Restic repository encryption key and needs to be kept secure. These files are used as input to into a8n-backup with the output being an encrypted base64 blob that does not need to be kept. The encryption keys are also output. This is clumsy and very much open to human error. Managing these configurations in etcd and secrets in Vault is a primary goal.

There is a one-to-one correspondence between the client configuration files and the encrypted payload that is added to TRMM. This can get out of hand very fast. Each backup configuration has a config file and corresponding payload, and if the custom fields do not have a good naming convention, things will get messy. This is susceptible to human error.

Procedures

Here are the procedures to get up and running.

1. git clone https://gitlab.com/a8n/backup.git
2. Copy manage/templates/agent.hcl somewhere and edit as necessary. Keep this in a safe place.
3. cd cmd/backup and then go build to build the backup tool. This is used for management purposes.
4. Run this command to create the encrypted payload, build the agent and upload it to the file server.
   • ./backup manage --data ~/secure/config/dir/ --agent-config agent-client-location-app.hcl
5. Add *.trmm.json.base64 as a custom fields in TRMM. Name the custom field something like a8n-backup-{client}-{application}. If it helps, add more fields: a8n-backup-{client}-{site}-{location}-{application}. This helps identify the contents of the custom field as there is no other indication for what the field represents.
   • Note: base64 --decode <*.trmm.json.base64 will show the JSON.
6. Add manage/scripts/a8n-backup-linux.sh script to the script library in TRMM.
7. The first argument needs to be the encrypted payload. Add this to the Script Arguments.
   • {{client.a8n-backup-client-site-location-application}}
8. The second argument is the command. General backups will use backup.

Housekeeping

Save/keep these items in a safe place like a password manager or encrypted directory.

• The unencrypted HCL config. In particular:
  • Restic encryption key. This is also referred to as the restic password.
  • Storage provider encryption password or api key.
• (Optional) TRMM private key to generate more backup configs.

How it works

The management command will generate an two encryption key pairs, one for TRMM and one for the agent. The agent configuration is encrypted with TRMM's private key and the Agent's public key. The agent is then compiled with the Agent's private key. TRMM's public key is included in the trmm.json base64 payload and exported as an env var. The Agent will use TRMM's public key and its private key to decrypt the settings. The necessary variables are exported as env vars for Restic to use. Then Restic is run.

GoHttpServer is used for the file server. This is protected by a single HTTP basic auth user. Issue #111 is to support more than one basic auth user. The upload and download sections because they can have separate credentials. It's possible to run two servers, one that is accessible to admins to upload files and one that is accessible to the agents to download files.

Documentation

Overview

Package backup integrates with TRMM to provide backups.

Package backup integrates with TRMM to provide backups.

Package backup integrates with TRMM to provide backups.

Package backup integrates with TRMM to provide backups.

Package backup integrates with TRMM to provide backups.

Index

• Constants
• Variables
• func SetLogging(l *LoggingConfig)
• type Backup
  • func NewBackup() *Backup
  • func (b *Backup) GetCLI() (err error)
  • func (b *Backup) Run() (err error)
• type CLI
  • func NewCLI() *CLI
  • func (cli *CLI) ParseCLI() (err error)
  • func (cli *CLI) ValidateCLI() (err error)
• type LoggingConfig

Constants

const (
	URL          = "https://gitlab.com/a8n/backup"
	AUTHORS      = "David Randall"
	LICENSE      = "PostgreSQL"
	COPYRIGHT    = "(c) 2022 The a8n Authors"
	ORGANIZATION = "Nice Guy IT, LLC"
	YEAR         = "2022"
)

Variables

var (
	// NAME is used as the -o target in the build process.
	NAME = "a8n-backup"
	// VERSION TODO: Add API version, GUI version, and possibly other versions.
	VERSION    = "unknown version"
	BUILD_TIME = "unknown time"
	BUILD_ENV  = "unknown env"
	GIT_COMMIT = "unknown git commit"
	GIT_TAG    = "unknown git tag"
	GIT_BRANCH = "unknown git branch"
)

External variables pulled in during the build process

var LOG_COLOR = true

Functions

func SetLogging

func SetLogging(l *LoggingConfig)

Types

type Backup

type Backup struct {
	CLI    *CLI
	Manage *manage.Manage
	Agent  *manage.Agent
	Crypt  *manage.Secrets
	TRMM   *manage.TRMM
}

Backup is the backup config

func NewBackup

func NewBackup() *Backup

NewBackup starts a new backup request.

func (*Backup) GetCLI

func (b *Backup) GetCLI() (err error)

GetCLI gets the configuration from the cli and env vars.

func (*Backup) Run

func (b *Backup) Run() (err error)

Run the command specified on the command line

type CLI

type CLI struct {
	// Manage is the user facing function for managing the backup configs
	Manage struct {
		ConfigDir string `kong:"name='data',group='manage',required='',placeholder='/secure/dir',help='Directory of the config files.'"`
		// FIXME: existingfile or existingdir with a default value will check for the file/dir even if another
		// command is run. Remove the default value for now.
		// https://github.com/alecthomas/kong/issues/26
		AgentConfig string `` /* 135-byte string literal not displayed */

	} `kong:"cmd='',group='manage',help='Manage the backup configuration.'"`

	// Currently not used
	TRMM struct {
		Client string `kong:"name='client',optional,help='TRMM Client'"`
		Site   string `kong:"name='site',optional,help='TRMM Site'"`
		Agent  string `kong:"name='agent',optional,help='TRMM AgentConfig'"`
		Script string `kong:"name='script',arg='script',help='Script to output.'"`
	} `kong:"cmd='',group='trmm',help='Tactical RMM functions.'"`

	// Run will run the backup
	Agent struct {
		Backup struct {
			Paths []string `kong:"arg='',optional='',group='agent',help='restic command'"`
		} `kong:"cmd='',group='agent',help='Run the backup'"`
		Restic struct {
			Command []string `kong:"arg='',group='agent',help='restic command'"`
		} `kong:"cmd='',group='agent',help='Run a restic command'"`
	} `kong:"cmd='',group='agent',help='Commands to be run on the agents.'"`

	// Logging options
	Logging struct {
		// https://github.com/rs/zerolog#leveled-logging
		Level string `kong:"enum='trace,debug,info,warn,error',default='info',help='Log level'"`
		Type  string `kong:"enum='json,console',default='console',help='Log type'"`
	} `kong:"embed='',prefix='log-'"`

	// Version
	Version struct {
	} `kong:"cmd='',group='version',name='version'"`

	// License
	License struct {
	} `kong:"cmd='',group='license',name='license'"`

	// FileRepo
	FileRepo struct {
	} `kong:"cmd='',group='filerepo',name='filerepo'"`
}

CLI contains the command line configuration

func NewCLI

func NewCLI() *CLI

NewCLI starts a new configuration.

func (*CLI) ParseCLI

func (cli *CLI) ParseCLI() (err error)

ParseCLI command line options

func (*CLI) ValidateCLI

func (cli *CLI) ValidateCLI() (err error)

ValidateCLI will validate the config (outside Kong) and set sane defaults. Conflicts with "Validate" since it will be called by Kong. See https://github.com/alecthomas/kong#validation Kong checks if the file exists with the type='existingfile' struct tag

type LoggingConfig

type LoggingConfig struct {
	// https://github.com/rs/zerolog#leveled-logging
	Level string `enum:"debug,info,warn,error" default:"info"`
	Type  string `enum:"json,console" default:"console"`
}