keysync

package module
v0.0.0-...-b8ca463 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 4, 2023 License: Apache-2.0 Imports: 34 Imported by: 0

README

Deprecated

As of 9/18/23 this project is now deprecated and no longer maintained; we recommend using HashiCorp Vault as a more robust and actively supported alternative.

Keysync

license report

Keysync is a production-ready program for accessing secrets in Keywhiz.

It is a replacement for the now-deprecated FUSE-based keywhiz-fs.

Getting Started

Building

Keysync must be built with Go 1.11+. You can build keysync from source:

$ git clone https://github.com/square/keysync
$ cd keysync
$ go build github.com/square/keysync/cmd/keysync

This will generate a binary called ./keysync

Dependencies

Keysync uses Go modules to manage dependencies. If you've cloned the repo into GOPATH, you should export GO111MODULE=on before running any go commands. All deps should be automatically fetched when using go build and go test. Add go mod tidy before committing.

Testing

Entire test suite:

go test ./...

Short, unit tests only:

go test -short ./...
Running locally

Keysync requires access to Keywhiz to work properly. Assuming you run Keywhiz locally on default port (4444), you can start keysync with:

./keysync --config keysync-config.yaml

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func BackupFromConfig 

func BackupFromConfig(cfg *Config) (backup.Backup, error)

func NewAPIServer 

func NewAPIServer(syncer *Syncer, backup backup.Backup, port uint16, baseLogger *logrus.Entry, metrics *sqmetrics.SquareMetrics)

NewAPIServer is the constructor for an APIServer

Types

type APIServer 

type APIServer struct {
	// contains filtered or unexported fields
}

APIServer holds state needed for responding to HTTP api requests

type BackupBundleClient 

type BackupBundleClient struct {
	// contains filtered or unexported fields
}

BackupBundleClient is a secrets client that reads from a Keywhiz backup bundle.

func (BackupBundleClient) Logger 

func (c BackupBundleClient) Logger() *logrus.Entry

Logger returns the logger for this client.

func (BackupBundleClient) RebuildClient 

func (c BackupBundleClient) RebuildClient() error

RebuildClient for bundle clients is a no-op.

func (BackupBundleClient) Secret 

func (c BackupBundleClient) Secret(name string) (secret *Secret, err error)

Secret returns secret with the given name from the bundle.

func (BackupBundleClient) SecretList 

func (c BackupBundleClient) SecretList() (map[string]Secret, error)

SecretList returns all secrets in a bundle (unlike the real Keywhiz interface, it will return secrets' contents as well).

func (BackupBundleClient) SecretListWithContents 

func (c BackupBundleClient) SecretListWithContents(secrets []string) (map[string]Secret, error)

SecretListWithContents returns the requested secrets from a bundle.

type Client 

type Client interface {
	Secret(name string) (secret *Secret, err error)
	SecretList() (map[string]Secret, error)
	SecretListWithContents(secrets []string) (map[string]Secret, error)
	Logger() *logrus.Entry
	RebuildClient() error
}

Client represents an interface to a secrets storage backend.

func NewBackupBundleClient 

func NewBackupBundleClient(path string, logger *logrus.Entry) (Client, error)

NewBackupBundleClient creates a new BackupBundleClient instance given a backup JSON file.

func NewClient 

func NewClient(cfg *ClientConfig, caFile string, serverURL *url.URL, logger *logrus.Entry, metricsHandle *sqmetrics.SquareMetrics) (client Client, err error)

NewClient produces a ready-to-use client struct given client config and CA file with the list of trusted certificate authorities.

type ClientConfig 

type ClientConfig struct {
	Key        string `yaml:"key"`       // Mandatory: Path to PEM key to use
	Cert       string `yaml:"cert"`      // Optional: PEM Certificate (If cert isn't in key file)
	User       string `yaml:"user"`      // Optional: User and Group are defaults for files without metadata
	DirName    string `yaml:"directory"` // Optional: What directory under SecretsDir this client is in. Defaults to the client name.
	Group      string `yaml:"group"`     // Optional: If unspecified, the global defaults are used.
	MaxRetries uint16
	Timeout    string
	MinBackoff string
	MaxBackoff string
}

The ClientConfig describes a single Keywhiz client. There are typically many of these per keysync instance.

type Config 

type Config struct {
	ClientsDir    string            `yaml:"client_directory"`  // A directory of configuration files
	SecretsDir    string            `yaml:"secrets_directory"` // The directory secrets will be written to
	CaFile        string            `yaml:"ca_file"`           // The CA to trust (PEM) for Keywhiz communication
	YamlExt       string            `yaml:"yaml_ext"`          // The filename extension of the yaml config files
	PollInterval  string            `yaml:"poll_interval"`     // If specified, poll at the given interval, otherwise, exit after syncing
	ClientTimeout string            `yaml:"client_timeout"`    // If specified, timeout client connections after specified duration, otherwise use default.
	MinBackoff    string            `yaml:"min_backoff"`       // If specified, wait time before first retry, otherwise, use default.
	MaxBackoff    string            `yaml:"max_backoff"`       // If specified, max wait time before retries, otherwise, use default.
	MaxRetries    uint16            `yaml:"max_retries"`       // If specified, retry each HTTP call after non-200 response
	Server        string            `yaml:"server"`            // The server to connect to (host:port)
	Debug         bool              `yaml:"debug"`             // Enable debugging output
	DefaultUser   string            `yaml:"default_user"`      // Default user to own files
	DefaultGroup  string            `yaml:"default_group"`     // Default group to own files
	APIPort       uint16            `yaml:"api_port"`          // Port for API to listen on
	SentryDSN     string            `yaml:"sentry_dsn"`        // Sentry DSN
	SentryCaFile  string            `yaml:"sentry_ca_file"`    // The CA to trust (PEM) for Sentry communication
	FsType        output.Filesystem `yaml:"filesystem_type"`   // Enforce writing this type of filesystem. Use value from statfs.
	ChownFiles    bool              `yaml:"chown_files"`       // Do we chown files? Set to false when running without CAP_CHOWN.
	MetricsPrefix string            `yaml:"metrics_prefix"`    // Prefix metric names with this
	Monitor       MonitorConfig     `yaml:"monitor"`           // Config for monitoring/alerts
	BackupPath    string            `yaml:"backup_path"`       // If specified, back up secrets as an encrypted tarball to this location
	BackupKeyPath string            `yaml:"backup_key_path"`   // write wrapped key encrypting the backup to this location
	BackupPubkey  string            `yaml:"backup_pubkey"`     // Public key to wrap backup keys to, from keyunwrap --generate
}

Config is the main yaml configuration file passed to the keysync binary

func LoadConfig 

func LoadConfig(configFile string) (*Config, error)

LoadConfig loads the "global" keysync configuration file. This would generally be called on startup.

func (*Config) LoadClients 

func (config *Config) LoadClients() (map[string]ClientConfig, error)

LoadClients looks in directory for files with suffix, and tries to load them as Yaml files describing clients for Keysync to load We filter by the yaml extension so we can keep configs and keys in the same directory

type KeywhizHTTPClient 

type KeywhizHTTPClient struct {
	// contains filtered or unexported fields
}

KeywhizHTTPClient is a client that reads from a Keywhiz server over HTTP (v2 API).

func (KeywhizHTTPClient) Logger 

func (c KeywhizHTTPClient) Logger() *logrus.Entry

Logger returns the underlying logger for this client

func (KeywhizHTTPClient) RawSecret 

func (c KeywhizHTTPClient) RawSecret(name string) ([]byte, error)

RawSecret returns raw JSON from requesting a secret.

func (KeywhizHTTPClient) RawSecretList 

func (c KeywhizHTTPClient) RawSecretList() ([]byte, error)

RawSecretList returns raw JSON from requesting a listing of secrets.

func (KeywhizHTTPClient) RawSecretListWithContents 

func (c KeywhizHTTPClient) RawSecretListWithContents(secrets []string) ([]byte, error)

RawSecretListWithContents returns raw JSON from requesting a listing of secrets with their contents.

func (*KeywhizHTTPClient) RebuildClient 

func (c *KeywhizHTTPClient) RebuildClient() error

RebuildClient reloads certificates from disk. It should be called periodically to ensure up-to-date client certificates are used. This is important if you're using short-lived certificates that are routinely replaced.

func (KeywhizHTTPClient) Secret 

func (c KeywhizHTTPClient) Secret(name string) (secret *Secret, err error)

Secret returns an unmarshalled Secret struct after requesting a secret.

func (KeywhizHTTPClient) SecretList 

func (c KeywhizHTTPClient) SecretList() (map[string]Secret, error)

SecretList returns a map of unmarshalled Secret structs without their contents after requesting a listing of secrets. The map keys are the names of the secrets

func (KeywhizHTTPClient) SecretListWithContents 

func (c KeywhizHTTPClient) SecretListWithContents(secrets []string) (map[string]Secret, error)

SecretList returns a map of unmarshalled Secret structs, including their contents, associated with the given list of secrets. The map keys are the names of the secrets. All secrets must be accessible to this client, or the entire request will fail.

func (KeywhizHTTPClient) ServerStatus 

func (c KeywhizHTTPClient) ServerStatus() (data []byte, err error)

ServerStatus returns raw JSON from the server's _status endpoint

type MonitorConfig 

type MonitorConfig struct {
	MinCertLifetime     time.Duration `yaml:"min_cert_lifetime"`     // If specified, warn if cert does not have given min lifetime.
	MinSecretsCount     int           `yaml:"min_secrets_count"`     // If specified, warn if client has less than minimum number of secrets
	AlertEmailServer    string        `yaml:"alert_email_server"`    // For alert emails: SMTP server host:port to use for sending email
	AlertEmailRecipient string        `yaml:"alert_email_recipient"` // For alert emails: Recipient of alert emails
	AlertEmailSender    string        `yaml:"alert_email_sender"`    // For alert emails: Sender (from) for alert emails
}

The MonitorConfig has extra settings for monitoring/alerts.

type Output 

type Output interface {
	// Validate returns true if the secret is persisted already
	Validate(secret *Secret, state secretState) bool
	// Write a secret
	Write(secret *Secret) (*secretState, error)
	// Remove a secret
	Remove(name string) error
	// Remove all secrets and the containing directory (eg, when the client config is removed)
	// Returns a count of deleted files
	RemoveAll() (uint, error)
	// Cleanup unknown files (eg, ones deleted in Keywhiz while keysync was not running)
	// Returns a count of deleted files
	Cleanup(map[string]Secret) (uint, error)
}

Output is an interface that encapsulates what it means to store secrets

type OutputCollection 

type OutputCollection interface {
	NewOutput(clientConfig ClientConfig, logger *logrus.Entry) (Output, error)
	// Cleanup unknown clients (eg, ones deleted while keysync was not running)
	// Returns a count of deleted clients
	Cleanup(map[string]struct{}, *logrus.Entry) (uint, []error)
}

OutputCollection handles a collection of outputs.

type OutputDir 

type OutputDir struct {
	WriteDirectory    string
	DefaultOwnership  ownership.Ownership
	EnforceFilesystem output.Filesystem // What filesystem type do we expect to write to?
	ChownFiles        bool              // Do we chown the file? (Needs root or CAP_CHOWN).
	Logger            *logrus.Entry
}

OutputDir implements Output to files, which is the typical keysync usage to a tmpfs.

func (*OutputDir) Cleanup 

func (out *OutputDir) Cleanup(secrets map[string]Secret) (uint, error)

func (*OutputDir) Remove 

func (out *OutputDir) Remove(name string) error

func (*OutputDir) RemoveAll 

func (out *OutputDir) RemoveAll() (uint, error)

func (*OutputDir) Validate 

func (out *OutputDir) Validate(secret *Secret, state secretState) bool

Validate verifies the secret is written to disk with the correct content, permissions, and ownership

func (*OutputDir) Write 

func (out *OutputDir) Write(secret *Secret) (*secretState, error)

Write puts a Secret into OutputDir

type OutputDirCollection 

type OutputDirCollection struct {
	Config *Config
}

func (OutputDirCollection) Cleanup 

func (c OutputDirCollection) Cleanup(known map[string]struct{}, logger *logrus.Entry) (uint, []error)

func (OutputDirCollection) NewOutput 

func (c OutputDirCollection) NewOutput(clientConfig ClientConfig, logger *logrus.Entry) (Output, error)

type Secret 

type Secret struct {
	Name             string
	Content          content   `json:"secret"`
	Length           uint64    `json:"secretLength"`
	Checksum         string    `json:"checksum"`
	CreatedAt        time.Time `json:"creationDate"`
	UpdatedAt        time.Time `json:"updateDate"`
	FilenameOverride *string   `json:"filename"`
	Mode             string
	Owner            string
	Group            string
}

Secret represents data returned after processing a server request.

json tags after fields indicate to json decoder the key name in JSON

func ParseSecret 

func ParseSecret(data []byte) (s *Secret, err error)

ParseSecret deserializes raw JSON into a Secret struct.

func ParseSecretList 

func ParseSecretList(data []byte) (secrets []Secret, err error)

ParseSecretList deserializes raw JSON into a list of Secret structs.

func (Secret) Filename 

func (s Secret) Filename() (string, error)

Filename returns the expected filename of a secret. The filename metadata overrides the name, but it can't be path, so keysync can't delete or write arbitrary files outside its secrets directory.

func (Secret) ModeValue 

func (s Secret) ModeValue() (os.FileMode, error)

ModeValue function helps by converting a textual mode to the expected value for fuse.

func (Secret) OwnershipValue 

func (s Secret) OwnershipValue(fallback ownership.Ownership) (ret ownership.Ownership)

OwnershipValue returns the ownership for a given secret, falling back to the values given as an argument if they're not present in the secret

type SecretDeleted 

type SecretDeleted struct{}

SecretDeleted is returned as an error when the server 404s.

func (SecretDeleted) Error 

func (e SecretDeleted) Error() string

type StatusResponse 

type StatusResponse struct {
	Ok      bool     `json:"ok"`
	Message string   `json:"message,omitempty"`
	Updated *Updated `json:"updated,omitempty"`
}

StatusResponse from API endpoints

type Syncer 

type Syncer struct {
	// contains filtered or unexported fields
}

A Syncer manages a collection of clients, handling downloads and writing out updated secrets. Construct one using the NewSyncer and AddClient functions

func NewSyncer 

func NewSyncer(config *Config, outputCollection OutputCollection, logger *logrus.Entry, metricsHandle *sqmetrics.SquareMetrics) (*Syncer, error)

NewSyncer instantiates the main stateful object in Keysync.

func NewSyncerFromFile 

func NewSyncerFromFile(config *Config, clientConfig ClientConfig, bundle string, logger *logrus.Entry, metricsHandle *sqmetrics.SquareMetrics) (*Syncer, error)

NewSyncerFromFile instantiates a syncer that reads from a file/bundle instead of an HTTP server.

func (*Syncer) LoadClients 

func (s *Syncer) LoadClients() (*pendingCleanup, error)

LoadClients gets configured clients, This function returns clients that have been deconfigured, which are expected to be cleaned up

func (*Syncer) Run 

func (s *Syncer) Run() error

Run the main sync loop.

func (*Syncer) RunOnce 

func (s *Syncer) RunOnce() (Updated, []error)

RunOnce runs the syncer once, for all clients, without sleeps.

type Updated 

type Updated struct {
	Added   uint
	Changed uint
	Deleted uint
}

Updated secrets during a sync. How many secrets were added, changed, or deleted this sync.

func (*Updated) Add 

func (u *Updated) Add(rhs Updated)

Add in another update count

func (*Updated) Total 

func (u *Updated) Total() uint

Total of changed secrets

Source Files

View all Source files

Directories

Path Synopsis
package backup handles reading and writing encrypted .tar files from the secretsDirectory to a backupPath using the key backupKey.
 package backup handles reading and writing encrypted .tar files from the secretsDirectory to a backupPath using the key backupKey.
cmd
keyrestore
This is the main entry point for Keysync.
 This is the main entry point for Keysync.
keysync
This is the main entry point for Keysync.
 This is the main entry point for Keysync.
keyunwrap
monitor

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.