etcd

package module
v0.0.0-...-380b48d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 31, 2024 License: MIT Imports: 32 Imported by: 0

README

caddy-etcd [DEPRECATED]

Build Status

DEPRECATION NOTICE: This etcd storage backend is deprecated and not recommended for production use. The implementation introduces significant operational complexity and potential reliability challenges. For clustered storage needs, please consider using the S3 storage backend (github.com/techknowlogick/certmagic-s3) which provides better reliability and simpler operations while meeting Caddy's distributed storage requirements.

This was a clustering plugin for Caddy that stored Caddy-managed certificates and other assets in etcd rather than the filesystem. It implemented a virtual filesystem on top of etcd storage to allow multiple Caddy instances to share configuration information and TLS certificates without sharing a filesystem.

Configuration

Caddy clustering plugins can be configured through environment variables or JSON configuration. To enable this plugin, you must set CADDY_CLUSTERING="etcd" when using environment variables, or include the appropriate JSON configuration.

Environment Variables Configuration

The table below lists the available environment variable options:

Environment Variable Function Default
CADDY_CLUSTERING_ETCD_SERVERS Comma or semicolon separated list of etcd servers (full URLs required, e.g.: https://127.0.0.1:2379) http://127.0.0.1:2379
CADDY_CLUSTERING_ETCD_PREFIX Key prefix for Caddy-managed resources in etcd /caddy
CADDY_CLUSTERING_ETCD_TIMEOUT Lock timeout for Caddy resources (Go duration format: 5m, 30s) 5m
CADDY_CLUSTERING_ETCD_CADDYFILE Path to local Caddyfile for initial cluster configuration
CADDY_CLUSTERING_ETCD_CADDYFILE_LOADER Enable/disable Caddyfile loading from etcd enable
TLS Configuration
Environment Variable Function Default
CADDY_CLUSTERING_ETCD_TLS_CERT Path to TLS client certificate
CADDY_CLUSTERING_ETCD_TLS_KEY Path to TLS client key
CADDY_CLUSTERING_ETCD_TLS_CA Path to CA certificate for server verification
CADDY_CLUSTERING_ETCD_TLS_SERVER_NAME Expected server name for verification
CADDY_CLUSTERING_ETCD_TLS_SKIP_VERIFY Skip TLS verification (not recommended) false
Authentication
Environment Variable Function Default
CADDY_CLUSTERING_ETCD_USERNAME Username for etcd authentication
CADDY_CLUSTERING_ETCD_PASSWORD Password for etcd authentication
Connection Settings
Environment Variable Function Default
CADDY_CLUSTERING_ETCD_DIAL_TIMEOUT Timeout for establishing connection 5s
CADDY_CLUSTERING_ETCD_KEEPALIVE_TIME Interval between keepalive probes 5s
CADDY_CLUSTERING_ETCD_KEEPALIVE_TIMEOUT Time to wait for keepalive response 5s
CADDY_CLUSTERING_ETCD_AUTO_SYNC_INTERVAL Interval for endpoint auto-sync 5m
CADDY_CLUSTERING_ETCD_REQUEST_TIMEOUT Timeout for individual requests 5s
JSON Configuration

The plugin can be configured using Caddy's JSON config format. Here's an example:

{
  "storage": {
    "module": "etcd",
    "config": {
      "endpoints": ["https://127.0.0.1:2379"],
      "key_prefix": "/caddy",
      "lock_timeout": "5m",
      "tls": {
        "cert_file": "/path/to/cert.pem",
        "key_file": "/path/to/key.pem",
        "ca_file": "/path/to/ca.pem",
        "server_name": "etcd.example.com"
      },
      "auth": {
        "username": "etcd_user",
        "password": "etcd_password"
      },
      "connection": {
        "dial_timeout": "5s",
        "keepalive_time": "5s",
        "request_timeout": "5s"
      }
    }
  }
}
Caddyfile Examples

While the plugin itself is configured through environment variables or JSON, here are some example Caddyfile configurations that work well with etcd clustering:

# Basic HTTPS server with automatic certificates
https://example.com {
    respond "Hello from clustered Caddy!"
}

# Multiple domains sharing certificates
(common) {
    tls {
        client_auth {
            mode require
            trusted_ca_cert_file /path/to/ca.pem
        }
    }
}

https://site1.example.com {
    import common
    respond "Site 1"
}

https://site2.example.com {
    import common
    respond "Site 2"
}

Breaking Changes

Version 2.0
  • Removed support for legacy environment variable names
  • Changed default connection timeout from 3s to 5s
  • TLS configuration now requires both cert and key files when enabled
  • Changed key prefix format in etcd storage

Building Caddy with this Plugin

This plugin requires caddy to be built with go modules. It cannot be built by the build server on caddyserver.com because it currently lacks module support.

This project uses mage to build the caddy binary. To build, first download mage:

go get -u github.com/magefile/mage

You must have the following binaries available on your system to run the build:

go >= 1.11
sed
git

Then build by running

mage build

See the magefile.go for other customizations, such as including other plugins in your custom build.

Testing

The project includes both unit tests and integration tests:

Unit Tests
go test -v -cover -race -mod vendor
Integration Tests

Integration tests require Docker and include:

  • Automatic test environment setup/teardown
  • Single node and cluster testing
  • TLS and authentication testing
  • Failure scenario testing

To run integration tests:

./test-docker.sh detach && \
go test -v -cover -race -mod vendor -tags=integration && \
docker stop etcd

Features

Current Features:

  • Distributed certificate storage and sharing
  • Mutual TLS authentication support
  • Username/password authentication
  • Automatic connection management and failover
  • Distributed locking with configurable timeouts
  • Atomic transactions for data consistency
  • Configurable connection parameters
  • Comprehensive test coverage
  • JSON configuration support
  • Environment variable configuration
  • Automatic endpoint discovery
  • High availability support
  • Metrics and monitoring support

Planned Features:

  • Dynamic Caddyfile configuration updates
  • Automatic backup and restore
  • Enhanced monitoring and alerting
  • Rate limiting and circuit breaking
  • Custom storage policies

Documentation

Overview

Package etcd implements a distributed storage backend for Caddy using etcd.

Deprecated: This storage backend is deprecated and not recommended for production use. The etcd implementation introduces significant operational complexity and potential reliability challenges. For clustered storage needs, please consider using the S3 storage backend (github.com/techknowlogick/certmagic-s3) which provides better reliability and simpler operations while meeting Caddy's distributed storage requirements.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrLockTimeout   = fmt.Errorf("lock timeout exceeded")
	ErrNoConnection  = fmt.Errorf("etcd cluster is not available")
	ErrClusterDown   = fmt.Errorf("etcd cluster is not available")
	ErrInvalidConfig = fmt.Errorf("invalid configuration")
)

Error types for specific failure cases

Functions

func FilterExactPrefix 

func FilterExactPrefix(prefix string, cut string) func(*mvccpb.KeyValue) bool

FilterExactPrefix returns a filter function that matches only terminal nodes (files) with the exact path prefix FilterExactPrefix returns a filter function that matches only terminal nodes (files) with the exact path prefix. It trims the base prefix (cut) before matching.

Parameters:

  • prefix: The exact prefix to match after trimming (e.g. /path/to/dir)
  • cut: The base prefix to remove before matching (e.g. /caddy)

Returns a filter function that matches only files directly under the given prefix, not in subdirectories. For example, with prefix "/foo", it matches "/foo/file.txt" but not "/foo/bar/file.txt"

func FilterPrefix 

func FilterPrefix(prefix string, cut string) func(*mvccpb.KeyValue) bool

FilterPrefix returns a filter function that matches keys with the given prefix FilterPrefix returns a filter function that matches keys with the given prefix. It trims the base prefix (cut) from each key before checking the match prefix.

Parameters:

  • prefix: The prefix to match against after trimming the base prefix
  • cut: The base prefix to remove before matching (e.g. /caddy)

Returns a filter function that can be used with List() to filter by key prefix

func FilterRemoveDirectories 

func FilterRemoveDirectories() func(*mvccpb.KeyValue) bool

FilterRemoveDirectories returns a filter function that removes directory entries A key is considered a directory if it has no value FilterRemoveDirectories returns a filter function that removes directory entries. A key is considered a directory if it has no value stored in etcd. This is useful for getting only leaf nodes (files) in a directory structure.

Returns a filter function that can be used with List() to exclude directories

func IsConnectionError 

func IsConnectionError(e error) bool

IsConnectionError checks if the error is a ConnectionError IsConnectionError checks if the given error is related to etcd connection issues. Returns true if the error is of type ConnectionError, false otherwise.

func IsFailedChecksumError 

func IsFailedChecksumError(e error) bool

IsFailedChecksumError checks to see if error is of type FailedChecksum IsFailedChecksumError checks if the given error indicates a checksum validation failure. Returns true if the error is of type FailedChecksum, false otherwise.

func IsLockError 

func IsLockError(e error) bool

IsLockError checks if the error is a LockError IsLockError checks if the given error is related to lock acquisition failure. Returns true if the error is of type LockError, false otherwise.

func IsNotExistError 

func IsNotExistError(e error) bool

IsNotExistError checks to see if error is of type NotExist IsNotExistError checks if the given error indicates a key does not exist. Returns true if the error is of type NotExist, false otherwise.

Types

type AuthConfig 

type AuthConfig struct {
	Username string `json:"username,omitempty"` // Username for etcd authentication
	Password string `json:"password,omitempty"` // Password for etcd authentication
}

AuthConfig holds authentication credentials for etcd access. Both username and password must be provided together if authentication is enabled.

type Cluster 

type Cluster struct {
	// contains filtered or unexported fields
}

Cluster implements certmagic.Storage using etcd as the storage backend. It provides distributed locking, atomic operations, and consistent storage across multiple Caddy instances in a cluster.

func (Cluster) CaddyModule 

func (Cluster) CaddyModule() caddy.ModuleInfo

CaddyModule returns the Caddy module information.

func (Cluster) CertMagicStorage 

func (c Cluster) CertMagicStorage() (certmagic.Storage, error)

CertMagicStorage converts c to a certmagic.Storage instance.

func (Cluster) Delete 

func (c Cluster) Delete(_ context.Context, key string) error

Delete deletes the value at the given key.

func (Cluster) Exists 

func (c Cluster) Exists(_ context.Context, key string) bool

Exists returns true if the key exists and is accessible.

func (Cluster) List 

func (c Cluster) List(_ context.Context, prefix string, recursive bool) ([]string, error)

List returns all keys that match prefix.

func (Cluster) Load 

func (c Cluster) Load(_ context.Context, key string) ([]byte, error)

Load retrieves the value at the given key.

func (Cluster) Lock 

func (c Cluster) Lock(_ context.Context, key string) error

Lock acquires a lock at the given key.

func (*Cluster) Provision 

func (c *Cluster) Provision(ctx caddy.Context) error

Provision sets up the storage backend.

func (Cluster) Stat 

func (c Cluster) Stat(_ context.Context, key string) (certmagic.KeyInfo, error)

Stat returns information about the given key.

func (Cluster) Store 

func (c Cluster) Store(_ context.Context, key string, value []byte) error

Store saves the given value at the given key.

func (Cluster) Unlock 

func (c Cluster) Unlock(_ context.Context, key string) error

Unlock releases the lock at the given key.

func (*Cluster) UnmarshalCaddyfile 

func (c *Cluster) UnmarshalCaddyfile(d *caddyfile.Dispenser) error

UnmarshalCaddyfile implements caddyfile.Unmarshaler. Syntax: 

storage etcd {
    prefix <key_prefix>
    endpoints {
        <endpoint1>
        <endpoint2>
        ...
    }
    timeout <duration>
    caddyfile <path>
    disable_caddyfile_load
    auth {
        username <username>
        password <password>
    }
    tls {
        cert <path>
        key <path>
        ca <path>
        server_name <name>
        insecure_skip_verify
    }
    connection {
        dial_timeout <duration>
        keepalive_time <duration>
        keepalive_timeout <duration>
        auto_sync_interval <duration>
        request_timeout <duration>
        reject_old_cluster <bool>
    }
}

func (*Cluster) Validate 

func (c *Cluster) Validate() error

Validate implements caddy.Validator and validates the configuration.

type ClusterConfig 

type ClusterConfig struct {
	KeyPrefix        string           `json:"key_prefix,omitempty"`
	ServerIP         []string         `json:"endpoints,omitempty"`
	LockTimeout      Duration         `json:"lock_timeout,string,omitempty"`
	CaddyFile        []byte           `json:"-"` // Not exposed in JSON
	CaddyFilePath    string           `json:"-"` // Not exposed in JSON
	DisableCaddyLoad bool             `json:"disable_caddyfile_load,omitempty"`
	TLS              TLSConfig        `json:"tls,omitempty"`
	Auth             AuthConfig       `json:"auth,omitempty"`
	Connection       ConnectionConfig `json:"connection,omitempty"`
	// contains filtered or unexported fields
}

func NewClusterConfig 

func NewClusterConfig(opts ...ConfigOption) (*ClusterConfig, error)

NewClusterConfig returns a new configuration with options passed as functional options and validates the configuration

func (*ClusterConfig) Validate 

func (c *ClusterConfig) Validate() error

Validate checks if the configuration is valid

type ConfigOption 

type ConfigOption func(c *ClusterConfig) error

ConfigOption represents a functional option for ClusterConfig

func ConfigOptsFromEnvironment 

func ConfigOptsFromEnvironment() ([]ConfigOption, error)

ConfigOptsFromEnvironment reads environment variables and returns options that can be applied via NewClusterConfig. It validates required pairs of environment variables and returns any validation errors.

func WithAutoSyncInterval 

func WithAutoSyncInterval(s string) ConfigOption

WithAutoSyncInterval sets the auto sync interval for etcd connections

func WithCaddyFile 

func WithCaddyFile(s string) ConfigOption

WithCaddyFile sets the path to the bootstrap Caddyfile to load on initial start if configuration information is not already present in etcd. The first cluster instance will load this file and store it in etcd. Subsequent members of the cluster will prioritize configuration from etcd even if this file is present. This function will not error even if the Caddyfile is not present. If a caddyfile cannot be read from etcd, from this file, or from the default loader, caddy will start with an empty default configuration.

func WithDialTimeout 

func WithDialTimeout(s string) ConfigOption

WithDialTimeout sets the dial timeout for etcd connections

func WithDisableCaddyfileLoad 

func WithDisableCaddyfileLoad(s string) ConfigOption

WithDisableCaddyfileLoad will skip all attempts at loading the caddyfile from etcd and force caddy to fall back to other enabled caddyfile loader plugins or the default loader

func WithKeepAliveTime 

func WithKeepAliveTime(s string) ConfigOption

WithKeepAliveTime sets the keepalive time for etcd connections

func WithKeepAliveTimeout 

func WithKeepAliveTimeout(s string) ConfigOption

WithKeepAliveTimeout sets the keepalive timeout for etcd connections

func WithPassword 

func WithPassword(s string) ConfigOption

WithPassword sets the password for etcd authentication

func WithPrefix 

func WithPrefix(s string) ConfigOption

WithPrefix sets the etcd namespace for caddy data. Default is `/caddy`. Prefixes are normalized to use `/` as a path separator.

func WithRejectOldCluster 

func WithRejectOldCluster(s string) ConfigOption

WithRejectOldCluster sets whether to reject connecting to old clusters

func WithRequestTimeout 

func WithRequestTimeout(s string) ConfigOption

WithRequestTimeout sets the request timeout for etcd operations

func WithServers 

func WithServers(s string) ConfigOption

WithServers sets the etcd server endpoints. Multiple endpoints are assumed to be separated by a comma, and consist of a full URL, including scheme and port (i.e., http://127.0.0.1:2379) The default config uses port 2379 on localhost.

func WithTLSCA 

func WithTLSCA(s string) ConfigOption

WithTLSCA sets the CA certificate file for TLS

func WithTLSCert 

func WithTLSCert(s string) ConfigOption

WithTLSCert sets the client certificate file for TLS

func WithTLSKey 

func WithTLSKey(s string) ConfigOption

WithTLSKey sets the client key file for TLS

func WithTLSServerName 

func WithTLSServerName(s string) ConfigOption

WithTLSServerName sets the server name for TLS verification

func WithTLSSkipVerify 

func WithTLSSkipVerify(s string) ConfigOption

WithTLSSkipVerify sets whether to skip TLS verification

func WithTimeout 

func WithTimeout(s string) ConfigOption

WithTimeout sets the time locks should be considered abandoned. Locks that exist longer than this setting will be overwritten by the next client that acquires the lock. The default is 5 minutes. This option takes standard Go duration formats such as 5m, 1h, etc.

func WithUsername 

func WithUsername(s string) ConfigOption

WithUsername sets the username for etcd authentication

type ConnectionConfig 

type ConnectionConfig struct {
	DialTimeout      Duration `json:"dial_timeout,string,omitempty"`       // Timeout for establishing initial connection
	KeepAliveTime    Duration `json:"keepalive_time,string,omitempty"`     // Time between keepalive probes
	KeepAliveTimeout Duration `json:"keepalive_timeout,string,omitempty"`  // Time to wait for keepalive response
	AutoSyncInterval Duration `json:"auto_sync_interval,string,omitempty"` // Interval for endpoint auto-synchronization
	RequestTimeout   Duration `json:"request_timeout,string,omitempty"`    // Timeout for individual requests
	RejectOldCluster bool     `json:"reject_old_cluster,omitempty"`        // Whether to reject connecting to old clusters
}

ConnectionConfig holds etcd connection settings including timeouts, keepalive parameters, and other connection-related configuration options.

type ConnectionError 

type ConnectionError struct {
	Endpoints []string
	Err       error
}

ConnectionError represents connection-related errors

func (ConnectionError) Error 

func (e ConnectionError) Error() string

func (ConnectionError) Unwrap 

func (e ConnectionError) Unwrap() error

type Duration 

type Duration struct {
	time.Duration
}

Duration wraps time.Duration to provide JSON marshaling/unmarshaling

func (*Duration) UnmarshalJSON 

func (d *Duration) UnmarshalJSON(b []byte) error

type FailedChecksum 

type FailedChecksum struct {
	Key string
}

FailedChecksum error is returned when the data returned by Load does not match the SHA1 checksum stored in its metadata node

func (FailedChecksum) Error 

func (e FailedChecksum) Error() string

type Lock 

type Lock struct {
	Token    string           // Random token identifying the client holding the lock
	Obtained string           // UTC timestamp when the lock was obtained
	Key      string           // The key being locked
	LeaseID  clientv3.LeaseID // Lease ID associated with this lock
}

Lock represents a distributed lock in etcd. Features:

  • Unique token per client for lock ownership
  • Automatic lock extension for same client
  • Configurable lock timeouts
  • Automatic cleanup of stale locks
  • Safe concurrent access across cluster

Note: The implementation assumes a single client does not attempt to acquire the same lock from multiple goroutines simultaneously. Such usage may result in race conditions where the last write wins.

type LockError 

type LockError struct {
	Key     string
	Timeout time.Duration
	Err     error
}

LockError represents a locking-related error

func (LockError) Error 

func (e LockError) Error() string

func (LockError) Unwrap 

func (e LockError) Unwrap() error

type Metadata 

type Metadata struct {
	Path      string    // Full path to the node
	Size      int       // Size of the value in bytes
	Timestamp time.Time // Last modification time
	Hash      [20]byte  // SHA1 hash of the value
	IsDir     bool      // Whether this node represents a directory
}

Metadata stores information about a particular node that represents a file in etcd

func NewMetadata 

func NewMetadata(key string, data []byte) Metadata

NewMetadata returns metadata information given a path and a file to be stored at the path. Typically, one metadata node is stored for each file node in etcd.

type NotExist 

type NotExist struct {
	Key string
}

NotExist is returned when a key lookup fails when calling Load or Metadata

func (NotExist) Error 

func (e NotExist) Error() string

func (NotExist) Is 

func (e NotExist) Is(target error) bool

type Service 

type Service interface {
	Store(key string, value []byte) error
	Load(key string) ([]byte, error)
	Delete(key string) error
	Metadata(key string) (*Metadata, error)
	Lock(key string) error
	Unlock(key string) error
	List(path string, filters ...func(*mvccpb.KeyValue) bool) ([]string, error)
	// contains filtered or unexported methods
}

func NewService 

func NewService(c *ClusterConfig) (Service, error)

NewService returns a new low level service to store and load values in etcd. The service implements filesystem-like semantics on top of etcd's key/value storage, with support for:

  • Atomic transactions for data consistency
  • Metadata tracking for each stored value
  • Directory-like operations with recursive listing
  • Distributed locking with configurable timeouts
  • Automatic connection management and retries
  • Data integrity verification via checksums

The service uses exponential backoff for retries and transactions to handle temporary failures. While best efforts are made to maintain consistency, prolonged etcd unavailability may impact the system's ability to recover to a fully coherent state.

type TLSConfig 

type TLSConfig struct {
	CertFile   string `json:"cert_file,omitempty"`   // Path to client certificate file
	KeyFile    string `json:"key_file,omitempty"`    // Path to client key file
	CAFile     string `json:"ca_file,omitempty"`     // Path to CA certificate for server verification
	ServerName string `json:"server_name,omitempty"` // Expected server name for verification
	SkipVerify bool   `json:"skip_verify,omitempty"` // Whether to skip TLS verification (not recommended)
}

ClusterConfig maintains configuration for connecting to and interacting with etcd. It includes settings for:

  • Server endpoints and key prefixes
  • TLS security and authentication
  • Connection timeouts and keepalive settings
  • Lock timeouts and operational parameters
  • Optional Caddyfile loading configuration

TLSConfig holds TLS-related configuration for secure etcd connections. It includes paths to certificates and keys, as well as verification options.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.