secretly

README

Secretly

Secretly was created to allow Go applications to easily interface with popular secret management services and reduce secret ingestion boiler-plate. In-memory secret caching is included to reduce the number of operations against the secret management service, when dealing with secrets that store map data in the form of JSON and YAML.

Below is a list of the currently supported secret management services:

• Google Cloud Platform's (GCP) Secret Manager
• Amazon Web Services' (AWS) Secrets Manager
• Vault KV Secrets Engine
  • KV Secrets Engine - Version 1
  • KV Secrets Engine - Version 2
• Microsoft Azure Key Vault

If there's a secret management service missing that you'd like to see included, create a Feature Request!

Usage

See the brief overview below or check out our examples.

Overview

Tag Support

Available Tags

• type - The secret content's structure.
  • Valid Values: "text", "json", "yaml"
  • Default: "text"
• name - The secret's name
• key - The specific field to extract from the secret's content. Note: Requires type "json" or "yaml".
  • Default: The struct field name (split if split_words is true).
• version - The version of the secret to retrieve.
  • Default: 0 (translates to the latest version within the client wrappers, e.g. with GCP Secret Manager, 0 -> "latest")
• split_words - If the field name is used as the secret name and/or key, split it with underscores. If set to true and a process option is provided that combines name and key, the name and key will be separated with an underscore.
  • Default: false

Below is an example structure definition detailing default behavior, and the available tags:

type Specification struct {
    // The latest version of a secret named "TextSecret" that stores text data.
    TextSecret string `type:"text"`

    // The first version of a secret named "TextSecretVersion" that stores text data.
    // Rather than retrieving the latest version, retrieve version 1.
    TextSecretWithVersion1 string `type:"text" version:"1"`

    // The latest version of a secret named "Split_Text_Secret" that stores text data.
    SplitTextSecret string `type:"text" split_words:"true"`

    // The latest version of a secret named "Json_Secret" that stores json data
    // including a key "Json_Secret_Key".
    JsonSecretKey int `type:"json" name:"Json_Secret" split_words:"true"`

    // The latest version of a secret named "Yaml_Secret" that stores yaml data
    // including a key "Yaml_Secret_Key".
    YamlSecretExplicitKey float64 `type:"json" name:"Yaml_Secret" key:"Yaml_Secret_Key"`

    // Ignored.
    IgnoredField string `ignored:"true"`

    // Also ignored.
    ignoredField string

    // The fields from the nested struct are also processed.
    SubSpecification SubSpecification
}

type SubSpecification struct {
    // The latest version of a secret named "SubTextSecret" that stores text data.
    // Since the type is not specified, the default type, text, is used.
    SubTextSecret string

    // The latest version of a secret named "SubJsonSecretAndKey" that stores yaml data
    // including a key "SubJsonSecretAndKey".
    SubJsonSecretAndKey string `type:"json"`
}

Supported Field types

• text - Plain text. Any secret value can be read as plain text.

  Example secrets that stores text data:

  sensitive data
  

• json - JSON map. The secret stores JSON data; read a specific field from the JSON map. Note: If you want to read the entire json object, use the text type.

  Example secret that stores a JSON map:

  {
      "sensitive-field-1": "sensitive data"
  }
  

• yaml - YAML map. The secret stores YAML data; read a specific field from the YAML map. Note: If you want to read the entire yaml mapping, use the text type.

  Example secret that stores a YAML map:

  sensitive-field-1: sensitive data
  

Secret Versioning

Secretly provides two options for specifying secret versions other than the version tag:

1. Read secret versions (and all other field values) from a patch file:

   • Supported patch file types:
     • JSON (ext: .json)
     • YAML (ext: .yaml OR .yml)

   Example of reading secret versions from a JSON patch file:

   • versions.json

     {
         "My-DB-Credentials_username": {
             "version": "latest"
         },
         "My-DB-Credentialspassword": {
             "version": "5"
         }
     }
     

   • example.go

     ...
     
     type Secrets struct {
         DatabaseUsername string `type:"yaml" name:"My-DB-Credentials" key:"username" split_words:"true"`
     
         DatabasePassword string `type:"yaml" name:"My-DB-Credentials" key:"password"`
     }
     
     func example(client secretly.Client) Secrets {
         var s Secrets
     
         err := client.Process(&s, secretly.ApplyPatch("versions.json"))
         if err != nil {
             log.Fatal(err)
         }
     
         return s
     }
     
     ...
     

2. Read secret versions from environment variables:

   Example of reading secret versions from environment variables:

   • Export environment variables:

     export EXAMPLE_MY_DB_CREDENTIALS_USERNAME_VERSION=latest
     export EXAMPLE_MYDBCREDENTIALSPASSWORD_VERSION=5
     

   • example.go

     ...
     
     type Secrets struct {
         DatabaseUsername string `type:"yaml" name:"My-DB-Credentials" key:"username" split_words:"true"`
     
         DatabasePassword string `type:"yaml" name:"My-DB-Credentials" key:"password"`
     }
     
     func example(client secretly.Client) Secrets {
         var s Secrets
     
         err := client.Process(&s, secretly.WithVersionsFromEnv("EXAMPLE"))
         if err != nil {
             log.Fatal(err)
         }
     
         return s
     }
     
     ...
     

References

• envconfig

  The API for secretly was inspired by and borrows from envconfig. A simple yet powerful package for managing configuration data from environment variables.

Documentation

Overview

Package secretly provides a client wrapper for popular secret management services.

The secretly package's client interface exposes convenience methods for retrieving secrets.

Index

• Constants
• Variables
• func NewNoOpSecretCache() noOpSecretCache
• func NewSecretCache() secretCache
• func Process(client Client, spec any, opts ...ProcessOption) error
• type Client
• type Config
• type Field
  • func NewField(fValue reflect.Value, fStructField reflect.StructField) (Field, error)
  • func (f *Field) Name() string
  • func (f *Field) Set(b []byte) error
• type ProcessOption
  • func ApplyPatch(filePath string) ProcessOption
  • func WithVersionsFromEnv(prefix string) ProcessOption
• type SecretCache
• type StructTagError
  • func (e StructTagError) Error() string
  • func (e StructTagError) Unwrap() error

Constants

const (
	// Defaults.
	DefaultType    = "text"
	DefaultVersion = "0"
)

Default values for optional field tags.

Variables

var (
	ErrInvalidSpecification        = errors.New("specification must be a struct pointer")
	ErrInvalidSecretType           = errors.New("invalid secret type")
	ErrInvalidSecretVersion        = errors.New("invalid secret version")
	ErrSecretTypeDoesNotSupportKey = errors.New("secret type does not support \"key\"")
)

var (
	ErrInvalidJSONSecret = errors.New("secret is not valid json")
	ErrInvalidYAMLSecret = errors.New("secret is not valid yaml")
	ErrSecretMissingKey  = errors.New("secret is missing provided key")
)

var ErrInvalidFileType = errors.New("invalid file type")

Functions

func NewNoOpSecretCache

func NewNoOpSecretCache() noOpSecretCache

NewNoOpSecretCache constructs a no-op secret cache, meant to be used for disabling secret caching.

func NewSecretCache

func NewSecretCache() secretCache

NewSecretCache constructs a secretCache.

func Process

func Process(client Client, spec any, opts ...ProcessOption) error

Process interprets the provided specification, resolving the described secrets with the provided secret management Client.

Types

type Client

type Client interface {
	// Process resolves the provided specification.
	// ProcessOptions can be provided
	// to add additional processing for the fields,
	// like reading version info from the env or a file.
	//
	// (*Client).Process is a convenience
	// for calling secretly.Process with the Client.
	Process(spec any, opts ...ProcessOption) error

	// GetSecret retrieves the latest secret version for name
	// from the secret management service.
	GetSecret(ctx context.Context, name string) ([]byte, error)

	// GetSecretWithVersion retrieves the specific secret version for name
	// from the secret management service.
	GetSecretWithVersion(ctx context.Context, name, version string) ([]byte, error)
}

Client describes a secretly secret manager client wrapper.

type Config

type Config struct {
	// DisableCaching disables the secret caching feature.
	// By default, secret caching is enabled.
	// With this set to true,
	// repeated gets to the same secret version will reach out
	// to the secret manager client.
	DisableCaching bool
}

Config provides configuration to change the behavior of secretly client wrappers.

type Field

type Field struct {
	SecretType    string
	SecretName    string
	SecretVersion string
	MapKeyName    string // NOTE: Only used when secretType is "json" or "yaml"
	SplitWords    bool
	Value         reflect.Value
}

Field represents a field in a struct, exposing its secretly tag values and reference to the underlying value.

func NewField

func NewField(fValue reflect.Value, fStructField reflect.StructField) (Field, error)

NewField constructs a field referencing the provided reflect.Value with the tags from the reflect.StructField applied

func (*Field) Name

func (f *Field) Name() string

Name returns the resolved name of the field. If the secret type is "json" or "yaml", the secret name and key name are combined. If "split_words" is true, the combination of secret name and key name are transformed into uppercase, snake case.

func (*Field) Set

func (f *Field) Set(b []byte) error

Set sets the field's reflect.Value with b.

type ProcessOption

type ProcessOption func([]Field) error

ProcessOptions are optional modifiers for secret processing.

func ApplyPatch

func ApplyPatch(filePath string) ProcessOption

ApplyPatch returns an ProcessOption which overwrites the specified/default field values with the provided patch. Can be used to overwrite any of the configurable field values.

Types of patch files are determined by their extensions. Accepted patch file types are:

1. JSON (.json)
2. YAML (.yaml,.yml)

func WithVersionsFromEnv

func WithVersionsFromEnv(prefix string) ProcessOption

WithVersionsFromEnv returns an ProcessOption which overwrites the specified/default secret versions with versions from the environment. Environment variables are to be named with the following logic:

if prefix
	uppercase( prefix + "_" + field.Name() ) + "_VERSION"
else
	uppercase( field.Name() ) + "_VERSION"

type SecretCache

type SecretCache interface {
	// Add adds a secret with its version to the cache.
	Add(name, version string, content []byte)

	// Get gets the secret version from the cache.
	// A bool is returned to indicate a cache hit or miss.
	Get(name, version string) ([]byte, bool)
}

SecretCache describes a secret cache, which are used to limit calls to the upstream secret manager service.

type StructTagError

type StructTagError struct {
	Name string
	Key  string
	Err  error
}

StructTagError describes an error resulting from an issue with a struct tag.

func (StructTagError) Error

func (e StructTagError) Error() string

func (StructTagError) Unwrap

func (e StructTagError) Unwrap() error