gcpvault

README

gcpvault

gcpvault provides a function for securely retrieving secrets from HashiCorp Vault while running on the Google Cloud Platform or in your local development environment.

To use this library, users must follow the Vault instructions for enabling GCP authentication: https://www.vaultproject.io/docs/auth/gcp.html.

Under the hood, when deployed to Google Cloud this tool will use the default application credentials to login to Vault and retrieve the specified secrets.

Since the login to Vault can be a heavy and relatively slow operation, we recommend users of the legacy Google App Engine Standard Environment (Go <=1.9) call this library during start up requests for manual scaling systems or in warm up requests for users of automatic scaling to prevent exposing public traffic to such latencies.

Local Development

For local development, users should use a Github personal access tokens or some similar method to login to Vault before injecting their Vault login token into the local environment.

Unit Testing

For mocking out the services required for interacting with Vault, a gcpvaulttest package has been included to provide httptest.Servers for each dependency.

Examples

Check out the examples directory for examples on how to use this package.

#Vault Token Caching

The library has an option to enable Vault Token Caching. Currently, Redis or GCS is supported for token storage. To enable token caching, one the following environment variables should set:

TOKEN_CACHE_STORAGE_REDIS - Host and port for Redis '10.200.30.4:6379'

TOKEN_CACHE_STORAGE_GCS - GCS bucket location where token can be stored for caching purposes. Care should be taken to make sure bucket permissions are set such that vault token is not leaked to the world.

Additional optional environment variables that control cache.

TOKEN_CACHE_REFRESH_THRESHOLD - How long before the token expiration should it be regenerated (in seconds). Default is 300 seconds.

TOKEN_CACHE_KEY_NAME - The object name to store. Default value is token-cache.

TOKEN_CACHE_CTX_TIMEOUT - This value is in seconds. Default value is 30 seconds.

TOKEN_CACHE_STORAGE_REDIS_DB - Database for Redis. Default is 0.

TOKEN_CACHE_REFRESH_RANDOM_OFFSET - Random refresh offset in seconds to avoid all the instances refreshing at once. Default is 1/2 the duration in seconds of the TOKEN_CACHE_REFRESH_THRESHOLD.

Documentation

Overview

Package gcpvault provides tools for securely retrieving secrets from Vault while running on Google Cloud infrastructure.

To use this library, users must follow the instructions for enabling GCP authentication: https://www.vaultproject.io/docs/auth/gcp.html

For local development, users should use something like Github personal access tokens to log into vault before injecting their Vault login token into the local environment.

Index

• Constants
• func GetSecrets(ctx context.Context, cfg Config) (map[string]interface{}, error)
• func GetVersionedSecrets(ctx context.Context, cfg Config) (map[string]interface{}, error)
• func PutSecrets(ctx context.Context, cfg Config, secrets map[string]interface{}) error
• func PutVersionedSecrets(ctx context.Context, cfg Config, secrets map[string]interface{}) error
• type Config
• type Token
• type TokenCache
• type TokenCacheGCS
  • func (t TokenCacheGCS) GetToken(ctx context.Context) (*Token, error)
  • func (t TokenCacheGCS) SaveToken(ctx context.Context, token Token) error
• type TokenCacheRedis
  • func (t TokenCacheRedis) GetToken(ctx context.Context) (*Token, error)
  • func (t TokenCacheRedis) SaveToken(ctx context.Context, token Token) error

Constants

const (
	CachedTokenRefreshThresholdDefault   = 300
	TokenCacheCtxTimeoutDefault          = 30
	TokenCacheRefreshRandomOffsetDefault = 60
	TokenCacheKeyNameDefault             = "token-cache"
	TokenCacheMaxRetriesDefault          = 3
)

Functions

func GetSecrets

func GetSecrets(ctx context.Context, cfg Config) (map[string]interface{}, error)

GetSecrets will use GCP Auth to access any secrets under the given SecretPath in Vault.

This is comparable to the `vault read` command.

Under the hood, this uses a JWT signed with the default Google application credentials to login to Vault via https://godoc.org/github.com/hashicorp/vault/api#Logical.Write and to read secrets via https://godoc.org/github.com/hashicorp/vault/api#Logical.Read. For more details about enabling GCP Auth and Vault visit: https://www.vaultproject.io/docs/auth/gcp.html

The map[string]interface{} returned is the actual contents of the secret referenced in the Config.SecretPath.

This is using the Vault API client's 'default config' to log in so users can provide additional environment variables to fine tune their Vault experience. For more information about configuring the Vault API client, view the code behind: https://godoc.org/github.com/hashicorp/vault/api#Config.ReadEnvironment

If running in a local development environment (via 'goapp test' or dev_appserver.py) this tool will expect the LocalToken to be set in some way.

func GetVersionedSecrets

func GetVersionedSecrets(ctx context.Context, cfg Config) (map[string]interface{}, error)

GetVersionedSecrets reads versioned secrets from Vault. This is comparable to the `vault kv get` command.

func PutSecrets

func PutSecrets(ctx context.Context, cfg Config, secrets map[string]interface{}) error

PutSecrets writes secrets to Vault at the configured path. This is comparable to the `vault write` command.

func PutVersionedSecrets

func PutVersionedSecrets(ctx context.Context, cfg Config, secrets map[string]interface{}) error

PutVersionedSecrets writes versioned secrets to Vault at the configured path. This is comparable to the `vault kv put` command.

Types

type Config

type Config struct {
	// SecretPath is the location of the secrets we wish to fetch from Vault.
	SecretPath string `envconfig:"VAULT_SECRET_PATH"`

	// VaultAddress is the location of the Vault server.
	VaultAddress string `envconfig:"VAULT_ADDR"`

	// Role is the role given to your service account when it was registered
	// with your Vault server. More information about creating roles for your service
	// account can be found here:
	// https://www.vaultproject.io/docs/auth/gcp.html#2-roles
	Role string `envconfig:"VAULT_GCP_IAM_ROLE"`

	// LocalToken is a Vault auth token obtained from logging into Vault via some outside
	// method like the command line tool. Users are only expected to pass this token
	// in local development scenarios.
	// This token can also be set in the `VAULT_TOKEN` environment variable and the
	// underlying Vault API client will use it.
	LocalToken string `envconfig:"VAULT_LOCAL_TOKEN"`

	// AuthPath is the path the GCP authentication method is mounted at.
	// Defaults to 'auth/gcp'.
	AuthPath string `envconfig:"VAULT_GCP_PATH"`

	// MaxRetries sets the number of retries that will be used in the case of certain
	// errors. The underlying Vault client will pull this value out of the environment
	// on it's own, but we're including it here so users can apply the same number of
	// attempts towards signing the JWT with Google's IAM services.
	MaxRetries int `envconfig:"VAULT_MAX_RETRIES"`

	// IAMAddress is the location of the GCP IAM server.
	// This should only used for testing.
	IAMAddress string `envconfig:"IAM_ADDR"`

	// MetadataAddress is the location of the GCP metadata
	// This should only used for testing.
	MetadataAddress string `envconfig:"METADATA_ADDR"`

	// HTTPClient can be optionally set if users wish to have more control over outbound
	// HTTP requests made by this library. If not set, an http.Client with a 1s
	// IdleConnTimeout will be used.
	HTTPClient *http.Client

	TokenCache TokenCache
	// How long before the token expiration should it be regenerated (in seconds).
	// Default is 300 seconds.
	TokenCacheRefreshThreshold int `envconfig:"TOKEN_CACHE_REFRESH_THRESHOLD"`
	//Random refresh offset in seconds to avoid all the instances refreshing at once. Default is 1/2 the duration in seconds of the TOKEN_CACHE_REFRESH_THRESHOLD.
	TokenCacheRefreshRandomOffset int `envconfig:"TOKEN_CACHE_REFRESH_RANDOM_OFFSET"`
	// this value is in seconds. Default value is 30 seconds
	TokenCacheCtxTimeout int `envconfig:"TOKEN_CACHE_CTX_TIMEOUT"`
	// the object name to store. Default value is 'token-cache'
	TokenCacheKeyName string `envconfig:"TOKEN_CACHE_KEY_NAME"`
	// GCS bucket location where token can be stored for caching purposes
	TokenCacheStorageGCS string `envconfig:"TOKEN_CACHE_STORAGE_GCS"`
	// Host and port for Redis '10.200.30.4:6379'
	TokenCacheStorageRedis string `envconfig:"TOKEN_CACHE_STORAGE_REDIS"`
	//Database for Redis. Default is 0
	TokenCacheStorageRedisDB int `envconfig:"TOKEN_CACHE_STORAGE_REDIS_DB"`
}

Config contains fields for configuring access and secrets retrieval from a Vault server.

type Token

type Token struct {
	Token   string
	Expires time.Time
}

type TokenCache

type TokenCache interface {
	GetToken(ctx context.Context) (*Token, error)
	SaveToken(ctx context.Context, token Token) error
}

type TokenCacheGCS

type TokenCacheGCS struct {
	// contains filtered or unexported fields
}

func (TokenCacheGCS) GetToken

func (t TokenCacheGCS) GetToken(ctx context.Context) (*Token, error)

func (TokenCacheGCS) SaveToken

func (t TokenCacheGCS) SaveToken(ctx context.Context, token Token) error

type TokenCacheRedis

type TokenCacheRedis struct {
	// contains filtered or unexported fields
}

func (TokenCacheRedis) GetToken

func (t TokenCacheRedis) GetToken(ctx context.Context) (*Token, error)

func (TokenCacheRedis) SaveToken

func (t TokenCacheRedis) SaveToken(ctx context.Context, token Token) error