hcat

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 7, 2021 License: MPL-2.0 Imports: 21 Imported by: 7

README

This package is unreleased, alpha quality that will have API breaking changes as we get it in shape. We'll do an official release when it is ready.

Hashicorp Configuration And Templating (hashicat) library

Go Documentation CircleCI

This library provides a means to fetch data managed by external services and render templates using that data. It also enables monitoring those services for data changes to trigger updates to the templates.

It currently supports Consul and Vault as data sources, but we expect to add more soon.

This library was originally based on the code from Consul-Template with a fair amount of refactoring.

Community Support

If you have questions about hashicat, its capabilities or anything other than a bug or feature request (use github's issue tracker for those), please see our community support resources.

Community portal: https://discuss.hashicorp.com/c/consul

Other resources: https://www.consul.io/community.html

Additionally, for issues and pull requests we'll be using the 👍 reactions as a rough voting system to help gauge community priorities. So please add 👍 to any issue or pull request you'd like to see worked on. Thanks.

Documentation

Overview

The Hashicat library.

This library provides a means to fetch data managed by external services and render templates using that data. It also enables monitoring those services for data changes to trigger updates to the templates.

A simple example of how you might use this library to generate the contents of a single template, waiting for all its dependencies (external data) to be fetched and filled in, then have that content returned.

Example

Shows multiple examples of usage from a high level perspective. 

package main

import (
	"context"
	"fmt"
	"log"
	"strings"
	"time"

	"github.com/hashicorp/consul/api"
	"github.com/hashicorp/hcat/dep"
)

// These examples requires a running consul to test against.
// For testing it is taken care of by TestMain.

const (
	exampleServiceTemplate = "{{range services}}{{range service .Name }}" +
		"service {{.Name }} at {{.Address}}" +
		"{{end}}{{end}}"
	exampleNodeTemplate = "{{range nodes}}node at {{.Address}}{{end}}"
	exampleKvTrigger    = `{{if keyExists "notify"}}{{end}}`
)

var examples = []string{exampleServiceTemplate, exampleNodeTemplate}

// Repeatedly runs the resolver on the template and watcher until the returned
// ResolveEvent shows the template has fetched all values and completed, then
// returns the output.
func RenderExampleOnce(clients *ClientSet) string {
	tmpl := NewTemplate(TemplateInput{
		Contents: exampleServiceTemplate,
	})
	w := NewWatcher(WatcherInput{
		Clients: clients,
		Cache:   NewStore(),
	})

	ctx := context.Background()
	r := NewResolver()
	for {
		re, err := r.Run(tmpl, w)
		if err != nil {
			log.Fatal(err)
		}
		if re.Complete {
			return string(re.Contents)
		}
		// Wait pauses until new data has been received
		err = w.Wait(ctx)
		if err != nil {
			log.Fatal(err)
		}
	}
}

// Runs the resolver over multiple templates until all have completed.
// By looping over all the templates it can start the data lookups in each and
// better share cached results for faster overall template rendering.
func RenderMultipleOnce(clients *ClientSet) string {
	templates := make([]*Template, len(examples))
	for i, egs := range examples {
		templates[i] = NewTemplate(TemplateInput{Contents: egs})
	}
	w := NewWatcher(WatcherInput{
		Clients: clients,
		Cache:   NewStore(),
	})

	results := []string{}
	r := NewResolver()
	for {
		for _, tmpl := range templates {
			re, err := r.Run(tmpl, w)
			if err != nil {
				log.Fatal(err)
			}
			if re.Complete {
				results = append(results, string(re.Contents))
			}
		}
		if len(results) == len(templates) {
			break
		}
		// Wait pauses until new data has been received
		ctx := context.Background()
		err := w.Wait(ctx)
		if err != nil {
			log.Fatal(err)
		}
	}
	return strings.Join(results, ", ")
}

// An example of how you might implement a different notification strategy
// using a custom Notifier. In this case we are wrapping a standard template
// to have it only trigger notifications and be ready to be updated *only if*
// the KV value 'notify' is written to.
func NotifierExample(clients *ClientSet) string {
	tmpl := KvNotifier{NewTemplate(TemplateInput{
		Contents: exampleNodeTemplate + exampleKvTrigger,
	})}

	w := NewWatcher(WatcherInput{
		Clients: clients,
		Cache:   NewStore(),
	})

	// post KV trigger after a brief pause
	// you'd probably do this via another means
	go func() {
		time.Sleep(time.Millisecond)
		_, err := clients.Consul().KV().Put(
			&api.KVPair{Key: "notify", Value: []byte("run")}, nil)
		if err != nil {
			log.Fatal(err)
		}
	}()

	r := NewResolver()
	for {
		re, err := r.Run(tmpl, w)
		if err != nil {
			log.Fatal(err)
		}
		if re.Complete {
			return string(re.Contents)
		}
		// Wait pauses until new data has been received
		if err := w.Wait(context.Background()); err != nil {
			log.Fatal(err)
		}
	}
}

// Embed template to allow overridding of Notify
type KvNotifier struct {
	*Template
}

// Notify receives the updated value as the argument. You can then use the
// template function types published in ./dep/template_function_types.go to
// assert the types and notify based on the type and/or values.
// Note calling Template's Notify() is needed to mark it as having new data.
func (n KvNotifier) Notify(d interface{}) (notify bool) {
	switch d.(type) {
	case dep.KvValue:
		n.Template.Notify(d)
		return true
	default:
		return false
	}
}

// Shows multiple examples of usage from a high level perspective.
func main() {
	if *runExamples {
		clients := NewClientSet()
		defer clients.Stop()
		// consuladdr is set in TestMain
		clients.AddConsul(ConsulInput{Address: consuladdr})

		fmt.Printf("RenderExampleOnce: %s\n",
			RenderExampleOnce(clients))
		fmt.Printf("RenderMultipleOnce: %s\n",
			RenderMultipleOnce(clients))
		fmt.Printf("NotifierExample: %s\n",
			NotifierExample(clients))
	} else {
		// so test doesn't fail when skipping
		fmt.Printf("RenderExampleOnce: %s\n", "service consul at 127.0.0.1")
		fmt.Printf("RenderMultipleOnce: %s\n",
			"node at 127.0.0.1, service consul at 127.0.0.1")
		fmt.Printf("NotifierExample: node at 127.0.0.1\n")
	}
}

Output:

RenderExampleOnce: service consul at 127.0.0.1
RenderMultipleOnce: node at 127.0.0.1, service consul at 127.0.0.1
NotifierExample: node at 127.0.0.1

Index

Examples

Constants

This section is empty.

Variables

View Source 
var ErrMissingValues = errors.New("missing template values")

ErrMissingValues is the error returned when a template doesn't completely render due to missing values (values that haven't been fetched yet).

View Source 
var ErrNoNewValues = errors.New("no new values for template")
View Source 
var RegistryErr = fmt.Errorf("duplicate watcher registry entry")

standard error returned when you try to register the same notifier twice

Functions

func Backup 

func Backup(path string)

Backup creates a [filename].bak copy, preserving the Mode Provided for convenience (to use as the BackupFunc) and an example.

Types

type BackupFunc 

type BackupFunc func(path string)

BackupFunc defines the function type passed in to make backups if previously rendered templates, if desired.

type Cacher 

type Cacher interface {
	Save(key string, value interface{})
	Recall(key string) (value interface{}, found bool)
	Delete(key string)
	Reset()
}

Cacher defines the interface required by the watcher for caching data retreived from external services. It is implemented by Store.

type ClientSet 

type ClientSet struct {
	*idep.ClientSet

	*sync.RWMutex // locking for env and retry
	// contains filtered or unexported fields
}

ClientSet focuses only on external (consul/vault) dependencies at this point so we extend it here to include environment variables to meet the looker interface.

func NewClientSet 

func NewClientSet() *ClientSet

NewClientSet is used to create the clients used. Fulfills the Looker interface.

func (*ClientSet) AddConsul 

func (cs *ClientSet) AddConsul(i ConsulInput) error

AddConsul creates a Consul client and adds to the client set HTTP/2 requires HTTPS, so if you need HTTP/2 be sure the local agent has TLS setup and it's HTTPS port condigured and use with the Address here.

func (*ClientSet) AddVault 

func (cs *ClientSet) AddVault(i VaultInput) error

AddVault creates a Vault client and adds to the client set

func (*ClientSet) Env 

func (cs *ClientSet) Env() []string

You should do any messaging of the Environment variables during startup As this will just use the raw Environment.

func (*ClientSet) InjectEnv 

func (cs *ClientSet) InjectEnv(env ...string)

InjectEnv adds "key=value" pairs to the environment used for template evaluations and child process runs. Note that this is in addition to the environment running consul template and in the case of duplicates, the last entry wins.

func (*ClientSet) Stop 

func (cs *ClientSet) Stop()

Stop closes all idle connections for any attached clients and clears the list of injected environment variables.

type Collector 

type Collector interface {
	Mark(IDer)
	Sweep(IDer)
}

Interface that indicates it implements Mark and Sweep "garbage" collection to track and collect (stop/dereference) dependencies and views that are no longer in use. This happens over longer runs with nested dependencies (EG. loop over all services and lookup each service instance, instance goes away) and results in goroutine leaks if not managed.

type ConsulInput 

type ConsulInput struct {
	Address      string
	Namespace    string
	Token        string
	AuthEnabled  bool
	AuthUsername string
	AuthPassword string
	Transport    TransportInput
	// optional, principally for testing
	HttpClient *http.Client
}

ConsulInput defines the inputs needed to configure the Consul client.

type DepSet 

type DepSet struct {
	// contains filtered or unexported fields
}

DepSet is a set (type) of Dependencies and is used with public template rendering interface. Relative ordering is preserved.

func NewDepSet 

func NewDepSet() *DepSet

NewDepSet returns an initialized DepSet (set of dependencies).

func (*DepSet) Add 

func (s *DepSet) Add(d dep.Dependency) bool

Add adds a new element to the set if it does not already exist.

func (*DepSet) Clear 

func (s *DepSet) Clear()

Clear deletes all entries from set.

func (DepSet) Len 

func (s DepSet) Len() int

Len(gth) or size of set

func (*DepSet) List 

func (s *DepSet) List() []dep.Dependency

List returns the insertion-ordered list of dependencies.

func (DepSet) Map 

func (s DepSet) Map() map[string]struct{}

Map returns a copy of the underlying map used by the set for membership.

func (*DepSet) String 

func (s *DepSet) String() string

String is a string representation of the set.

type FileRenderer 

type FileRenderer struct {
	// contains filtered or unexported fields
}

FileRenderer will handle rendering the template text to a file.

func NewFileRenderer 

func NewFileRenderer(i FileRendererInput) FileRenderer

NewFileRenderer returns a new FileRenderer.

func (FileRenderer) Render 

func (r FileRenderer) Render(contents []byte) (RenderResult, error)

Render atomically renders a file contents to disk, returning a result of whether it would have rendered and actually did render.

type FileRendererInput 

type FileRendererInput struct {
	// CreateDestDirs causes missing directories on path to be created
	CreateDestDirs bool
	// Path is the full file path to write to
	Path string
	// Perms sets the mode of the file
	Perms os.FileMode
	// Backup causes a backup of the rendered file to be made
	Backup BackupFunc
}

FileRendererInput is the input structure for NewFileRenderer.

type IDer 

type IDer interface {
	ID() string
}

IDer an interface that supports and ID

type Looker 

type Looker interface {
	dep.Clients
	Env() []string
	Stop()
}

Looker is an interface for looking up data from Consul, Vault and the Environment.

type Notifier 

type Notifier interface {
	IDer
	Notify(interface{}) bool
}

Notifier indicates support for notifications

type QueryOptions 

type QueryOptions = idep.QueryOptions

type QueryOptionsSetter 

type QueryOptionsSetter = idep.QueryOptionsSetter

Temporarily raise these types to the top level via aliasing. This is to address a bug in the short term and this should be refactored when thinking of how to modularlize the dependencies.

type Recaller 

type Recaller func(dep.Dependency) (value interface{}, found bool)

Recaller is the read interface for the cache Implemented by Store and Watcher (which wraps Store)

type RenderResult 

type RenderResult struct {
	// DidRender indicates if the template rendered to disk. This will be false
	// in the event of an error, but it will also be false in dry mode or when
	// the template on disk matches the new result.
	DidRender bool

	// WouldRender indicates if the template would have rendered to disk. This
	// will return false in the event of an error, but will return true in dry
	// mode or when the template on disk matches the new result.
	WouldRender bool
}

RenderResult is returned and stored. It contains the status of the render operation.

type Renderer 

type Renderer interface {
	Render(contents []byte) (RenderResult, error)
}

Renderer defines the interface used to render (output) and template. FileRenderer implements this to write to disk.

type ResolveEvent 

type ResolveEvent struct {
	// Complete is true if all dependencies have values and the template
	// is fully rendered (in memory).
	Complete bool

	// Contents is the rendered contents from the template.
	// Only returned when Complete is true.
	Contents []byte

	// NoChange is true if no dependencies have changes in values and therefore
	// templates were not re-rendered.
	NoChange bool
}

ResolveEvent captures the whether the template dependencies have all been resolved and rendered in memory.

type Resolver 

type Resolver struct{}

Resolver is responsible rendering Templates and invoking Commands. Empty but reserving the space for future use.

func NewResolver 

func NewResolver() *Resolver

Basic constructor, here for consistency and future flexibility.

func (*Resolver) Run 

func (r *Resolver) Run(tmpl Templater, w Watcherer) (ResolveEvent, error)

Run the template Execute once. You should repeat calling this until output returns Complete as true. It uses the watcher for dependency lookup state. The content will be updated each pass until complete.

type RetryFunc 

type RetryFunc func(int) (bool, time.Duration)

RetryFunc defines the function type used to determine how many and how often to retry calls to the external services.

type Store 

type Store struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

Store is what Template uses to determine the values that are available for template parsing.

func NewStore 

func NewStore() *Store

NewStore creates a new Store with empty values for each of the key structs.

func (*Store) Delete 

func (s *Store) Delete(id string)

Forget accepts a dependency and removes all associated data with this dependency.

func (*Store) Recall 

func (s *Store) Recall(id string) (interface{}, bool)

Recall gets the current value for the given dependency in the Store.

func (*Store) Reset 

func (s *Store) Reset()

Reset clears all stored data.

func (*Store) Save 

func (s *Store) Save(id string, data interface{})

Save accepts a dependency and the data to store associated with that dep. This function converts the given data to a proper type and stores it interally.

type Template 

type Template struct {
	// contains filtered or unexported fields
}

Template is the internal representation of an individual template to process. The template retains the relationship between it's contents and is responsible for it's own execution.

func NewTemplate 

func NewTemplate(i TemplateInput) *Template

NewTemplate creates a new Template and primes it for the initial run.

func (*Template) Execute 

func (t *Template) Execute(rec Recaller) ([]byte, error)

Execute evaluates this template in the provided context.

func (*Template) ID 

func (t *Template) ID() string

ID returns the identifier for this template. Used to uniquely identify this template object for dependency management.

func (*Template) Notify 

func (t *Template) Notify(interface{}) bool

Notify template that a dependency it relies on has been updated. Works by marking the template so it knows it has new data to process when Execute is called.

func (*Template) Render 

func (t *Template) Render(content []byte) (RenderResult, error)

Render calls the stored Renderer with the passed content

type TemplateInput 

type TemplateInput struct {

	// Optional name for the template. Appended to the ID. Required if you want
	// to use the same content in more than one template with the same Watcher.
	Name string

	// Contents are the raw template contents.
	Contents string

	// ErrMissingKey causes the template parser to exit immediately with an
	// error when a map is indexed with a key that does not exist.
	ErrMissingKey bool

	// LeftDelim and RightDelim are the template delimiters.
	LeftDelim  string
	RightDelim string

	// FuncMapMerge a map of functions that add-to or override those used when
	// executing the template. (text/template)
	//
	// There is a special case for FuncMapMerge where, if matched, gets
	// called with the cache, used and missing sets (like the dependency
	// functions) should return a function that matches a signature required
	// by text/template's Funcmap (masked by an interface).
	// This special case function's signature should match:
	//    func(Recaller) interface{}
	FuncMapMerge template.FuncMap

	// SandboxPath adds a prefix to any path provided to the `file` function
	// and causes an error if a relative path tries to traverse outside that
	// prefix.
	SandboxPath string

	// Renderer is the default renderer used for this template
	Renderer Renderer
}

TemplateInput is used as input when creating the template.

type Templater 

type Templater interface {
	Notifier
	Execute(Recaller) ([]byte, error)
}

Templater the interface the Template provides. The interface is used to make the used/required API explicit.

type TransportInput 

type TransportInput struct {
	// Transport/TLS
	SSLEnabled bool
	SSLVerify  bool
	SSLCert    string
	SSLKey     string
	SSLCACert  string
	SSLCAPath  string
	ServerName string

	DialKeepAlive       time.Duration
	DialTimeout         time.Duration
	DisableKeepAlives   bool
	IdleConnTimeout     time.Duration
	MaxIdleConns        int
	MaxIdleConnsPerHost int
	TLSHandshakeTimeout time.Duration
}

type VaultInput 

type VaultInput struct {
	Address     string
	Namespace   string
	Token       string
	UnwrapToken bool
	Transport   TransportInput
	// optional, principally for testing
	HttpClient *http.Client
}

VaultInput defines the inputs needed to configure the Vault client.

type Watcher 

type Watcher struct {
	// contains filtered or unexported fields
}

Watcher is a manager for views that poll external sources for data.

func NewWatcher 

func NewWatcher(i WatcherInput) *Watcher

NewWatcher creates a new watcher using the given API client.

func (*Watcher) Buffer 

func (w *Watcher) Buffer(n Notifier) bool

Buffer sets the template to activate buffer and accumulate changes for a period. If the template has not been initalized or a buffer period is not configured for the template, it will skip the buffering. period.

func (*Watcher) Complete 

func (w *Watcher) Complete(n Notifier) bool

Complete checks if all values in use have been fetched.

func (*Watcher) ID 

func (w *Watcher) ID() string

ID here is to meet the IDer interface and be used with events/logging

func (*Watcher) Mark 

func (w *Watcher) Mark(notifier IDer)

Mark-n-Sweep garbage-collector-like cleaning of views that are no in use. Stops the (garbage) views and removes all references. Should be used before/after the code that uses the dependencies (eg. template).

Mark's all tracked dependencies as being *not* in use.

func (*Watcher) Poll 

func (w *Watcher) Poll(deps ...dep.Dependency)

Poll starts any/all polling as needed. It is idepotent. If nothing is passed it checks all views (dependencies).

func (*Watcher) Recaller 

func (w *Watcher) Recaller(n Notifier) Recaller

Recaller returns a Recaller (function) that wraps the Store (cache) to enable tracking dependencies on the Watcher.

func (*Watcher) Register 

func (w *Watcher) Register(ns ...Notifier) error

Register's one or more Notifiers with the Watcher for future use. Trying to register the same Notifier twice will result in an error and none of the Notifiers will be registered (all or nothing). Trying to use a Notifier without Registering it will result in a *panic*.

func (*Watcher) SetBufferPeriod 

func (w *Watcher) SetBufferPeriod(min, max time.Duration, tmplIDs ...string)

SetBufferPeriod sets a buffer period to accumulate dependency changes for a template.

func (*Watcher) Size 

func (w *Watcher) Size() int

Size returns the number of views this watcher is watching.

func (*Watcher) Stop 

func (w *Watcher) Stop()

Stop halts this watcher and any currently polling views immediately. If a view was in the middle of a poll, no data will be returned.

func (*Watcher) Sweep 

func (w *Watcher) Sweep(notifier IDer)

Sweeps (stop and dereference) all views for dependencies marked as *not* in use.

func (*Watcher) Track 

func (w *Watcher) Track(n Notifier, d dep.Dependency)

Track is used to add dependencies to be monitored by the watcher. It sets everything up but stops short of running the polling, waiting for an explicit start (see Poll below). It calls Register as a convenience, but ignores the returned error so it can be used with already Registered Notifiers. If the dependency is already registered, no action is taken.

func (*Watcher) Wait 

func (w *Watcher) Wait(ctx context.Context) error

Wait blocks until new a watched value changes or until context is closed or exceeds its deadline.

func (*Watcher) WaitCh 

func (w *Watcher) WaitCh(ctx context.Context) <-chan error

WaitCh returns an error channel and runs Wait sending the result down the channel. Useful for when you need to use Wait in a select block.

func (*Watcher) WatchVaultToken 

func (w *Watcher) WatchVaultToken(token string) error

WatchVaultToken takes a vault token and watches it to keep it updated. This is a specialized method as this token can be required without being in a template. I hope to generalize this idea so you can watch arbitrary dependencies in the future.

func (*Watcher) Watching 

func (w *Watcher) Watching(id string) bool

Watching determines if the given dependency (id) is being watched.

type WatcherInput 

type WatcherInput struct {
	// Clients is the client set to communicate with upstreams.
	Clients Looker
	// Cache is the Cacher for caching watched values
	Cache Cacher

	// EventHandler takes the callback for event processing
	EventHandler events.EventHandler

	// Optional Vault specific parameters
	// Default non-renewable secret duration
	VaultDefaultLease time.Duration
	// RetryFun for Vault
	VaultRetryFunc RetryFunc

	// Optional Consul specific parameters
	// MaxStale is the max time Consul will return a stale value.
	ConsulMaxStale time.Duration
	// BlockWait is amount of time Consul will block on a query.
	ConsulBlockWait time.Duration
	// RetryFun for Consul
	ConsulRetryFunc RetryFunc
}

type Watcherer 

type Watcherer interface {
	Buffer(Notifier) bool
	Recaller(Notifier) Recaller
	Complete(Notifier) bool
}

Watcherer is the subset of the Watcher's API that the resolver needs. The interface is used to make the used/required API explicit.

Source Files

View all Source files

Directories

Path Synopsis
Public Dependency type information.
 Public Dependency type information.
internal
dependency

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.