discovery

README

SDP Discovery Libraries

Code to help with all things related to discovering system state using State Description Protocol. Allows users to easily create software that discovers system state, for example:

• Source containers for running with srcman
• Agents for discovering local state on servers and other devices

This library is currently under development and documentation can be found on pkg.go.dev

Documentation

Index

• Constants
• func GetCacheDuration(s Source) time.Duration
• func IsWildcard(s string) bool
• type CacheDefiner
• type Engine
  • func (e *Engine) AddSources(sources ...Source)
  • func (e *Engine) Connect() error
  • func (e *Engine) ExecuteRequest(req *sdp.ItemRequest) ([]*sdp.Item, error)
  • func (e *Engine) FilterSources(typ string, context string) []Source
  • func (e *Engine) Find(typ string, context string) ([]*sdp.Item, error)
  • func (e *Engine) Get(typ string, context string, query string) (*sdp.Item, error)
  • func (e *Engine) IsNATSConnected() bool
  • func (e *Engine) NewItemRequestHandler(sources []Source) func(req *sdp.ItemRequest)
  • func (e *Engine) Search(typ string, context string, query string) ([]*sdp.Item, error)
  • func (e *Engine) SetupThrottle()
  • func (e *Engine) Sources() []Source
  • func (e *Engine) Start() error
  • func (e *Engine) Stop() error
• type GetFindMutex
  • func (g *GetFindMutex) FindLock(itemContext string, typ string)
  • func (g *GetFindMutex) FindUnlock(itemContext string, typ string)
  • func (g *GetFindMutex) GetLock(itemContext string, typ string)
  • func (g *GetFindMutex) GetUnlock(itemContext string, typ string)
• type HiddenSource
• type NATSOptions
• type NilPublisher
  • func (n NilPublisher) Publish(subject string, v interface{}) error
• type RequestTracker
  • func (r *RequestTracker) Execute() ([]*sdp.Item, error)
  • func (r *RequestTracker) LinkedItems() []*sdp.Item
• type SearchableSource
• type Source
• type Throttle
  • func (t *Throttle) Lock()
  • func (t *Throttle) Unlock()

Constants

const AllContexts = "*"

const WILDCARD = "*"

WILDCARD Used for requests that are relevant to many contexts or types

Functions

func GetCacheDuration

func GetCacheDuration(s Source) time.Duration

GetCacheDuration Gets the cache duration for a specific source, or a default value

func IsWildcard

func IsWildcard(s string) bool

IsWildcard checks if a string is the wildcard. Use this instead of implementing the wildcard check everwhere so that if we need to change the woldcard at a later date we can do so here

Types

type CacheDefiner

type CacheDefiner interface {
	DefaultCacheDuration() time.Duration
}

CacheDefiner Some backends may implement the CacheDefiner interface which means that they specify a custom default cache interval. The config will always take precedence however

type Engine

type Engine struct {
	// Descriptive name of this engine. Used as responder name in SDP responses
	Name string

	// Options for connecting to NATS
	NATSOptions *NATSOptions

	// The maximum number of queries that can be executing in parallel. Defaults
	// to the number of CPUs
	MaxParallelExecutions int
	// contains filtered or unexported fields
}

Engine is the main discovery engine. This is where all of the Sources and sources are stored and is responsible for calling out to the right sources to discover everything

Note that an engine that does not have a connected NATS connection will simply not communicate over NATS

func (*Engine) AddSources

func (e *Engine) AddSources(sources ...Source)

AddSources Adds a source to this engine

func (*Engine) Connect

func (e *Engine) Connect() error

Connect Connects to NATS

func (*Engine) ExecuteRequest

func (e *Engine) ExecuteRequest(req *sdp.ItemRequest) ([]*sdp.Item, error)

ExecuteRequest Executes a single request and returns the results without any linking

func (*Engine) FilterSources

func (e *Engine) FilterSources(typ string, context string) []Source

FilterSources returns the set of sources that match the supplied type and context. Supports wildcards in either position

func (*Engine) Find

func (e *Engine) Find(typ string, context string) ([]*sdp.Item, error)

Find executes Find() on all sources for a given type, returning the merged results. Only returns an error if all sources fail, in which case returns the first error

func (*Engine) Get

func (e *Engine) Get(typ string, context string, query string) (*sdp.Item, error)

Get Runs a get query against known sources in priority order. If nothing was found, returns the first error

func (*Engine) IsNATSConnected

func (e *Engine) IsNATSConnected() bool

IsNATSConnected returns whether the engine is connected to NATS

func (*Engine) NewItemRequestHandler

func (e *Engine) NewItemRequestHandler(sources []Source) func(req *sdp.ItemRequest)

NewItemRequestHandler Returns a function whose job is to handle a single request. This includes responses, linking etc.

func (*Engine) Search

func (e *Engine) Search(typ string, context string, query string) ([]*sdp.Item, error)

Search executes Search() on all sources for a given type, returning the merged results. Only returns an error if all sources fail, in which case returns the first error

func (*Engine) SetupThrottle

func (e *Engine) SetupThrottle()

SetupThrottle Sets up the throttling based on MaxParallelExecutions, including ensuring that it's not set to zero

func (*Engine) Sources

func (e *Engine) Sources() []Source

Sources Returns a slice of all known sources

func (*Engine) Start

func (e *Engine) Start() error

Start performs all of the initialisation steps required for the engine to work. Note that this creates NATS subscriptions for all available sources so modifying the Sources value after an engine has been started will not have any effect until the engine is restarted

func (*Engine) Stop

func (e *Engine) Stop() error

Stop Stops the engine running, draining all connections

type GetFindMutex

type GetFindMutex struct {
	// contains filtered or unexported fields
}

GetFindMutex A modified version of a RWMutex. Many get locks can be held but only one Find lock. A waiting Find lock (even if it hasn't been locked, just if someone is waiting) blocks all other get locks until it unlocks.

The intended usage of this is that it will allow a source which is trying to process many requests at once, to process a FIND request before any GET requests, since it's likely that once FIND has been run, subsequent GET requests will be able to be served from cache

func (*GetFindMutex) FindLock

func (g *GetFindMutex) FindLock(itemContext string, typ string)

FindLock An exclusive lock. Ensure that all GetLocks have been unlocked and stops any more from being obtained. Provide a type and context to ensure that the lock is only help for that type and context combination rather than locking the whole engine

func (*GetFindMutex) FindUnlock

func (g *GetFindMutex) FindUnlock(itemContext string, typ string)

FindUnlock Unlocks a FindLock

func (*GetFindMutex) GetLock

func (g *GetFindMutex) GetLock(itemContext string, typ string)

GetLock Gets a lock that can be held by an unlimited number of goroutines, these locks are only blocked by FindLocks. A type and context must be provided since a Get in one type (or context) should not be blocked by a Find in another

func (*GetFindMutex) GetUnlock

func (g *GetFindMutex) GetUnlock(itemContext string, typ string)

GetUnlock Unlocks the GetLock. This must be called once for each GetLock otherwise it will be impossible to ever obtain a FindLock

type HiddenSource

type HiddenSource interface {
	Hidden() bool
}

HiddenSource Sources that define a `Hidden()` method are able to tell whether or not the items they produce should be marked as hidden within the metadata. Hidden items will not be shown in GUIs or stored in databases and are used for gathering data as part of other proccesses such as remotely executed secondary sources

type NATSOptions

type NATSOptions struct {
	// The list of URLs to use for connecting to NATS
	URLs []string

	// The name given to the connection, useful in logging
	ConnectionName string

	// How long to wait when trying to connect to each NATS server
	ConnectTimeout time.Duration

	// How many times to retry if there was an error when connecting
	NumRetries int

	// Path to the customer CA file to use when using TLS (if required)
	CAFile string

	// Path to the NKey seed file
	NkeyFile string

	// Path to the JWT
	JWTFile string

	// The name of the queue to join when subscribing to subjects
	QueueName string
}

type NilPublisher

type NilPublisher struct{}

When testing this library, or running without a real NATS connection, it is necessary to create a fake publisher rather than pass in a nil pointer. This is due to the fact that the NATS libraries will panic if a method is called on a nil pointer

func (NilPublisher) Publish

func (n NilPublisher) Publish(subject string, v interface{}) error

Publish Logs an error rather than publishing

type RequestTracker

type RequestTracker struct {
	// The list of requests to track
	Requests []*sdp.ItemRequest

	// The engine that this is connected to, used for sending NATS messages
	Engine *Engine
	// contains filtered or unexported fields
}

RequestTracker is used for tracking the progress of a single request (or a set of related requests). This is required because a single request could have a link depth that results in many requests being executed meaning that we need to not only track the first request, but also all other requests and items that result from linking

func (*RequestTracker) Execute

func (r *RequestTracker) Execute() ([]*sdp.Item, error)

func (*RequestTracker) LinkedItems

func (r *RequestTracker) LinkedItems() []*sdp.Item

type SearchableSource

type SearchableSource interface {
	Source
	Search(itemContext string, query string) ([]*sdp.Item, error)
}

SearchableItemSource Is a source of items that supports searching

type Source

type Source interface {
	// Type The type of items that this source is capable of finding
	Type() string

	// Descriptive name for the source, used in logging and metadata
	Name() string

	// List of contexts that this source is capable of find items for. If the
	// source supports all contexts the special value `AllContexts` ("*")
	// should be used
	Contexts() []string

	// Get Get a single item with a given context and query. The item returned
	// should have a UniqueAttributeValue that matches the `query` parameter
	Get(itemContext string, query string) (*sdp.Item, error)

	// Find Finds all items in a given context
	Find(itemContext string) ([]*sdp.Item, error)

	// Weight Returns the priority weighting of items returned by this source.
	// This is used to resolve conflicts where two sources of the same type
	// return an item for a GET request. In this instance only one item can be
	// sen on, so the one with the higher weight value will win.
	Weight() int
}

Source is capable of finding information about items

type Throttle

type Throttle struct {
	NumParallel int
	// contains filtered or unexported fields
}

Throttle limits the number of processes that can be executing at once to `NumParallel`. Users should call `Lock()` to obtain a lock and `Unlock()` once their work is done

func (*Throttle) Lock

func (t *Throttle) Lock()

func (*Throttle) Unlock

func (t *Throttle) Unlock()