liveness

README

Conjure Liveness Module

The liveness module is designed to keep track of all the live hosts in the network by scanning the internet on a regular basis. The module provides cached liveness detection and uncached liveness detection. Cached liveness detection stores live hosts from previous PhantomIsLive calls to improve performance, non-responsive hosts and stale cache entries are re-scanned each time. Uncached liveness detection directly visits an IP address to check its liveness status on each call.

The validity of cached liveness detection was tested in a week-long survey of the internet where we measured the change in liveness across the network. We observed a clear trend in the stability of discovered hosts over time and as such chose a 2 hour default cache period.

Usage

To check if an IP address is live in the network, call PhantomIsLive(addr string, port uint16) which return a bool and an error message if applicable.

Network Survey Result

Survey data for 3 weeks
Survey data for first 48 hours

Percentages of the number of current live hosts divided by the number of cached live hosts(marked green) went down drastically in the first 24 hours of scanning the network which indicates that caching every discoverable live hosts is not an effective way to represent the current liveness status of the addresses in the network over time. Instead, we decided to cache individual IP addresses that are passed to PhantomIsLive for checking its liveness status. Every cached address gets a timestamp when its liveness status is checked, cached addresses with expired timestamp will no longer be considered as live hosts by the module. Expiration duration can be set in the config file.

Caching Flow Diagram

Documentation

Index

• Variables
• type CachedLivenessTester
  • func (blt *CachedLivenessTester) ClearExpiredCache()
  • func (blt *CachedLivenessTester) Init(conf *Config) error
  • func (blt *CachedLivenessTester) PhantomIsLive(addr string, port uint16) (bool, error)
  • func (blt *CachedLivenessTester) PrintAndReset(logger *log.Logger)
  • func (blt *CachedLivenessTester) PrintStats(logger *log.Logger)
  • func (s CachedLivenessTester) Reset()
  • func (blt *CachedLivenessTester) Stop()
• type Config
  • func (c *Config) LivenessConfig() *Config
• type Stats
• type Tester
  • func New(c *Config) (Tester, error)
• type UncachedLivenessTester
  • func (blt *UncachedLivenessTester) PhantomIsLive(addr string, port uint16) (bool, error)
  • func (s UncachedLivenessTester) PrintAndReset(logger *log.Logger)
  • func (s UncachedLivenessTester) PrintStats(logger *log.Logger)
  • func (s UncachedLivenessTester) Reset()

Variables

var (
	// ErrCachedPhantom provides a constant expected error returned for cached
	// liveness hits
	ErrCachedPhantom = errors.New("cached live host")

	// ErrNotImplemented indicates that a feature will be implemented at some point
	// but is not yet completed.
	ErrNotImplemented = errors.New("not supported yet")

	// NotLive is an error returned when a liveness test reaches timeout, giving
	// confidence that the host in question is not live
	NotLive = errors.New("reached statistical timeout")

	// ErrLiveHost indicates that an error occurred or a successful connection
	// was formed with the host in question indicating that the host is live.
	ErrLiveHost = errors.New("phantom picked up the connection")
)

Types

type CachedLivenessTester

type CachedLivenessTester struct {
	// contains filtered or unexported fields
}

CachedLivenessTester implements LivenessTester interface with caching, PhantomIsLive will check historical results first before using the network to determine phantom liveness.

func (*CachedLivenessTester) ClearExpiredCache

func (blt *CachedLivenessTester) ClearExpiredCache()

ClearExpiredCache cleans out stale entries in the cache.

func (*CachedLivenessTester) Init

func (blt *CachedLivenessTester) Init(conf *Config) error

Init parses cache expiry duration and initializes the Cache.

func (*CachedLivenessTester) PhantomIsLive

func (blt *CachedLivenessTester) PhantomIsLive(addr string, port uint16) (bool, error)

PhantomIsLive first checks the cached set of addresses for a fresh entry. If one is available and this is returned immediately and no network probes are sent. If the host was not recently measured, the entry is stale, or there is no entry then network probes are sent and the result is then added to the cache.

Lock on mutex is taken for lookup, then for cache update. Do NOT hold mutex while scanning for liveness as this will make cache extremely slow.

func (*CachedLivenessTester) PrintAndReset

func (blt *CachedLivenessTester) PrintAndReset(logger *log.Logger)

PrintAndReset implements the Stats interface extending from the stats struct to add logging for the cache capacity

func (*CachedLivenessTester) PrintStats

func (blt *CachedLivenessTester) PrintStats(logger *log.Logger)

PrintStats implements the Stats interface extending from the stats struct to add logging for the cache capacity

func (CachedLivenessTester) Reset

func (s CachedLivenessTester) Reset()

func (*CachedLivenessTester) Stop

func (blt *CachedLivenessTester) Stop()

Stop end periodic scanning using running in separate goroutine. If periodic scanning is not running this will do nothing.

type Config

type Config struct {
	// CacheDuration specifies the duration that a phantom IP identified as
	// "LIVE" using a liveness test is cached, preventing further lookups to the
	// address. Empty string disables caching for live phantom hosts.
	CacheDuration string `toml:"cache_expiration_time"`

	// CacheCapacity specifies the cache capacity to use for phantom IPs
	// identified as "LIVE". CacheDuration must be set otherwise no caching
	// occurs for live hosts.
	//
	// If unset or 0 no capacity is set and a map is used for the cache
	// otherwise cache will have finite capacity and implement LRU eviction.
	CacheCapacity int `toml:"cache_capacity"`

	// CacheDurationNonLive specifies the duration that a phantom IP identified
	// as "NOT LIVE" using a liveness test is cached, preventing further lookups
	// to the address. This should generally be shorter to be responsive to
	// remain responsive to hosts that become live. Empty string disables
	// caching for non-live phantom hosts.
	CacheDurationNonLive string `toml:"cache_expiration_nonlive"`

	// CacheCapacityNonLive specifies the cache capacity to use for phantom IPs
	// identified as "NOT LIVE". CacheDurationNonLive must be set otherwise no
	// caching occurs for non-live hosts.
	//
	// If unset or 0 no capacity is set and a map is used for the cache
	// otherwise cache will have finite capacity and implement LRU eviction.
	CacheCapacityNonLive int `toml:"cache_capacity_nonlive"`
}

Config provides all params relating to liveness testing construction

func (*Config) LivenessConfig

func (c *Config) LivenessConfig() *Config

LivenessConfig identity function for reflection in composed Config type

type Stats

type Stats interface {
	PrintAndReset(logger *log.Logger)
	PrintStats(logger *log.Logger)
	Reset()
}

Stats provides an interface to write out the collected metrics about liveness tester usage

type Tester

type Tester interface {
	Stats
	PhantomIsLive(addr string, port uint16) (bool, error)
}

Tester provides a generic interface for testing hosts in phantom subnets for liveness. This prevents potential interference in connection creation.

func New

func New(c *Config) (Tester, error)

New provides a builder for the proper tester based on config.

type UncachedLivenessTester

type UncachedLivenessTester struct {
	// contains filtered or unexported fields
}

UncachedLivenessTester implements LivenessTester interface without caching, PhantomIsLive will always use the network to determine phantom liveness.

func (*UncachedLivenessTester) PhantomIsLive

func (blt *UncachedLivenessTester) PhantomIsLive(addr string, port uint16) (bool, error)

PhantomIsLive sends 4 TCP syn packets to determine if the host will respond to traffic and potentially interfere with a connection if used as a phantom address. Measurement results are uncached, meaning endpoints are re-scanned every time.

func (UncachedLivenessTester) PrintAndReset

func (s UncachedLivenessTester) PrintAndReset(logger *log.Logger)

func (UncachedLivenessTester) PrintStats

func (s UncachedLivenessTester) PrintStats(logger *log.Logger)

func (UncachedLivenessTester) Reset

func (s UncachedLivenessTester) Reset()