gw_cache

README

A cache for API gateways

gateway_cache implements a simple cache from HTTP requests to their responses. An essential quality is that the cache can receive repeated requests before computing the first one. In such a case, the repeated requests will wait without contention for other different requests until the response is ready.

Once the response is ready, the runtime will free the retained repeated requests, and the flow will continue in a usual way.

Declaration

The user can use the cache from any GO HTTP middleware.

To use it, declare something like that:

var cache \*Cache.CacheDriver = Cache.New(cacheCapacity, capFactor, cacheTTL,
                                          toKey, preProcess, callServices)

• cacheCapacity: The maximum number of entries that the cache can store. Once this limit reaches and one wants to insert a new entry, the runtime selects the LRU entry for eviction.
• capFactor: a multiplicative factor for increasing the physical size of the internal hash table. Using a physical size larger than the logical size reduces the possibility of an undesired table resizing when a new request arrives.
• cacheTTL: a soft duration time for the cache entry. By soft, we mean that the duration does not avoid eviction.
• toKey: a function used for transforming the request to a string. This function is used for mapping request to cache entries, which at the end contain the response.
• preProcess: an optional function used for executing some validations or transformations on the request.
• callServices: this is the core function in charge of calling the microservices, gathering the request, assembly them in an HTPP response, and eventually compressing it. The result is store in the cache entry.

Usage

In your request handler, you should put a line like this one:

gzipResponse, predictError := cache.RetrieveFromCacheOrCompute(request)  

The first result is the request itself already prepared by the callServices function. The second result is an error indication. If its value is not nil, then the first parameter is nil.

If it is the first request, then the flow blocks, but the process coded in callServices is triggered. The following repeated requests before to get the result block too, but they do not cause contention on other requests that are not related to the original one.

Once the result is gotten (by callServices function), all the retained requests are unblocked, and the flow continues as usual.

If the result is already in the cache, then the cache retrieves the result, return it, and the flow continues in a usual way without blocking.

Notes on its implementation

Parameters selection

The most important thing to know is the maximum number of simultaneous request that a gateway could receive. Once this number is known, the cache recommended cache capacity should be at least 30 % larger. The larger the capacity is, the more performant the cache should be.

Liveness danger

The cache could start to reject requests if it receives more different requests in a short time than its capacity.

Possible performance issue with internal hash Table

Internally, the cache handles a GO map from a string version of a request to a cache entry containing control information and the request and its response. The GO maps are implemented through a hash table that eventually could require a resize when the load factor becomes high (about 80 % of occupation). A new request arrival could cause, since this the only moment when caching insertions occur. Consenqtely, this could slow down the response time. The proper way for avoiding this event is to ensure that the table size is always enough. That implies that the size should always be greater than the number of inserted entries. This is the reason for the capFactor parameter received by the constructor.

We advise using a capFactor of at least 0.5 or more.

Documentation

Index

• Constants
• type CacheDriver
  • func New(capacity int, capFactor float64, ttl time.Duration, ...) *CacheDriver
  • func NewWithCompression(capacity int, capFactor float64, ttl time.Duration, ...) (cache *CacheDriver)
  • func (cache *CacheDriver) Capacity() int
  • func (cache *CacheDriver) Clean() error
  • func (cache *CacheDriver) Contains(keyVal interface{}) (bool, error)
  • func (cache *CacheDriver) ExtendedCapacity() int
  • func (cache *CacheDriver) GetState() (string, error)
  • func (cache *CacheDriver) HitCount() int
  • func (cache *CacheDriver) LazyRemove(keyVal interface{}) error
  • func (cache *CacheDriver) MissCount() int
  • func (cache *CacheDriver) NewCacheIt() *CacheIt
  • func (cache *CacheDriver) NumEntries() int
  • func (cache *CacheDriver) RetrieveFromCacheOrCompute(request interface{}, other ...interface{}) (interface{}, *RequestError)
  • func (cache *CacheDriver) Set(capacity int, ttl time.Duration) error
  • func (cache *CacheDriver) Touch(keyVal interface{}) error
  • func (cache *CacheDriver) Ttl() time.Duration
• type CacheEntry
• type CacheIt
  • func (it *CacheIt) GetCurr() *CacheEntry
  • func (it *CacheIt) HasCurr() bool
  • func (it *CacheIt) Next() *CacheEntry
• type CacheState
• type RequestError

Constants

const (
	AVAILABLE = iota
	COMPUTING
	COMPUTED
	FAILED5xx
	FAILED4xx
	FAILED5XXMISSHANDLERERROR
)

State that a cache entry could have

const (
	Status4xx = iota
	Status4xxCached
	Status5xx
	Status5xxCached
	StatusUser
)

Types

type CacheDriver

type CacheDriver struct {
	// contains filtered or unexported fields
}

func New

func New(capacity int, capFactor float64, ttl time.Duration,
	toMapKey func(key interface{}) (string, error),
	preProcessRequest func(request interface{}, other ...interface{}) (interface{}, *RequestError),
	callUServices func(request, payload interface{}, other ...interface{}) (interface{}, *RequestError),
) *CacheDriver

New Creates a new cache. Parameters are:

capacity: maximum number of entries that cache can manage without evicting the least recently used

capFactor is a number in (0.1, 3] that indicates how long the cache should be oversize in order to avoid rehashing

ttl: time to live of a cache entry

toMapKey is a function in charge of transforming the request into a string

preProcessRequest is an optional function that could be used for validation, transforming the request in a more suitable form, etc.

callUService: is responsible for calling to the service and building a byte sequence corresponding to the service response

func NewWithCompression

func NewWithCompression(capacity int, capFactor float64, ttl time.Duration,
	toMapKey func(key interface{}) (string, error),
	valueToBytes func(value interface{}) ([]byte, error),
	bytesToValue func([]byte) (interface{}, error),
	preProcessRequest func(request interface{}, other ...interface{}) (interface{}, *RequestError),
	callUServices func(request, payload interface{},
		other ...interface{}) (interface{}, *RequestError),
) (cache *CacheDriver)

NewWithCompression Creates a new cache with compressed entries.

The constructor is some similar to the version that does not compress. The difference is that in order to compress, the cache needs a serialized representation of what will be stored into the cache. For that reason, the constructor receives two additional functions. The first function, ValueToBytes transforms the value into a byte slice (type []byte). The second function, bytesToValue, takes a serialized representation of the value stored into the cache, and it transforms it to the original representation.

Parameters are:

capacity: maximum number of entries that cache can manage without evicting the least recently used

capFactor is a number in (0.1, 3] that indicates how long the cache should be oversize in order to avoid rehashing

ttl: time to live of a cache entry

toMapKey is a function in charge of transforming the request into a string

valueToBytes transforms the value into a []byte

bytesToValue transforms a []byte into the original value representation

valueToBytes

preProcessRequest is an optional function that could be used for validation, transforming the request in a more suitable form, etc.

callUService: is responsible for calling to the service and building a byte sequence corresponding to the service response

func (*CacheDriver) Capacity

func (cache *CacheDriver) Capacity() int

func (*CacheDriver) Clean

func (cache *CacheDriver) Clean() error

Clean Try to clean the cache. All the entries are deleted and counters reset. Fails if any entry is in COMPUTING state.

Uses internal lock

func (*CacheDriver) Contains

func (cache *CacheDriver) Contains(keyVal interface{}) (bool, error)

Contains returns true if the cache contains keyVal. It does not update the entry timestamp and consequently it does not change the eviction order.

func (*CacheDriver) ExtendedCapacity

func (cache *CacheDriver) ExtendedCapacity() int

func (*CacheDriver) GetState

func (cache *CacheDriver) GetState() (string, error)

GetState Return a json containing the cache state. Use the internal mutex. Be careful with a deadlock

func (*CacheDriver) HitCount

func (cache *CacheDriver) HitCount() int

func (*CacheDriver) LazyRemove

func (cache *CacheDriver) LazyRemove(keyVal interface{}) error

LazyRemove removes the entry with keyVal from the cache. It does not remove the entry immediately, but it marks it as removed.

func (*CacheDriver) MissCount

func (cache *CacheDriver) MissCount() int

func (*CacheDriver) NewCacheIt

func (cache *CacheDriver) NewCacheIt() *CacheIt

func (*CacheDriver) NumEntries

func (cache *CacheDriver) NumEntries() int

func (*CacheDriver) RetrieveFromCacheOrCompute

func (cache *CacheDriver) RetrieveFromCacheOrCompute(request interface{},
	other ...interface{}) (interface{}, *RequestError)

RetrieveFromCacheOrCompute Search Request in the cache. If the request is already computed, then it immediately returns the cached entry. If the request is the first, then it blocks until the result is ready. If the request is not the first but the result is not still ready, then it blocks until the result is ready

func (*CacheDriver) Set

func (cache *CacheDriver) Set(capacity int, ttl time.Duration) error

func (*CacheDriver) Touch

func (cache *CacheDriver) Touch(keyVal interface{}) error

func (*CacheDriver) Ttl

func (cache *CacheDriver) Ttl() time.Duration

type CacheEntry

type CacheEntry struct {
	// contains filtered or unexported fields
}

CacheEntry Every cache entry has this information

type CacheIt

type CacheIt struct {
	// contains filtered or unexported fields
}

CacheIt Iterator on cache entries. Go from MUR to LRU

func (*CacheIt) GetCurr

func (it *CacheIt) GetCurr() *CacheEntry

func (*CacheIt) HasCurr

func (it *CacheIt) HasCurr() bool

func (*CacheIt) Next

func (it *CacheIt) Next() *CacheEntry

type CacheState

type CacheState struct {
	MissCount  int
	HitCount   int
	TTL        time.Duration
	Capacity   int
	NumEntries int
}

type RequestError

type RequestError struct {
	Error    error
	Code     int
	UserCode int
}