dedupe

Documentation

Overview

Package dedupe implements a generic request/response proxy that issues a single request instead of multiple redundant requests.

To initialize a Deduper, define the function that handles the request, and pass it in to the constructor:

dd := deduper.New(f, 0, 0)

Requests can then be issued:

dd.Request(ctx.TODO(), objectA)
dd.Request(ctx.TODO(), objectB)

If objectA and objectB have the same DedupeKey() (and arrive at dd at approximately the same time, see the documentation for New for more information), a single call to f is made.

To support anycast behavior (where multiple requests are sent out to various services, and the first response that we get unblocks all waiters), requests can define BroadcastKeys.

Index

• Constants
• type CancelFunc
• type Deduper
  • func New(f RequestFunc, dedupeLifetime, responseValidity time.Duration) Deduper
• type Request
• type RequestFunc
• type Response
• type ResponseChannel

Constants

const (
	DefaultDedupeLifetime   = 5 * time.Second
	DefaultResponseValidity = 1 * time.Second
)

Types

type CancelFunc

type CancelFunc func()

CancelFunc can be called to cancel a request ahead of time, freeing up internal resources (notification lists, goroutines, etc.).

type Deduper

type Deduper interface {
	// Request passes a request that is subject to deduplication. This function
	// returns immediately, and callers should wait on the returned channel for the
	// result. The second return value is a cancellation function that can be used
	// to free up resources associated with the request. It is safe to call Request
	// from multiple goroutines.
	//
	// Objects written to the channel might share the same address space, so
	// callers should copy the value drained from the channel if they want to have
	// exclusive ownership.
	//
	// Note this method explicitly takes no context, when waiting on the response
	// you should always also read from the ctx.Done() channel.
	Request(ctx context.Context, req Request) (<-chan Response, CancelFunc, opentracing.Span)
}

A Deduper issues a single request instead of multiple identical requests. Responses get broadcast to all waiters. For more information, see the package level documentation.

The zero value is a valid Deduper object. Members variables should only be set during initialization; setting them after the first Request is undefined behavior.

func New

func New(f RequestFunc, dedupeLifetime, responseValidity time.Duration) Deduper

New allocates a new Deduper.

f is the function to call when a new request needs to be sent out.

dedupeLifetime is the timeout for network requests. For dedupeLifetime time after a fresh network request is sent out (for a DedupeKey), no new network requests are sent out. Once the request completes, all callers of Request are notified. If 0, dedupeLifetime defaults to DefaultDedupeLifetime.

responseValidity is the time after a successful network request where no new network requests for the same broadcast key are sent out. The result is immediately returned from an internal cache for this period. If 0, responseValidity defaults to DefaultResponseValidity.

type Request

type Request interface {
	// Two requests are considered identical if they return the same
	// DedupeKey.
	DedupeKey() string
	// When a reply arrives for a request, the reply is delivered to all
	// requests sharing the same broadcast key. If two requests have the same
	// DedupeKey, they must have the same BroadcastKey.  BroadcastKeys allow
	// applications to implement anycast request/responses.  For this, define
	// different DedupeKey for the same message sent to different remote
	// servers, while keeping the BroadcastKey the same.
	BroadcastKey() string
}

type RequestFunc

type RequestFunc func(ctx context.Context, request Request) Response

RequestFunc performs a request/response exchange, with the response written on the channel. To support proper clean-up, RequestFunc must write exactly one value to response. The Response is transparently passed by the Deduper to all callers waiting on the same BroadcastKey. To avoid leaks, RequestFunc must take ctx into account and correctly time out/terminate if the context is Done.

When sending out a fresh request, the Deduper calls RequestFunc in a goroutine, and selects on ctx.Done() and on the response channel.

Use TestRequestFunc to verify that an implementation is valid for use with Deduper.

type Response

type Response struct {
	Data  interface{}
	Error error
}

Response represents the outcome of a request. It is passed along channels by the Deduper.

type ResponseChannel

type ResponseChannel chan Response