stream

Documentation

Index

• func Deduplicate(in <-chan []byte, window time.Duration, ...) <-chan []byte
• func Execute(ctx context.Context, pf PeerFactory, rcf ReadCloserFactory, ...) <-chan []byte
• type PeerFactory
• type ReadCloserFactory
  • func HTTPReadCloserFactory(client *http.Client, addr2url func(string) string) ReadCloserFactory

Functions

func Deduplicate

func Deduplicate(in <-chan []byte, window time.Duration, ticker func(time.Duration) *time.Ticker) <-chan []byte

Deduplicate and order records within the given time window. A smaller window may cause duplicate or out-of-order messages. A larger window will cause higher end-to-end latency. The ticker is used every window / 10 to flush the buffer. The returned chan is closed when the in chan is closed.

func Execute

func Execute(
	ctx context.Context,
	pf PeerFactory,
	rcf ReadCloserFactory,
	sleep func(time.Duration),
	ticker func(time.Duration) *time.Ticker,
) <-chan []byte

Execute creates and maintains streams of records to multiple peers. It muxes the streams to the returned chan of records. The chan will be closed when the context is canceled. It's designed to be invoked once per user stream request.

Incoming records are muxed onto the provided sink chan. The sleep func is used to backoff between retries of a single peer. The ticker func is used to regularly resolve peers.

Types

type PeerFactory

type PeerFactory func() []string

PeerFactory should return the current set of peer addresses. Each address will be converted to an io.Reader via the ReadCloserFactory. The PeerFactory is periodically invoked to get the latest set of peers.

type ReadCloserFactory

type ReadCloserFactory func(context.Context, string) (io.ReadCloser, error)

ReadCloserFactory converts a peer address to an io.ReadCloser. ReadClosers must exit with context.Canceled when the context is canceled. Other errors will cause the managing goroutine to remanufacture.

func HTTPReadCloserFactory

func HTTPReadCloserFactory(client *http.Client, addr2url func(string) string) ReadCloserFactory

HTTPReadCloserFactory returns a ReadCloserFactory that converts the addr to a URL via the addr2url function, makes a GET request via the client, and returns the response body as the ReadCloser.