fetcher

Documentation

Index

• Constants
• Variables
• type Fetcher
  • func New(c context.Context, o Options) *Fetcher
  • func (f *Fetcher) Descriptor() *logpb.LogStreamDescriptor
  • func (f *Fetcher) NextLogEntry() (*logpb.LogEntry, error)
  • func (f *Fetcher) Reader() io.Reader
• type LogRequest
• type Options
• type Source

Constants

const (
	// DefaultDelay is the default Delay value.
	DefaultDelay = 5 * time.Second

	// DefaultBufferBytes is the default number of bytes to buffer.
	DefaultBufferBytes = int64(1024 * 1024) // 1MB
)

Variables

var ErrIncompleteStream = errors.New("stream has not yet terminated")

ErrIncompleteStream is returned by Fetcher if Options.RequireCompleteStream was true and the underlying Stream is still incomplete (i.e. has not yet been terminated by the client, or archived).

Types

type Fetcher

type Fetcher struct {
	// contains filtered or unexported fields
}

A Fetcher buffers LogEntry records by querying the Source for log data. It attmepts to maintain a steady stream of records by prefetching available records in advance of consumption.

A Fetcher is not goroutine-safe.

func New

func New(c context.Context, o Options) *Fetcher

New instantiates a new Fetcher instance.

The Fetcher can be cancelled by cancelling the supplied context.

func (*Fetcher) Descriptor

func (f *Fetcher) Descriptor() *logpb.LogStreamDescriptor

Descriptor returns the last-known Descriptor from the underlying Stream, if known.

If the underlying stream doesn't have this information, this returns nil.

func (*Fetcher) NextLogEntry

func (f *Fetcher) NextLogEntry() (*logpb.LogEntry, error)

NextLogEntry returns the next buffered LogEntry, blocking until it becomes available.

If the end of the log stream is encountered, NextLogEntry will return io.EOF.

If the Fetcher is cancelled, a context.Canceled error will be returned.

func (*Fetcher) Reader

func (f *Fetcher) Reader() io.Reader

Reader returns an io.Reader (a *renderer.Renderer) for this Fetcher.

type LogRequest

type LogRequest struct {
	// Index is the starting log index to request.
	Index types.MessageIndex
	// Count, if >0, is the maximum number of log entries to request.
	Count int
	// Bytes, if >0, is the maximum number of log bytes to request. At least one
	// log must be returned regardless of the byte limit.
	Bytes int64
}

LogRequest is a structure used by the Fetcher to request a range of logs from its Source.

type Options

type Options struct {
	// Source is the log stream source.
	Source Source

	// Index is the starting stream index to retrieve.
	Index types.MessageIndex
	// Count is the total number of logs to retrieve. If zero, the full stream
	// will be fetched.
	Count int64

	// Count is the minimum amount of LogEntry records to buffer. If the buffered
	// amount dips below Count, more logs will be fetched.
	//
	// If zero, no count target will be applied.
	BufferCount int

	// BufferBytes is the target number of LogEntry bytes to buffer. If the
	// buffered amount dips below Bytes, more logs will be fetched.
	//
	// If zero, no byte target will be applied unless Count is also zero, in which
	// case DefaultBufferBytes byte constraint will be applied.
	BufferBytes int64

	// PrefetchFactor constrains the amount of additional data to fetch when
	// refilling the buffer. Effective Count and Bytes values are multiplied
	// by PrefetchFactor to determine the amount of logs to request.
	PrefetchFactor int

	// Delay is the amount of time to wait in between unsuccessful log requests.
	Delay time.Duration

	// Set this to immediately bail out with ErrIncompleteStream if the stream
	// isn't complete yet. This can be useful when you believe the stream to
	// already be terminal, but haven't done an RPC with LogDog yet to actually
	// confirm this.
	RequireCompleteStream bool
	// contains filtered or unexported fields
}

Options is the set of configuration parameters for a Fetcher.

type Source

type Source interface {
	// LogEntries populates the supplied LogRequest with available sequential
	// log entries as available.
	//
	// This may optionally block pending new log entries, but may also return zero
	// log entries if none are available yet.
	//
	// Upon success, the requested logs and terminal message index is returned. If
	// no terminal index is known, a value <0 will be returned.
	LogEntries(context.Context, *LogRequest) ([]*logpb.LogEntry, types.MessageIndex, error)

	// Descriptor returns the stream's descriptor, if the source knows it.
	//
	// If the source doesn't have this information, this should return nil.
	Descriptor() *logpb.LogStreamDescriptor
}

Source is the source of log stream and log information.

The Source is responsible for handling retries, backoff, and transient errors. An error from the Source will shut down the Fetcher.