Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DefaultOptions ¶
func DefaultOptions() *options
Types ¶
type Manager ¶
type Manager struct {
// we need lock to access pbqMap, since deregister will be called inside pbq
// and each pbq will be inside a go routine, and also entire PBQ could be managed
// through a go routine (depends on the orchestrator)
sync.RWMutex
// contains filtered or unexported fields
}
Manager helps in managing the lifecycle of PBQ instances
func NewManager ¶
func NewManager(ctx context.Context, vertexName string, pipelineName string, vr int32, storeProvider wal.Manager, windowType window.Type, opts ...PBQOption) (*Manager, error)
NewManager returns new instance of manager We don't intend this to be called by multiple routines.
func (*Manager) CreateNewPBQ ¶
func (m *Manager) CreateNewPBQ(ctx context.Context, partitionID partition.ID) (ReadWriteCloser, error)
CreateNewPBQ creates new pbq for a partition
func (*Manager) GetPBQ ¶
func (m *Manager) GetPBQ(partitionID partition.ID) ReadWriteCloser
GetPBQ returns pbq for the given ID
func (*Manager) ListPartitions ¶
ListPartitions returns all the pbq instances
type PBQ ¶
PBQ Buffer queue which is backed with a persisted store, each partition will have a PBQ associated with it
func (*PBQ) Close ¶
Close is used by the writer to indicate close of context we should flush pending messages to store
func (*PBQ) GC ¶
GC cleans up the PBQ and also the store associated with it. GC is invoked after the Reader (ProcessAndForward) has finished forwarding the output to ISB.
func (*PBQ) ReadCh ¶
func (p *PBQ) ReadCh() <-chan *window.TimedWindowRequest
ReadCh exposes read channel to read the window requests from the PBQ close on read channel indicates COB
type PBQOption ¶
type PBQOption func(options *options) error
func WithChannelBufferSize ¶
WithChannelBufferSize sets buffer size option
func WithReadBatchSize ¶
WithReadBatchSize sets read batch size option
func WithReadTimeout ¶
WithReadTimeout sets read timeout option
type ReadWriteCloser ¶
type ReadWriteCloser interface {
Reader
WriteCloser
}
ReadWriteCloser is an unified interface to PBQ read and write interfaces. Close is only for Writer.
type Reader ¶
type Reader interface {
// ReadCh exposes channel to read from PBQ
ReadCh() <-chan *window.TimedWindowRequest
// GC does garbage collection, it deletes all the persisted data from the store
GC() error
}
Reader provides methods to read from PBQ.
type WriteCloser ¶
type WriteCloser interface {
// Write writes the TimedWindowRequest to PBQ
// persist to decide if the data should be persisted or not
// during replay persist will be false
Write(ctx context.Context, msg *window.TimedWindowRequest, persist bool) error
// CloseOfBook (cob) closes PBQ, no writes will be accepted after cob
CloseOfBook()
// Close to handle context close on writer
// Any pending data can be flushed to the persistent store at this point.
Close() error
}
WriteCloser provides methods to write data to the PQB and close the PBQ. No data can be written to PBQ after cob.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package partition is a tuple containing (start, end) time and an optional slot.
|
Package partition is a tuple containing (start, end) time and an optional slot. |
|
aligned/fs
Package fs implements write-ahead-log on FileSystem.
|
Package fs implements write-ahead-log on FileSystem. |
|
unaligned/fs
Package fs implements write-ahead-log on FileSystem.
|
Package fs implements write-ahead-log on FileSystem. |