queue

package
v0.0.0-...-85b91c5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 25, 2023 License: MIT Imports: 18 Imported by: 16

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrAlreadyInQueue = fmt.Errorf("already in queue")

ErrAlreadyInQueue is returned when trying to push data to the queue that is already in the queue

Functions

func IsErrInvalidConfiguration 

func IsErrInvalidConfiguration(err error) bool

IsErrInvalidConfiguration checks if an error is an ErrInvalidConfiguration

func RegisteredTypesAsString 

func RegisteredTypesAsString() []string

RegisteredTypesAsString provides the list of requested types of queues

Types

type ByteFIFO 

type ByteFIFO interface {
	// Len returns the length of the fifo
	Len(ctx context.Context) int64
	// PushFunc pushes data to the end of the fifo and calls the callback if it is added
	PushFunc(ctx context.Context, data []byte, fn func() error) error
	// Pop pops data from the start of the fifo
	Pop(ctx context.Context) ([]byte, error)
	// Close this fifo
	Close() error
	// PushBack pushes data back to the top of the fifo
	PushBack(ctx context.Context, data []byte) error
}

ByteFIFO defines a FIFO that takes a byte array

type ByteFIFOQueue 

type ByteFIFOQueue struct {
	*WorkerPool
	// contains filtered or unexported fields
}

ByteFIFOQueue is a Queue formed from a ByteFIFO and WorkerPool

func NewByteFIFOQueue 

func NewByteFIFOQueue(typ Type, byteFIFO ByteFIFO, handle HandlerFunc, cfg, exemplar interface{}) (*ByteFIFOQueue, error)

NewByteFIFOQueue creates a new ByteFIFOQueue

func (*ByteFIFOQueue) Flush 

func (q *ByteFIFOQueue) Flush(timeout time.Duration) error

Flush flushes the ByteFIFOQueue

func (*ByteFIFOQueue) IsEmpty 

func (q *ByteFIFOQueue) IsEmpty() bool

IsEmpty checks if the queue is empty

func (*ByteFIFOQueue) IsShutdown 

func (q *ByteFIFOQueue) IsShutdown() <-chan struct{}

IsShutdown returns a channel which is closed when this Queue is shutdown

func (*ByteFIFOQueue) IsTerminated 

func (q *ByteFIFOQueue) IsTerminated() <-chan struct{}

IsTerminated returns a channel which is closed when this Queue is terminated

func (*ByteFIFOQueue) Name 

func (q *ByteFIFOQueue) Name() string

Name returns the name of this queue

func (*ByteFIFOQueue) NumberInQueue 

func (q *ByteFIFOQueue) NumberInQueue() int64

NumberInQueue returns the number in the queue

func (*ByteFIFOQueue) Push 

func (q *ByteFIFOQueue) Push(data Data) error

Push pushes data to the fifo

func (*ByteFIFOQueue) PushBack 

func (q *ByteFIFOQueue) PushBack(data Data) error

PushBack pushes data to the fifo

func (*ByteFIFOQueue) PushFunc 

func (q *ByteFIFOQueue) PushFunc(data Data, fn func() error) error

PushFunc pushes data to the fifo

func (*ByteFIFOQueue) Run 

func (q *ByteFIFOQueue) Run(atShutdown, atTerminate func(func()))

Run runs the bytefifo queue

func (*ByteFIFOQueue) Shutdown 

func (q *ByteFIFOQueue) Shutdown()

Shutdown processing from this queue

func (*ByteFIFOQueue) Terminate 

func (q *ByteFIFOQueue) Terminate()

Terminate this queue and close the queue

type ByteFIFOQueueConfiguration 

type ByteFIFOQueueConfiguration struct {
	WorkerPoolConfiguration
	Workers     int
	WaitOnEmpty bool
}

ByteFIFOQueueConfiguration is the configuration for a ByteFIFOQueue

type ByteFIFOUniqueQueue 

type ByteFIFOUniqueQueue struct {
	ByteFIFOQueue
}

ByteFIFOUniqueQueue represents a UniqueQueue formed from a UniqueByteFifo

func NewByteFIFOUniqueQueue 

func NewByteFIFOUniqueQueue(typ Type, byteFIFO UniqueByteFIFO, handle HandlerFunc, cfg, exemplar interface{}) (*ByteFIFOUniqueQueue, error)

NewByteFIFOUniqueQueue creates a new ByteFIFOUniqueQueue

func (*ByteFIFOUniqueQueue) Has 

func (q *ByteFIFOUniqueQueue) Has(data Data) (bool, error)

Has checks if the provided data is in the queue

type ChannelQueue 

type ChannelQueue struct {
	*WorkerPool
	// contains filtered or unexported fields
}

ChannelQueue implements Queue

A channel queue is not persistable and does not shutdown or terminate cleanly It is basically a very thin wrapper around a WorkerPool

func (*ChannelQueue) Flush 

func (q *ChannelQueue) Flush(timeout time.Duration) error

Flush flushes the channel with a timeout - the Flush worker will be registered as a flush worker with the manager

func (*ChannelQueue) FlushWithContext 

func (q *ChannelQueue) FlushWithContext(ctx context.Context) error

FlushWithContext is very similar to CleanUp but it will return as soon as the dataChan is empty

func (*ChannelQueue) Name 

func (q *ChannelQueue) Name() string

Name returns the name of this queue

func (*ChannelQueue) Push 

func (q *ChannelQueue) Push(data Data) error

Push will push data into the queue

func (*ChannelQueue) Run 

func (q *ChannelQueue) Run(atShutdown, atTerminate func(func()))

Run starts to run the queue

func (*ChannelQueue) Shutdown 

func (q *ChannelQueue) Shutdown()

Shutdown processing from this queue

func (*ChannelQueue) Terminate 

func (q *ChannelQueue) Terminate()

Terminate this queue and close the queue

type ChannelQueueConfiguration 

type ChannelQueueConfiguration struct {
	WorkerPoolConfiguration
	Workers int
}

ChannelQueueConfiguration is the configuration for a ChannelQueue

type ChannelUniqueQueue 

type ChannelUniqueQueue struct {
	*WorkerPool
	// contains filtered or unexported fields
}

ChannelUniqueQueue implements UniqueQueue

It is basically a thin wrapper around a WorkerPool but keeps a store of what has been pushed within a table.

Please note that this Queue does not guarantee that a particular task cannot be processed twice or more at the same time. Uniqueness is only guaranteed whilst the task is waiting in the queue.

func (*ChannelUniqueQueue) Flush 

func (q *ChannelUniqueQueue) Flush(timeout time.Duration) error

Flush flushes the channel with a timeout - the Flush worker will be registered as a flush worker with the manager

func (*ChannelUniqueQueue) FlushWithContext 

func (q *ChannelUniqueQueue) FlushWithContext(ctx context.Context) error

FlushWithContext is very similar to CleanUp but it will return as soon as the dataChan is empty

func (*ChannelUniqueQueue) Has 

func (q *ChannelUniqueQueue) Has(data Data) (bool, error)

Has checks if the data is in the queue

func (*ChannelUniqueQueue) Name 

func (q *ChannelUniqueQueue) Name() string

Name returns the name of this queue

func (*ChannelUniqueQueue) Push 

func (q *ChannelUniqueQueue) Push(data Data) error

Push will push data into the queue if the data is not already in the queue

func (*ChannelUniqueQueue) PushFunc 

func (q *ChannelUniqueQueue) PushFunc(data Data, fn func() error) error

PushFunc will push data into the queue

func (*ChannelUniqueQueue) Run 

func (q *ChannelUniqueQueue) Run(atShutdown, atTerminate func(func()))

Run starts to run the queue

func (*ChannelUniqueQueue) Shutdown 

func (q *ChannelUniqueQueue) Shutdown()

Shutdown processing from this queue

func (*ChannelUniqueQueue) Terminate 

func (q *ChannelUniqueQueue) Terminate()

Terminate this queue and close the queue

type ChannelUniqueQueueConfiguration 

type ChannelUniqueQueueConfiguration ChannelQueueConfiguration

ChannelUniqueQueueConfiguration is the configuration for a ChannelUniqueQueue

type Data 

type Data interface{}

Data defines an type of queuable data

type DummyByteFIFO 

type DummyByteFIFO struct{}

DummyByteFIFO represents a dummy fifo

func (*DummyByteFIFO) Close 

func (*DummyByteFIFO) Close() error

Close returns nil

func (*DummyByteFIFO) Len 

func (*DummyByteFIFO) Len(ctx context.Context) int64

Len is always 0

func (*DummyByteFIFO) Pop 

func (*DummyByteFIFO) Pop(ctx context.Context) ([]byte, error)

Pop returns nil

func (*DummyByteFIFO) PushBack 

func (*DummyByteFIFO) PushBack(ctx context.Context, data []byte) error

PushBack pushes data back to the top of the fifo

func (*DummyByteFIFO) PushFunc 

func (*DummyByteFIFO) PushFunc(ctx context.Context, data []byte, fn func() error) error

PushFunc returns nil

type DummyQueue 

type DummyQueue struct{}

DummyQueue represents an empty queue

func (*DummyQueue) Flush 

func (*DummyQueue) Flush(time.Duration) error

Flush always returns nil

func (*DummyQueue) FlushWithContext 

func (*DummyQueue) FlushWithContext(context.Context) error

FlushWithContext always returns nil

func (*DummyQueue) Has 

func (*DummyQueue) Has(Data) (bool, error)

Has always returns false as this queue never does anything

func (*DummyQueue) IsEmpty 

func (*DummyQueue) IsEmpty() bool

IsEmpty asserts that the queue is empty

func (*DummyQueue) Push 

func (*DummyQueue) Push(Data) error

Push fakes a push of data to the queue

func (*DummyQueue) PushFunc 

func (*DummyQueue) PushFunc(Data, func() error) error

PushFunc fakes a push of data to the queue with a function. The function is never run.

func (*DummyQueue) Run 

func (*DummyQueue) Run(_, _ func(func()))

Run does nothing

type DummyUniqueByteFIFO 

type DummyUniqueByteFIFO struct {
	DummyByteFIFO
}

DummyUniqueByteFIFO represents a dummy unique fifo

func (*DummyUniqueByteFIFO) Has 

func (*DummyUniqueByteFIFO) Has(ctx context.Context, data []byte) (bool, error)

Has always returns false

type ErrInvalidConfiguration 

type ErrInvalidConfiguration struct {
	// contains filtered or unexported fields
}

ErrInvalidConfiguration is called when there is invalid configuration for a queue

func (ErrInvalidConfiguration) Error 

func (err ErrInvalidConfiguration) Error() string

type Flushable 

type Flushable interface {
	// Flush will add a flush worker to the pool - the worker should be autoregistered with the manager
	Flush(time.Duration) error
	// FlushWithContext is very similar to Flush
	// NB: The worker will not be registered with the manager.
	FlushWithContext(ctx context.Context) error
	// IsEmpty will return if the managed pool is empty and has no work
	IsEmpty() bool
}

Flushable represents a pool or queue that is flushable

type HandlerFunc 

type HandlerFunc func(...Data) (unhandled []Data)

HandlerFunc is a function that takes a variable amount of data and processes it

type Immediate 

type Immediate struct {
	// contains filtered or unexported fields
}

Immediate represents an direct execution queue

func (*Immediate) Flush 

func (*Immediate) Flush(time.Duration) error

Flush always returns nil

func (*Immediate) FlushWithContext 

func (*Immediate) FlushWithContext(context.Context) error

FlushWithContext always returns nil

func (*Immediate) Has 

func (*Immediate) Has(Data) (bool, error)

Has always returns false as this queue never does anything

func (*Immediate) IsEmpty 

func (*Immediate) IsEmpty() bool

IsEmpty asserts that the queue is empty

func (*Immediate) Push 

func (q *Immediate) Push(data Data) error

Push fakes a push of data to the queue

func (*Immediate) PushFunc 

func (q *Immediate) PushFunc(data Data, f func() error) error

PushFunc fakes a push of data to the queue with a function. The function is never run.

func (*Immediate) Run 

func (*Immediate) Run(_, _ func(func()))

Run does nothing

type LevelQueue 

type LevelQueue struct {
	*ByteFIFOQueue
}

LevelQueue implements a disk library queue

type LevelQueueByteFIFO 

type LevelQueueByteFIFO struct {
	// contains filtered or unexported fields
}

LevelQueueByteFIFO represents a ByteFIFO formed from a LevelQueue

func NewLevelQueueByteFIFO 

func NewLevelQueueByteFIFO(connection, prefix string) (*LevelQueueByteFIFO, error)

NewLevelQueueByteFIFO creates a ByteFIFO formed from a LevelQueue

func (*LevelQueueByteFIFO) Close 

func (fifo *LevelQueueByteFIFO) Close() error

Close this fifo

func (*LevelQueueByteFIFO) Len 

func (fifo *LevelQueueByteFIFO) Len(ctx context.Context) int64

Len returns the length of the fifo

func (*LevelQueueByteFIFO) Pop 

func (fifo *LevelQueueByteFIFO) Pop(ctx context.Context) ([]byte, error)

Pop pops data from the start of the fifo

func (*LevelQueueByteFIFO) PushBack 

func (fifo *LevelQueueByteFIFO) PushBack(ctx context.Context, data []byte) error

PushBack pushes data to the top of the fifo

func (*LevelQueueByteFIFO) PushFunc 

func (fifo *LevelQueueByteFIFO) PushFunc(ctx context.Context, data []byte, fn func() error) error

PushFunc will push data into the fifo

type LevelQueueConfiguration 

type LevelQueueConfiguration struct {
	ByteFIFOQueueConfiguration
	DataDir          string
	ConnectionString string
	QueueName        string
}

LevelQueueConfiguration is the configuration for a LevelQueue

type LevelUniqueQueue 

type LevelUniqueQueue struct {
	*ByteFIFOUniqueQueue
}

LevelUniqueQueue implements a disk library queue

type LevelUniqueQueueByteFIFO 

type LevelUniqueQueueByteFIFO struct {
	// contains filtered or unexported fields
}

LevelUniqueQueueByteFIFO represents a ByteFIFO formed from a LevelUniqueQueue

func NewLevelUniqueQueueByteFIFO 

func NewLevelUniqueQueueByteFIFO(connection, prefix string) (*LevelUniqueQueueByteFIFO, error)

NewLevelUniqueQueueByteFIFO creates a new ByteFIFO formed from a LevelUniqueQueue

func (*LevelUniqueQueueByteFIFO) Close 

func (fifo *LevelUniqueQueueByteFIFO) Close() error

Close this fifo

func (*LevelUniqueQueueByteFIFO) Has 

func (fifo *LevelUniqueQueueByteFIFO) Has(ctx context.Context, data []byte) (bool, error)

Has returns whether the fifo contains this data

func (*LevelUniqueQueueByteFIFO) Len 

func (fifo *LevelUniqueQueueByteFIFO) Len(ctx context.Context) int64

Len returns the length of the fifo

func (*LevelUniqueQueueByteFIFO) Pop 

func (fifo *LevelUniqueQueueByteFIFO) Pop(ctx context.Context) ([]byte, error)

Pop pops data from the start of the fifo

func (*LevelUniqueQueueByteFIFO) PushBack 

func (fifo *LevelUniqueQueueByteFIFO) PushBack(ctx context.Context, data []byte) error

PushBack pushes data to the top of the fifo

func (*LevelUniqueQueueByteFIFO) PushFunc 

func (fifo *LevelUniqueQueueByteFIFO) PushFunc(ctx context.Context, data []byte, fn func() error) error

PushFunc pushes data to the end of the fifo and calls the callback if it is added

type LevelUniqueQueueConfiguration 

type LevelUniqueQueueConfiguration struct {
	ByteFIFOQueueConfiguration
	DataDir          string
	ConnectionString string
	QueueName        string
}

LevelUniqueQueueConfiguration is the configuration for a LevelUniqueQueue

type ManagedPool 

type ManagedPool interface {
	// AddWorkers adds a number of worker as group to the pool with the provided timeout. A CancelFunc is provided to cancel the group
	AddWorkers(number int, timeout time.Duration) context.CancelFunc
	// NumberOfWorkers returns the total number of workers in the pool
	NumberOfWorkers() int
	// MaxNumberOfWorkers returns the maximum number of workers the pool can dynamically grow to
	MaxNumberOfWorkers() int
	// SetMaxNumberOfWorkers sets the maximum number of workers the pool can dynamically grow to
	SetMaxNumberOfWorkers(int)
	// BoostTimeout returns the current timeout for worker groups created during a boost
	BoostTimeout() time.Duration
	// BlockTimeout returns the timeout the internal channel can block for before a boost would occur
	BlockTimeout() time.Duration
	// BoostWorkers sets the number of workers to be created during a boost
	BoostWorkers() int
	// SetPoolSettings sets the user updatable settings for the pool
	SetPoolSettings(maxNumberOfWorkers, boostWorkers int, timeout time.Duration)
	// NumberInQueue returns the total number of items in the pool
	NumberInQueue() int64
	// Done returns a channel that will be closed when the Pool's baseCtx is closed
	Done() <-chan struct{}
}

ManagedPool is a simple interface to get certain details from a worker pool

type ManagedQueue 

type ManagedQueue struct {
	QID           int64
	Type          Type
	Name          string
	Configuration interface{}
	ExemplarType  string
	Managed       interface{}

	PoolWorkers map[int64]*PoolWorkers
	// contains filtered or unexported fields
}

ManagedQueue represents a working queue with a Pool of workers.

Although a ManagedQueue should really represent a Queue this does not necessarily have to be the case. This could be used to describe any queue.WorkerPool.

func (*ManagedQueue) AddWorkers 

func (q *ManagedQueue) AddWorkers(number int, timeout time.Duration) context.CancelFunc

AddWorkers adds workers to the queue if it has registered an add worker function

func (*ManagedQueue) BlockTimeout 

func (q *ManagedQueue) BlockTimeout() time.Duration

BlockTimeout returns the timeout til the next boost

func (*ManagedQueue) BoostTimeout 

func (q *ManagedQueue) BoostTimeout() time.Duration

BoostTimeout returns the timeout of the next boost

func (*ManagedQueue) BoostWorkers 

func (q *ManagedQueue) BoostWorkers() int

BoostWorkers returns the number of workers for a boost

func (*ManagedQueue) CancelWorkers 

func (q *ManagedQueue) CancelWorkers(pid int64)

CancelWorkers cancels pooled workers with pid

func (*ManagedQueue) Flush 

func (q *ManagedQueue) Flush(timeout time.Duration) error

Flush flushes the queue with a timeout

func (*ManagedQueue) Flushable 

func (q *ManagedQueue) Flushable() bool

Flushable returns true if the queue is flushable

func (*ManagedQueue) IsEmpty 

func (q *ManagedQueue) IsEmpty() bool

IsEmpty returns if the queue is empty

func (*ManagedQueue) IsPaused 

func (q *ManagedQueue) IsPaused() bool

IsPaused reveals if the queue is paused

func (*ManagedQueue) MaxNumberOfWorkers 

func (q *ManagedQueue) MaxNumberOfWorkers() int

MaxNumberOfWorkers returns the maximum number of workers for the pool

func (*ManagedQueue) NumberInQueue 

func (q *ManagedQueue) NumberInQueue() int64

NumberInQueue returns the number of items in the queue

func (*ManagedQueue) NumberOfWorkers 

func (q *ManagedQueue) NumberOfWorkers() int

NumberOfWorkers returns the number of workers in the queue

func (*ManagedQueue) Pausable 

func (q *ManagedQueue) Pausable() bool

Pausable returns whether the queue is Pausable

func (*ManagedQueue) Pause 

func (q *ManagedQueue) Pause()

Pause pauses the queue

func (*ManagedQueue) RegisterWorkers 

func (q *ManagedQueue) RegisterWorkers(number int, start time.Time, hasTimeout bool, timeout time.Time, cancel context.CancelFunc, isFlusher bool) int64

RegisterWorkers registers workers to this queue

func (*ManagedQueue) RemoveWorkers 

func (q *ManagedQueue) RemoveWorkers(pid int64)

RemoveWorkers deletes pooled workers with pid

func (*ManagedQueue) Resume 

func (q *ManagedQueue) Resume()

Resume resumes the queue

func (*ManagedQueue) SetPoolSettings 

func (q *ManagedQueue) SetPoolSettings(maxNumberOfWorkers, boostWorkers int, timeout time.Duration)

SetPoolSettings sets the setable boost values

func (*ManagedQueue) Workers 

func (q *ManagedQueue) Workers() []*PoolWorkers

Workers returns the poolworkers

type ManagedQueueList 

type ManagedQueueList []*ManagedQueue

ManagedQueueList implements the sort.Interface

func (ManagedQueueList) Len 

func (l ManagedQueueList) Len() int

func (ManagedQueueList) Less 

func (l ManagedQueueList) Less(i, j int) bool

func (ManagedQueueList) Swap 

func (l ManagedQueueList) Swap(i, j int)

type Manager 

type Manager struct {
	Queues map[int64]*ManagedQueue
	// contains filtered or unexported fields
}

Manager is a queue manager

func GetManager 

func GetManager() *Manager

GetManager returns a Manager and initializes one as singleton if there's none yet

func (*Manager) Add 

func (m *Manager) Add(managed interface{},
	t Type,
	configuration,
	exemplar interface{},
) int64

Add adds a queue to this manager

func (*Manager) FlushAll 

func (m *Manager) FlushAll(baseCtx context.Context, timeout time.Duration) error

FlushAll flushes all the flushable queues attached to this manager

func (*Manager) GetManagedQueue 

func (m *Manager) GetManagedQueue(qid int64) *ManagedQueue

GetManagedQueue by qid

func (*Manager) ManagedQueues 

func (m *Manager) ManagedQueues() []*ManagedQueue

ManagedQueues returns the managed queues

func (*Manager) Remove 

func (m *Manager) Remove(qid int64)

Remove a queue from the Manager

type Mappable 

type Mappable interface {
	MapTo(v interface{}) error
}

Mappable represents an interface that can MapTo another interface

type Named 

type Named interface {
	Name() string
}

Named represents a queue with a name

type NewQueueFunc 

type NewQueueFunc func(handler HandlerFunc, config, exemplar interface{}) (Queue, error)

NewQueueFunc is a function that creates a queue

type Pausable 

type Pausable interface {
	// IsPaused will return if the pool or queue is paused
	IsPaused() bool
	// Pause will pause the pool or queue
	Pause()
	// Resume will resume the pool or queue
	Resume()
	// IsPausedIsResumed will return a bool indicating if the pool or queue is paused and a channel that will be closed when it is resumed
	IsPausedIsResumed() (paused, resumed <-chan struct{})
}

Pausable represents a pool or queue that is Pausable

type PersistableChannelQueue 

type PersistableChannelQueue struct {
	// contains filtered or unexported fields
}

PersistableChannelQueue wraps a channel queue and level queue together The disk level queue will be used to store data at shutdown and terminate - and will be restored on start up.

func (*PersistableChannelQueue) Flush 

func (q *PersistableChannelQueue) Flush(timeout time.Duration) error

Flush flushes the queue and blocks till the queue is empty

func (*PersistableChannelQueue) FlushWithContext 

func (q *PersistableChannelQueue) FlushWithContext(ctx context.Context) error

FlushWithContext flushes the queue and blocks till the queue is empty

func (*PersistableChannelQueue) IsEmpty 

func (q *PersistableChannelQueue) IsEmpty() bool

IsEmpty checks if a queue is empty

func (*PersistableChannelQueue) IsPaused 

func (q *PersistableChannelQueue) IsPaused() bool

IsPaused returns if the pool is paused

func (*PersistableChannelQueue) IsPausedIsResumed 

func (q *PersistableChannelQueue) IsPausedIsResumed() (<-chan struct{}, <-chan struct{})

IsPausedIsResumed returns if the pool is paused and a channel that is closed when it is resumed

func (*PersistableChannelQueue) Name 

func (q *PersistableChannelQueue) Name() string

Name returns the name of this queue

func (*PersistableChannelQueue) Pause 

func (q *PersistableChannelQueue) Pause()

Pause pauses the WorkerPool

func (*PersistableChannelQueue) Push 

func (q *PersistableChannelQueue) Push(data Data) error

Push will push the indexer data to queue

func (*PersistableChannelQueue) PushBack 

func (q *PersistableChannelQueue) PushBack(data Data) error

PushBack will push the indexer data to queue

func (*PersistableChannelQueue) Resume 

func (q *PersistableChannelQueue) Resume()

Resume resumes the WorkerPool

func (*PersistableChannelQueue) Run 

func (q *PersistableChannelQueue) Run(atShutdown, atTerminate func(func()))

Run starts to run the queue

func (*PersistableChannelQueue) Shutdown 

func (q *PersistableChannelQueue) Shutdown()

Shutdown processing this queue

func (*PersistableChannelQueue) Terminate 

func (q *PersistableChannelQueue) Terminate()

Terminate this queue and close the queue

type PersistableChannelQueueConfiguration 

type PersistableChannelQueueConfiguration struct {
	Name         string
	DataDir      string
	BatchLength  int
	QueueLength  int
	Timeout      time.Duration
	MaxAttempts  int
	Workers      int
	MaxWorkers   int
	BlockTimeout time.Duration
	BoostTimeout time.Duration
	BoostWorkers int
}

PersistableChannelQueueConfiguration is the configuration for a PersistableChannelQueue

type PersistableChannelUniqueQueue 

type PersistableChannelUniqueQueue struct {
	// contains filtered or unexported fields
}

PersistableChannelUniqueQueue wraps a channel queue and level queue together

Please note that this Queue does not guarantee that a particular task cannot be processed twice or more at the same time. Uniqueness is only guaranteed whilst the task is waiting in the queue.

func (*PersistableChannelUniqueQueue) Flush 

func (q *PersistableChannelUniqueQueue) Flush(timeout time.Duration) error

Flush flushes the queue

func (*PersistableChannelUniqueQueue) FlushWithContext 

func (q *PersistableChannelUniqueQueue) FlushWithContext(ctx context.Context) error

FlushWithContext flushes the queue

func (*PersistableChannelUniqueQueue) Has 

func (q *PersistableChannelUniqueQueue) Has(data Data) (bool, error)

Has will test if the queue has the data

func (*PersistableChannelUniqueQueue) IsEmpty 

func (q *PersistableChannelUniqueQueue) IsEmpty() bool

IsEmpty checks if a queue is empty

func (*PersistableChannelUniqueQueue) IsPaused 

func (q *PersistableChannelUniqueQueue) IsPaused() bool

IsPaused will return if the pool or queue is paused

func (*PersistableChannelUniqueQueue) IsPausedIsResumed 

func (q *PersistableChannelUniqueQueue) IsPausedIsResumed() (paused, resumed <-chan struct{})

IsPausedIsResumed will return a bool indicating if the pool or queue is paused and a channel that will be closed when it is resumed

func (*PersistableChannelUniqueQueue) Name 

func (q *PersistableChannelUniqueQueue) Name() string

Name returns the name of this queue

func (*PersistableChannelUniqueQueue) Pause 

func (q *PersistableChannelUniqueQueue) Pause()

Pause will pause the pool or queue

func (*PersistableChannelUniqueQueue) Push 

func (q *PersistableChannelUniqueQueue) Push(data Data) error

Push will push the indexer data to queue

func (*PersistableChannelUniqueQueue) PushBack 

func (q *PersistableChannelUniqueQueue) PushBack(data Data) error

PushBack will push the indexer data to queue

func (*PersistableChannelUniqueQueue) PushFunc 

func (q *PersistableChannelUniqueQueue) PushFunc(data Data, fn func() error) error

PushFunc will push the indexer data to queue

func (*PersistableChannelUniqueQueue) Resume 

func (q *PersistableChannelUniqueQueue) Resume()

Resume will resume the pool or queue

func (*PersistableChannelUniqueQueue) Run 

func (q *PersistableChannelUniqueQueue) Run(atShutdown, atTerminate func(func()))

Run starts to run the queue

func (*PersistableChannelUniqueQueue) Shutdown 

func (q *PersistableChannelUniqueQueue) Shutdown()

Shutdown processing this queue

func (*PersistableChannelUniqueQueue) Terminate 

func (q *PersistableChannelUniqueQueue) Terminate()

Terminate this queue and close the queue

type PersistableChannelUniqueQueueConfiguration 

type PersistableChannelUniqueQueueConfiguration struct {
	Name         string
	DataDir      string
	BatchLength  int
	QueueLength  int
	Timeout      time.Duration
	MaxAttempts  int
	Workers      int
	MaxWorkers   int
	BlockTimeout time.Duration
	BoostTimeout time.Duration
	BoostWorkers int
}

PersistableChannelUniqueQueueConfiguration is the configuration for a PersistableChannelUniqueQueue

type PoolWorkers 

type PoolWorkers struct {
	PID        int64
	Workers    int
	Start      time.Time
	Timeout    time.Time
	HasTimeout bool
	Cancel     context.CancelFunc
	IsFlusher  bool
}

PoolWorkers represents a group of workers working on a queue

type PoolWorkersList 

type PoolWorkersList []*PoolWorkers

PoolWorkersList implements the sort.Interface for PoolWorkers

func (PoolWorkersList) Len 

func (l PoolWorkersList) Len() int

func (PoolWorkersList) Less 

func (l PoolWorkersList) Less(i, j int) bool

func (PoolWorkersList) Swap 

func (l PoolWorkersList) Swap(i, j int)

type PushBackable 

type PushBackable interface {
	// PushBack pushes data back to the top of the fifo
	PushBack(Data) error
}

PushBackable queues can be pushed back to

type Queue 

type Queue interface {
	Flushable
	Run(atShutdown, atTerminate func(func()))
	Push(Data) error
}

Queue defines an interface of a queue-like item

Queues will handle their own contents in the Run method

func CreateQueue 

func CreateQueue(name string, handle HandlerFunc, exemplar interface{}) Queue

CreateQueue for name with provided handler and exemplar

func NewChannelQueue 

func NewChannelQueue(handle HandlerFunc, cfg, exemplar interface{}) (Queue, error)

NewChannelQueue creates a memory channel queue

func NewChannelUniqueQueue 

func NewChannelUniqueQueue(handle HandlerFunc, cfg, exemplar interface{}) (Queue, error)

NewChannelUniqueQueue create a memory channel queue

func NewDummyQueue 

func NewDummyQueue(handler HandlerFunc, opts, exemplar interface{}) (Queue, error)

NewDummyQueue creates a new DummyQueue

func NewImmediate 

func NewImmediate(handler HandlerFunc, opts, exemplar interface{}) (Queue, error)

NewImmediate creates a new false queue to execute the function when push

func NewLevelQueue 

func NewLevelQueue(handle HandlerFunc, cfg, exemplar interface{}) (Queue, error)

NewLevelQueue creates a ledis local queue

func NewLevelUniqueQueue 

func NewLevelUniqueQueue(handle HandlerFunc, cfg, exemplar interface{}) (Queue, error)

NewLevelUniqueQueue creates a ledis local queue

Please note that this Queue does not guarantee that a particular task cannot be processed twice or more at the same time. Uniqueness is only guaranteed whilst the task is waiting in the queue.

func NewPersistableChannelQueue 

func NewPersistableChannelQueue(handle HandlerFunc, cfg, exemplar interface{}) (Queue, error)

NewPersistableChannelQueue creates a wrapped batched channel queue with persistable level queue backend when shutting down This differs from a wrapped queue in that the persistent queue is only used to persist at shutdown/terminate

func NewPersistableChannelUniqueQueue 

func NewPersistableChannelUniqueQueue(handle HandlerFunc, cfg, exemplar interface{}) (Queue, error)

NewPersistableChannelUniqueQueue creates a wrapped batched channel queue with persistable level queue backend when shutting down This differs from a wrapped queue in that the persistent queue is only used to persist at shutdown/terminate

func NewQueue 

func NewQueue(queueType Type, handlerFunc HandlerFunc, opts, exemplar interface{}) (Queue, error)

NewQueue takes a queue Type, HandlerFunc, some options and possibly an exemplar and returns a Queue or an error

func NewRedisQueue 

func NewRedisQueue(handle HandlerFunc, cfg, exemplar interface{}) (Queue, error)

NewRedisQueue creates single redis or cluster redis queue

func NewRedisUniqueQueue 

func NewRedisUniqueQueue(handle HandlerFunc, cfg, exemplar interface{}) (Queue, error)

NewRedisUniqueQueue creates single redis or cluster redis queue.

Please note that this Queue does not guarantee that a particular task cannot be processed twice or more at the same time. Uniqueness is only guaranteed whilst the task is waiting in the queue.

func NewWrappedQueue 

func NewWrappedQueue(handle HandlerFunc, cfg, exemplar interface{}) (Queue, error)

NewWrappedQueue will attempt to create a queue of the provided type, but if there is a problem creating this queue it will instead create a WrappedQueue with delayed startup of the queue instead and a channel which will be redirected to the queue

func NewWrappedUniqueQueue 

func NewWrappedUniqueQueue(handle HandlerFunc, cfg, exemplar interface{}) (Queue, error)

NewWrappedUniqueQueue will attempt to create a unique queue of the provided type, but if there is a problem creating this queue it will instead create a WrappedUniqueQueue with delayed startup of the queue instead and a channel which will be redirected to the queue

Please note that this Queue does not guarantee that a particular task cannot be processed twice or more at the same time. Uniqueness is only guaranteed whilst the task is waiting in the queue.

type RedisByteFIFO 

type RedisByteFIFO struct {
	// contains filtered or unexported fields
}

RedisByteFIFO represents a ByteFIFO formed from a redisClient

func NewRedisByteFIFO 

func NewRedisByteFIFO(config RedisByteFIFOConfiguration) (*RedisByteFIFO, error)

NewRedisByteFIFO creates a ByteFIFO formed from a redisClient

func (*RedisByteFIFO) Close 

func (fifo *RedisByteFIFO) Close() error

Close this fifo

func (*RedisByteFIFO) Len 

func (fifo *RedisByteFIFO) Len(ctx context.Context) int64

Len returns the length of the fifo

func (*RedisByteFIFO) Pop 

func (fifo *RedisByteFIFO) Pop(ctx context.Context) ([]byte, error)

Pop pops data from the start of the fifo

func (*RedisByteFIFO) PushBack 

func (fifo *RedisByteFIFO) PushBack(ctx context.Context, data []byte) error

PushBack pushes data to the top of the fifo

func (*RedisByteFIFO) PushFunc 

func (fifo *RedisByteFIFO) PushFunc(ctx context.Context, data []byte, fn func() error) error

PushFunc pushes data to the end of the fifo and calls the callback if it is added

type RedisByteFIFOConfiguration 

type RedisByteFIFOConfiguration struct {
	ConnectionString string
	QueueName        string
}

RedisByteFIFOConfiguration is the configuration for the RedisByteFIFO

type RedisQueue 

type RedisQueue struct {
	*ByteFIFOQueue
}

RedisQueue redis queue

type RedisQueueConfiguration 

type RedisQueueConfiguration struct {
	ByteFIFOQueueConfiguration
	RedisByteFIFOConfiguration
}

RedisQueueConfiguration is the configuration for the redis queue

type RedisUniqueByteFIFO 

type RedisUniqueByteFIFO struct {
	RedisByteFIFO
	// contains filtered or unexported fields
}

RedisUniqueByteFIFO represents a UniqueByteFIFO formed from a redisClient

func NewRedisUniqueByteFIFO 

func NewRedisUniqueByteFIFO(config RedisUniqueByteFIFOConfiguration) (*RedisUniqueByteFIFO, error)

NewRedisUniqueByteFIFO creates a UniqueByteFIFO formed from a redisClient

func (*RedisUniqueByteFIFO) Has 

func (fifo *RedisUniqueByteFIFO) Has(ctx context.Context, data []byte) (bool, error)

Has returns whether the fifo contains this data

func (*RedisUniqueByteFIFO) Pop 

func (fifo *RedisUniqueByteFIFO) Pop(ctx context.Context) ([]byte, error)

Pop pops data from the start of the fifo

func (*RedisUniqueByteFIFO) PushBack 

func (fifo *RedisUniqueByteFIFO) PushBack(ctx context.Context, data []byte) error

PushBack pushes data to the top of the fifo

func (*RedisUniqueByteFIFO) PushFunc 

func (fifo *RedisUniqueByteFIFO) PushFunc(ctx context.Context, data []byte, fn func() error) error

PushFunc pushes data to the end of the fifo and calls the callback if it is added

type RedisUniqueByteFIFOConfiguration 

type RedisUniqueByteFIFOConfiguration struct {
	RedisByteFIFOConfiguration
	SetName string
}

RedisUniqueByteFIFOConfiguration is the configuration for the RedisUniqueByteFIFO

type RedisUniqueQueue 

type RedisUniqueQueue struct {
	*ByteFIFOUniqueQueue
}

RedisUniqueQueue redis queue

type RedisUniqueQueueConfiguration 

type RedisUniqueQueueConfiguration struct {
	ByteFIFOQueueConfiguration
	RedisUniqueByteFIFOConfiguration
}

RedisUniqueQueueConfiguration is the configuration for the redis queue

type Shutdownable 

type Shutdownable interface {
	Shutdown()
	Terminate()
}

Shutdownable represents a queue that can be shutdown

type Type 

type Type string

Type is a type of Queue 

const ChannelQueueType Type = "channel"

ChannelQueueType is the type for channel queue 

const ChannelUniqueQueueType Type = "unique-channel"

ChannelUniqueQueueType is the type for channel queue 

const DummyQueueType Type = "dummy"

DummyQueueType is the type for the dummy queue 

const ImmediateType Type = "immediate"

ImmediateType is the type to execute the function when push 

const LevelQueueType Type = "level"

LevelQueueType is the type for level queue 

const LevelUniqueQueueType Type = "unique-level"

LevelUniqueQueueType is the type for level queue 

const PersistableChannelQueueType Type = "persistable-channel"

PersistableChannelQueueType is the type for persistable queue 

const PersistableChannelUniqueQueueType Type = "unique-persistable-channel"

PersistableChannelUniqueQueueType is the type for persistable queue 

const RedisQueueType Type = "redis"

RedisQueueType is the type for redis queue 

const RedisUniqueQueueType Type = "unique-redis"

RedisUniqueQueueType is the type for redis queue 

const WrappedQueueType Type = "wrapped"

WrappedQueueType is the type for a wrapped delayed starting queue 

const WrappedUniqueQueueType Type = "unique-wrapped"

WrappedUniqueQueueType is the type for a wrapped delayed starting queue

func RegisteredTypes 

func RegisteredTypes() []Type

RegisteredTypes provides the list of requested types of queues

type UniqueByteFIFO 

type UniqueByteFIFO interface {
	ByteFIFO
	// Has returns whether the fifo contains this data
	Has(ctx context.Context, data []byte) (bool, error)
}

UniqueByteFIFO defines a FIFO that Uniques its contents

type UniqueQueue 

type UniqueQueue interface {
	Queue
	PushFunc(Data, func() error) error
	Has(Data) (bool, error)
}

UniqueQueue defines a queue which guarantees only one instance of same data is in the queue. Instances with same identity will be discarded if there is already one in the line.

This queue is particularly useful for preventing duplicated task of same purpose - please note that this does not guarantee that a particular task cannot be processed twice or more at the same time. Uniqueness is only guaranteed whilst the task is waiting in the queue.

Users of this queue should be careful to push only the identifier of the data

func CreateUniqueQueue 

func CreateUniqueQueue(name string, handle HandlerFunc, exemplar interface{}) UniqueQueue

CreateUniqueQueue for name with provided handler and exemplar

type WorkerPool 

type WorkerPool struct {
	// contains filtered or unexported fields
}

WorkerPool represent a dynamically growable worker pool for a provided handler function. They have an internal channel which they use to detect if there is a block and will grow and shrink in response to demand as per configuration.

func NewWorkerPool 

func NewWorkerPool(handle HandlerFunc, config WorkerPoolConfiguration) *WorkerPool

NewWorkerPool creates a new worker pool

func (*WorkerPool) AddWorkers 

func (p *WorkerPool) AddWorkers(number int, timeout time.Duration) context.CancelFunc

AddWorkers adds workers to the pool - this allows the number of workers to go above the limit

func (*WorkerPool) BlockTimeout 

func (p *WorkerPool) BlockTimeout() time.Duration

BlockTimeout returns the timeout til the next boost

func (*WorkerPool) BoostTimeout 

func (p *WorkerPool) BoostTimeout() time.Duration

BoostTimeout returns the timeout of the next boost

func (*WorkerPool) BoostWorkers 

func (p *WorkerPool) BoostWorkers() int

BoostWorkers returns the number of workers for a boost

func (*WorkerPool) CleanUp 

func (p *WorkerPool) CleanUp(ctx context.Context)

CleanUp will drain the remaining contents of the channel This should be called after AddWorkers context is closed

func (*WorkerPool) Done 

func (p *WorkerPool) Done() <-chan struct{}

Done returns when this worker pool's base context has been cancelled

func (*WorkerPool) Flush 

func (p *WorkerPool) Flush(timeout time.Duration) error

Flush flushes the channel with a timeout - the Flush worker will be registered as a flush worker with the manager

func (*WorkerPool) FlushWithContext 

func (p *WorkerPool) FlushWithContext(ctx context.Context) error

FlushWithContext is very similar to CleanUp but it will return as soon as the dataChan is empty NB: The worker will not be registered with the manager.

func (*WorkerPool) HasNoWorkerScaling 

func (p *WorkerPool) HasNoWorkerScaling() bool

HasNoWorkerScaling will return true if the queue has no workers, and has no worker boosting

func (*WorkerPool) IsEmpty 

func (p *WorkerPool) IsEmpty() bool

IsEmpty returns if true if the worker queue is empty

func (*WorkerPool) IsPaused 

func (p *WorkerPool) IsPaused() bool

IsPaused returns if the pool is paused

func (*WorkerPool) IsPausedIsResumed 

func (p *WorkerPool) IsPausedIsResumed() (<-chan struct{}, <-chan struct{})

IsPausedIsResumed returns if the pool is paused and a channel that is closed when it is resumed

func (*WorkerPool) MaxNumberOfWorkers 

func (p *WorkerPool) MaxNumberOfWorkers() int

MaxNumberOfWorkers returns the maximum number of workers automatically added to the pool

func (*WorkerPool) NumberInQueue 

func (p *WorkerPool) NumberInQueue() int64

NumberInQueue returns the number of items in the queue

func (*WorkerPool) NumberOfWorkers 

func (p *WorkerPool) NumberOfWorkers() int

NumberOfWorkers returns the number of current workers in the pool

func (*WorkerPool) Pause 

func (p *WorkerPool) Pause()

Pause pauses the WorkerPool

func (*WorkerPool) Push 

func (p *WorkerPool) Push(data Data)

Push pushes the data to the internal channel

func (*WorkerPool) Resume 

func (p *WorkerPool) Resume()

Resume resumes the WorkerPool

func (*WorkerPool) SetMaxNumberOfWorkers 

func (p *WorkerPool) SetMaxNumberOfWorkers(newMax int)

SetMaxNumberOfWorkers sets the maximum number of workers automatically added to the pool Changing this number will not change the number of current workers but will change the limit for future additions

func (*WorkerPool) SetPoolSettings 

func (p *WorkerPool) SetPoolSettings(maxNumberOfWorkers, boostWorkers int, timeout time.Duration)

SetPoolSettings sets the setable boost values

func (*WorkerPool) Wait 

func (p *WorkerPool) Wait()

Wait for WorkerPool to finish

type WorkerPoolConfiguration 

type WorkerPoolConfiguration struct {
	Name         string
	QueueLength  int
	BatchLength  int
	BlockTimeout time.Duration
	BoostTimeout time.Duration
	BoostWorkers int
	MaxWorkers   int
}

WorkerPoolConfiguration is the basic configuration for a WorkerPool

type WrappedQueue 

type WrappedQueue struct {
	// contains filtered or unexported fields
}

WrappedQueue wraps a delayed starting queue

func (*WrappedQueue) Flush 

func (q *WrappedQueue) Flush(timeout time.Duration) error

Flush flushes the queue and blocks till the queue is empty

func (*WrappedQueue) FlushWithContext 

func (q *WrappedQueue) FlushWithContext(ctx context.Context) error

FlushWithContext implements the final part of Flushable

func (*WrappedQueue) IsEmpty 

func (q *WrappedQueue) IsEmpty() bool

IsEmpty checks whether the queue is empty

func (*WrappedQueue) IsPaused 

func (q *WrappedQueue) IsPaused() bool

IsPaused will return if the pool or queue is paused

func (*WrappedQueue) IsPausedIsResumed 

func (q *WrappedQueue) IsPausedIsResumed() (paused, resumed <-chan struct{})

IsPausedIsResumed will return a bool indicating if the pool or queue is paused and a channel that will be closed when it is resumed

func (*WrappedQueue) Name 

func (q *WrappedQueue) Name() string

Name returns the name of the queue

func (*WrappedQueue) Pause 

func (q *WrappedQueue) Pause()

Pause will pause the pool or queue

func (*WrappedQueue) Push 

func (q *WrappedQueue) Push(data Data) error

Push will push the data to the internal channel checking it against the exemplar

func (*WrappedQueue) Resume 

func (q *WrappedQueue) Resume()

Resume will resume the pool or queue

func (*WrappedQueue) Run 

func (q *WrappedQueue) Run(atShutdown, atTerminate func(func()))

Run starts to run the queue and attempts to create the internal queue

func (*WrappedQueue) Shutdown 

func (q *WrappedQueue) Shutdown()

Shutdown this queue and stop processing

func (*WrappedQueue) Terminate 

func (q *WrappedQueue) Terminate()

Terminate this queue and close the queue

type WrappedQueueConfiguration 

type WrappedQueueConfiguration struct {
	Underlying  Type
	Timeout     time.Duration
	MaxAttempts int
	Config      interface{}
	QueueLength int
	Name        string
}

WrappedQueueConfiguration is the configuration for a WrappedQueue

type WrappedUniqueQueue 

type WrappedUniqueQueue struct {
	*WrappedQueue
	// contains filtered or unexported fields
}

WrappedUniqueQueue wraps a delayed starting unique queue

func (*WrappedUniqueQueue) Has 

func (q *WrappedUniqueQueue) Has(data Data) (bool, error)

Has checks if the data is in the queue

func (*WrappedUniqueQueue) IsEmpty 

func (q *WrappedUniqueQueue) IsEmpty() bool

IsEmpty checks whether the queue is empty

func (*WrappedUniqueQueue) Push 

func (q *WrappedUniqueQueue) Push(data Data) error

Push will push the data to the internal channel checking it against the exemplar

func (*WrappedUniqueQueue) PushFunc 

func (q *WrappedUniqueQueue) PushFunc(data Data, fn func() error) error

PushFunc will push the data to the internal channel checking it against the exemplar

type WrappedUniqueQueueConfiguration 

type WrappedUniqueQueueConfiguration struct {
	Underlying  Type
	Timeout     time.Duration
	MaxAttempts int
	Config      interface{}
	QueueLength int
	Name        string
}

WrappedUniqueQueueConfiguration is the configuration for a WrappedUniqueQueue

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.