partitioner

Documentation

Index

• type BasePartitioner
  • func (b *BasePartitioner) Close()
  • func (b *BasePartitioner) Rebalance(n *cluster.Notification)
• type Config
• type DefaultPartitioner
  • func (d *DefaultPartitioner) GetBuffer(msg *sarama.ConsumerMessage) (*buffer.Buffer, error)
  • func (d *DefaultPartitioner) GetKey(f *buffer.Flush) string
• type IsoDateFieldPartitioner
  • func (t *IsoDateFieldPartitioner) GetBuffer(msg *sarama.ConsumerMessage) (*buffer.Buffer, error)
  • func (t *IsoDateFieldPartitioner) GetKey(f *buffer.Flush) string
• type Partitioner
  • func New(c *Config) Partitioner
• type TimeFieldPartitioner
  • func (t *TimeFieldPartitioner) GetBuffer(msg *sarama.ConsumerMessage) (*buffer.Buffer, error)
  • func (t *TimeFieldPartitioner) GetKey(f *buffer.Flush) string

Types

type BasePartitioner

type BasePartitioner struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

BasePartitioner serves as a base struct for every Partitioner instance.

func (*BasePartitioner) Close

func (b *BasePartitioner) Close()

Close closes all the current subscriptions buffers.

func (*BasePartitioner) Rebalance

func (b *BasePartitioner) Rebalance(n *cluster.Notification)

Rebalance is called when the consumer group rebalances the topics and partitions among the cluster. This method holds the lock until it finishes looping through all the consumers.

type Config

type Config struct {
	Type                string // Type of the selected partitioner.
	BaseFolder          string
	TopicNamePrefix     string
	FieldKeyName        string
	DefaultBufferConfig *buffer.Config
}

Config for partitioner.

type DefaultPartitioner

type DefaultPartitioner struct {
	BasePartitioner
}

DefaultPartitioner partitions the data by topic and partition.

func (*DefaultPartitioner) GetBuffer

func (d *DefaultPartitioner) GetBuffer(msg *sarama.ConsumerMessage) (*buffer.Buffer, error)

GetBuffer returns a buffer that can be used to write the message to.

func (*DefaultPartitioner) GetKey

func (d *DefaultPartitioner) GetKey(f *buffer.Flush) string

GetKey returns the file path used after a file is flushed.

type IsoDateFieldPartitioner

type IsoDateFieldPartitioner struct {
	BasePartitioner
	// contains filtered or unexported fields
}

IsoDateFieldPartitioner partitions the data in hourly buckets based on a given timestamp. The field must be a unix timestamp in milliseconds.

func (*IsoDateFieldPartitioner) GetBuffer

func (t *IsoDateFieldPartitioner) GetBuffer(msg *sarama.ConsumerMessage) (*buffer.Buffer, error)

GetBuffer returns a buffer that can be used to write the message to.

func (*IsoDateFieldPartitioner) GetKey

func (t *IsoDateFieldPartitioner) GetKey(f *buffer.Flush) string

GetKey returns the file path used after a file is flushed.

type Partitioner

type Partitioner interface {
	// GetBuffer returns a buffer that can be used to write the message to.
	// The buffer is decided based on the Partitioner class used.
	GetBuffer(msg *sarama.ConsumerMessage) (*buffer.Buffer, error)

	// GetKey returns the file path used after a file is flushed.
	GetKey(f *buffer.Flush) string

	// Rebalance is called when the consumer group rebalances the topics and
	// partitions among the cluster. This method holds the lock until it finishes looping
	// through all the consumers.
	Rebalance(n *cluster.Notification)

	// Close takes care of closing all the buffers.
	Close()
}

Partitioner.

func New

func New(c *Config) Partitioner

NewPartitioner creates a new object that satisfies the Partitioner interface. Uses a Worker instance for additional configuration fields.

type TimeFieldPartitioner

type TimeFieldPartitioner struct {
	BasePartitioner
	// contains filtered or unexported fields
}

TimeFieldPartitioner partitions the data in hourly buckets based on a given timestamp. The field must be a unix timestamp in milliseconds.

func (*TimeFieldPartitioner) GetBuffer

func (t *TimeFieldPartitioner) GetBuffer(msg *sarama.ConsumerMessage) (*buffer.Buffer, error)

GetBuffer returns a buffer that can be used to write the message to.

func (*TimeFieldPartitioner) GetKey

func (t *TimeFieldPartitioner) GetKey(f *buffer.Flush) string

GetKey returns the file path used after a file is flushed.