rebalancers

Documentation

Index

• type FrequencyRebalancer
  • func NewFrequencyRebalancer(brokers []admin.BrokerInfo, picker pickers.Picker, ...) *FrequencyRebalancer
  • func (f *FrequencyRebalancer) Rebalance(topic string, curr []admin.PartitionAssignment, brokersToRemove []int) ([]admin.PartitionAssignment, error)
• type Rebalancer

Types

type FrequencyRebalancer

type FrequencyRebalancer struct {
	// contains filtered or unexported fields
}

FrequencyRebalancer is a Rebalancer that rebalances to achieve in-topic balance among all available brokers. The algorithm used is:

for each replica position index:
  while true:
    get counts for each broker in that position
    partition the set of brokers into two sets, a "lower" one and an "upper one", based on
      on the sorted frequency counts (with brokers to be removed treated as the highest
      frequencies)
    for each lower broker, upper broker combination:
      try to replace the upper broker with the lower one
      if replacement made, continue to next while loop iteration
    if no replacement made, break out of while loop, continue to next partition index

Replacements are made if:

1. The replacement improves the broker balance for the index OR the replacement improves the broker balance for the topic as a whole AND
2. The replacement is consistent with the placement strategy for the topic (e.g., balanced leaders, in-rack, etc.)

The picker passed in to the rebalancer is used to sort the partitions for each broker (if it appears more than once for the current index) and also to break ties when sorting and partitioning the brokers.

func NewFrequencyRebalancer

func NewFrequencyRebalancer(
	brokers []admin.BrokerInfo,
	picker pickers.Picker,
	placementConfig config.TopicPlacementConfig,
) *FrequencyRebalancer

NewFrequencyRebalancer creates a new FrequencyRebalancer instance.

func (*FrequencyRebalancer) Rebalance

func (f *FrequencyRebalancer) Rebalance(
	topic string,
	curr []admin.PartitionAssignment,
	brokersToRemove []int,
) ([]admin.PartitionAssignment, error)

Rebalance rebalances the argument partition assignments according to the algorithm described earlier.

type Rebalancer

type Rebalancer interface {
	Rebalance(
		topic string,
		currAssignments []admin.PartitionAssignment,
		brokersToRemove []int,
	) ([]admin.PartitionAssignment, error)
}

Rebalancer is an interface for structs that figure out how to reassign replicas in existing topic partitions in order to ensure that all brokers are evenly represented. It also supports removing brokers (e.g., if they are to be removed from the cluster).