nodeselection

package
v1.105.9-rc Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 4, 2024 License: AGPL-3.0 Imports: 22 Imported by: 5

Documentation

Overview

Package uploadselection implements node selection logic for uploads.

Index

Constants

View Source 
const (
	// AutoExcludeSubnet is placement annotation key to turn off subnet
	// restrictions.
	AutoExcludeSubnet = "autoExcludeSubnet"

	// AutoExcludeSubnetOFF is the value of AutoExcludeSubnet to disable subnet
	// restrictions.
	AutoExcludeSubnetOFF = "off"

	// Location is the placement annotation key for meaningful and
	// human-readable descriptions of placements.
	Location = "location"
)

Variables

View Source 
var EeaCountriesWithoutEu = []location.CountryCode{
	location.Iceland,
	location.Liechtenstein,
	location.Norway,
}

EeaCountriesWithoutEu defined the EEA countries.

View Source 
var ErrNotEnoughNodes = errs.Class("not enough nodes")

ErrNotEnoughNodes is when selecting nodes failed with the given parameters.

View Source 
var ErrPlacement = errs.Class("placement")

ErrPlacement is used for placement definition related parsing errors.

View Source 
var (

	// Error represents an uploadselection error.
	Error = errs.Class("uploadselection")
)
View Source 
var EuCountries = []location.CountryCode{
	location.Austria,
	location.Belgium,
	location.Bulgaria,
	location.Croatia,
	location.Cyprus,
	location.Czechia,
	location.Denmark,
	location.Estonia,
	location.Finland,
	location.France,
	location.Germany,
	location.Greece,
	location.Hungary,
	location.Ireland,
	location.Italy,
	location.Lithuania,
	location.Latvia,
	location.Luxembourg,
	location.Malta,
	location.Netherlands,
	location.Poland,
	location.Portugal,
	location.Romania,
	location.Slovenia,
	location.Slovakia,
	location.Spain,
	location.Sweden,
}

EuCountries defines the member countries of European Union.

View Source 
var LastNetAttribute = mustCreateNodeAttribute("last_net")

LastNetAttribute is used for subnet based declumping/selection.

Functions

func GetAnnotation added in v1.86.1 

func GetAnnotation(filter NodeFilter, name string) string

GetAnnotation retrieves annotation from AnnotatedNodeFilter.

Types

type AllowedNodesFilter added in v1.97.1 

type AllowedNodesFilter []storj.NodeID

AllowedNodesFilter is a special filter which enables only the selected nodes.

func AllowedNodesFromFile added in v1.97.1 

func AllowedNodesFromFile(file string) (AllowedNodesFilter, error)

AllowedNodesFromFile loads a list of allowed NodeIDs from a text file. One ID per line.

func (AllowedNodesFilter) Match added in v1.97.1 

func (n AllowedNodesFilter) Match(node *SelectedNode) bool

Match implements NodeFilter.

type AnnotatedNodeFilter added in v1.86.1 

type AnnotatedNodeFilter struct {
	Filter      NodeFilter
	Annotations []Annotation
}

AnnotatedNodeFilter is just a NodeFilter with additional annotations.

func (AnnotatedNodeFilter) GetAnnotation added in v1.87.1 

func (a AnnotatedNodeFilter) GetAnnotation(name string) string

GetAnnotation implements NodeFilterWithAnnotation.

func (AnnotatedNodeFilter) Match added in v1.88.2 

func (a AnnotatedNodeFilter) Match(node *SelectedNode) bool

Match implements NodeFilter.

func (AnnotatedNodeFilter) String added in v1.89.2 

func (a AnnotatedNodeFilter) String() string

type Annotation added in v1.87.1 

type Annotation struct {
	Key   string
	Value string
}

Annotation can be used as node filters in 'XX && annotation('...')' like struct.

func (Annotation) GetAnnotation added in v1.87.1 

func (a Annotation) GetAnnotation(name string) string

GetAnnotation implements NodeFilterWithAnnotation.

func (Annotation) Match added in v1.88.2 

func (a Annotation) Match(node *SelectedNode) bool

Match implements NodeFilter.

func (Annotation) String added in v1.89.2 

func (a Annotation) String() string

type AnyFilter added in v1.86.1 

type AnyFilter struct{}

AnyFilter matches all the nodes.

func (AnyFilter) Match added in v1.88.2 

func (a AnyFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type AttributeFilter added in v1.100.2 

type AttributeFilter struct {
	// contains filtered or unexported fields
}

AttributeFilter selects nodes based on dynamic node attributes (eg. vetted=true or tag:owner=...).

func NewAttributeFilter added in v1.100.2 

func NewAttributeFilter(attr string, value interface{}) (*AttributeFilter, error)

NewAttributeFilter creates new attribute filter. Value should be string or stringNotMatch.

func (*AttributeFilter) Match added in v1.100.2 

func (a *AttributeFilter) Match(node *SelectedNode) bool

Match implements NodeFilter.

type ConfigurablePlacementRule added in v1.95.1 

type ConfigurablePlacementRule struct {
	PlacementRules string
}

ConfigurablePlacementRule is a string configuration includes all placement rules in the form of id1:def1,id2:def2...

func (ConfigurablePlacementRule) Parse added in v1.95.1 

func (c ConfigurablePlacementRule) Parse(defaultPlacement func() (Placement, error), environment *PlacementConfigEnvironment) (PlacementDefinitions, error)

Parse creates the PlacementDefinitions from the string rules. defaultPlacement is used to create the placement if no placement has been set.

func (*ConfigurablePlacementRule) Set added in v1.95.1 

func (c *ConfigurablePlacementRule) Set(s string) error

Set implements pflag.Value.

func (*ConfigurablePlacementRule) String added in v1.95.1 

func (c *ConfigurablePlacementRule) String() string

String implements pflag.Value.

func (*ConfigurablePlacementRule) Type added in v1.95.1 

func (c *ConfigurablePlacementRule) Type() string

Type implements pflag.Value.

type CountryFilter added in v1.84.1 

type CountryFilter struct {
	// contains filtered or unexported fields
}

CountryFilter can select nodes based on the condition of the country code.

func NewCountryFilter added in v1.84.1 

func NewCountryFilter(permit location.Set) *CountryFilter

NewCountryFilter creates a new CountryFilter.

func NewCountryFilterFromString added in v1.89.2 

func NewCountryFilterFromString(countries []string) (*CountryFilter, error)

NewCountryFilterFromString parses country definitions like 'hu','!hu','*','none' and creates a CountryFilter.

func (*CountryFilter) Match added in v1.88.2 

func (p *CountryFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

func (*CountryFilter) String added in v1.89.2 

func (p *CountryFilter) String() string

type ExcludeAllFilter added in v1.84.1 

type ExcludeAllFilter struct{}

ExcludeAllFilter will never select any node.

func (ExcludeAllFilter) Match added in v1.88.2 

func (ExcludeAllFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type ExcludeFilter added in v1.86.1 

type ExcludeFilter struct {
	// contains filtered or unexported fields
}

ExcludeFilter excludes only the matched nodes.

func NewExcludeFilter added in v1.86.1 

func NewExcludeFilter(filter NodeFilter) ExcludeFilter

NewExcludeFilter creates filter, nodes matching the given filter will be excluded.

func (ExcludeFilter) Match added in v1.88.2 

func (e ExcludeFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

func (ExcludeFilter) String added in v1.89.2 

func (e ExcludeFilter) String() string

type ExcludedIDs added in v1.84.1 

type ExcludedIDs []storj.NodeID

ExcludedIDs can blacklist NodeIDs.

func (ExcludedIDs) Match added in v1.88.2 

func (e ExcludedIDs) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type ExcludedNetworks added in v1.84.1 

type ExcludedNetworks []string

ExcludedNetworks will exclude nodes with specified networks.

func (ExcludedNetworks) Match added in v1.88.2 

func (e ExcludedNetworks) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type ExcludedNodeNetworks added in v1.86.1 

type ExcludedNodeNetworks []*SelectedNode

ExcludedNodeNetworks exclude nodes which has same net as the one of the specified.

func (ExcludedNodeNetworks) Match added in v1.88.2 

func (e ExcludedNodeNetworks) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type Invariant added in v1.96.2 

type Invariant func(pieces metabase.Pieces, nodes []SelectedNode) intset.Set

Invariant checks the current placement, and identifies the pieces which should be moved. Used by repair jobs.

func AllGood added in v1.96.2 

func AllGood() Invariant

AllGood is an invariant, which accepts all piece sets as good.

func ClumpingByAnyTag added in v1.96.2 

func ClumpingByAnyTag(key string, maxAllowed int) Invariant

ClumpingByAnyTag tries to limit the number of nodes with the same tag value.

func ClumpingByAttribute added in v1.96.2 

func ClumpingByAttribute(attr NodeAttribute, maxAllowed int) Invariant

ClumpingByAttribute allows only one selected piece by attribute groups.

func InvariantFromString added in v1.100.2 

func InvariantFromString(expr string) (Invariant, error)

InvariantFromString parses complex invariants (~declumping rules) from config lines.

type NodeAttribute added in v1.95.1 

type NodeAttribute func(SelectedNode) string

NodeAttribute returns a string (like last_net or tag value) for each SelectedNode.

func AnyNodeTagAttribute added in v1.96.2 

func AnyNodeTagAttribute(tagName string) NodeAttribute

AnyNodeTagAttribute selects a tag value from node, accepts any signer.

func CreateNodeAttribute added in v1.95.1 

func CreateNodeAttribute(attr string) (NodeAttribute, error)

CreateNodeAttribute creates the NodeAttribute selected based on a string definition.

func NodeTagAttribute added in v1.96.2 

func NodeTagAttribute(signer storj.NodeID, tagName string) NodeAttribute

NodeTagAttribute selects a tag value from node.

type NodeFilter added in v1.84.1 

type NodeFilter interface {
	Match(node *SelectedNode) bool
}

NodeFilter can decide if a Node should be part of the selection or not.

func FilterFromString added in v1.100.2 

func FilterFromString(expr string) (NodeFilter, error)

FilterFromString parses complex node filter expressions from config lines.

type NodeFilterFunc added in v1.84.1 

type NodeFilterFunc func(node *SelectedNode) bool

NodeFilterFunc is helper to use func as NodeFilter.

func (NodeFilterFunc) Match added in v1.88.2 

func (n NodeFilterFunc) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type NodeFilterWithAnnotation added in v1.87.1 

type NodeFilterWithAnnotation interface {
	NodeFilter
	GetAnnotation(name string) string
}

NodeFilterWithAnnotation is a NodeFilter with additional annotations.

func WithAnnotation added in v1.86.1 

func WithAnnotation(filter NodeFilter, name string, value string) NodeFilterWithAnnotation

WithAnnotation adds annotations to a NodeFilter.

type NodeFilters added in v1.84.1 

type NodeFilters []NodeFilter

NodeFilters is a collection of multiple node filters (all should vote with true).

func (NodeFilters) GetAnnotation added in v1.87.1 

func (n NodeFilters) GetAnnotation(name string) string

GetAnnotation implements NodeFilterWithAnnotation.

func (NodeFilters) Match added in v1.88.2 

func (n NodeFilters) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

func (NodeFilters) String added in v1.89.2 

func (n NodeFilters) String() string

func (NodeFilters) WithCountryFilter added in v1.84.1 

func (n NodeFilters) WithCountryFilter(permit location.Set) NodeFilters

WithCountryFilter is a helper to create a new filter with additional CountryFilter.

func (NodeFilters) WithExcludedIDs added in v1.84.1 

func (n NodeFilters) WithExcludedIDs(ds []storj.NodeID) NodeFilters

WithExcludedIDs is a helper to create a new filter with additional WithExcludedIDs.

type NodeSelector added in v1.95.1 

type NodeSelector func(requester storj.NodeID, n int, excluded []storj.NodeID, alreadySelected []*SelectedNode) ([]*SelectedNode, error)

NodeSelector pick random nodes based on a specific algorithm. Nodes from excluded should never be used. Same is true for alreadySelected, but it may also trigger other restrictions (for example, when a last_net is already selected, all the nodes from the same net should be excluded as well.

type NodeSelectorInit added in v1.95.1 

type NodeSelectorInit func([]*SelectedNode, NodeFilter) NodeSelector

NodeSelectorInit initializes a stateful NodeSelector when node cache is refreshed.

func AttributeGroupSelector added in v1.96.2 

func AttributeGroupSelector(attribute NodeAttribute) NodeSelectorInit

AttributeGroupSelector first selects a group with equal chance (like last_net) and choose node from the group randomly.

func BalancedGroupBasedSelector added in v1.97.2 

func BalancedGroupBasedSelector(attribute NodeAttribute) NodeSelectorInit

BalancedGroupBasedSelector first selects a group with equal chance (like last_net) and choose one single node randomly. . One group can be tried multiple times, and if the node is already selected, it will be ignored.

func BestOfN added in v1.106.1 

func BestOfN(tracker UploadSuccessTracker, ratio float64, delegate NodeSelectorInit) NodeSelectorInit

BestOfN selects more nodes than the required one, and choose the fastest from those.

func ChoiceOfN added in v1.106.1 

func ChoiceOfN(tracker UploadSuccessTracker, n int64, delegate NodeSelectorInit) NodeSelectorInit

ChoiceOfN will perform the selection for n*x nodes and choose the best node from groups of n size. n is an int64 type due to a mito scripting shortcoming but really an int16 should be fine. NOTE: it may break other pre-conditions, like the results of the balanced selector...

func ChoiceOfTwo added in v1.106.1 

func ChoiceOfTwo(tracker UploadSuccessTracker, delegate NodeSelectorInit) NodeSelectorInit

ChoiceOfTwo will repeat the selection and choose the better node pair-wise. NOTE: it may break other pre-conditions, like the results of the balanced selector...

func FilterBest added in v1.106.1 

func FilterBest(tracker UploadSuccessTracker, selection string, uplink string, delegate NodeSelectorInit) NodeSelectorInit

FilterBest is a selector, which keeps only the best nodes (based on percentage, or fixed number of nodes). this selector will permanently ban the worst nodes for the period of nodeselection cache refresh.

func FilterSelector added in v1.97.1 

func FilterSelector(loadTimeFilter NodeFilter, init NodeSelectorInit) NodeSelectorInit

FilterSelector is a specific selector, which can filter out nodes from the upload selection. Note: this is different from the generic filter attribute of the NodeSelectorInit, as that is applied to all node selection (upload/download/repair).

func FilteredSelector added in v1.96.2 

func FilteredSelector(preFilter NodeFilter, init NodeSelectorInit) NodeSelectorInit

FilteredSelector uses another selector on the filtered list of nodes.

func RandomSelector added in v1.96.2 

func RandomSelector() NodeSelectorInit

RandomSelector selects any nodes with equal chance.

func SelectorFromString added in v1.100.2 

func SelectorFromString(expr string, environment *PlacementConfigEnvironment) (NodeSelectorInit, error)

SelectorFromString parses complex node selection rules from config lines.

func UnvettedSelector added in v1.96.2 

func UnvettedSelector(newNodeFraction float64, init NodeSelectorInit) NodeSelectorInit

UnvettedSelector selects new nodes first based on newNodeFraction, and selects old nodes for the remaining.

type NodeTag added in v1.84.1 

type NodeTag struct {
	NodeID   storj.NodeID
	SignedAt time.Time
	Signer   storj.NodeID
	Name     string
	Value    []byte
}

NodeTag is a tag associated with a node (approved by signer).

type NodeTags added in v1.84.1 

type NodeTags []NodeTag

NodeTags is a collection of multiple NodeTag.

func (NodeTags) FindBySignerAndName added in v1.84.1 

func (n NodeTags) FindBySignerAndName(signer storj.NodeID, name string) (NodeTag, error)

FindBySignerAndName selects first tag with same name / NodeID.

type NoopTracker added in v1.105.3 

type NoopTracker struct {
}

NoopTracker doesn't tracker uploads at all. Always returns with zero.

func (NoopTracker) Get added in v1.105.3 

func (n NoopTracker) Get(uplink storj.NodeID) func(node storj.NodeID) float64

Get implements UploadSuccessTracker.

type OrFilter added in v1.91.2 

type OrFilter []NodeFilter

OrFilter will include the node, if at lest one of the filters are matched.

func (OrFilter) Match added in v1.91.2 

func (n OrFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

func (OrFilter) String added in v1.91.2 

func (n OrFilter) String() string

type Placement added in v1.95.1 

type Placement struct {
	// the unique ID of the placement
	ID storj.PlacementConstraint
	// meaningful identifier/label for Humans. Will be used on UI.
	Name string
	// binding condition for filtering out nodes
	NodeFilter NodeFilter
	// selector is the method how the nodes are selected from the full node space (eg. pick a subnet first, and pick a node from the subnet)
	Selector NodeSelectorInit
	// checked by repair job, applied to the full selection. Out of placement items will be replaced by new, selected by the Selector.
	Invariant Invariant
}

Placement defined all the custom behavior metadata of a specific placement group.

func (Placement) GetAnnotation added in v1.95.1 

func (p Placement) GetAnnotation(name string) string

GetAnnotation implements NodeFilterWithAnnotation. Deprecated: use Name instead.

func (Placement) Match added in v1.95.1 

func (p Placement) Match(node *SelectedNode) bool

Match implements NodeFilter.

type PlacementConfigEnvironment added in v1.105.3 

type PlacementConfigEnvironment struct {
	// contains filtered or unexported fields
}

PlacementConfigEnvironment includes all generic functions and variables, which can be used in the configuration.

func NewPlacementConfigEnvironment added in v1.105.3 

func NewPlacementConfigEnvironment(tracker UploadSuccessTracker) *PlacementConfigEnvironment

NewPlacementConfigEnvironment creates PlacementConfigEnvironment.

type PlacementDefinitions added in v1.95.1 

type PlacementDefinitions map[storj.PlacementConstraint]Placement

PlacementDefinitions can include the placement definitions for each known identifier.

func LoadConfig added in v1.96.2 

func LoadConfig(configFile string, environment *PlacementConfigEnvironment) (PlacementDefinitions, error)

LoadConfig loads the placement yaml file and creates the Placement definitions.

func NewPlacementDefinitions added in v1.95.1 

func NewPlacementDefinitions(placements ...Placement) PlacementDefinitions

NewPlacementDefinitions creates a PlacementDefinition with a default placement.

func TestPlacementDefinitions added in v1.96.2 

func TestPlacementDefinitions() PlacementDefinitions

TestPlacementDefinitions creates placements for testing. Only 0 placement is defined with subnetfiltering.

func TestPlacementDefinitionsWithFraction added in v1.96.2 

func TestPlacementDefinitionsWithFraction(newNodeFraction float64) PlacementDefinitions

TestPlacementDefinitionsWithFraction creates placements for testing. Similar to TestPlacementDefinitions, but also selects newNodes based on fraction.

func (PlacementDefinitions) AddLegacyStaticRules added in v1.95.1 

func (d PlacementDefinitions) AddLegacyStaticRules()

AddLegacyStaticRules initializes all the placement rules defined earlier in static golang code.

func (PlacementDefinitions) AddPlacement added in v1.96.2 

func (d PlacementDefinitions) AddPlacement(id storj.PlacementConstraint, placement Placement)

AddPlacement registers a new placement.

func (PlacementDefinitions) AddPlacementFromString added in v1.95.1 

func (d PlacementDefinitions) AddPlacementFromString(definitions string) error

AddPlacementFromString parses placement definition form string representations from id:definition;id:definition;... Deprecated: we will switch to the YAML based configuration.

func (PlacementDefinitions) AddPlacementRule added in v1.95.1 

func (d PlacementDefinitions) AddPlacementRule(id storj.PlacementConstraint, filter NodeFilter)

AddPlacementRule registers a new placement.

func (PlacementDefinitions) CreateFilters added in v1.95.1 

func (d PlacementDefinitions) CreateFilters(constraint storj.PlacementConstraint) (filter NodeFilter)

CreateFilters implements PlacementCondition.

func (PlacementDefinitions) SupportedPlacements added in v1.95.1 

func (d PlacementDefinitions) SupportedPlacements() (res []storj.PlacementConstraint)

SupportedPlacements returns all the IDs, which have associated placement rules.

type PlacementRules added in v1.95.1 

type PlacementRules func(constraint storj.PlacementConstraint) (filter NodeFilter)

PlacementRules can crate filter based on the placement identifier.

type RandomOrder added in v1.86.1 

type RandomOrder struct {
	// contains filtered or unexported fields
}

RandomOrder as an iterator of a pseudo-random permutation set.

func NewRandomOrder added in v1.86.1 

func NewRandomOrder(n int) RandomOrder

NewRandomOrder creates new iterator, returns number between [0,n) in pseudo-random order.

func (*RandomOrder) At added in v1.86.1 

func (r *RandomOrder) At() uint64

At returns the current number in the permutations.

func (*RandomOrder) Finished added in v1.104.4 

func (r *RandomOrder) Finished() bool

Finished returns true, if there is no more permutation.

func (*RandomOrder) Next added in v1.86.1 

func (r *RandomOrder) Next() bool

Next generates the next number.

func (*RandomOrder) Reset added in v1.104.4 

func (r *RandomOrder) Reset()

Reset makes it possible to reuse the RandomOrder: the full pseudo-random permutations can be read again.

type SelectedNode added in v1.84.1 

type SelectedNode struct {
	ID          storj.NodeID
	Address     *pb.NodeAddress
	Email       string
	Wallet      string
	LastNet     string
	LastIPPort  string
	CountryCode location.CountryCode
	Exiting     bool
	Suspended   bool
	Online      bool
	Vetted      bool
	Tags        NodeTags
}

SelectedNode is used as a result for creating orders limits.

func (*SelectedNode) Clone added in v1.84.1 

func (node *SelectedNode) Clone() *SelectedNode

Clone returns a deep clone of the selected node.

type State 

type State map[storj.PlacementConstraint]NodeSelector

State includes a stateful selector (indexed nodes) for each placement.

func NewState 

func NewState(nodes []*SelectedNode, placements PlacementDefinitions) State

NewState initializes the State for each placement.

func (State) Select 

func (s State) Select(requester storj.NodeID, p storj.PlacementConstraint, count int, excluded []storj.NodeID, alreadySelected []*SelectedNode) ([]*SelectedNode, error)

Select picks the required nodes given a specific placement.

type TagFilter added in v1.84.1 

type TagFilter struct {
	// contains filtered or unexported fields
}

TagFilter matches nodes with specific tags.

func NewTagFilter added in v1.84.1 

func NewTagFilter(id storj.NodeID, name string, value []byte, match ValueMatch) TagFilter

NewTagFilter creates a new tag filter.

func (TagFilter) Match added in v1.88.2 

func (t TagFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

func (TagFilter) String added in v1.89.2 

func (t TagFilter) String() string

type UploadSuccessTracker added in v1.105.3 

type UploadSuccessTracker interface {
	Get(uplink storj.NodeID) func(node storj.NodeID) float64
}

UploadSuccessTracker can give hints about the frequency of the long-tail cancellation per node.

type ValueMatch added in v1.89.2 

type ValueMatch func(a []byte, b []byte) bool

ValueMatch defines how to compare tag value with the defined one.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.