nodeselection

package
v1.116.1-rc Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 25, 2024 License: AGPL-3.0 Imports: 23 Imported by: 5

Documentation

Overview

Package uploadselection implements node selection logic for uploads.

Index

Constants

View Source 
const (
	// AutoExcludeSubnet is placement annotation key to turn off subnet
	// restrictions.
	AutoExcludeSubnet = "autoExcludeSubnet"

	// AutoExcludeSubnetOFF is the value of AutoExcludeSubnet to disable subnet
	// restrictions.
	AutoExcludeSubnetOFF = "off"

	// Location is the placement annotation key for meaningful and
	// human-readable descriptions of placements.
	Location = "location"
)

Variables

View Source 
var EeaCountriesWithoutEu = []location.CountryCode{
	location.Iceland,
	location.Liechtenstein,
	location.Norway,
}

EeaCountriesWithoutEu defined the EEA countries.

View Source 
var ErrNotEnoughNodes = errs.Class("not enough nodes")

ErrNotEnoughNodes is when selecting nodes failed with the given parameters.

View Source 
var ErrPlacement = errs.Class("placement")

ErrPlacement is used for placement definition related parsing errors.

View Source 
var (

	// Error represents an uploadselection error.
	Error = errs.Class("uploadselection")
)
View Source 
var EuCountries = []location.CountryCode{
	location.Austria,
	location.Belgium,
	location.Bulgaria,
	location.Croatia,
	location.Cyprus,
	location.Czechia,
	location.Denmark,
	location.Estonia,
	location.Finland,
	location.France,
	location.Germany,
	location.Greece,
	location.Hungary,
	location.Ireland,
	location.Italy,
	location.Lithuania,
	location.Latvia,
	location.Luxembourg,
	location.Malta,
	location.TheNetherlands,
	location.Poland,
	location.Portugal,
	location.Romania,
	location.Slovenia,
	location.Slovakia,
	location.Spain,
	location.Sweden,
}

EuCountries defines the member countries of European Union.

View Source 
var LastNetAttribute = mustCreateNodeAttribute("last_net")

LastNetAttribute is used for subnet based declumping/selection.

Functions

func EqualSelector added in v1.107.1 

func EqualSelector(nodeAttribute NodeAttribute, attribute string) func(SelectedNode) bool

EqualSelector returns a function that compares the node attribute with the given attribute.

func GetAnnotation added in v1.86.1 

func GetAnnotation(filter NodeFilter, name string) string

GetAnnotation retrieves annotation from AnnotatedNodeFilter.

func RoundWithProbability added in v1.108.1 

func RoundWithProbability(r float64) int

RoundWithProbability is like math.Round, but instead of rounding 2.6 to 3 all the time, it will round up to 3 with 60% chance, and to 2 with 40% chance.

Types

type AllowedNodesFilter added in v1.97.1 

type AllowedNodesFilter []storj.NodeID

AllowedNodesFilter is a special filter which enables only the selected nodes.

func AllowedNodesFromFile added in v1.97.1 

func AllowedNodesFromFile(file string) (AllowedNodesFilter, error)

AllowedNodesFromFile loads a list of allowed NodeIDs from a text file. One ID per line.

func (AllowedNodesFilter) Match added in v1.97.1 

func (n AllowedNodesFilter) Match(node *SelectedNode) bool

Match implements NodeFilter.

type AnnotatedNodeFilter added in v1.86.1 

type AnnotatedNodeFilter struct {
	Filter      NodeFilter
	Annotations []Annotation
}

AnnotatedNodeFilter is just a NodeFilter with additional annotations.

func (AnnotatedNodeFilter) GetAnnotation added in v1.87.1 

func (a AnnotatedNodeFilter) GetAnnotation(name string) string

GetAnnotation implements NodeFilterWithAnnotation.

func (AnnotatedNodeFilter) Match added in v1.88.2 

func (a AnnotatedNodeFilter) Match(node *SelectedNode) bool

Match implements NodeFilter.

func (AnnotatedNodeFilter) String added in v1.89.2 

func (a AnnotatedNodeFilter) String() string

type Annotation added in v1.87.1 

type Annotation struct {
	Key   string
	Value string
}

Annotation can be used as node filters in 'XX && annotation('...')' like struct.

func (Annotation) GetAnnotation added in v1.87.1 

func (a Annotation) GetAnnotation(name string) string

GetAnnotation implements NodeFilterWithAnnotation.

func (Annotation) Match added in v1.88.2 

func (a Annotation) Match(node *SelectedNode) bool

Match implements NodeFilter.

func (Annotation) String added in v1.89.2 

func (a Annotation) String() string

type AnyFilter added in v1.86.1 

type AnyFilter struct{}

AnyFilter matches all the nodes.

func (AnyFilter) Match added in v1.88.2 

func (a AnyFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type AttributeFilter added in v1.100.2 

type AttributeFilter struct {
	// contains filtered or unexported fields
}

AttributeFilter selects nodes based on dynamic node attributes (eg. vetted=true or tag:owner=...).

func NewAttributeFilter added in v1.100.2 

func NewAttributeFilter(attr string, args ...any) (*AttributeFilter, error)

NewAttributeFilter creates new attribute filter. NewAttributeFilter can be called in two ways, either like `NewAttributeFilter("attr", "value")`, or `NewAttributeFilter("attr", "<", 3). The first style, 2 arguments, is a string equality test. The second style, 3 arguments, is a more generic form of the first. String equality can be specified, but also inequalities as well. If value is stringNotMatch, then the test is inverted.

func (*AttributeFilter) Match added in v1.100.2 

func (a *AttributeFilter) Match(node *SelectedNode) bool

Match implements NodeFilter.

type ConfigurablePlacementRule added in v1.95.1 

type ConfigurablePlacementRule struct {
	PlacementRules string
}

ConfigurablePlacementRule is a string configuration includes all placement rules in the form of id1:def1,id2:def2...

func (ConfigurablePlacementRule) Parse added in v1.95.1 

func (c ConfigurablePlacementRule) Parse(defaultPlacement func() (Placement, error), environment *PlacementConfigEnvironment) (PlacementDefinitions, error)

Parse creates the PlacementDefinitions from the string rules. defaultPlacement is used to create the placement if no placement has been set.

func (*ConfigurablePlacementRule) Set added in v1.95.1 

func (c *ConfigurablePlacementRule) Set(s string) error

Set implements pflag.Value.

func (*ConfigurablePlacementRule) String added in v1.95.1 

func (c *ConfigurablePlacementRule) String() string

String implements pflag.Value.

func (*ConfigurablePlacementRule) Type added in v1.95.1 

func (c *ConfigurablePlacementRule) Type() string

Type implements pflag.Value.

type CountryFilter added in v1.84.1 

type CountryFilter struct {
	// contains filtered or unexported fields
}

CountryFilter can select nodes based on the condition of the country code.

func NewContinentFilterFromString added in v1.112.2 

func NewContinentFilterFromString(continent string) (*CountryFilter, error)

NewContinentFilterFromString parses country definitions like 'SA','!NA'.

func NewCountryFilter added in v1.84.1 

func NewCountryFilter(permit location.Set) *CountryFilter

NewCountryFilter creates a new CountryFilter.

func NewCountryFilterFromString added in v1.89.2 

func NewCountryFilterFromString(countries []string) (*CountryFilter, error)

NewCountryFilterFromString parses country definitions like 'hu','!hu','*','none' and creates a CountryFilter.

func (*CountryFilter) Match added in v1.88.2 

func (p *CountryFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

func (*CountryFilter) String added in v1.89.2 

func (p *CountryFilter) String() string

type DownloadSelector added in v1.109.1 

type DownloadSelector func(requester storj.NodeID, possibleNodes map[storj.NodeID]*SelectedNode, needed int) (map[storj.NodeID]*SelectedNode, error)

DownloadSelector will take a map of possible nodes to choose for a download. It returns a new map of nodes to consider for selecting for the download. It is always true that 0 <= len(result) <= len(possibleNodes), and every element in result will have come from possibleNodes. 'needed' is a hint to the selector of how many nodes are needed for return ideally, so many selectors will try to return at least 'needed' nodes. 

var DefaultDownloadSelector DownloadSelector = func(_ storj.NodeID, possibleNodes map[storj.NodeID]*SelectedNode, _ int) (map[storj.NodeID]*SelectedNode, error) {
	return possibleNodes, nil
}

DefaultDownloadSelector is a DownloadSelector that returns the set of possibleNodes unchanged. 

var ExcludeAllDownloadSelector DownloadSelector = func(storj.NodeID, map[storj.NodeID]*SelectedNode, int) (map[storj.NodeID]*SelectedNode, error) {
	return map[storj.NodeID]*SelectedNode{}, nil
}

ExcludeAllDownloadSelector is a DownloadSelector that always returns an empty map.

func DownloadBest added in v1.109.1 

func DownloadBest(tracker UploadSuccessTracker) DownloadSelector

DownloadBest will take a set of nodes and will return just the best nodes.

func DownloadChoiceOfN added in v1.109.1 

func DownloadChoiceOfN(tracker UploadSuccessTracker, n int64) DownloadSelector

DownloadChoiceOfN will take a set of nodes and winnow it down using choice of n. n is an int64 type due to a mito scripting shortcoming but really an int16 should be fine.

func DownloadSelectorFromString added in v1.109.1 

func DownloadSelectorFromString(expr string, environment *PlacementConfigEnvironment) (DownloadSelector, error)

DownloadSelectorFromString parses complex node download selection expressions from config lines.

type ECParameters added in v1.107.1 

type ECParameters struct {
	Minimum int
	Success int
	Total   int
	Repair  int
}

ECParameters can be used to override certain part of the RS parameters.

type ExcludeAllFilter added in v1.84.1 

type ExcludeAllFilter struct{}

ExcludeAllFilter will never select any node.

func (ExcludeAllFilter) Match added in v1.88.2 

func (ExcludeAllFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type ExcludeFilter added in v1.86.1 

type ExcludeFilter struct {
	// contains filtered or unexported fields
}

ExcludeFilter excludes only the matched nodes.

func NewExcludeFilter added in v1.86.1 

func NewExcludeFilter(filter NodeFilter) ExcludeFilter

NewExcludeFilter creates filter, nodes matching the given filter will be excluded.

func (ExcludeFilter) Match added in v1.88.2 

func (e ExcludeFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

func (ExcludeFilter) String added in v1.89.2 

func (e ExcludeFilter) String() string

type ExcludedIDs added in v1.84.1 

type ExcludedIDs []storj.NodeID

ExcludedIDs can blacklist NodeIDs.

func (ExcludedIDs) Match added in v1.88.2 

func (e ExcludedIDs) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type ExcludedNetworks added in v1.84.1 

type ExcludedNetworks []string

ExcludedNetworks will exclude nodes with specified networks.

func (ExcludedNetworks) Match added in v1.88.2 

func (e ExcludedNetworks) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type ExcludedNodeNetworks added in v1.86.1 

type ExcludedNodeNetworks []*SelectedNode

ExcludedNodeNetworks exclude nodes which has same net as the one of the specified.

func (ExcludedNodeNetworks) Match added in v1.88.2 

func (e ExcludedNodeNetworks) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type Invariant added in v1.96.2 

type Invariant func(pieces metabase.Pieces, nodes []SelectedNode) intset.Set

Invariant checks the current placement, and identifies the pieces which should be moved. Used by repair jobs.

func AllGood added in v1.96.2 

func AllGood() Invariant

AllGood is an invariant, which accepts all piece sets as good.

func ClumpingByAnyTag added in v1.96.2 

func ClumpingByAnyTag(key string, maxAllowed int) Invariant

ClumpingByAnyTag tries to limit the number of nodes with the same tag value.

func ClumpingByAttribute added in v1.96.2 

func ClumpingByAttribute(attr NodeAttribute, maxAllowed int) Invariant

ClumpingByAttribute allows only one selected piece by attribute groups.

func InvariantFromString added in v1.100.2 

func InvariantFromString(expr string) (Invariant, error)

InvariantFromString parses complex invariants (~declumping rules) from config lines.

type NodeAttribute added in v1.95.1 

type NodeAttribute func(SelectedNode) string

NodeAttribute returns a string (like last_net or tag value) for each SelectedNode. can be used to group / label nodes.

func AnyNodeTagAttribute added in v1.96.2 

func AnyNodeTagAttribute(tagName string) NodeAttribute

AnyNodeTagAttribute selects a tag value from node, accepts any signer.

func CreateNodeAttribute added in v1.95.1 

func CreateNodeAttribute(attr string) (NodeAttribute, error)

CreateNodeAttribute creates the NodeAttribute selected based on a string definition.

func IfSelector added in v1.107.1 

func IfSelector(condition func(SelectedNode) bool, conditionTrue, conditionFalse NodeAttribute) NodeAttribute

IfSelector selects the first node attribute if the condition is true, otherwise the second node attribute.

func NodeTagAttribute added in v1.96.2 

func NodeTagAttribute(signer storj.NodeID, tagName string) NodeAttribute

NodeTagAttribute selects a tag value from node.

func Subnet 

func Subnet(bits int64) NodeAttribute

Subnet can return the IP network of the node for any netmask length.

type NodeFilter added in v1.84.1 

type NodeFilter interface {
	Match(node *SelectedNode) bool
}

NodeFilter can decide if a Node should be part of the selection or not.

func FilterFromString added in v1.100.2 

func FilterFromString(expr string) (NodeFilter, error)

FilterFromString parses complex node filter expressions from config lines.

type NodeFilterFunc added in v1.84.1 

type NodeFilterFunc func(node *SelectedNode) bool

NodeFilterFunc is helper to use func as NodeFilter.

func (NodeFilterFunc) Match added in v1.88.2 

func (n NodeFilterFunc) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

type NodeFilterWithAnnotation added in v1.87.1 

type NodeFilterWithAnnotation interface {
	NodeFilter
	GetAnnotation(name string) string
}

NodeFilterWithAnnotation is a NodeFilter with additional annotations.

func WithAnnotation added in v1.86.1 

func WithAnnotation(filter NodeFilter, name string, value string) NodeFilterWithAnnotation

WithAnnotation adds annotations to a NodeFilter.

type NodeFilters added in v1.84.1 

type NodeFilters []NodeFilter

NodeFilters is a collection of multiple node filters (all should vote with true).

func (NodeFilters) GetAnnotation added in v1.87.1 

func (n NodeFilters) GetAnnotation(name string) string

GetAnnotation implements NodeFilterWithAnnotation.

func (NodeFilters) Match added in v1.88.2 

func (n NodeFilters) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

func (NodeFilters) String added in v1.89.2 

func (n NodeFilters) String() string

func (NodeFilters) WithCountryFilter added in v1.84.1 

func (n NodeFilters) WithCountryFilter(permit location.Set) NodeFilters

WithCountryFilter is a helper to create a new filter with additional CountryFilter.

func (NodeFilters) WithExcludedIDs added in v1.84.1 

func (n NodeFilters) WithExcludedIDs(ds []storj.NodeID) NodeFilters

WithExcludedIDs is a helper to create a new filter with additional WithExcludedIDs.

type NodeSelector added in v1.95.1 

type NodeSelector func(requester storj.NodeID, n int, excluded []storj.NodeID, alreadySelected []*SelectedNode) ([]*SelectedNode, error)

NodeSelector pick random nodes based on a specific algorithm. Nodes from excluded should never be used. Same is true for alreadySelected, but it may also trigger other restrictions (for example, when a last_net is already selected, all the nodes from the same net should be excluded as well.

type NodeSelectorInit added in v1.95.1 

type NodeSelectorInit func([]*SelectedNode, NodeFilter) NodeSelector

NodeSelectorInit initializes a stateful NodeSelector when node cache is refreshed.

func AttributeGroupSelector added in v1.96.2 

func AttributeGroupSelector(attribute NodeAttribute) NodeSelectorInit

AttributeGroupSelector first selects a group with equal chance (like last_net) and choose node from the group randomly.

func BalancedGroupBasedSelector added in v1.97.2 

func BalancedGroupBasedSelector(attribute NodeAttribute, uploadFilter NodeFilter) NodeSelectorInit

BalancedGroupBasedSelector first selects a group with equal chance (like last_net) and choose one single node randomly. . One group can be tried multiple times, and if the node is already selected, it will be ignored.

func BestOfN added in v1.106.1 

func BestOfN(tracker ScoreNode, ratio float64, delegate NodeSelectorInit) NodeSelectorInit

BestOfN selects more nodes than the required one, and choose the fastest from those.

func ChoiceOfN added in v1.106.1 

func ChoiceOfN(tracker ScoreNode, n int64, delegate NodeSelectorInit) NodeSelectorInit

ChoiceOfN will perform the selection for n*x nodes and choose the best node from groups of n size. n is an int64 type due to a mito scripting shortcoming but really an int16 should be fine. NOTE: it may break other pre-conditions, like the results of the balanced selector...

func ChoiceOfNSelection added in v1.113.5 

func ChoiceOfNSelection(n int64, delegate NodeSelectorInit, scoreSource ...ScoreSelection) NodeSelectorInit

ChoiceOfNSelection is similar to ChoiceOfN, but doesn't break the pre-conditions of the original selector. it chooses from selections, without mixing nodes. scoreSources are ar judging the selections in order.

func ChoiceOfTwo added in v1.106.1 

func ChoiceOfTwo(tracker ScoreNode, delegate NodeSelectorInit) NodeSelectorInit

ChoiceOfTwo will repeat the selection and choose the better node pair-wise. NOTE: it may break other pre-conditions, like the results of the balanced selector...

func DualSelector added in v1.108.1 

func DualSelector(fraction float64, first NodeSelectorInit, second NodeSelectorInit) NodeSelectorInit

DualSelector selects fraction of nodes with first, and remaining with the second selector.

func EnoughFast added in v1.108.1 

func EnoughFast(tracker UploadSuccessTracker, ratio float64, splitLine float64, selectionRatio float64, delegate NodeSelectorInit) NodeSelectorInit

EnoughFast will select `ratio` times more nodes. The fastest nodes (under splitLine) will be used the selectionRation nodes, remaining wil be chosen from the second part.

func FilterBest added in v1.106.1 

func FilterBest(tracker UploadSuccessTracker, selection string, uplink string, delegate NodeSelectorInit) NodeSelectorInit

FilterBest is a selector, which keeps only the best nodes (based on percentage, or fixed number of nodes). this selector will permanently ban the worst nodes for the period of nodeselection cache refresh.

func FilterSelector added in v1.97.1 

func FilterSelector(loadTimeFilter NodeFilter, init NodeSelectorInit) NodeSelectorInit

FilterSelector is a specific selector, which can filter out nodes from the upload selection. Note: this is different from the generic filter attribute of the NodeSelectorInit, as that is applied to all node selection (upload/download/repair).

func FilteredSelector added in v1.96.2 

func FilteredSelector(preFilter NodeFilter, init NodeSelectorInit) NodeSelectorInit

FilteredSelector uses another selector on the filtered list of nodes.

func RandomSelector added in v1.96.2 

func RandomSelector() NodeSelectorInit

RandomSelector selects any nodes with equal chance.

func SelectorFromString added in v1.100.2 

func SelectorFromString(expr string, environment *PlacementConfigEnvironment) (NodeSelectorInit, error)

SelectorFromString parses complex node selection rules from config lines.

func UnvettedSelector added in v1.96.2 

func UnvettedSelector(newNodeFraction float64, init NodeSelectorInit) NodeSelectorInit

UnvettedSelector selects new nodes first based on newNodeFraction, and selects old nodes for the remaining.

func WeightedSelector added in v1.116.3 

func WeightedSelector(value NodeValue, defaultValue int64, initFilter NodeFilter) NodeSelectorInit

WeightedSelector selects randomly from nodes, but supporting custom probabilities. Some nodes can be selected more often than others. The implementation is based on Walker's alias method: https://www.youtube.com/watch?v=retAwpUv42E

type NodeTag added in v1.84.1 

type NodeTag struct {
	NodeID   storj.NodeID
	SignedAt time.Time
	Signer   storj.NodeID
	Name     string
	Value    []byte
}

NodeTag is a tag associated with a node (approved by signer).

type NodeTags added in v1.84.1 

type NodeTags []NodeTag

NodeTags is a collection of multiple NodeTag.

func (NodeTags) FindBySignerAndName added in v1.84.1 

func (n NodeTags) FindBySignerAndName(signer storj.NodeID, name string) (NodeTag, error)

FindBySignerAndName selects first tag with same name / NodeID.

type NodeValue added in v1.116.3 

type NodeValue func(node SelectedNode) int64

NodeValue returns a numerical value for each node.

func CreateNodeValue added in v1.116.3 

func CreateNodeValue(attr string) (NodeValue, error)

CreateNodeValue creates a NodeValue from a string definition.

type NoopTracker added in v1.105.3 

type NoopTracker struct {
}

NoopTracker doesn't tracker uploads at all. Always returns with zero.

func (NoopTracker) Get added in v1.105.3 

func (n NoopTracker) Get(uplink storj.NodeID) func(node *SelectedNode) float64

Get implements UploadSuccessTracker.

type OrFilter added in v1.91.2 

type OrFilter []NodeFilter

OrFilter will include the node, if at lest one of the filters are matched.

func (OrFilter) Match added in v1.91.2 

func (n OrFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

func (OrFilter) String added in v1.91.2 

func (n OrFilter) String() string

type Placement added in v1.95.1 

type Placement struct {
	// the unique ID of the placement
	ID storj.PlacementConstraint
	// meaningful identifier/label for Humans. Will be used on UI.
	Name string
	// binding condition for filtering out nodes
	NodeFilter NodeFilter
	// Selector is the method how the nodes are selected from the full node space (eg. pick a subnet first, and pick a node from the subnet)
	Selector NodeSelectorInit
	// checked by repair job, applied to the full selection. Out of placement items will be replaced by new, selected by the Selector.
	Invariant Invariant
	// DownloadSelector is the method for how the nodes are selected for
	// downloading (e.g., at random from the uploaded set, or filtered down with
	// choice of 2).
	DownloadSelector DownloadSelector

	// EC defines erasure coding parameter overrides.
	EC ECParameters `yaml:"ec"`
}

Placement defined all the custom behavior metadata of a specific placement group.

func (Placement) GetAnnotation added in v1.95.1 

func (p Placement) GetAnnotation(name string) string

GetAnnotation implements NodeFilterWithAnnotation. Deprecated: use Name instead.

func (Placement) Match added in v1.95.1 

func (p Placement) Match(node *SelectedNode) bool

Match implements NodeFilter.

type PlacementConfigEnvironment added in v1.105.3 

type PlacementConfigEnvironment struct {
	// contains filtered or unexported fields
}

PlacementConfigEnvironment includes all generic functions and variables, which can be used in the configuration.

func NewPlacementConfigEnvironment added in v1.105.3 

func NewPlacementConfigEnvironment(tracker UploadSuccessTracker) *PlacementConfigEnvironment

NewPlacementConfigEnvironment creates PlacementConfigEnvironment.

type PlacementDefinitions added in v1.95.1 

type PlacementDefinitions map[storj.PlacementConstraint]Placement

PlacementDefinitions can include the placement definitions for each known identifier.

func LoadConfig added in v1.96.2 

func LoadConfig(configFile string, environment *PlacementConfigEnvironment) (PlacementDefinitions, error)

LoadConfig loads the placement yaml file and creates the Placement definitions.

func NewPlacementDefinitions added in v1.95.1 

func NewPlacementDefinitions(placements ...Placement) PlacementDefinitions

NewPlacementDefinitions creates a PlacementDefinition with a default placement.

func TestPlacementDefinitions added in v1.96.2 

func TestPlacementDefinitions() PlacementDefinitions

TestPlacementDefinitions creates placements for testing. Only 0 placement is defined with subnetfiltering.

func TestPlacementDefinitionsWithFraction added in v1.96.2 

func TestPlacementDefinitionsWithFraction(newNodeFraction float64) PlacementDefinitions

TestPlacementDefinitionsWithFraction creates placements for testing. Similar to TestPlacementDefinitions, but also selects newNodes based on fraction.

func (PlacementDefinitions) AddLegacyStaticRules added in v1.95.1 

func (d PlacementDefinitions) AddLegacyStaticRules()

AddLegacyStaticRules initializes all the placement rules defined earlier in static golang code.

func (PlacementDefinitions) AddPlacement added in v1.96.2 

func (d PlacementDefinitions) AddPlacement(id storj.PlacementConstraint, placement Placement)

AddPlacement registers a new placement.

func (PlacementDefinitions) AddPlacementFromString added in v1.95.1 

func (d PlacementDefinitions) AddPlacementFromString(definitions string) error

AddPlacementFromString parses placement definition form string representations from id:definition;id:definition;... Deprecated: we will switch to the YAML based configuration.

func (PlacementDefinitions) AddPlacementRule added in v1.95.1 

func (d PlacementDefinitions) AddPlacementRule(id storj.PlacementConstraint, filter NodeFilter, downloadSelector DownloadSelector)

AddPlacementRule registers a new placement.

func (PlacementDefinitions) CreateFilters added in v1.95.1 

func (d PlacementDefinitions) CreateFilters(constraint storj.PlacementConstraint) (filter NodeFilter, selector DownloadSelector)

CreateFilters implements PlacementCondition.

func (PlacementDefinitions) SupportedPlacements added in v1.95.1 

func (d PlacementDefinitions) SupportedPlacements() (res []storj.PlacementConstraint)

SupportedPlacements returns all the IDs, which have associated placement rules.

type PlacementRules added in v1.95.1 

type PlacementRules func(constraint storj.PlacementConstraint) (filter NodeFilter, selector DownloadSelector)

PlacementRules can crate filter based on the placement identifier.

type RandomOrder added in v1.86.1 

type RandomOrder struct {
	// contains filtered or unexported fields
}

RandomOrder as an iterator of a pseudo-random permutation set.

func NewRandomOrder added in v1.86.1 

func NewRandomOrder(n int) RandomOrder

NewRandomOrder creates new iterator, returns number between [0,n) in pseudo-random order.

func (*RandomOrder) At added in v1.86.1 

func (r *RandomOrder) At() uint64

At returns the current number in the permutations.

func (*RandomOrder) Finished added in v1.104.4 

func (r *RandomOrder) Finished() bool

Finished returns true, if there is no more permutation.

func (*RandomOrder) Next added in v1.86.1 

func (r *RandomOrder) Next() bool

Next generates the next number.

func (*RandomOrder) Reset added in v1.104.4 

func (r *RandomOrder) Reset()

Reset makes it possible to reuse the RandomOrder: the full pseudo-random permutations can be read again.

type ScoreNode added in v1.113.5 

type ScoreNode interface {
	Get(uplink storj.NodeID) func(node *SelectedNode) float64
}

ScoreNode can help to assign a score to a node. The higher score is better. float.Nan is valid, if no information is available.

func Desc added in v1.113.5 

func Desc(original ScoreNode) ScoreNode

Desc is a score node, which reverses the score of the original node.

func Min added in v1.113.5 

func Min(sources ...interface{}) ScoreNode

Min is a ScoreNode, which uses the lowest values. Parameters can be ScoreNodes or static float64/int64.

func PieceCount added in v1.113.5 

func PieceCount(divider int64) ScoreNode

PieceCount scores the node based on the piece count.

type ScoreNodeFunc added in v1.113.5 

type ScoreNodeFunc func(uplink storj.NodeID, node *SelectedNode) float64

ScoreNodeFunc implements ScoreNode interface with a single func.

func (ScoreNodeFunc) Get added in v1.113.5 

func (s ScoreNodeFunc) Get(id storj.NodeID) func(node *SelectedNode) float64

Get implements ScoreNode.

type ScoreSelection added in v1.113.5 

type ScoreSelection func(uplink storj.NodeID, selected []*SelectedNode) float64

ScoreSelection can help to choose between two selections with assigning a score. The higher score is better.

func LastBut added in v1.113.5 

func LastBut(attr ScoreNode, skip int64) ScoreSelection

LastBut scores a selection based on the worst node (but skip the worst n nodes).

func MaxGroup added in v1.113.5 

func MaxGroup(attr NodeAttribute) ScoreSelection

MaxGroup returns with the size of the biggest group in the node selection.

func Median added in v1.113.5 

func Median(attr ScoreNode) ScoreSelection

Median scores the selection based on the median of the attribute.

type SelectedNode added in v1.84.1 

type SelectedNode struct {
	ID          storj.NodeID
	Address     *pb.NodeAddress
	Email       string
	Wallet      string
	LastNet     string
	LastIPPort  string
	CountryCode location.CountryCode
	Exiting     bool
	Suspended   bool
	Online      bool
	Vetted      bool
	Tags        NodeTags
	PieceCount  int64
	// free disk space in bytes
	FreeDisk int64
}

SelectedNode is used as a result for creating orders limits.

func (*SelectedNode) Clone added in v1.84.1 

func (node *SelectedNode) Clone() *SelectedNode

Clone returns a deep clone of the selected node.

type State 

type State map[storj.PlacementConstraint]NodeSelector

State includes a stateful selector (indexed nodes) for each placement.

func NewState 

func NewState(nodes []*SelectedNode, placements PlacementDefinitions) State

NewState initializes the State for each placement.

func (State) Select 

func (s State) Select(requester storj.NodeID, p storj.PlacementConstraint, count int, excluded []storj.NodeID, alreadySelected []*SelectedNode) ([]*SelectedNode, error)

Select picks the required nodes given a specific placement.

type TagFilter added in v1.84.1 

type TagFilter struct {
	// contains filtered or unexported fields
}

TagFilter matches nodes with specific tags.

func NewTagFilter added in v1.84.1 

func NewTagFilter(id storj.NodeID, name string, value []byte, match ValueMatch) TagFilter

NewTagFilter creates a new tag filter.

func (TagFilter) Match added in v1.88.2 

func (t TagFilter) Match(node *SelectedNode) bool

Match implements NodeFilter interface.

func (TagFilter) String added in v1.89.2 

func (t TagFilter) String() string

type UploadSuccessTracker added in v1.105.3 

type UploadSuccessTracker interface {
	// Get gives a Score to the node based on the upload success rate. Can be math.NaN (no information). Higher is better.
	Get(uplink storj.NodeID) func(node *SelectedNode) float64
}

UploadSuccessTracker can give hints about the frequency of the long-tail cancellation per node.

type ValueMatch added in v1.89.2 

type ValueMatch func(a []byte, b []byte) bool

ValueMatch defines how to compare tag value with the defined one.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.