config

package
v0.7.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 22, 2023 License: Apache-2.0, MIT Imports: 16 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// DefaultPathName is the default config dir name.
	DefaultPathName = ".storetheindex"
	// DefaultPathRoot is the path to the default config dir location.
	DefaultPathRoot = "~/" + DefaultPathName
	// DefaultConfigFile is the filename of the configuration file.
	DefaultConfigFile = "config"
	// EnvDir is the environment variable used to change the path root.
	EnvDir = "STORETHEINDEX_PATH"

	Version = 2
)
View Source 
const (
	IdentityTag     = "Identity"
	PrivKeyTag      = "PrivKey"
	PrivKeySelector = IdentityTag + "." + PrivKeyTag

	PrivateKeyPathEnvVar = "STORETHEINDEX_PRIV_KEY_PATH"
)

Variables

View Source 
var (
	ErrInitialized    = errors.New("configuration file already exists")
	ErrNotInitialized = errors.New("not initialized")
)
View Source 
var ErrInvalidPeerAddr = errors.New("invalid peer address")

ErrInvalidPeerAddr signals an address is not a valid peer address.

Functions

func Filename 

func Filename(configRoot string) (string, error)

Filename returns the configuration file path given a configuration root directory. If the configuration root directory is empty, use the default.

func Marshal 

func Marshal(value interface{}) ([]byte, error)

Marshal configuration with JSON.

func Path 

func Path(configRoot, configFile string) (string, error)

Path returns the config file path relative to the configuration root. If an empty string is provided for `configRoot`, the default root is used. If configFile is an absolute path, then configRoot is ignored.

func PathRoot 

func PathRoot() (string, error)

PathRoot returns the default configuration root directory.

Types

type Addresses 

type Addresses struct {
	// Admin is the admin http listen address. Set to "none" to disable this
	// server for both http and libp2p.
	Admin string
	// Finder is the finder http isten address. Set to "none" to disable this
	// server for both http and libp2p.
	Finder string
	// Ingest is the index data ingestion http listen address. Set to "none"
	// to disable this server for both http and libp2p.
	Ingest string
	// P2PMaddr is the libp2p host multiaddr for all servers. Set to "none" to
	// disable libp2p hosting.
	P2PAddr string
	// NoResourceManager disables the libp2p resource manager when true.
	NoResourceManager bool
}

Addresses stores the (string) multiaddr addresses for the node.

func NewAddresses 

func NewAddresses() Addresses

NewAddresses returns Addresses with values set to their defaults.

type Bootstrap 

type Bootstrap struct {
	// Peers is the local node's bootstrap peer addresses.
	Peers []string
	// MinimumPeers governs whether to bootstrap more connections. If the node
	// has less open connections than this number, it will open connections to
	// the bootstrap nodes. Set to 0 to disable bootstrapping.
	MinimumPeers int
}

Bootstrap configures other nodes to connect to for the purpose of exchanging gossip pubsub. The nodes listed here must be running pubsub and must also be subscribed to the indexer/ingest topic. The peers can be other indexers, or IPFS nodes with pubsub enabled and subscribed to the topic.

func NewBootstrap 

func NewBootstrap() Bootstrap

NewBootstrap returns Bootstrap with values set to their defaults.

func (Bootstrap) PeerAddrs 

func (b Bootstrap) PeerAddrs() ([]peer.AddrInfo, error)

PeerAddrs returns the bootstrap peers as a list of AddrInfo.

func (*Bootstrap) SetPeers 

func (b *Bootstrap) SetPeers(addrs []peer.AddrInfo)

SetPeers sets the bootstrap peers from a list of AddrInfo.

type ByteSize added in v0.5.14 

type ByteSize uint64

ByteSize wraps uint64 to provide json serialization and deserialization.

NOTE: the zero value encodes to an empty string.

func (ByteSize) MarshalText added in v0.5.14 

func (d ByteSize) MarshalText() ([]byte, error)

func (ByteSize) String added in v0.5.14 

func (d ByteSize) String() string

func (*ByteSize) UnmarshalText added in v0.5.14 

func (d *ByteSize) UnmarshalText(text []byte) error

type Config 

type Config struct {
	Version   int       // config version
	Identity  Identity  // peer identity
	Addresses Addresses // addresses to listen on
	Bootstrap Bootstrap // Peers to connect to for gossip
	Datastore Datastore // datastore config
	Discovery Discovery // provider pubsub peers
	Finder    Finder    // finder code configuration
	Indexer   Indexer   // indexer code configuration
	Ingest    Ingest    // ingestion related configuration.
	Logging   Logging   // logging configuration.
	Peering   Peering   // peering service configuration.
}

Config is used to load config files.

func Init 

func Init(out io.Writer) (*Config, error)

func InitWithIdentity 

func InitWithIdentity(identity Identity) (*Config, error)

func Load 

func Load(filePath string) (*Config, error)

Load reads the json-serialized config at the specified path.

func (*Config) Save 

func (c *Config) Save(filePath string) error

Save writes the json-serialized config to the specified path.

func (*Config) String 

func (c *Config) String() string

String returns a pretty-printed json config.

func (*Config) UpgradeConfig 

func (c *Config) UpgradeConfig(filePath string) error

UpgradeConfig upgrades (or downgrades) the config file to the current version. If the config file is at the current version a backup is still created and the config rewritten with any unconfigured values set to their defaults.

type Datastore 

type Datastore struct {
	// Dir is the directory where the datastore is kept. If this is not an
	// absolute path then the location is relative to the indexer repo
	// directory.
	Dir string
	// Type is the type of datastore.
	Type string
	// TmpDir is the directory where the datastore for persisted temporaty data
	// is kept. This datastore contains temporary items such as synced
	// advertisement data and data-transfer session state. If this is not an
	// absolute path then the location is relative to the indexer repo.
	TmpDir string
	// TmpType is the type of datastore for temporary persisted data.
	TmpType string
}

Datastore tracks the configuration of the datastore.

func NewDatastore 

func NewDatastore() Datastore

NewDatastore returns Datastore with values set to their defaults.

type Discovery 

type Discovery struct {
	// FilterIPs, when true, removes any private, loopback, or unspecified IP
	// addresses from provider and publisher addresses.
	FilterIPs bool
	// Policy configures which publishers are allowed and blocked, rate-limited,
	// and allowed to publish on behalf of others providers.
	Policy Policy
	// PollInterval is the amount of time to wait without getting any updates
	// for a provider, before sending a request for the latest advertisement.
	// If there is no response after at least one poll attempt, then the
	// provider is considered inactive and is not returned in find responses.
	// Values are a number ending in "s", "m", "h" for seconds. minutes, hours.
	PollInterval Duration
	// PollRetryAfter is the amount of time from one poll attempt, without a
	// response, to the next poll attempt, and is also the time between checks
	// for providers to poll. This value must be smaller than PollStopAfter for
	// there to be more than one poll attempt for a provider.
	PollRetryAfter Duration
	// PollStopAfter is the amount of time, from the start of polling, to
	// continuing polling for the latest advertisement without getting a
	// response. After this time elapses with no updates from the provider, the
	// provider's data is removed from the indexer and must be re-fetched if
	// the provider returns.
	PollStopAfter Duration
	// DeactivateAfter is the duration after which an unseen provider will be
	// excluded from find queries as well as providers list. Note, an inactive
	// provider will not be deleted until PollStopAfter has elapses since last
	// it was seen. If unspecified, providers will be removed once PollStopAfter
	// has elapsed without entering inactive state grace period. If set to less
	// than PollStopAfter this parameter will have no effect.
	DeactivateAfter Duration
	// PollOverrides configures polling for specific providers.
	PollOverrides []Polling
	// RemoveOldAssignments, if true, removes persisted assignments of previous
	// versions. When false, previous versions of persisted assignments are
	// migrated. Only applies if UseAssigner is true.
	RemoveOldAssignments bool
	// UnassignedPublishers, if true, allows unassigned publishers to remain
	// unassigned until the assigner service assigns them, preferring this
	// indexer to assign the publishers to. If false (the default), assigns all
	// unassigned publishers known to this indexer.
	//
	// When configuring an existing indexer to use an assigner service, set
	// this to false if the known publishers should be immediately assigned to
	// this indexer to make them eligible for handoff if this indexer is
	// frozen. Set to true if the publishers known to this indexer may already
	// be assigned to another indexer, possibly making the assignment to this
	// indexer unnecessary.
	//
	// In general, it is safer to leave this set to false to guarantee that the
	// indexer is assigned the publishers it knows about. Otherwise, if the
	// indexer is frozen the unassigned publishers may not be handed off to
	// another indexer and an update from that publisher will result in
	// indexing all that publisher's data.
	UnassignedPublishers bool
	// UseAssigner configures the indexer to work with an assigner service.
	// This also requires that Policy.Allow is false, making Policy.Except into
	// a list of allowed peers. Peers listed in Policy.Except in the
	// configuration file are included in the set of assigned peers. This
	// allows an indexer to be pre-configured with assigned peers.
	UseAssigner bool
}

Discovery configures policy and polling settings that determines how information about available index data is discovered.

func NewDiscovery 

func NewDiscovery() Discovery

NewDiscovery returns Discovery with values set to their defaults.

type Duration 

type Duration time.Duration

Duration wraps time.Duration to provide json serialization and deserialization.

NOTE: the zero value encodes to an empty string.

func (Duration) MarshalText 

func (d Duration) MarshalText() ([]byte, error)

func (Duration) String 

func (d Duration) String() string

func (*Duration) UnmarshalText 

func (d *Duration) UnmarshalText(text []byte) error

type FileStore added in v0.5.9 

type FileStore struct {
	// Type of file store to use: "", "local", "s3"
	Type string
	// Configuration for storing files in local filesystem.
	Local LocalFileStore
	// Configuration for storing files in S3.
	S3 S3FileStore
}

FileStore configures a particular file store implementation.

type Finder 

type Finder struct {
	// ApiReadTimeout sets the HTTP server's maximum duration for reading the
	// entire request, including the body. A value of zero sets the default,
	// and a negative value means there will be no timeout.
	ApiReadTimeout Duration
	// ApiWriteTimeout sets the HTTP server's maximum duration before timing
	// out writes of the response. A value of zero sets the default and a
	// negative value means there will be no timeout.
	ApiWriteTimeout Duration
	// MaxConnections is maximum number of simultaneous connections that the
	// HTTP server will accept. A value of zero sets the default and a negative
	// value means there is no limit.
	MaxConnections int
	// Webpage is a domain to display when the homepage of the finder is
	// accessed over HTTP.
	Webpage string
}

func NewFinder 

func NewFinder() Finder

type Identity 

type Identity struct {
	// PeerID is the peer ID of the indexer that must match the given PrivKey.
	// If unspecified, it is automatically generated from the PrivKey.
	PeerID string
	// PrivKey represents the peer identity of the indexer.
	//
	// If unset, the key is loaded from the file at the path specified via
	// STORETHEINDEX_PRIV_KEY_PATH environment variable.
	PrivKey string `json:",omitempty"`
}

Identity tracks the configuration of the local node's identity.

func CreateIdentity 

func CreateIdentity(out io.Writer) (Identity, error)

CreateIdentity initializes a new identity.

func (Identity) Decode 

func (i Identity) Decode() (peer.ID, ic.PrivKey, error)

func (Identity) DecodePrivateKey 

func (i Identity) DecodePrivateKey(passphrase string) (ic.PrivKey, error)

DecodePrivateKey is a helper to decode the user's PrivateKey.

type Indexer 

type Indexer struct {
	// CacheSize is the maximum number of CIDs that cache can hold. Setting to
	// -1 disables the cache.
	CacheSize int
	// ConfigCheckInterval is the time between config file update checks.
	ConfigCheckInterval Duration
	// DHBatchSize configures the batch size when sending batches of merge
	// requests to the DHStore service. A value < 1 results in the default
	// size.
	DHBatchSize int
	// DHStoreURL is the base URL for the DHStore service. This value is required
	// if ValueStoreType is "dhstore".
	DHStoreURL string
	// DHStoreClusterURLs provide addional URLs that the core will send delete
	// requests to. Deletes will be send to the dhstoreURL as well as to all
	// dhstoreClusterURLs. This is required as deletes need to be applied to
	// all nodes until consistent hashing is implemented. dhstoreURL shouldn't
	// be included in this list.
	DHStoreClusterURLs []string
	// DHStoreHttpClientTimeout is a timeout for the DHStore http client
	DHStoreHttpClientTimeout Duration
	// FreezeAtPercent is the percent used, of the file system that
	// ValueStoreDir is on, at which to trigger the indexer to enter frozen
	// mode. A zero value uses the default. A negative value disables freezing.
	FreezeAtPercent float64
	// IndexCountTotalAddend is a value that is added to the index count total,
	// to account for uncounted indexes that have existed in the value store
	// before provider index counts were tracked. This value is reloadable.
	IndexCountTotalAddend uint64
	// ShutdownTimeout is the duration that a graceful shutdown has to complete
	// before the daemon process is terminated. If unset or zero, configures no
	// shutdown timeout. This value is reloadable.
	ShutdownTimeout Duration
	// ValueStoreDir is the directory where value store is kept, if the value
	// store type requires local storage. If this is not an absolute path then
	// the location is relative to the indexer repo directory.
	ValueStoreDir string
	// ValueStoreType specifies type of valuestore to use, such as "dhstore" or
	// "pebble". If no set, then defaults to "dhstore" if DHStoreURL is
	// configures, otherwise defaults to "pebble".
	ValueStoreType string
	// PebbleDisableWAL sets whether to disable write-ahead-log in Pebble which
	// can offer better performance in specific cases. Enabled by default. This
	// option only applies when ValueStoreType is set to "pebble".
	PebbleDisableWAL bool
	// PebbleBlockCacheSize is a size of pebble block cache in bytes
	PebbleBlockCacheSize ByteSize
	// UnfreezeOnStart tells that indexer to unfreeze itself on startup if it
	// is frozen. This reverts the indexer to the state it was in before it was
	// frozen. It only retains the most recent provider and publisher
	// addresses.
	UnfreezeOnStart bool
}

Indexer holds configuration for the indexer core. Setting any of these items to their zero-value configures the default value.

func NewIndexer 

func NewIndexer() Indexer

NewIndexer returns Indexer with values set to their defaults.

type Ingest 

type Ingest struct {
	// AdvertisementDepthLimit is the total maximum recursion depth limit when
	// syncing advertisements. The value -1 means no limit and zero means use
	// the default value. Limiting the depth of advertisements can be done if
	// there is a need to prevent an indexer from ingesting long chains of
	// advertisements.
	//
	// A chain of advertisements is synced by separate requests to the provider
	// for each advertisement. These requests are done in groups (segments) of
	// size set by SyncSegmentDepthLimit. AdvertisementDepthLimit sets the
	// limit on the total number of advertisements across all segments.
	AdvertisementDepthLimit int
	// AdvertisementMirror configures if, how, and where to store content
	// advertisements data in CAR files. The mirror may be readable, writable,
	// both, or neither. If the mirror is neither readable or writable, or a
	// storage type is not specified, then the mirror is not used.
	AdvertisementMirror Mirror
	// EntriesDepthLimit is the total maximum recursion depth limit when
	// syncing advertisement entries. The value -1 means no limit and zero
	// means use the default value. The purpose is to prevent overload from
	// extremely long entry chains resulting from publisher misconfiguration.
	//
	// A chain of multihash entries chunks is synced by separate requests to
	// the provider for each chunk. These requests are done in groups
	// (segments) of size set by SyncSegmentDepthLimit. EntriesDepthLimit sets
	// the limit on the total number of entries chunks across all segments.
	EntriesDepthLimit int
	// GsMaxInRequests is the maximum number of incoming in-progress graphsync
	// requests. Default is 1024.
	GsMaxInRequests uint64
	// GsMaxOutRequests is the maximum number of outgoing in-progress graphsync
	// requests. Default is 1024.
	GsMaxOutRequests uint64
	// HttpSyncRetryMax sets the maximum number of times HTTP sync requests
	// should be retried.
	HttpSyncRetryMax int
	// HttpSyncRetryWaitMax sets the maximum time to wait before retrying a
	// failed HTTP sync.
	HttpSyncRetryWaitMax Duration
	// HttpSyncRetryWaitMin sets the minimum time to wait before retrying a
	// failed HTTP sync.
	HttpSyncRetryWaitMin Duration
	// HttpSyncTimeout sets the time limit for HTTP sync requests.
	HttpSyncTimeout Duration
	// IngestWorkerCount sets how many ingest worker goroutines to spawn. This
	// controls how many concurrent ingest from different providers we can handle.
	IngestWorkerCount int
	// MinimumKeyLengt causes any multihash, that has a digest length less than
	// this, to be ignored.
	MinimumKeyLength int
	// PubSubTopic sets the topic name to which to subscribe for ingestion
	// announcements.
	PubSubTopic string
	// ResendDirectAnnounce determines whether or not to re-publish direct
	// announce messages over gossip pubsub. When a single indexer receives an
	// announce message via HTTP, enabling this lets the indexers re-publish
	// the announce so that other indexers can also receive it. This is always
	// false if configured to use an assigner.
	ResendDirectAnnounce bool
	// SyncSegmentDepthLimit is the depth limit of a single sync in a series of
	// calls that collectively sync advertisements or their entries. The value
	// -1 disables the segmentation where the sync will be done in a single call
	// and zero means use the default value.
	SyncSegmentDepthLimit int
	// SyncTimeout is the maximum amount of time allowed for a sync to complete
	// before it is canceled. This can be a sync of a chain of advertisements
	// or a chain of advertisement entries. The value is an integer string
	// ending in "s", "m", "h" for seconds. minutes, hours.
	SyncTimeout Duration
}

Ingest tracks the configuration related to the ingestion protocol.

func NewIngest 

func NewIngest() Ingest

NewIngest returns Ingest with values set to their defaults.

type LocalFileStore added in v0.5.9 

type LocalFileStore struct {
	// Path to filesystem directory where files are stored.
	BasePath string
}

type Logging 

type Logging struct {
	// Level sets the log level for all loggers that do not have a setting in
	// Loggers. The default value is "info".
	Level string
	// Loggers sets log levels for individual loggers.
	Loggers map[string]string
}

Logging configures overall and logger-specific log levels. Level values are case-insensitive and include the following in order of importance: "fatal", "panic", "dpanic", ""error", "warn", "info", "debug"

func NewLogging 

func NewLogging() Logging

NewLogging returns Logging with values set to their defaults.

type Mirror added in v0.5.14 

type Mirror struct {
	// Read specifies to read advertisement content from the mirror.
	Read bool
	// Write specified to write advertisement content to the mirrir.
	Write bool
	// Compress specifies how to compress files. One of: "gzip", "none".
	// Defaults to "gzip" if unspecified.
	Compress string
	// Storage configures the backing file store for the mirror.
	Storage FileStore
}

type Peering 

type Peering struct {
	// Peers lists the nodes to attempt to stay connected with.
	Peers []string
}

Peering configures the peering service. Peering is similar to bootstrapping, but peering maintains connection with all peers configured in the peering service.

func NewPeering 

func NewPeering() Peering

NewPeering returns Peering with values set to their defaults.

func (Peering) PeerAddrs 

func (p Peering) PeerAddrs() ([]peer.AddrInfo, error)

PeerAddrs returns the peering peers as a list of AddrInfo.

type Policy 

type Policy struct {
	// Allow is either false or true, and determines whether a peer is allowed
	// (true) or is blocked (false), by default. If a peer if blocked, then it
	// cannot publish advertisements to this indexer or be listed as a provider
	// by this indexer.
	Allow bool
	// Except is a list of peer IDs that are exceptions to the Allow policy.
	// If Allow is true, then all peers are allowed except those listed in
	// Except. If Allow is false, then no peers are allowed except those listed
	// in Except. in other words, Allow=true means that Except is a deny-list
	// and Allow=false means that Except is an allow-list.
	Except []string

	// Publish determines whether or not peers are allowed to publish
	// advertisements for a provider with a differen peer ID.
	Publish bool
	// PublisherExcept is a list of peer IDs that are exceptions to the Publish
	// policy. If Publish is false, then all allowed peers cannot publish
	// advertisements for providers with a different peer ID, unless listed in
	// PublishExcept. If Publish is true, then all allowed peers can publish
	// advertisements for any provider, unless listed in PublishExcept.
	PublishExcept []string
}

Policy configures which peers are allowed to be providers, publish advertisements and publish on behalf of other providers. The Allow policy determines which peers the indexer will request advertisements from and index content for. The Publish policy determines if a publisher may supply an advertisement that has a provider that is different from the publisher.

Publishers and providers are not the same. Publishers are peers that supply data to the indexer. Providers are the peers that appear in advertisements and are where retrieval clients get content from.

func NewPolicy 

func NewPolicy() Policy

NewPolicy returns Policy with values set to their defaults.

type Polling 

type Polling struct {
	// ProviderID identifies the provider that this override applies to.
	ProviderID string
	// Interval overrides Discovery.PollInterval.
	Interval Duration
	// RetryAfter overrides Discovery.PollRetryAfter.
	RetryAfter Duration
	// StopAfter overrides Discovery.PollStopAfter.
	StopAfter Duration
	// DeactivateAfter overrides Discovery.DeactivateAfter.
	DeactivateAfter Duration
}

Polling is a set of polling parameters that is applied to a specific provider. The values override the matching Poll values in the Discovery config.

type S3FileStore added in v0.5.9 

type S3FileStore struct {
	BucketName string

	// ## Optional Overrides ##
	//
	// These values are generally set by the environment and should only be
	// provided when necessary to override values from the environment, or when the
	// environment is not configured.
	Endpoint  string
	Region    string
	AccessKey string
	SecretKey string
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.