Documentation
¶
Index ¶
- Constants
- Variables
- func Marshal(value interface{}) ([]byte, error)
- func Path(configRoot, configFile string) (string, error)
- func PathRoot() (string, error)
- type Addresses
- type Bootstrap
- type ByteSize
- type Config
- type Datastore
- type Discovery
- type Duration
- type Finder
- type Identity
- type Indexer
- type Ingest
- type Logging
- type Mirror
- type Peering
- type Policy
- type Polling
- type PublishersPolicy
Constants ¶
View Source
const ( // DefaultPathName is the default config dir name. DefaultPathName = ".storetheindex" // DefaultPathRoot is the path to the default config dir location. DefaultPathRoot = "~/" + DefaultPathName // DefaultConfigFile is the filename of the configuration file. DefaultConfigFile = "config" // EnvDir is the environment variable used to change the path root. EnvDir = "STORETHEINDEX_PATH" Version = 2 )
View Source
const ( IdentityTag = "Identity" PrivKeyTag = "PrivKey" PrivKeySelector = IdentityTag + "." + PrivKeyTag PrivateKeyPathEnvVar = "STORETHEINDEX_PRIV_KEY_PATH" )
Variables ¶
View Source
var ( ErrInitialized = errors.New("configuration file already exists") ErrNotInitialized = errors.New("not initialized") )
View Source
var ErrInvalidPeerAddr = errors.New("invalid peer address")
ErrInvalidPeerAddr signals an address is not a valid peer address.
Functions ¶
Types ¶
type Addresses ¶
type Addresses struct {
// Admin is the admin http listen address. Set to "none" to disable this
// server for both http and libp2p.
Admin string
// Finder is the finder http isten address. Set to "none" to disable this
// server for both http and libp2p.
Finder string
// Ingest is the index data ingestion http listen address. Set to "none"
// to disable this server for both http and libp2p.
Ingest string
// P2PMaddr is the libp2p host multiaddr for all servers. Set to "none" to
// disable libp2p hosting.
P2PAddr string
// NoResourceManager disables the libp2p resource manager when true.
NoResourceManager bool
}
Addresses stores the (string) multiaddr addresses for the node.
func NewAddresses ¶
func NewAddresses() Addresses
NewAddresses returns Addresses with values set to their defaults.
type Bootstrap ¶
type Bootstrap struct {
// Peers is the local node's bootstrap peer addresses.
Peers []string
// MinimumPeers governs whether to bootstrap more connections. If the node
// has less open connections than this number, it will open connections to
// the bootstrap nodes. Set to 0 to disable bootstrapping.
MinimumPeers int
}
Bootstrap configures other nodes to connect to for the purpose of exchanging gossip pubsub. The nodes listed here must be running pubsub and must also be subscribed to the indexer/ingest topic. The peers can be other indexers, or IPFS nodes with pubsub enabled and subscribed to the topic.
func NewBootstrap ¶
func NewBootstrap() Bootstrap
NewBootstrap returns Bootstrap with values set to their defaults.
type ByteSize ¶ added in v0.5.14
type ByteSize uint64
ByteSize wraps uint64 to provide json serialization and deserialization.
NOTE: the zero value encodes to an empty string.
func (ByteSize) MarshalText ¶ added in v0.5.14
func (*ByteSize) UnmarshalText ¶ added in v0.5.14
type Config ¶
type Config struct {
Version int // config version
Identity Identity // peer identity
Addresses Addresses // addresses to listen on
Bootstrap Bootstrap // Peers to connect to for gossip
Datastore Datastore // datastore config
Discovery Discovery // provider pubsub peers
Finder Finder // finder code configuration
Indexer Indexer // indexer code configuration
Ingest Ingest // ingestion related configuration.
Logging Logging // logging configuration.
Peering Peering // peering service configuration.
}
Config is used to load config files.
func InitWithIdentity ¶
func (*Config) UpgradeConfig ¶
UpgradeConfig upgrades (or downgrades) the config file to the current version. If the config file is at the current version a backup is still created and the config rewritten with any unconfigured values set to their defaults.
type Datastore ¶
type Datastore struct {
// Dir is the directory where the datastore is kept. If this is not an
// absolute path then the location is relative to the indexer repo
// directory.
Dir string
// Type is the type of datastore.
Type string
// TmpDir is the directory where the datastore for persisted temporaty data
// is kept. This datastore contains temporary items such as synced
// advertisement data and data-transfer session state. If this is not an
// absolute path then the location is relative to the indexer repo.
TmpDir string
// TmpType is the type of datastore for temporary persisted data.
TmpType string
// RemoveTmpAtStart causes all temproary data to be removed at startup.
RemoveTmpAtStart bool
}
Datastore tracks the configuration of the datastore.
func NewDatastore ¶
func NewDatastore() Datastore
NewDatastore returns Datastore with values set to their defaults.
type Discovery ¶
type Discovery struct {
// FilterIPs, when true, removes any private, loopback, or unspecified IP
// addresses from provider and publisher addresses.
FilterIPs bool
// IgnoreBadAdsTime is the period of time to ignore publishers that have
// published unindexable advertisement chains. Default is 2 hours.
IgnoreBadAdsTime Duration
// Policy configures which publishers are allowed and blocked, rate-limited,
// and allowed to publish on behalf of others providers.
Policy Policy
// PollInterval is the amount of time to wait without getting any updates
// for a provider, before sending a request for the latest advertisement.
// If there is no response after at least one poll attempt, then the
// provider is considered inactive and is not returned in find responses.
// Values are a number ending in "s", "m", "h" for seconds. minutes, hours.
PollInterval Duration
// PollRetryAfter is the amount of time from one poll attempt, without a
// response, to the next poll attempt, and is also the time between checks
// for providers to poll. This value must be smaller than PollStopAfter for
// there to be more than one poll attempt for a provider.
PollRetryAfter Duration
// PollStopAfter is the amount of time, from the start of polling, to
// continuing polling for the latest advertisement without getting a
// response. After this time elapses with no updates from the provider, the
// provider's data is removed from the indexer and must be re-fetched if
// the provider returns.
PollStopAfter Duration
// DeactivateAfter is the duration after which an unseen provider will be
// excluded from find queries as well as providers list. Note, an inactive
// provider will not be deleted until PollStopAfter has elapses since last
// it was seen. If unspecified, providers will be removed once PollStopAfter
// has elapsed without entering inactive state grace period. If set to less
// than PollStopAfter this parameter will have no effect.
DeactivateAfter Duration
// PollOverrides configures polling for specific providers.
PollOverrides []Polling
// RemoveOldAssignments, if true, removes persisted assignments of previous
// versions. When false, previous versions of persisted assignments are
// migrated. Only applies if UseAssigner is true.
RemoveOldAssignments bool
// UnassignedPublishers, if true, allows unassigned publishers to remain
// unassigned until the assigner service assigns them, preferring this
// indexer to assign the publishers to. If false (the default), assigns all
// unassigned publishers known to this indexer.
//
// When configuring an existing indexer to use an assigner service, set
// this to false if the known publishers should be immediately assigned to
// this indexer to make them eligible for handoff if this indexer is
// frozen. Set to true if the publishers known to this indexer may already
// be assigned to another indexer, possibly making the assignment to this
// indexer unnecessary.
//
// In general, it is safer to leave this set to false to guarantee that the
// indexer is assigned the publishers it knows about. Otherwise, if the
// indexer is frozen the unassigned publishers may not be handed off to
// another indexer and an update from that publisher will result in
// indexing all that publisher's data.
UnassignedPublishers bool
// UseAssigner configures the indexer to work with an assigner service.
// This also requires that Policy.Allow is false, making Policy.Except into
// a list of allowed peers. Peers listed in Policy.Except in the
// configuration file are included in the set of assigned peers. This
// allows an indexer to be pre-configured with assigned peers.
UseAssigner bool
}
Discovery configures policy and polling settings that determines how information about available index data is discovered.
func NewDiscovery ¶
func NewDiscovery() Discovery
NewDiscovery returns Discovery with values set to their defaults.
type Duration ¶
Duration wraps time.Duration to provide json serialization and deserialization.
NOTE: the zero value encodes to an empty string.
func (Duration) MarshalText ¶
func (*Duration) UnmarshalText ¶
type Finder ¶
type Finder struct {
// ApiReadTimeout sets the HTTP server's maximum duration for reading the
// entire request, including the body. A value of zero sets the default,
// and a negative value means there will be no timeout.
ApiReadTimeout Duration
// ApiWriteTimeout sets the HTTP server's maximum duration before timing
// out writes of the response. A value of zero sets the default and a
// negative value means there will be no timeout.
ApiWriteTimeout Duration
// MaxConnections is maximum number of simultaneous connections that the
// HTTP server will accept. A value of zero sets the default and a negative
// value means there is no limit.
MaxConnections int
// Webpage is a domain to display when the homepage of the finder is
// accessed over HTTP.
Webpage string
}
type Identity ¶
type Identity struct {
// PeerID is the peer ID of the indexer that must match the given PrivKey.
// If unspecified, it is automatically generated from the PrivKey.
PeerID string
// PrivKey represents the peer identity of the indexer.
//
// If unset, the key is loaded from the file at the path specified via
// STORETHEINDEX_PRIV_KEY_PATH environment variable.
PrivKey string `json:",omitempty"`
}
Identity tracks the configuration of the local node's identity.
func CreateIdentity ¶
CreateIdentity initializes a new identity.
type Indexer ¶
type Indexer struct {
// CacheSize is the maximum number of CIDs that cache can hold. Setting to
// -1 disables the cache.
CacheSize int
// ConfigCheckInterval is the time between config file update checks.
ConfigCheckInterval Duration
// DHBatchSize configures the batch size when sending batches of merge
// requests to the DHStore service. A value < 1 results in the default
// size.
DHBatchSize int
// DHStoreURL is the base URL for the DHStore service. This value is required
// if ValueStoreType is "dhstore".
DHStoreURL string
// DHStoreClusterURLs provide addional URLs that the core will send delete
// requests to. Deletes will be send to the dhstoreURL as well as to all
// dhstoreClusterURLs. This is required as deletes need to be applied to
// all nodes until consistent hashing is implemented. dhstoreURL shouldn't
// be included in this list.
DHStoreClusterURLs []string
// DHStoreHttpClientTimeout is a timeout for the DHStore http client
DHStoreHttpClientTimeout Duration
// FreezeAtPercent is the percent used, of the file system that
// ValueStoreDir is on, at which to trigger the indexer to enter frozen
// mode. A zero value uses the default. A negative value disables freezing.
FreezeAtPercent float64
// ShutdownTimeout is the duration that a graceful shutdown has to complete
// before the daemon process is terminated. If unset or zero, configures no
// shutdown timeout. This value is reloadable.
ShutdownTimeout Duration
// ValueStoreDir is the directory where value store is kept, if the value
// store type requires local storage. If this is not an absolute path then
// the location is relative to the indexer repo directory.
ValueStoreDir string
// ValueStoreType specifies type of valuestore to use, such as "dhstore" or
// "pebble". If no set, then defaults to "dhstore" if DHStoreURL is
// configures, otherwise defaults to "pebble".
ValueStoreType string
// PebbleDisableWAL sets whether to disable write-ahead-log in Pebble which
// can offer better performance in specific cases. Enabled by default. This
// option only applies when ValueStoreType is set to "pebble".
PebbleDisableWAL bool
// PebbleBlockCacheSize is a size of pebble block cache in bytes
PebbleBlockCacheSize ByteSize
// UnfreezeOnStart tells that indexer to unfreeze itself on startup if it
// is frozen. This reverts the indexer to the state it was in before it was
// frozen. It only retains the most recent provider and publisher
// addresses.
UnfreezeOnStart bool
}
Indexer holds configuration for the indexer core. Setting any of these items to their zero-value configures the default value.
func NewIndexer ¶
func NewIndexer() Indexer
NewIndexer returns Indexer with values set to their defaults.
type Ingest ¶
type Ingest struct {
// AdvertisementDepthLimit is the total maximum recursion depth limit when
// syncing advertisements. The value -1 means no limit and zero means use
// the default value. Limiting the depth of advertisements can be done if
// there is a need to prevent an indexer from ingesting long chains of
// advertisements.
//
// A chain of advertisements is synced by separate requests to the provider
// for each advertisement. These requests are done in groups (segments) of
// size set by SyncSegmentDepthLimit. AdvertisementDepthLimit sets the
// limit on the total number of advertisements across all segments.
AdvertisementDepthLimit int
// AdvertisementMirror configures if, how, and where to store content
// advertisements data in CAR files. The mirror may be readable, writable,
// both, or neither. If the mirror is neither readable or writable, or a
// storage type is not specified, then the mirror is not used.
AdvertisementMirror Mirror
// EntriesDepthLimit is the total maximum recursion depth limit when
// syncing advertisement entries. The value -1 means no limit and zero
// means use the default value. The purpose is to prevent overload from
// extremely long entry chains resulting from publisher misconfiguration.
//
// A chain of multihash entries chunks is synced by separate requests to
// the provider for each chunk. These requests are done in groups
// (segments) of size set by SyncSegmentDepthLimit. EntriesDepthLimit sets
// the limit on the total number of entries chunks across all segments.
EntriesDepthLimit int
// FirstSyncDepth sets the advertisement chain depth to sync on the first
// sync with a new provider. To sync a new provider with only the most
// recent advertisement, set this to 1. A value of 0, the default, means
// unlimited depth.
FirstSyncDepth int
// GsMaxInRequests is the maximum number of incoming in-progress graphsync
// requests. Default is 1024.
GsMaxInRequests uint64
// GsMaxOutRequests is the maximum number of outgoing in-progress graphsync
// requests. Default is 1024.
GsMaxOutRequests uint64
// HttpSyncRetryMax sets the maximum number of times HTTP sync requests
// should be retried. A value of zero, the default, means no retry.
HttpSyncRetryMax int
// HttpSyncRetryWaitMax sets the maximum time to wait before retrying a
// failed HTTP sync.
HttpSyncRetryWaitMax Duration
// HttpSyncRetryWaitMin sets the minimum time to wait before retrying a
// failed HTTP sync.
HttpSyncRetryWaitMin Duration
// HttpSyncTimeout sets the time limit for HTTP sync requests.
HttpSyncTimeout Duration
// IngestWorkerCount sets how many ingest worker goroutines to spawn. This
// controls how many concurrent ingest from different providers we can handle.
IngestWorkerCount int
// MaxAsyncConcurrency sets the maximum number of concurrent asynchrouous
// syncs (started by announce messages). Set -1 for unlimited, 0 for
// default. This value is reloadable.
MaxAsyncConcurrency int
// MinimumKeyLengt causes any multihash, that has a digest length less than
// this, to be ignored.
MinimumKeyLength int
// OverwriteMirrorOnResync overwrites the advertisement when resyncing.
OverwriteMirrorOnResync bool
// PubSubTopic sets the topic name to which to subscribe for ingestion
// announcements.
PubSubTopic string
// ResendDirectAnnounce determines whether or not to re-publish direct
// announce messages over gossip pubsub. When a single indexer receives an
// announce message via HTTP, enabling this lets the indexers re-publish
// the announce so that other indexers can also receive it. This is always
// false if configured to use an assigner.
ResendDirectAnnounce bool
// Skip500EntriesError, when true, skips advertisements for which the
// publisher returns a 500 status code and an error message "failed to sync
// first entry". This value is reloadable.
Skip500EntriesError bool
// SyncSegmentDepthLimit is the depth limit of a single sync in a series of
// calls that collectively sync advertisements or their entries. The value
// -1 disables the segmentation where the sync will be done in a single call
// and zero means use the default value.
SyncSegmentDepthLimit int
// SyncTimeout is the maximum amount of time allowed for a sync to complete
// before it is canceled. This can be a sync of a chain of advertisements
// or a chain of advertisement entries. The value is an integer string
// ending in "s", "m", "h" for seconds. minutes, hours.
SyncTimeout Duration
}
Ingest tracks the configuration related to the ingestion protocol.
type Logging ¶
type Logging struct {
// Level sets the log level for all loggers that do not have a setting in
// Loggers. The default value is "info".
Level string
// Loggers sets log levels for individual loggers.
Loggers map[string]string
}
Logging configures overall and logger-specific log levels. Level values are case-insensitive and include the following in order of importance: "fatal", "panic", "dpanic", ""error", "warn", "info", "debug"
func NewLogging ¶
func NewLogging() Logging
NewLogging returns Logging with values set to their defaults.
type Mirror ¶ added in v0.5.14
type Mirror struct {
// Read specifies to read advertisement content from the Retrieval mirror.
Read bool
// Write specifies to write advertisement content to the Storage mirror.
Write bool
// Compress specifies how to compress files. One of: "gzip", "none".
// Defaults to "gzip" if unspecified.
Compress string
// Retrieval configures the backing file store for mirror read operations.
Retrieval filestore.Config
// Storage configures the backing file store for mirror write operations.
Storage filestore.Config
}
Mirror configures if, how, and where to store content advertisements data in CAR files. The mirror may be readable, writable, both, or neither. If the mirror is neither readable or writable, or a storage type is not specified, then the mirror is not used.
func NewMirror ¶ added in v0.8.10
func NewMirror() Mirror
NewMirror returns Mirror with values set to their defaults.
func (*Mirror) PopulateUnset ¶ added in v0.8.10
func (c *Mirror) PopulateUnset()
PopulateUnset replaces zero-values in the config with default values.
type Peering ¶
type Peering struct {
// Peers lists the nodes to attempt to stay connected with.
Peers []string
}
Peering configures the peering service. Peering is similar to bootstrapping, but peering maintains connection with all peers configured in the peering service.
func NewPeering ¶
func NewPeering() Peering
NewPeering returns Peering with values set to their defaults.
type Policy ¶
type Policy struct {
// Allow is either false or true, and determines whether a peer is allowed
// (true) or is blocked (false), by default. If a peer if blocked, then it
// cannot publish advertisements to this indexer or be listed as a provider
// by this indexer.
Allow bool
// Except is a list of peer IDs that are exceptions to the Allow policy.
// If Allow is true, then all peers are allowed except those listed in
// Except. If Allow is false, then no peers are allowed except those listed
// in Except. in other words, Allow=true means that Except is a deny-list
// and Allow=false means that Except is an allow-list.
Except []string
// Publish is the default Allow policy when a provider has no policy in
// PublisherForProviders. It determines if any peers are allowed to publish
// advertisements for a provider with a differen peer ID.
Publish bool
// PublisherExcept is a list of peer IDs that are exceptions to the default
// Publish policy.
PublishExcept []string
// PublishersForProvider is a list of policies specifying which publishers
// are allowed to publish advertisements on behalf of a specified provider.
PublishersForProvider []PublishersPolicy
}
Policy configures which peers are allowed to be providers, publish advertisements and publish on behalf of other providers. The Allow policy determines which peers the indexer will request advertisements from and index content for. The Publish policy determines if a publisher may supply an advertisement that has a provider that is different from the publisher.
Publishers and providers are not the same. Publishers are peers that supply data to the indexer. Providers are the peers that appear in advertisements and are where retrieval clients get content from.
type Polling ¶
type Polling struct {
// ProviderID identifies the provider that this override applies to.
ProviderID string
// Interval overrides Discovery.PollInterval.
Interval Duration
// RetryAfter overrides Discovery.PollRetryAfter.
RetryAfter Duration
// StopAfter overrides Discovery.PollStopAfter.
StopAfter Duration
// DeactivateAfter overrides Discovery.DeactivateAfter.
DeactivateAfter Duration
}
Polling is a set of polling parameters that is applied to a specific provider. The values override the matching Poll values in the Discovery config.
type PublishersPolicy ¶ added in v0.8.22
type PublishersPolicy struct {
// Provider is the provider peer ID this policy pertains to.
Provider string
// Allow determines whether a peer is allowed or blocked from publishing
// advertisements on behalf of a provider with a differen peer ID.
Allow bool
// Except is a list of peer IDs that are exceptions to the Allow
// policy.
Except []string
}
Source Files
Click to show internal directories.
Click to hide internal directories.