Documentation
¶
Index ¶
- Constants
- Variables
- func Filename(configRoot string) (string, error)
- func Marshal(value interface{}) ([]byte, error)
- func Path(configRoot, configFile string) (string, error)
- func PathRoot() (string, error)
- type Addresses
- type Bootstrap
- type Config
- type Datastore
- type Discovery
- type Duration
- type FileStore
- type Finder
- type Identity
- type Indexer
- type Ingest
- type LocalFileStore
- type Logging
- type Peering
- type Policy
- type Polling
- type RateLimit
- type S3FileStore
Constants ¶
View Source
const ( // DefaultPathName is the default config dir name. DefaultPathName = ".storetheindex" // DefaultPathRoot is the path to the default config dir location. DefaultPathRoot = "~/" + DefaultPathName // DefaultConfigFile is the filename of the configuration file. DefaultConfigFile = "config" // EnvDir is the environment variable used to change the path root. EnvDir = "STORETHEINDEX_PATH" Version = 2 )
View Source
const ( IdentityTag = "Identity" PrivKeyTag = "PrivKey" PrivKeySelector = IdentityTag + "." + PrivKeyTag PrivateKeyPathEnvVar = "STORETHEINDEX_PRIV_KEY_PATH" )
Variables ¶
View Source
var ( ErrInitialized = errors.New("configuration file already exists") ErrNotInitialized = errors.New("not initialized") )
View Source
var ErrInvalidPeerAddr = errors.New("invalid peer address")
ErrInvalidPeerAddr signals an address is not a valid peer address.
Functions ¶
func Filename ¶
Filename returns the configuration file path given a configuration root directory. If the configuration root directory is empty, use the default.
Types ¶
type Addresses ¶
type Addresses struct {
// Admin is the admin http listen address. Set to "none" to disable this
// server for both http and libp2p.
Admin string
// Finder is the finder http isten address. Set to "none" to disable this
// server for both http and libp2p.
Finder string
// Ingest is the index data ingestion http listen address. Set to "none"
// to disable this server for both http and libp2p.
Ingest string
// P2PMaddr is the libp2p host multiaddr for all servers. Set to "none" to
// disable libp2p hosting.
P2PAddr string
// NoResourceManager disables the libp2p resource manager when true.
NoResourceManager bool
}
Addresses stores the (string) multiaddr addresses for the node.
func NewAddresses ¶
func NewAddresses() Addresses
NewAddresses returns Addresses with values set to their defaults.
type Bootstrap ¶
type Bootstrap struct {
// Peers is the local node's bootstrap peer addresses.
Peers []string
// MinimumPeers governs whether to bootstrap more connections. If the node
// has less open connections than this number, it will open connections to
// the bootstrap nodes. Set to 0 to disable bootstrapping.
MinimumPeers int
}
Bootstrap configures other nodes to connect to for the purpose of exchanging gossip pubsub. The nodes listed here must be running pubsub and must also be subscribed to the indexer/ingest topic. The peers can be other indexers, or IPFS nodes with pubsub enabled and subscribed to the topic.
func NewBootstrap ¶
func NewBootstrap() Bootstrap
NewBootstrap returns Bootstrap with values set to their defaults.
type Config ¶
type Config struct {
Version int // config version
Identity Identity // peer identity
Addresses Addresses // addresses to listen on
Bootstrap Bootstrap // Peers to connect to for gossip
Datastore Datastore // datastore config
Discovery Discovery // provider pubsub peers
Finder Finder // finder code configuration
Indexer Indexer // indexer code configuration
Ingest Ingest // ingestion related configuration.
Logging Logging // logging configuration.
Peering Peering // peering service configuration.
}
Config is used to load config files.
func InitWithIdentity ¶
func (*Config) UpgradeConfig ¶
UpgradeConfig upgrades (or downgrades) the config file to the current version. If the config file is at the current version a backup is still created and the config rewritten with any unconfigured values set to their defaults.
type Datastore ¶
type Datastore struct {
// Dir is the directory where the datastore is kept. If this is not an
// absolute path then the location is relative to the indexer repo
// directory.
Dir string
// DirAdvertisements specifies to keep advertisements in a separate
// datastore directory, using a separate datastore instance. If this is not
// set or is set to the same value as Dir, then the same datastore instance
// is used to store advertisements. If this is not an absolute path then
// the location is relative to the indexer repo directory.
DirAdvertisements string
// Type is the type of datastore.
Type string
}
Datastore tracks the configuration of the datastore.
func NewDatastore ¶
func NewDatastore() Datastore
NewDatastore returns Datastore with values set to their defaults.
type Discovery ¶
type Discovery struct {
// FilterIPs, when true, removes any private, loopback, or unspecified IP
// addresses from provider and publisher addresses.
FilterIPs bool
// Policy configures which publishers are allowed and blocked, rate-limited,
// and allowed to publish on behalf of others providers.
Policy Policy
// PollInterval is the amount of time to wait without getting any updates
// for a provider, before sending a request for the latest advertisement.
// If there is no response after at least one poll attempt, then the
// provider is considered inactive and is not returned in find responses.
// Values are a number ending in "s", "m", "h" for seconds. minutes, hours.
PollInterval Duration
// PollRetryAfter is the amount of time from one poll attempt, without a
// response, to the next poll attempt, and is also the time between checks
// for providers to poll. This value must be smaller than PollStopAfter for
// there to be more than one poll attempt for a provider.
PollRetryAfter Duration
// PollStopAfter is the amount of time, from the start of polling, to
// continuing polling for the latest advertisement without getting a
// response. After this time elapses with no updates from the provider, the
// provider's data is removed from the indexer and must be re-fetched if
// the provider returns.
PollStopAfter Duration
// DeactivateAfter is the duration after which an unseen provider will be
// excluded from find queries as well as providers list. Note, an inactive
// provider will not be deleted until PollStopAfter has elapses since last
// it was seen. If unspecified, providers will be removed once PollStopAfter
// has elapsed without entering inactive state grace period. If set to less
// than PollStopAfter this parameter will have no effect.
DeactivateAfter Duration
// PollOverrides configures polling for specific providers.
PollOverrides []Polling
// RemoveOldAssignments, if true, removes persisted assignments of previous
// versions. When false, previous versions of persisted assignments are
// migrated. Only applies if UseAssigner is true.
RemoveOldAssignments bool
// UseAssigner configures the indexer to work with an assigner service.
// This also requires that Policy.Allow is false, making Policy.Except into
// a list of allowed peers. Peers listed in Policy.Except in the
// configuration file are included in the set of assigned peers. This
// allows an indexer to be pre-configured with assigned peers.
UseAssigner bool
}
Discovery configures policy and polling settings that determines how information about available index data is discovered.
func NewDiscovery ¶
func NewDiscovery() Discovery
NewDiscovery returns Discovery with values set to their defaults.
type Duration ¶
Duration wraps time.Duration to provide json serialization and deserialization.
NOTE: the zero value encodes to an empty string.
func (Duration) MarshalText ¶
func (*Duration) UnmarshalText ¶
type FileStore ¶ added in v0.5.9
type FileStore struct {
// Compress specifies how to compress files. One of: "gzip", "none".
// Defaults to "gzip" if unspecified.
Compress string
// Type of file store to use: "", "local", "s3"
Type string
// Configuration for storing files in local filesystem.
Local LocalFileStore
// Configuration for storing files in S3.
S3 S3FileStore
}
FileStore configures a particular file store implementation.
type Finder ¶
type Finder struct {
// ApiReadTimeout sets the HTTP server's maximum duration for reading the
// entire request, including the body. A value of zero sets the default,
// and a negative value means there will be no timeout.
ApiReadTimeout Duration
// ApiWriteTimeout sets the HTTP server's maximum duration before timing
// out writes of the response. A value of zero sets the default and a
// negative value means there will be no timeout.
ApiWriteTimeout Duration
// MaxConnections is maximum number of simultaneous connections that the
// HTTP server will accept. A value of zero sets the default and a negative
// value means there is no limit.
MaxConnections int
// Webpage is a domain to display when the homepage of the finder is
// accessed over HTTP.
Webpage string
}
type Identity ¶
type Identity struct {
// PeerID is the peer ID of the indexer that must match the given PrivKey.
// If unspecified, it is automatically generated from the PrivKey.
PeerID string
// PrivKey represents the peer identity of the indexer.
//
// If unset, the key is loaded from the file at the path specified via
// STORETHEINDEX_PRIV_KEY_PATH environment variable.
PrivKey string `json:",omitempty"`
}
Identity tracks the configuration of the local node's identity.
func CreateIdentity ¶
CreateIdentity initializes a new identity.
type Indexer ¶
type Indexer struct {
// CacheSize is the maximum number of CIDs that cache can hold. Setting to -1 disables the
// cache.
CacheSize int
// ConfigCheckInterval is the time between config file update checks.
ConfigCheckInterval Duration
// CorePutConcurrency is the number of core goroutines used to write
// individual multihashes within a Put. A value of 1 means no concurrency,
// and zero uses the default.
CorePutConcurrency int
// DHBatchSize configures the batch size when sending batches of merge
// requests to the DHStore service. A value < 1 results in the default
// size.
DHBatchSize int
// DHStoreURL is the base URL for the DHStore service. This option value
// tells the indexer core to use a DHStore service, if configured.
DHStoreURL string
// DHStoreHttpClientTimeout is a timeout for the DHStore http client
DHStoreHttpClientTimeout Duration
// FreezeAtPercent is the percent used, of the file system that
// ValueStoreDir is on, at which to trigger the indexer to enter frozen
// mode. A zero value uses the default. A negative value disables freezing.
FreezeAtPercent float64
// GCInterval configures the garbage collection interval for valuestores
// that support it.
GCInterval Duration
// GCTimeLimit configures the maximum amount of time a garbage collection
// cycle may run.
GCTimeLimit Duration
// IndexCountTotalAddend is a value that is added to the index count total,
// to account for uncounted indexes that have existed in the value store
// before provider index counts were tracked. This value is reloadable.
IndexCountTotalAddend uint64
// ShutdownTimeout is the duration that a graceful shutdown has to complete
// before the daemon process is terminated. If unset or zero, configures no
// shutdown timeout. This value is reloadable.
ShutdownTimeout Duration
// ValueStoreDir is the directory where value store is kept. If this is not
// an absolute path then the location is relative to the indexer repo
// directory.
ValueStoreDir string
// ValueStoreType specifies type of valuestore to use, such as "sth" or "pogreb".
ValueStoreType string
// STHBits is bits for bucket size in store the hash. Note: this should not be changed
// from its value at initialization or the datastore will be corrupted.
STHBits uint8
// STHBurstRate specifies how much unwritten data can accumulate before
// causing data to be flushed to disk.
STHBurstRate uint64
// STHFileCacheSize is the maximum number of open files that the STH file
// cache may have. A value of 0 uses the default, and a value of -1
// disables the file cache.
STHFileCacheSize int
// STHSyncInterval determines how frequently changes are flushed to disk.
STHSyncInterval Duration
// PebbleDisableWAL sets whether to disable write-ahead-log in Pebble which
// can offer better performance in specific cases. Enabled by default. This
// option only applies when ValueStoreType is set to "pebble".
PebbleDisableWAL bool
// UnfreezeOnStart tells that indexer to unfreeze itself on startup if it
// is frozen. This reverts the indexer to the state it was in before it was
// frozen. It only retains the most recent provider and publisher
// addresses.
UnfreezeOnStart bool
// VSNoNewMH, when true, prevents storing new multihashes in the
// valuestore. Existing data is still retrievable and metadata can be
// deleted and updated.
VSNoNewMH bool
}
Indexer holds configuration for the indexer core. Setting any of these items to their zero-value configures the default value.
func NewIndexer ¶
func NewIndexer() Indexer
NewIndexer returns Indexer with values set to their defaults.
type Ingest ¶
type Ingest struct {
// AdvertisementDepthLimit is the total maximum recursion depth limit when
// syncing advertisements. The value -1 means no limit and zero means use
// the default value. Limiting the depth of advertisements can be done if
// there is a need to prevent an indexer from ingesting long chains of
// advertisements.
//
// A chain of advertisements is synced by separate requests to the provider
// for each advertisement. These requests are done in groups (segments) of
// size set by SyncSegmentDepthLimit. AdvertisementDepthLimit sets the
// limit on the total number of advertisements across all segments.
AdvertisementDepthLimit int
// CarMirrorDestination configures if, how, and where to store ingested
// advertisements and entries in CAR files.
CarMirrorDestination FileStore
// EntriesDepthLimit is the total maximum recursion depth limit when
// syncing advertisement entries. The value -1 means no limit and zero
// means use the default value. The purpose is to prevent overload from
// extremely long entry chains resulting from publisher misconfiguration.
//
// A chain of multihash entries chunks is synced by separate requests to
// the provider for each chunk. These requests are done in groups
// (segments) of size set by SyncSegmentDepthLimit. EntriesDepthLimit sets
// the limit on the total number of entries chunks across all segments.
EntriesDepthLimit int
// HttpSyncRetryMax sets the maximum number of times HTTP sync requests
// should be retried.
HttpSyncRetryMax int
// HttpSyncRetryWaitMax sets the maximum time to wait before retrying a
// failed HTTP sync.
HttpSyncRetryWaitMax Duration
// HttpSyncRetryWaitMin sets the minimum time to wait before retrying a
// failed HTTP sync.
HttpSyncRetryWaitMin Duration
// HttpSyncTimeout sets the time limit for HTTP sync requests.
HttpSyncTimeout Duration
// IngestWorkerCount sets how many ingest worker goroutines to spawn. This
// controls how many concurrent ingest from different providers we can handle.
IngestWorkerCount int
// MinimumKeyLengt causes any multihash, that has a digest length less than
// this, to be ignored. If using storethehash, this value is automatically
// set to 4 if it was configured to be anything less.
MinimumKeyLength int
// PubSubTopic sets the topic name to which to subscribe for ingestion
// announcements.
PubSubTopic string
// RateLimit contains rate-limiting configuration.
RateLimit RateLimit
// ResendDirectAnnounce determines whether or not to re-publish direct
// announce messages over gossip pubsub. When a single indexer receives an
// announce message via HTTP, enabling this lets the indexers re-publish
// the announce so that other indexers can also receive it. This is always
// false if configured to use an assigner.
ResendDirectAnnounce bool
// StoreBatchSize is the number of entries in each write to the value
// store. Specifying a value less than 2 disables batching. This should be
// smaller than the maximum number of multihashes in an entry block to
// write concurrently to the value store.
StoreBatchSize int
// SyncSegmentDepthLimit is the depth limit of a single sync in a series of
// calls that collectively sync advertisements or their entries. The value
// -1 disables the segmentation where the sync will be done in a single call
// and zero means use the default value.
SyncSegmentDepthLimit int
// SyncTimeout is the maximum amount of time allowed for a sync to complete
// before it is canceled. This can be a sync of a chain of advertisements
// or a chain of advertisement entries. The value is an integer string
// ending in "s", "m", "h" for seconds. minutes, hours.
SyncTimeout Duration
}
Ingest tracks the configuration related to the ingestion protocol.
type LocalFileStore ¶ added in v0.5.9
type LocalFileStore struct {
// Path to filesystem directory where files are stored.
BasePath string
}
type Logging ¶
type Logging struct {
// Level sets the log level for all loggers that do not have a setting in
// Loggers. The default value is "info".
Level string
// Loggers sets log levels for individual loggers.
Loggers map[string]string
}
Logging configures overall and logger-specific log levels. Level values are case-insensitive and include the following in order of importance: "fatal", "panic", "dpanic", ""error", "warn", "info", "debug"
func NewLogging ¶
func NewLogging() Logging
NewLogging returns Logging with values set to their defaults.
type Peering ¶
type Peering struct {
// Peers lists the nodes to attempt to stay connected with.
Peers []string
}
Peering configures the peering service. Peering is similar to bootstrapping, but peering maintains connection with all peers configured in the peering service.
func NewPeering ¶
func NewPeering() Peering
NewPeering returns Peering with values set to their defaults.
type Policy ¶
type Policy struct {
// Allow is either false or true, and determines whether a peer is allowed
// (true) or is blocked (false), by default. If a peer if blocked, then it
// cannot publish advertisements to this indexer or be listed as a provider
// by this indexer.
Allow bool
// Except is a list of peer IDs that are exceptions to the Allow policy.
// If Allow is true, then all peers are allowed except those listed in
// Except. If Allow is false, then no peers are allowed except those listed
// in Except. in other words, Allow=true means that Except is a deny-list
// and Allow=false means that Except is an allow-list.
Except []string
// Publish determines whether or not peers are allowed to publish
// advertisements for a provider with a differen peer ID.
Publish bool
// PublisherExcept is a list of peer IDs that are exceptions to the Publish
// policy. If Publish is false, then all allowed peers cannot publish
// advertisements for providers with a different peer ID, unless listed in
// PublishExcept. If Publish is true, then all allowed peers can publish
// advertisements for any provider, unless listed in PublishExcept.
PublishExcept []string
}
Policy configures which peers are allowed to be providers, publish advertisements and publish on behalf of other providers. The Allow policy determines which peers the indexer will request advertisements from and index content for. The Publish policy determines if a publisher may supply an advertisement that has a provider that is different from the publisher.
Publishers and providers are not the same. Publishers are peers that supply data to the indexer. Providers are the peers that appear in advertisements and are where retrieval clients get content from.
type Polling ¶
type Polling struct {
// ProviderID identifies the provider that this override applies to.
ProviderID string
// Interval overrides Discovery.PollInterval.
Interval Duration
// RetryAfter overrides Discovery.PollRetryAfter.
RetryAfter Duration
// StopAfter overrides Discovery.PollStopAfter.
StopAfter Duration
// DeactivateAfter overrides Discovery.DeactivateAfter.
DeactivateAfter Duration
}
Polling is a set of polling parameters that is applied to a specific provider. The values override the matching Poll values in the Discovery config.
type RateLimit ¶
type RateLimit struct {
// Apply is either false or true, and determines whether a peer is subject
// to rate limiting (true) or not (false), by default.
Apply bool
// Except is a list of peer IDs that are exceptions to the Apply rule. If
// Apply is false then peers are not rate-limited unless they appear in the
// Except list. If Apply is true, then only the peers listed in Except are
// not rate-limited.
Except []string
// BlocksPerSecond is the number of blocks allowed to be transferred per
// second. An advertisement and a block of multihashes are both represented
// as a block, so this limit applies to both. Setting a value of 0 disables
// rate limiting, meaning that the rate is infinite.
BlocksPerSecond int
// BurstSize is the maximum number of blocks that can be received at once.
// After this, BlocksPerSecond additional blocks maybe received each
// second, and any more results in rate limiting. With HTTP ingestion, rate
// limiting waits until more blocks are allowed to be received. With
// graphsync, rate limiting terminates the session and resumes it when
// sufficient time has passed to be able to receive BurstSize blocks. A
// value of 0 results in 10 times BlocksPerSecond.
BurstSize int
}
func NewRateLimit ¶
func NewRateLimit() RateLimit
NewRateLimit returns RateLimit with values set to their defaults.
type S3FileStore ¶ added in v0.5.9
type S3FileStore struct {
BucketName string
// ## Optional Overrides ##
//
// These values are generally set by the environment and should only be
// provided when necessary to override values from the environment, or when the
// environment is not configured.
Endpoint string
Region string
AccessKey string
SecretKey string
}
Source Files
Click to show internal directories.
Click to hide internal directories.