bitswaphelpers

package
v0.16.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 24, 2023 License: Apache-2.0, MIT Imports: 22 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrAlreadyRegisterd = errors.New("already registered")

ErrAlreadyRegistered means something has already been registered for a retrieval id

View Source 
var ErrNotRegistered = errors.New("not registered")

ErrAlreadyRegistered means there is nothing registered for a retrieval id

View Source 
var ErrNotSupported = errors.New("not supported")

ErrNotSupported indicates an operation not supported by the MultiBlockstore

Functions

func NewByteCountingLinkSystem 

func NewByteCountingLinkSystem(lsys *linking.LinkSystem, bytesWritten func(count uint64)) *linking.LinkSystem

Types

type InProgressCids 

type InProgressCids struct {
	// contains filtered or unexported fields
}

func NewInProgressCids 

func NewInProgressCids() *InProgressCids

func (*InProgressCids) Dec 

func (ipc *InProgressCids) Dec(c cid.Cid, retrievalID types.RetrievalID)

func (*InProgressCids) Get 

func (ipc *InProgressCids) Get(c cid.Cid) []types.RetrievalID

func (*InProgressCids) Inc 

func (ipc *InProgressCids) Inc(c cid.Cid, retrievalID types.RetrievalID)

type IndexerRouting 

type IndexerRouting struct {
	routinghelpers.Null
	// contains filtered or unexported fields
}

IndexerRouting provides an interface that satisfies routing.Routing but only returns provider records based on a preset set of providers read from the context key. Bitswap will potentially make multiple FindProvidersAsync requests, and the cid passed will not always be the root As a result, we have to rely on the retrieval id within a context key Also while there is a delegated routing client that talks to the indexer, we use this cause we run it on top of the processing we're doing at a higher level with multiprotocol filtering

func NewIndexerRouting 

func NewIndexerRouting(toRetrievalID func(cid.Cid) []types.RetrievalID) *IndexerRouting

NewIndexerRouting makes a new indexer routing instance

func (*IndexerRouting) AddProviders 

func (ir *IndexerRouting) AddProviders(retrievalID types.RetrievalID, providers []types.RetrievalCandidate)

AddProviders adds provider records to the total list for a given retrieval id

func (*IndexerRouting) FindProvidersAsync 

func (ir *IndexerRouting) FindProvidersAsync(ctx context.Context, c cid.Cid, max int) <-chan peer.AddrInfo

FindProvidersAsync returns providers based on the retrieval id in a context key It returns a channel with up to `max` providers, keeping the others around for a future call TODO: there is a slight risk that go-bitswap, which dedups requests by CID across multiple sessions, could accidentally read the wrong retrieval id if two retrievals were running at the same time. Not sure how much of a risk this really is, cause when requests are deduped, both calls still receive the results. See go-bitswap ProviderQueryManager for more specifics

func (*IndexerRouting) RemoveProviders 

func (ir *IndexerRouting) RemoveProviders(retrievalID types.RetrievalID)

RemoveProviders removes all provider records for a given retrieval id

type MultiBlockstore 

type MultiBlockstore struct {
	// contains filtered or unexported fields
}

MultiBlockstore creates a blockstore based on one or more linkystems, extracting the target linksystem for each request from the retrieval id context key

func NewMultiblockstore 

func NewMultiblockstore() *MultiBlockstore

NewMultiblockstore returns a new MultiBlockstore

func (*MultiBlockstore) AddLinkSystem 

func (mbs *MultiBlockstore) AddLinkSystem(id types.RetrievalID, lsys *linking.LinkSystem) error

AddLinkSystem registers a linksystem to use for a given retrieval id

func (*MultiBlockstore) AllKeysChan 

func (mbs *MultiBlockstore) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)

AllKeysChan is unsupported

func (*MultiBlockstore) DeleteBlock 

func (mbs *MultiBlockstore) DeleteBlock(ctx context.Context, c cid.Cid) error

DeleteBlock is not supported

func (*MultiBlockstore) Get 

func (mbs *MultiBlockstore) Get(ctx context.Context, c cid.Cid) (blocks.Block, error)

Get returns a block only if the given ctx contains a retrieval ID as a value that references a known linksystem. If it does, it uses that linksystem to load the block

func (*MultiBlockstore) GetSize 

func (mbs *MultiBlockstore) GetSize(ctx context.Context, c cid.Cid) (int, error)

GetSize is unsupported

func (*MultiBlockstore) Has 

func (mbs *MultiBlockstore) Has(ctx context.Context, c cid.Cid) (bool, error)

Has is not supported

func (*MultiBlockstore) HashOnRead 

func (mbs *MultiBlockstore) HashOnRead(enabled bool)

HashOnRead is unsupported

func (*MultiBlockstore) Put 

func (mbs *MultiBlockstore) Put(ctx context.Context, blk blocks.Block) error

Put writes a block only if the given ctx contains a retrieval ID as a value that references a known linksystem. If it does, it uses that linksystem to save the block

func (*MultiBlockstore) PutMany 

func (mbs *MultiBlockstore) PutMany(ctx context.Context, blks []blocks.Block) error

PutMany puts a slice of blocks at the same time, with the same rules as Put

func (*MultiBlockstore) RemoveLinkSystem 

func (mbs *MultiBlockstore) RemoveLinkSystem(id types.RetrievalID)

RemoveLinkSystem unregisters the link system for a given retrieval id

type PreloadCachingStorage added in v0.8.0 

type PreloadCachingStorage struct {
	TraversalLinkSystem *linking.LinkSystem
	BitswapLinkSystem   *linking.LinkSystem
	// contains filtered or unexported fields
}

func NewPreloadCachingStorage added in v0.8.0 

func NewPreloadCachingStorage(
	parentLinkSystem linking.LinkSystem,
	cacheLinkSystem linking.LinkSystem,
	fetcher linking.BlockReadOpener,
	concurrency int,
) (*PreloadCachingStorage, error)

NewPreloadCachingStorage creates a new PreloadCachingStorage.

The parentLinkSystem is used directly by the traversal for both reads (existing blocks, or blocks already traversed and stored) and writes (new blocks discovered during traversal). Writes will be properly ordered according to the traversal.

The cacheLinkSystem is used by the preloader as a temporary space for preloaded blocks. Writes are not properly ordered, only according to the preloader's ability to fetch blocks in its queue. When a block is requested by the traversal and it is already in the cacheLinkSystem, it is copied to the parentLinkSystem.

The fetcher is used by both the preloader and the loader to fetch blocks. It should be able to fetch blocks in a thread-safe manner.

The concurrency parameter controls how many blocks can be preloaded at once (the preloader will fetch up to this many blocks at once using the fetcher).

func (*PreloadCachingStorage) GetStats added in v0.8.0 

func (cs *PreloadCachingStorage) GetStats() PreloadStats

GetStats returns the current stats for the PreloadCachingStorage.

func (*PreloadCachingStorage) Loader added in v0.8.0 

func (cs *PreloadCachingStorage) Loader(linkCtx linking.LinkContext, link ipld.Link) (io.Reader, error)

Loader is compatible with go-ipld-prime's StorageReadOpener. It is not intended to be used directly but is wired up as the StorageReadOpener for the TraversalLinkSystem.

func (*PreloadCachingStorage) Preloader added in v0.8.0 

func (cs *PreloadCachingStorage) Preloader(preloadCtx preload.PreloadContext, link preload.Link)

Preloader is compatible with go-ipld-prime's linking/preload.Loader and should be provided to a traversal Config as the Preloader field.

func (*PreloadCachingStorage) Start added in v0.8.0 

func (cs *PreloadCachingStorage) Start(ctx context.Context) error

Start the preloader; this will start the background goroutines that will fetch blocks from the fetcher and cache them as they are flagged by the preloader. If you Start(), you should also Stop().

func (*PreloadCachingStorage) Stop added in v0.8.0 

func (cs *PreloadCachingStorage) Stop()

Stop the preloader; this will stop the background goroutines.

type PreloadStats added in v0.8.0 

type PreloadStats struct {
	// LoadCount is the number of times the Loader was called
	LoadCount int
	// ActivePreloads is the number of links currently being preloaded
	ActivePreloads int // number of links currently being preloaded
	// NotFound is the number of links that were a decisive "not found" (by the
	// preloader)
	NotFound int
	// PreloadedHits is the number of times a link was loaded from the cache
	// (already full preloaded)
	PreloadedHits int
	// PreloadingHits is the number of times a link was loaded that was in the
	// queue to be preloaded but hadn't yet been fully loaded.
	PreloadingHits int
	// PreloadMisses is the number of times a link was not found in the cache or
	// queued to be preloaded. This should only happen once per traversal (the
	// root).
	PreloadMisses int
}

func (PreloadStats) PreloadedPercent added in v0.8.0 

func (s PreloadStats) PreloadedPercent() uint64

PreloadedPercent returns the percentage of loads that were hits in the preloaded cache. This is a good indicator of how much of the traversal was able to be satisfied by the preloaded cache, however it is not able to capture the number of blocks that were actively being loaded when they were requested.

func (PreloadStats) PreloadingPercent added in v0.8.0 

func (s PreloadStats) PreloadingPercent() uint64

PreloadingHitRate returns the fraction of loads that were hits in the queue of links to be preloaded. This means that the block was flagged as being needed by the preloader, but had not yet been preloaded, or was in the process of being preloaded when it was required by the traversal.

PreloadingHitRate() + PreloadedHitRate() should be close to a value of 1.0 (not exactly as the first block will be a complete preloader miss), together they provide a measure of the performance of the preloader for the traversal.

func (PreloadStats) Print added in v0.8.0 

func (s PreloadStats) Print()

Print prints the stats to stdout

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.