Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BlobIndexLookup ¶
type BlobIndexLookup interface {
// Find should:
// 1. attempt to read the sharded dag index from the cache from the encoded contextID
// 2. if not found, attempt to fetch the index from the provided URL. Store the result in cache
// 3. return the index
// 4. asyncronously, add records to the ProviderStore from the parsed blob index so that we can avoid future queries to IPNI for
// other multihashes in the index
Find(ctx context.Context, contextID types.EncodedContextID, provider model.ProviderResult, fetchURL url.URL, rng *metadata.Range) (blobindex.ShardedDagIndexView, error)
}
BlobIndexLookup is a read through cache for fetching blob indexes
type ClaimLookup ¶
type ClaimLookup interface {
// LookupClaim should:
// 1. attempt to read the claim from the cache from the encoded contextID
// 2. if not found, attempt to fetch the claim from the provided URL. Store the result in cache
// 3. return the claim
LookupClaim(ctx context.Context, claimCid cid.Cid, fetchURL url.URL) (delegation.Delegation, error)
}
ClaimLookup is used to get full claims from a claim cid
type IndexingService ¶
type IndexingService struct {
// contains filtered or unexported fields
}
IndexingService implements read/write logic for indexing data with IPNI, content claims, sharded dag indexes, and a cache layer
func Construct ¶
func Construct(sc ServiceConfig) (*IndexingService, func(context.Context), error)
func NewIndexingService ¶
func NewIndexingService(blobIndexLookup BlobIndexLookup, claimLookup ClaimLookup, providerIndex ProviderIndex, options ...Option) *IndexingService
NewIndexingService returns a new indexing service
func (*IndexingService) CacheClaim ¶
func (is *IndexingService) CacheClaim(ctx context.Context, claim delegation.Delegation) error
CacheClaim is used to cache a claim without publishing it to IPNI this is used cache a location commitment that come from a storage provider on blob/accept, without publishing, since the SP will publish themselves (a delegation for a location commitment is already generated on blob/accept) ideally however, IPNI would enable UCAN chains for publishing so that we could publish it directly from the storage service it doesn't for now, so we let SPs publish themselves them direct cache with us
func (*IndexingService) PublishClaim ¶
func (is *IndexingService) PublishClaim(ctx context.Context, claim delegation.Delegation) error
PublishClaim caches and publishes a content claim I imagine publish claim to work as follows For all claims except index, just use the publish API on IPNIIndex For index claims, let's assume they fail if a location claim for the index car cid is not already published The service should lookup the index cid location claim, and fetch the ShardedDagIndexView, then use the hashes inside to assemble all the multihashes in the index advertisement
func (*IndexingService) Query ¶
func (is *IndexingService) Query(ctx context.Context, q Query) (queryresult.QueryResult, error)
Query returns back relevant content claims for the given query using the following steps 1. Query the IPNIIndex for all matching records 2. For any index records, query the IPNIIndex for any location claims for that index cid 3. For any index claims, query the IPNIIndex for location claims for the index cid 4. Query the BlobIndexLookup to get the full ShardedDagIndex for any index claims 5. Query IPNIIndex for any location claims for any shards that contain the multihash based on the ShardedDagIndex 6. Read the requisite claims from the ClaimLookup 7. Return all discovered claims and sharded dag indexes
type Option ¶
type Option func(is *IndexingService)
Option configures an IndexingService
func WithConcurrency ¶
WithConcurrency causes the indexing service to process find queries parallel, with the given concurrency
type ProviderIndex ¶
type ProviderIndex interface {
// Find should do the following
// 1. Read from the IPNI Storage cache to get a list of providers
// a. If there is no record in cache, query IPNI, filter out any non-content claims metadata, and store
// the resulting records in the cache
// b. the are no records in the cache or IPNI, it can attempt to read from legacy systems -- Dynamo tables & content claims storage, synthetically constructing provider results
// 2. With returned provider results, filter additionally for claim type. If space dids are set, calculate an encodedcontextid's by hashing space DID and Hash, and filter for a matching context id
// Future TODO: kick off a conversion task to update the recrds
Find(context.Context, providerindex.QueryKey) ([]model.ProviderResult, error)
// Publish should do the following:
// 1. Write the entries to the cache with no expiration until publishing is complete
// 2. Generate an advertisement for the advertised hashes and publish/announce it
Publish(context.Context, []multihash.Multihash, model.ProviderResult)
}
ProviderIndex is a read/write interface to a local cache of providers that falls back to IPNI
Source Files
Directories