repo

package
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 13, 2018 License: GPL-3.0 Imports: 18 Imported by: 2

Documentation

Overview

Package repo represents a repository of qri information Analogous to a git repository, repo expects a rigid structure filled with rich types specific to qri. Lots of things in here take inspiration from the ipfs datastore interface: github.com/ipfs/go-datastore

Index

Constants

View Source 
const (
	// ChangeRequestStatusOpen is a request that hasn't been addressed yet
	ChangeRequestStatusOpen = "open"
	// ChangeRequestStatusAccepted have been merged into
	// the target history
	ChangeRequestStatusAccepted = "accepted"
	// ChangeRequestStatusDeclined will not be merged into
	// the target history
	ChangeRequestStatusDeclined = "declined"
)

Variables

View Source 
var (
	// ErrNotFound is the err implementers should return when stuff isn't found
	ErrNotFound = fmt.Errorf("repo: not found")
	// ErrPeernameRequired is for when a peername is missing-but-expected
	ErrPeernameRequired = fmt.Errorf("repo: peername is required")
	// ErrNameRequired is for when a name is missing-but-expected
	ErrNameRequired = fmt.Errorf("repo: name is required")
	// ErrPathRequired is for when a path is missing-but-expected
	ErrPathRequired = fmt.Errorf("repo: path is required")
	// ErrNameTaken is for when a name name is already taken
	ErrNameTaken = fmt.Errorf("repo: name already in use")
	// ErrRepoEmpty is for when the repo has no datasets
	ErrRepoEmpty = fmt.Errorf("repo: this repo contains no datasets")
)

Functions

func CanonicalizeDatasetRef added in v0.1.1 

func CanonicalizeDatasetRef(r Repo, ref *DatasetRef) error

CanonicalizeDatasetRef uses a repo to turn any local aliases into known canonical peername for a dataset and populates a missing path if the repo has path information for a peername/name combo if we provide any other shortcuts for names other than "me" in the future, it should be handled here.

func CanonicalizePeername added in v0.1.1 

func CanonicalizePeername(r Repo, peername *string) error

CanonicalizePeername uses a repo to replace aliases with canonical peernames. basically, this thing replaces "me" with the proper peername.

func CompareDatasetRef 

func CompareDatasetRef(a, b DatasetRef) error

CompareDatasetRef compares two Dataset References, returning an error describing any difference between the two references

func DataNodes 

func DataNodes(nodes map[string]*dsgraph.Node) (ds map[string]bool)

DataNodes returns a map[path]bool of all raw data nodes

func DatasetForQuery 

func DatasetForQuery(r Repo, qpath datastore.Key) (datastore.Key, error)

DatasetForQuery gives a corresponding dataset for a query path if one exists

func DatasetQueries 

func DatasetQueries(nodes map[string]*dsgraph.Node) (qs map[string]datastore.Key)

DatasetQueries returns a mapped subset of a list of nodes in the form: 

DatasetHash : QueryHash

func DatasetsQuery 

func DatasetsQuery(dss Datasets, q query.Query) (map[string]*dataset.Dataset, error)

DatasetsQuery is a convenience function to read all query results & parse into a map[string]*dataset.Dataset.

func Graph 

func Graph(r Repo) (map[string]*dsgraph.Node, error)

Graph generates a map of all paths on this repository pointing to dsgraph.Node structs with all links configured. This is potentially expensive to calculate. Best to do some caching.

func HasPath 

func HasPath(r Repo, path datastore.Key) (bool, error)

HasPath returns true if this repo already has a reference to a given path.

func QueriesMap 

func QueriesMap(nodes map[string]*dsgraph.Node) (qs map[string]datastore.Key)

QueriesMap returns a mapped subset of a list of nodes in the form: 

QueryHash : DatasetHash

func QueryPeers 

func QueryPeers(ps Peers, q query.Query) ([]*profile.Profile, error)

QueryPeers wraps a call to Query, transforming responses to a slice of Profile pointers

func WalkRepoDatasets 

func WalkRepoDatasets(r Repo, visit func(depth int, ref *DatasetRef, err error) (bool, error)) error

WalkRepoDatasets visits every dataset in the history of a user's namespace Yes, this potentially a very expensive function to call, use sparingly.

Types

type ChangeRequest 

type ChangeRequest struct {
	// status of this request. one of: open,accepted,declined
	Status string `json:"status"`
	// created marks the time this change request was created
	Created time.Time `json:"created"`
	// the dataset this change is aimed at. The
	// history of target must match the history
	// of change up until new history entries
	// TODO - should changes be targeting a mutable history?
	Target datastore.Key `json:"target"`
	// path to HEAD of the change history
	Path datastore.Key `json:"path"`
	// The actual change history. All relevant details must be stored
	// in the dataset itself. Title & description of the change goes
	// into this dataset's commit.
	Change *dataset.Dataset `json:"change"`
}

ChangeRequest is one or more proposed additions to the history of a given dataset

type ChangeRequestStore 

type ChangeRequestStore interface {
	// Put a change request into the store
	PutChangeRequest(path datastore.Key, cr *ChangeRequest) error
	// Get a change request by it's path
	GetChangeRequest(path datastore.Key) (*ChangeRequest, error)
	// accept an open change request
	// AcceptChangeRequest(path datastore.Key) error
	// decline an open change request
	// DeclineChangeRequest(path datastore.Key) error
	// get change requests for a given target
	ChangeRequestsForTarget(target datastore.Key, limit, offset int) ([]*ChangeRequest, error)
	// list change requests in this store
	ListChangeRequests(limit, offset int) ([]*ChangeRequest, error)
}

ChangeRequestStore is the interface for storying & manipulating Change Requests Qri repos should embed a ChangeRequestStore

type DatasetRef 

type DatasetRef struct {
	// Peername of dataset owner
	Peername string `json:"peername,omitempty"`
	// Unique name reference for this dataset
	Name string `json:"name,omitempty"`
	// Content-addressed path for this dataset
	Path string `json:"path,omitempty"`
	// Dataset is a pointer to the dataset being referenced
	Dataset *dataset.Dataset `json:"dataset,omitempty"`
}

DatasetRef encapsulates a reference to a dataset. This needs to exist to bind ways of referring to a dataset to a dataset itself, as datasets can't easily contain their own hash information, and names are unique on a per-repository basis. It's tempting to think this needs to be "bigger", supporting more fields, keep in mind that if the information is important at all, it should be stored as metadata within the dataset itself.

func ParseDatasetRef 

func ParseDatasetRef(ref string) (DatasetRef, error)

ParseDatasetRef decodes a dataset reference from a string value It’s possible to refer to a dataset in a number of ways. The full definition of a dataset reference is as follows: 

dataset_reference = peer_name/dataset_name@/network/hash

we swap in defaults as follows, all of which are represented as empty strings: 

network - defaults to /ipfs/
hash - tip of version history (latest known commit)

these defaults are currently enforced by convention. TODO - make Dataset Ref parsing the responisiblity of the Repo interface, replacing empty strings with actual defaults

through defaults the following should all parse: 

peer_name/dataset_name
dataset_name@hash
/network/hash
peername
hash

see tests for more exmples

dataset names & hashes are disambiguated by checking if the input parses to a valid multihash after base58 decoding

TODO - add validation that prevents peernames from being valid base58 multihashes. TODO - figure out how IPFS CID's play into this

func (DatasetRef) Equal added in v0.1.1 

func (r DatasetRef) Equal(b DatasetRef) bool

Equal returns true only if Peername Name and Path are equal

func (DatasetRef) IsEmpty added in v0.1.1 

func (r DatasetRef) IsEmpty() bool

IsEmpty returns true if none of it's fields are set

func (DatasetRef) IsPeerRef 

func (r DatasetRef) IsPeerRef() bool

IsPeerRef returns true if only Peername is set

func (DatasetRef) Match added in v0.1.1 

func (r DatasetRef) Match(b DatasetRef) bool

Match checks returns true if Peername and Name are equal, and/or path is equal

func (DatasetRef) String 

func (r DatasetRef) String() (s string)

String implements the Stringer interface for DatasetRef

type Datasets 

type Datasets interface {
	// Put a dataset in the store
	PutDataset(path datastore.Key, ds *dataset.Dataset) error
	// Put multiple datasets in the store
	PutDatasets([]*DatasetRef) error
	// Get a dataset from the store
	GetDataset(path datastore.Key) (*dataset.Dataset, error)
	// Remove a dataset from the store
	DeleteDataset(path datastore.Key) error
	// Query is extracted from the ipfs datastore interface:
	Query(query.Query) (query.Results, error)
}

Datasets is the minimum interface to act as a store of datasets. It's intended to look a *lot* like the ipfs datastore interface, but scoped only to datasets to make for easier consumption. Datasets stored here should be reasonably dereferenced to avoid additional lookups. All fields here work only with paths (which are datastore.Key's) to dereference a name, you'll need a Refstore interface oh golang, can we haz generics plz?

type MemChangeRequests 

type MemChangeRequests map[string]*ChangeRequest

MemChangeRequests is an in-memory map of change requests

func (MemChangeRequests) ChangeRequestsForTarget 

func (mcr MemChangeRequests) ChangeRequestsForTarget(target datastore.Key, limit, offset int) ([]*ChangeRequest, error)

ChangeRequestsForTarget get change requests for a given ("target") dataset

func (MemChangeRequests) DeleteChangeRequest 

func (mcr MemChangeRequests) DeleteChangeRequest(path datastore.Key) error

DeleteChangeRequest removes a change requres from the store by it's path

func (MemChangeRequests) GetChangeRequest 

func (mcr MemChangeRequests) GetChangeRequest(path datastore.Key) (*ChangeRequest, error)

GetChangeRequest fetches a change request by it's path

func (MemChangeRequests) ListChangeRequests 

func (mcr MemChangeRequests) ListChangeRequests(limit, offset int) ([]*ChangeRequest, error)

ListChangeRequests enumerates change requests in this store

func (MemChangeRequests) PutChangeRequest 

func (mcr MemChangeRequests) PutChangeRequest(path datastore.Key, cr *ChangeRequest) error

PutChangeRequest adds a change request to the store

type MemDatasets 

type MemDatasets struct {
	// contains filtered or unexported fields
}

MemDatasets is an in-memory implementation of the DatasetStore interface

func NewMemDatasets 

func NewMemDatasets(store cafs.Filestore) MemDatasets

NewMemDatasets creates a datasets instance from a cafs.Filstore

func (MemDatasets) DeleteDataset 

func (d MemDatasets) DeleteDataset(path datastore.Key) error

DeleteDataset removes a dataset from the store

func (MemDatasets) GetDataset 

func (d MemDatasets) GetDataset(path datastore.Key) (*dataset.Dataset, error)

GetDataset fetches a dataset from the store

func (MemDatasets) PutDataset 

func (d MemDatasets) PutDataset(path datastore.Key, ds *dataset.Dataset) error

PutDataset adds a dataset to the store, referenced by a given key

func (MemDatasets) PutDatasets 

func (d MemDatasets) PutDatasets(datasets []*DatasetRef) error

PutDatasets adds multiple dataset references to the store

func (MemDatasets) Query 

func (d MemDatasets) Query(q query.Query) (query.Results, error)

Query the store for dataset results

type MemPeers 

type MemPeers map[peer.ID]*profile.Profile

MemPeers is an in-memory implementation of the Peers interface

func (MemPeers) DeletePeer 

func (m MemPeers) DeletePeer(id peer.ID) error

DeletePeer removes a peer from this store

func (MemPeers) GetID 

func (m MemPeers) GetID(peername string) (peer.ID, error)

GetID gives the peer.ID for a given peername

func (MemPeers) GetPeer 

func (m MemPeers) GetPeer(id peer.ID) (*profile.Profile, error)

GetPeer give's peer info from the store for a given peer.ID

func (MemPeers) IPFSPeerID 

func (m MemPeers) IPFSPeerID(peername string) (peer.ID, error)

IPFSPeerID gives the IPFS peer.ID for a given peername

func (MemPeers) List 

func (m MemPeers) List() (map[string]*profile.Profile, error)

List hands the full list of peers back

func (MemPeers) PutPeer 

func (m MemPeers) PutPeer(id peer.ID, profile *profile.Profile) error

PutPeer adds a peer to this store

func (MemPeers) Query 

func (m MemPeers) Query(q query.Query) (query.Results, error)

Query grabs a set of peers from this store for given query params

type MemQueryLog 

type MemQueryLog []*QueryLogItem

MemQueryLog is an in-memory implementation of the QueryLog interface

func (MemQueryLog) ListQueryLogs 

func (ql MemQueryLog) ListQueryLogs(limit, offset int) ([]*QueryLogItem, error)

ListQueryLogs grabs a set of QueryLogItems from the store

func (*MemQueryLog) LogQuery 

func (ql *MemQueryLog) LogQuery(item *QueryLogItem) error

LogQuery adds a query entry to the store

func (*MemQueryLog) QueryLogItem 

func (ql *MemQueryLog) QueryLogItem(q *QueryLogItem) (*QueryLogItem, error)

QueryLogItem fills a partial QueryLogItem with all details from the store

type MemRefstore added in v0.1.1 

type MemRefstore []DatasetRef

MemRefstore is an in-memory implementation of the Namestore interface

func (*MemRefstore) DeleteRef added in v0.1.1 

func (r *MemRefstore) DeleteRef(del DatasetRef) error

DeleteRef removes a name from the store

func (MemRefstore) GetRef added in v0.1.1 

func (r MemRefstore) GetRef(get DatasetRef) (ref DatasetRef, err error)

GetRef completes a reference with , refs can have either Path or Peername & Name specified, GetRef should fill out the missing pieces

func (*MemRefstore) PutRef added in v0.1.1 

func (r *MemRefstore) PutRef(put DatasetRef) error

PutRef adds a reference to the namestore. Only complete references may be added

func (MemRefstore) RefCount added in v0.1.1 

func (r MemRefstore) RefCount() (int, error)

RefCount returns the total number of names in the store

func (MemRefstore) References added in v0.1.1 

func (r MemRefstore) References(limit, offset int) ([]DatasetRef, error)

References grabs a set of names from the Store's namespace

type MemRepo 

type MemRepo struct {
	MemDatasets
	*MemRefstore
	*MemQueryLog
	MemChangeRequests
	// contains filtered or unexported fields
}

MemRepo is an in-memory implementation of the Repo interface

func (*MemRepo) Analytics 

func (r *MemRepo) Analytics() analytics.Analytics

Analytics returns this repo's analytics store

func (*MemRepo) Cache 

func (r *MemRepo) Cache() Datasets

Cache gives this repo's ephemeral cache of Datasets

func (*MemRepo) CreateDataset 

func (r *MemRepo) CreateDataset(ds *dataset.Dataset, data cafs.File, pin bool) (path datastore.Key, err error)

CreateDataset initializes a dataset from a dataset pointer and data file

func (*MemRepo) Graph 

func (r *MemRepo) Graph() (map[string]*dsgraph.Node, error)

Graph gives the graph of objects in this repo

func (*MemRepo) Peers 

func (r *MemRepo) Peers() Peers

Peers gives this repo's Peer interface implementation

func (*MemRepo) Profile 

func (r *MemRepo) Profile() (*profile.Profile, error)

Profile returns the peer profile for this repository

func (*MemRepo) SaveProfile 

func (r *MemRepo) SaveProfile(p *profile.Profile) error

SaveProfile updates this repo's profile

func (*MemRepo) SetPrivateKey 

func (r *MemRepo) SetPrivateKey(pk crypto.PrivKey) error

SetPrivateKey sets this repos's internal private key reference

func (*MemRepo) Store 

func (r *MemRepo) Store() cafs.Filestore

Store returns the underlying cafs.Filestore for this repo

type NodeList 

type NodeList struct {
	Nodes map[string]*dsgraph.Node
}

NodeList is a collection of nodes

type Peers 

type Peers interface {
	List() (map[string]*profile.Profile, error)
	Query(query.Query) (query.Results, error)
	IPFSPeerID(peername string) (peer.ID, error)
	GetID(peername string) (peer.ID, error)
	PutPeer(id peer.ID, profile *profile.Profile) error
	GetPeer(id peer.ID) (*profile.Profile, error)
	DeletePeer(id peer.ID) error
}

Peers is a store of peer information It's named peers to disambiguate from the lib-p2p peerstore

type QueryLog 

type QueryLog interface {
	LogQuery(*QueryLogItem) error
	ListQueryLogs(limit, offset int) ([]*QueryLogItem, error)
	QueryLogItem(q *QueryLogItem) (*QueryLogItem, error)
}

QueryLog keeps logs

type QueryLogItem 

type QueryLogItem struct {
	Query       string
	Name        string
	Key         datastore.Key
	DatasetPath datastore.Key
	Time        time.Time
}

QueryLogItem is a list of details for logging a query

type Refstore added in v0.1.1 

type Refstore interface {
	// PutRef adds a reference to the store. References must be complete with
	// Peername, Name, and Path specified
	PutRef(ref DatasetRef) error
	GetRef(ref DatasetRef) (DatasetRef, error)
	DeleteRef(ref DatasetRef) error
	References(limit, offset int) ([]DatasetRef, error)
	RefCount() (int, error)
}

Refstore keeps a collection of dataset references

type Repo 

type Repo interface {
	// All repositories wraps a content-addressed filestore as the cannonical
	// record of this repository's data. Store gives direct access to the
	// cafs.Filestore instance any given repo is using.
	Store() cafs.Filestore
	// Graph returns a graph of this repositories data resources
	Graph() (map[string]*dsgraph.Node, error)
	// All Repos must keep a Refstore, defining a given peer's datasets
	Refstore
	// CreateDataset initializes a dataset from a dataset pointer and data file
	// It's not part of the Datasets interface because creating a dataset requires
	// access to this repos store & private key
	CreateDataset(ds *dataset.Dataset, data cafs.File, pin bool) (path datastore.Key, err error)
	// Repos also serve as a store of dataset information.
	// It's important that this store maintain sync with any underlying filestore.
	// (which is why we might want to kill this in favor of just having a cache?)
	// The behaviour of the embedded DatasetStore will typically differ from the cache,
	// by only returning saved/pinned/permanent datasets.
	Datasets
	// QueryLog keeps a log of queries that have been run
	QueryLog
	// ChangeRequets gives this repo's change request store
	ChangeRequestStore
	// A repository must maintain profile information about the owner of this dataset.
	// The value returned by Profile() should represent the peer.
	Profile() (*profile.Profile, error)
	// It must be possible to alter profile information.
	SaveProfile(*profile.Profile) error
	// SetPrivateKey sets an internal reference to the private key for this profile.
	// PrivateKey is used to tie peer actions to this profile. Repo implementations must
	// never expose this private key once set.
	SetPrivateKey(pk crypto.PrivKey) error
	// A repository must maintain profile information about encountered peers.
	// Decsisions regarding retentaion of peers is left to the the implementation
	// TODO - should rename this to "profiles" to separate from the networking
	// concept of a peer
	Peers() Peers
	// Cache keeps an ephemeral store of dataset information
	// that may be purged at any moment. Results of searching for datasets,
	// dataset references other peers have, etc, should all be stored here.
	Cache() Datasets
	// All repositories provide their own analytics information.
	// Our analytics implementation is under super-active development.
	Analytics() analytics.Analytics
}

Repo is the interface for working with a qri repository qri repos are stored graph of resources:datasets, known peers, analytics data, change requests, etc. Repos are connected to a single peer profile. Repos must wrap an underlying cafs.Filestore, which is intended to act as the canonical store of state across all peers that this repo may interact with.

func NewMemRepo 

func NewMemRepo(p *profile.Profile, store cafs.Filestore, ps Peers, a analytics.Analytics) (Repo, error)

NewMemRepo creates a new in-memory repository

type SearchParams 

type SearchParams struct {
	Q             string
	Limit, Offset int
}

SearchParams encapsulates parameters provided to Searchable.Search

type Searchable 

type Searchable interface {
	Search(p SearchParams) ([]DatasetRef, error)
}

Searchable is an opt-in interface for supporting repository search

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.