Documentation ¶
Overview ¶
Package logbook records and syncs dataset histories. As users work on datasets, they build of a log of operations. Each operation is a record of an action taken, like creating a dataset, or unpublishing a version. Each of these operations is wrtten to a log attributed to the user that performed the action, and stored in the logbook under the namespace of that dataset. The current state of a user's log is derived from iterating over all operations to produce the current state.
Example ¶
package main import ( "context" "encoding/base64" "fmt" "time" crypto "github.com/libp2p/go-libp2p-core/crypto" "github.com/qri-io/dataset" "github.com/qri-io/qfs" "github.com/qri-io/qri/dsref" "github.com/qri-io/qri/event" "github.com/qri-io/qri/logbook" ) func main() { // background context to play with ctx := context.Background() // logbooks are encrypted at rest, we need a private key to interact with // them, including to create a new logbook. This is a dummy Private Key // you should never, ever use in real life. demo only folks. testPk := `CAASpgkwggSiAgEAAoIBAQC/7Q7fILQ8hc9g07a4HAiDKE4FahzL2eO8OlB1K99Ad4L1zc2dCg+gDVuGwdbOC29IngMA7O3UXijycckOSChgFyW3PafXoBF8Zg9MRBDIBo0lXRhW4TrVytm4Etzp4pQMyTeRYyWR8e2hGXeHArXM1R/A/SjzZUbjJYHhgvEE4OZy7WpcYcW6K3qqBGOU5GDMPuCcJWac2NgXzw6JeNsZuTimfVCJHupqG/dLPMnBOypR22dO7yJIaQ3d0PFLxiDG84X9YupF914RzJlopfdcuipI+6gFAgBw3vi6gbECEzcohjKf/4nqBOEvCDD6SXfl5F/MxoHurbGBYB2CJp+FAgMBAAECggEAaVOxe6Y5A5XzrxHBDtzjlwcBels3nm/fWScvjH4dMQXlavwcwPgKhy2NczDhr4X69oEw6Msd4hQiqJrlWd8juUg6vIsrl1wS/JAOCS65fuyJfV3Pw64rWbTPMwO3FOvxj+rFghZFQgjg/i45uHA2UUkM+h504M5Nzs6Arr/rgV7uPGR5e5OBw3lfiS9ZaA7QZiOq7sMy1L0qD49YO1ojqWu3b7UaMaBQx1Dty7b5IVOSYG+Y3U/dLjhTj4Hg1VtCHWRm3nMOE9cVpMJRhRzKhkq6gnZmni8obz2BBDF02X34oQLcHC/Wn8F3E8RiBjZDI66g+iZeCCUXvYz0vxWAQQKBgQDEJu6flyHPvyBPAC4EOxZAw0zh6SF/r8VgjbKO3n/8d+kZJeVmYnbsLodIEEyXQnr35o2CLqhCvR2kstsRSfRz79nMIt6aPWuwYkXNHQGE8rnCxxyJmxV4S63GczLk7SIn4KmqPlCI08AU0TXJS3zwh7O6e6kBljjPt1mnMgvr3QKBgQD6fAkdI0FRZSXwzygx4uSg47Co6X6ESZ9FDf6ph63lvSK5/eue/ugX6p/olMYq5CHXbLpgM4EJYdRfrH6pwqtBwUJhlh1xI6C48nonnw+oh8YPlFCDLxNG4tq6JVo071qH6CFXCIank3ThZeW5a3ZSe5pBZ8h4bUZ9H8pJL4C7yQKBgFb8SN/+/qCJSoOeOcnohhLMSSD56MAeK7KIxAF1jF5isr1TP+rqiYBtldKQX9bIRY3/8QslM7r88NNj+aAuIrjzSausXvkZedMrkXbHgS/7EAPflrkzTA8fyH10AsLgoj/68mKr5bz34nuY13hgAJUOKNbvFeC9RI5g6eIqYH0FAoGAVqFTXZp12rrK1nAvDKHWRLa6wJCQyxvTU8S1UNi2EgDJ492oAgNTLgJdb8kUiH0CH0lhZCgr9py5IKW94OSM6l72oF2UrS6PRafHC7D9b2IV5Al9lwFO/3MyBrMocapeeyaTcVBnkclz4Qim3OwHrhtFjF1ifhP9DwVRpuIg+dECgYANwlHxLe//tr6BM31PUUrOxP5Y/cj+ydxqM/z6papZFkK6Mvi/vMQQNQkh95GH9zqyC5Z/yLxur4ry1eNYty/9FnuZRAkEmlUSZ/DobhU0Pmj8Hep6JsTuMutref6vCk2n02jc9qYmJuD7iXkdXDSawbEG6f5C4MUkJ38z1t1OjA==` data, err := base64.StdEncoding.DecodeString(testPk) if err != nil { panic(err) } pk, err := crypto.UnmarshalPrivateKey(data) if err != nil { panic(fmt.Errorf("error unmarshaling private key: %s", err.Error())) } // logbook relies on a qfs.Filesystem for read & write. create an in-memory // filesystem we can play with fs := qfs.NewMemFS() // Create a new journal for b5, passing in: // * the author private key to encrypt & decrypt the logbook // * author's current username // * an event bus (not used in this example) // * a qfs.Filesystem for reading & writing the logbook // * a base path on the filesystem to read & write the logbook to // Initializing a logbook ensures the author has an user opset that matches // their current state. It will error if a stored book can't be decrypted book, err := logbook.NewJournal(pk, "b5", event.NilBus, fs, "/mem/logbook.qfb") if err != nil { panic(err) // real programs don't panic } // create a name to store dataset versions in. NameInit will create a new // log under the logbook author's namespace with the given name, and an opset // that tracks operations by this author within that new namespace. // The entire logbook is persisted to the filestore after each operation initID, err := book.WriteDatasetInit(ctx, "world_bank_population") if err != nil { panic(err) } // pretend we've just created a dataset, these are the only fields the log // will care about ds := &dataset.Dataset{ Peername: "b5", Name: "world_bank_population", Commit: &dataset.Commit{ Timestamp: time.Date(2000, time.January, 1, 0, 0, 0, 0, time.UTC), Title: "initial commit", }, Path: "QmHashOfVersion1", PreviousPath: "", // TODO (b5) - at some point we may want to log parent versions as well, // need to model those properly first. } // create a log record of the version of a dataset. In practice this'll be // part of the overall save routine that created the above ds variable if err := book.WriteVersionSave(ctx, initID, ds); err != nil { panic(err) } // sometime later, we create another version ds2 := &dataset.Dataset{ Peername: "b5", Name: "world_bank_population", Commit: &dataset.Commit{ Timestamp: time.Date(2000, time.January, 2, 0, 0, 0, 0, time.UTC), Title: "added body data", }, Structure: &dataset.Structure{ Length: 100, }, Path: "QmHashOfVersion2", PreviousPath: "QmHashOfVersion1", } // once again, write to the log if err := book.WriteVersionSave(ctx, initID, ds2); err != nil { panic(err) } ref := dsref.Ref{ Username: "b5", Name: "world_bank_population", } // pretend we just published both saved versions of the dataset to the // registry we log that here. Providing a revisions arg of 2 means we've // published two consecutive revisions from head: the latest version, and the // one before it. "registry.qri.cloud" indicates we published to a remote // with that address if _, _, err := book.WriteRemotePush(ctx, initID, 2, "registry.qri.cloud"); err != nil { panic(err) } // pretend the user just deleted a dataset version, well, we need to log it! // VersionDelete accepts an argument of number of versions back from HEAD // more complex deletes that remove pieces of history may require either // composing multiple log operations book.WriteVersionDelete(ctx, initID, 1) // create another version ds3 := &dataset.Dataset{ Peername: "b5", Name: "world_bank_population", Commit: &dataset.Commit{ Timestamp: time.Date(2000, time.January, 3, 0, 0, 0, 0, time.UTC), Title: "added meta info", }, Structure: &dataset.Structure{ Length: 100, }, Path: "QmHashOfVersion3", // note that we're referring to version 1 here. version 2 no longer exists // this is happening outside of the log, but the log should reflect // contiguous history PreviousPath: "QmHashOfVersion1", } // once again, write to the log if err := book.WriteVersionSave(ctx, initID, ds3); err != nil { panic(err) } // now for the fun bit. When we ask for the state of the log, it will // play our opsets forward and get us the current state of tne log // we can also get the state of a log from the book: log, err := book.Items(ctx, ref, 0, 100) if err != nil { panic(err) } for _, info := range log { fmt.Println(info.SimpleRef().String()) } }
Output: b5/world_bank_population@QmHashOfVersion3 b5/world_bank_population@QmHashOfVersion1
Index ¶
- Constants
- Variables
- func DsrefAliasForLog(log *oplog.Log) (dsref.Ref, error)
- func ModelString(m uint32) string
- type Book
- func (book *Book) ActivePeerID(ctx context.Context) (id string, err error)
- func (book *Book) AllReferencedDatasetPaths(ctx context.Context) (map[string]struct{}, error)
- func (book *Book) Author() identity.Author
- func (book *Book) AuthorID() string
- func (book *Book) AuthorName() string
- func (book *Book) AuthorPubKey() crypto.PubKey
- func (book Book) BranchRef(ctx context.Context, ref dsref.Ref) (*oplog.Log, error)
- func (book *Book) ConstructDatasetLog(ctx context.Context, ref dsref.Ref, history []*dataset.Dataset) error
- func (book Book) DatasetRef(ctx context.Context, ref dsref.Ref) (*oplog.Log, error)
- func (book *Book) DeleteAuthor() error
- func (book Book) Items(ctx context.Context, ref dsref.Ref, offset, limit int) ([]DatasetLogItem, error)
- func (book Book) ListAllLogs(ctx context.Context) ([]*oplog.Log, error)
- func (book Book) Log(ctx context.Context, id string) (*oplog.Log, error)
- func (book Book) LogBytes(log *oplog.Log) ([]byte, error)
- func (book Book) LogEntries(ctx context.Context, ref dsref.Ref, offset, limit int) ([]LogEntry, error)
- func (book *Book) MergeLog(ctx context.Context, sender identity.Author, lg *oplog.Log) error
- func (book Book) PlainLogs(ctx context.Context) ([]PlainLog, error)
- func (book *Book) RefToInitID(ref dsref.Ref) (string, error)
- func (book *Book) RemoveLog(ctx context.Context, ref dsref.Ref) error
- func (book *Book) RenameAuthor() error
- func (book *Book) ResolveRef(ctx context.Context, ref *dsref.Ref) (string, error)
- func (book Book) SignLog(log *oplog.Log) error
- func (book Book) SummaryString(ctx context.Context) string
- func (book Book) UserDatasetBranchesLog(ctx context.Context, datasetInitID string) (*oplog.Log, error)
- func (book *Book) WriteAuthorRename(ctx context.Context, newName string) error
- func (book *Book) WriteDatasetDelete(ctx context.Context, initID string) error
- func (book *Book) WriteDatasetInit(ctx context.Context, dsName string) (string, error)
- func (book *Book) WriteDatasetRename(ctx context.Context, initID string, newName string) error
- func (book *Book) WriteRemoteDelete(ctx context.Context, initID string, revisions int, remoteAddr string) (l *oplog.Log, rollback func(ctx context.Context) error, err error)
- func (book *Book) WriteRemotePush(ctx context.Context, initID string, revisions int, remoteAddr string) (l *oplog.Log, rollback func(ctx context.Context) error, err error)
- func (book *Book) WriteVersionAmend(ctx context.Context, initID string, ds *dataset.Dataset) error
- func (book *Book) WriteVersionDelete(ctx context.Context, initID string, revisions int) error
- func (book *Book) WriteVersionSave(ctx context.Context, initID string, ds *dataset.Dataset) error
- type BookBuilder
- func (b *BookBuilder) AddForeign(ctx context.Context, t *testing.T, log *oplog.Log)
- func (b *BookBuilder) Commit(ctx context.Context, t *testing.T, initID, title, ipfsHash string) dsref.Ref
- func (b *BookBuilder) DatasetDelete(ctx context.Context, t *testing.T, initID string)
- func (b *BookBuilder) DatasetInit(ctx context.Context, t *testing.T, dsname string) string
- func (b *BookBuilder) DatasetRename(ctx context.Context, t *testing.T, initID, newName string) dsref.Ref
- func (b *BookBuilder) Delete(ctx context.Context, t *testing.T, initID string, num int) dsref.Ref
- func (b *BookBuilder) Logbook() *Book
- type BranchLog
- type DatasetLog
- type DatasetLogItem
- type LogEntry
- type PlainLog
- type PlainOp
- type UserLog
Examples ¶
Constants ¶
const ( // AuthorModel is the enum for an author model AuthorModel uint32 = iota // DatasetModel is the enum for a dataset model DatasetModel // BranchModel is the enum for a branch model BranchModel // CommitModel is the enum for a commit model CommitModel // PushModel is the enum for a push model PushModel // ACLModel is the enum for a acl model ACLModel )
const DefaultBranchName = "main"
DefaultBranchName is the default name all branch-level logbook data is read from and written to. we currently don't present branches as a user-facing feature in qri, but logbook supports them
Variables ¶
var ( // ErrNoLogbook indicates a logbook doesn't exist ErrNoLogbook = fmt.Errorf("logbook: does not exist") // ErrNotFound is a sentinel error for data not found in a logbook ErrNotFound = fmt.Errorf("logbook: not found") // ErrLogTooShort indicates a log is missing elements. Because logs are // append-only, passing a shorter log than the one on file is grounds // for rejection ErrLogTooShort = fmt.Errorf("logbook: log is too short") // ErrAccessDenied indicates insufficent privileges to perform a logbook // operation ErrAccessDenied = fmt.Errorf("access denied") // NewTimestamp generates the current unix nanosecond time. // This is mainly here for tests to override NewTimestamp = func() int64 { return time.Now().UnixNano() } )
Functions ¶
func DsrefAliasForLog ¶ added in v0.9.1
DsrefAliasForLog parses log data into a dataset alias reference, populating only the username and name components of a dataset. the passed in oplog must refer unambiguously to a dataset or branch. book.Log() returns exact log references
func ModelString ¶ added in v0.9.3
ModelString gets a unique string descriptor for an integral model identifier
Types ¶
type Book ¶
type Book struct {
// contains filtered or unexported fields
}
Book wraps a oplog.Book with a higher-order API specific to Qri
func NewJournal ¶ added in v0.9.3
func NewJournal(pk crypto.PrivKey, username string, bus event.Bus, fs qfs.Filesystem, location string) (*Book, error)
NewJournal initializes a logbook owned by a single author, reading any existing data at the given filesystem location. logbooks are encrypted at rest with the given private key
func (*Book) ActivePeerID ¶ added in v0.9.1
ActivePeerID returns the in-use PeerID of the logbook author TODO (b5) - remove the need for this context by caching the active PeerID at key load / save / mutation points
func (*Book) AllReferencedDatasetPaths ¶ added in v0.9.9
AllReferencedDatasetPaths scans an entire logbook looking for dataset paths
func (*Book) AuthorName ¶ added in v0.9.1
AuthorName returns the human-readable name of the author
func (*Book) AuthorPubKey ¶ added in v0.9.3
AuthorPubKey gives this book's author public key
func (Book) BranchRef ¶ added in v0.9.1
BranchRef gets a branch log for a dataset reference. Branch logs describe a line of commits
currently all logs are hardcoded to only accept one branch name. This function always returns
TODO(dustmop): Do not add new callers to this, transition away (preferring branchLog instead), and delete it.
func (*Book) ConstructDatasetLog ¶ added in v0.9.1
func (book *Book) ConstructDatasetLog(ctx context.Context, ref dsref.Ref, history []*dataset.Dataset) error
ConstructDatasetLog creates a sparse log from a connected dataset history where no prior log exists the given history MUST be ordered from oldest to newest commits TODO (b5) - this presently only works for datasets in an author's user namespace
func (Book) DatasetRef ¶ added in v0.9.1
DatasetRef gets a dataset log and all branches. Dataset logs describe activity affecting an entire dataset. Things like dataset name changes and access control changes are kept in the dataset log
currently all logs are hardcoded to only accept one branch name. This function will always return a single branch
TODO(dustmop): Do not add new callers to this, transition away (preferring datasetLog instead), and delete it.
func (*Book) DeleteAuthor ¶
DeleteAuthor removes an author, used on teardown
func (Book) Items ¶ added in v0.9.7
func (book Book) Items(ctx context.Context, ref dsref.Ref, offset, limit int) ([]DatasetLogItem, error)
Items collapses the history of a dataset branch into linear log items
func (Book) ListAllLogs ¶ added in v0.9.5
ListAllLogs lists all of the logs in the logbook
func (Book) LogEntries ¶ added in v0.9.1
func (book Book) LogEntries(ctx context.Context, ref dsref.Ref, offset, limit int) ([]LogEntry, error)
LogEntries returns a summarized "line-by-line" representation of a log for a given dataset reference
func (*Book) MergeLog ¶ added in v0.9.1
MergeLog adds a log to the logbook, merging with any existing log data
func (Book) PlainLogs ¶ added in v0.9.5
PlainLogs returns plain-old-data representations of the logs, intended for serialization
func (*Book) RefToInitID ¶ added in v0.9.9
RefToInitID converts a dsref to an initID by iterating the entire logbook looking for a match. This function is inefficient, iterating the entire set of operations in a log. Replacing this function call with mechanisms in dscache will fix this problem. TODO(dustmop): Don't depend on this function permanently, use a higher level resolver and convert all callers of this function to use that resolver's initID instead of converting a dsref yet again.
func (*Book) RenameAuthor ¶
RenameAuthor marks a change in author name
func (*Book) ResolveRef ¶ added in v0.9.9
ResolveRef finds the identifier & head path for a dataset reference implements resolve.NameResolver interface
func (Book) SignLog ¶ added in v0.9.9
SignLog populates the signature field of a log using the author's private key
func (Book) SummaryString ¶ added in v0.9.9
SummaryString prints the entire hierarchy of logbook model/ID/opcount/name in a single string
func (Book) UserDatasetBranchesLog ¶ added in v0.9.9
func (book Book) UserDatasetBranchesLog(ctx context.Context, datasetInitID string) (*oplog.Log, error)
UserDatasetBranchesLog gets a user's log and a dataset reference. the returned log will be a user log with only one dataset log containing all known branches:
user dataset branch branch ...
func (*Book) WriteAuthorRename ¶ added in v0.9.4
WriteAuthorRename adds an operation updating the author's username
func (*Book) WriteDatasetDelete ¶ added in v0.9.1
WriteDatasetDelete closes a dataset, marking it as deleted
func (*Book) WriteDatasetInit ¶ added in v0.9.1
WriteDatasetInit initializes a new dataset name within the author's namespace
func (*Book) WriteDatasetRename ¶ added in v0.9.1
WriteDatasetRename marks renaming a dataset
func (*Book) WriteRemoteDelete ¶ added in v0.9.9
func (book *Book) WriteRemoteDelete(ctx context.Context, initID string, revisions int, remoteAddr string) (l *oplog.Log, rollback func(ctx context.Context) error, err error)
WriteRemoteDelete adds an operation to a log marking an unpublish request for a count of sequential versions from HEAD
func (*Book) WriteRemotePush ¶ added in v0.9.9
func (book *Book) WriteRemotePush(ctx context.Context, initID string, revisions int, remoteAddr string) (l *oplog.Log, rollback func(ctx context.Context) error, err error)
WriteRemotePush adds an operation to a log marking the publication of a number of versions to a remote address. It returns a rollback function that removes the operation when called
func (*Book) WriteVersionAmend ¶
WriteVersionAmend adds an operation to a log when a dataset amends a commit TODO(dustmop): Currently unused by codebase, only called in tests.
func (*Book) WriteVersionDelete ¶
WriteVersionDelete adds an operation to a log marking a number of sequential versions from HEAD as deleted. Because logs are append-only, deletes are recorded as "tombstone" operations that mark removal.
type BookBuilder ¶ added in v0.9.7
BookBuilder builds a logbook in a convenient way
func NewLogbookTempBuilder ¶ added in v0.9.7
func NewLogbookTempBuilder(t *testing.T, privKey crypto.PrivKey, username string, fs qfs.Filesystem, rootPath string) BookBuilder
NewLogbookTempBuilder constructs a logbook tmp BookBuilder
func (*BookBuilder) AddForeign ¶ added in v0.9.7
AddForeign merges a foreign log into this book
func (*BookBuilder) Commit ¶ added in v0.9.7
func (b *BookBuilder) Commit(ctx context.Context, t *testing.T, initID, title, ipfsHash string) dsref.Ref
Commit adds a commit to a dataset
func (*BookBuilder) DatasetDelete ¶ added in v0.9.7
DatasetDelete deletes a dataset
func (*BookBuilder) DatasetInit ¶ added in v0.9.7
DatasetInit initializes a new dataset and return a reference to it
func (*BookBuilder) DatasetRename ¶ added in v0.9.7
func (b *BookBuilder) DatasetRename(ctx context.Context, t *testing.T, initID, newName string) dsref.Ref
DatasetRename changes the name of a dataset
func (*BookBuilder) Logbook ¶ added in v0.9.7
func (b *BookBuilder) Logbook() *Book
Logbook returns the built logbook
type BranchLog ¶ added in v0.9.9
type BranchLog struct {
// contains filtered or unexported fields
}
BranchLog is the bottom-level log representing a branch of a dataset history
type DatasetLog ¶ added in v0.9.9
type DatasetLog struct {
// contains filtered or unexported fields
}
DatasetLog is the mid-level log representing a single dataset
func (*DatasetLog) Append ¶ added in v0.9.9
func (dlog *DatasetLog) Append(op oplog.Op)
Append adds an op to the DatasetLog
func (*DatasetLog) InitID ¶ added in v0.9.9
func (dlog *DatasetLog) InitID() string
InitID returns the initID for the dataset
type DatasetLogItem ¶ added in v0.9.7
type DatasetLogItem struct { // Decription of a dataset reference dsref.VersionInfo // Title field from the commit CommitTitle string `json:"commitTitle,omitempty"` // Message field from the commit CommitMessage string `json:"commitMessage,omitempty"` }
DatasetLogItem is a line item in a dataset response
func ConvertLogsToItems ¶ added in v0.9.9
func ConvertLogsToItems(l *oplog.Log, ref dsref.Ref) []DatasetLogItem
ConvertLogsToItems collapses the history of a dataset branch into linear log items
type PlainLog ¶ added in v0.9.5
type PlainLog struct { Ops []PlainOp `json:"ops,omitempty"` Logs []PlainLog `json:"logs,omitempty"` }
PlainLog is a human-oriented representation of oplog.Log intended for serialization
func NewPlainLog ¶ added in v0.9.5
NewPlainLog converts an oplog to a plain log
type PlainOp ¶ added in v0.9.5
type PlainOp struct { // type of operation Type string `json:"type,omitempty"` // data model to operate on Model string `json:"model,omitempty"` // identifier of data this operation is documenting Ref string `json:"ref,omitempty"` // previous reference in a causal history Prev string `json:"prev,omitempty"` // references this operation relates to. usage is operation type-dependant Relations []string `json:"relations,omitempty"` // human-readable name for the reference Name string `json:"name,omitempty"` // identifier for author AuthorID string `json:"authorID,omitempty"` // operation timestamp, for annotation purposes only Timestamp time.Time `json:"timestamp,omitempty"` // size of the referenced value in bytes Size int64 `json:"size,omitempty"` // operation annotation for users. eg: commit title Note string `json:"note,omitempty"` }
PlainOp is a human-oriented representation of oplog.Op intended for serialization
type UserLog ¶ added in v0.9.9
type UserLog struct {
// contains filtered or unexported fields
}
UserLog is the top-level log representing users that make datasets
func (*UserLog) AddChild ¶ added in v0.9.9
AddChild adds a child log TODO(dustmop): Change this parameter to be a DatasetLog
Directories ¶
Path | Synopsis |
---|---|
Package logsync synchronizes logs between logbooks across networks
|
Package logsync synchronizes logs between logbooks across networks |
Package oplog is an operation-based replicated data type of append-only logs oplog has three main structures: logbook, log, and op A log is a sequence of operations attributed to a single author, designated by a private key.
|
Package oplog is an operation-based replicated data type of append-only logs oplog has three main structures: logbook, log, and op A log is a sequence of operations attributed to a single author, designated by a private key. |