README
¶
qri
Qri Backend and CLI
Welcome
| Question | Answer |
|---|---|
| "I want to learn about Qri" | Read the official documentation |
| "I want to download Qri" | Download Qri or brew install qri-io/qri/qri |
| "I have a question" | Create an issue and use the label 'question' |
| "I found a bug" | Create an issue and use the label 'bug' |
| "I want to help build the Qri backend" | Read the Contributing guides |
| "I want to build Qri from source" | Build Qri from source |
qri is a global dataset version control system (GDVCS) built on the distributed web
Breaking that down:
- global so that if anyone, anywhere has published work with the same or similar datasets, you can discover it.
- Specific to datasets because data deserves purpose-built tools
- version control to keep data in sync, attributing all changes to authors
- On the distributed web to make all of the data published on qri simultaneously available, letting peers work on data together.
If you’re unfamiliar with version control, particularly the distributed kind, well you're probably viewing this document on github — which is a version control system intended for code. Its underlying technology – git – popularized some magic sauce that has inspired a generation of programmers and popularized concepts at the heart of the distributed web. Qri is applying that family of concepts to four common data problems:
- Discovery Can I find data I’m looking for?
- Trust Can I trust what I’ve found?
- Friction Can I make this work with my other stuff?
- Sync How do I handle changes in data?
Because qri is global and content-addressed, adding data to qri also checks the entire network to see if someone has added it before. Since qri is focused solely on datasets, it can provide meaningful search results. Every change on qri is associated with a peer, creating an audit-able trail you can use to quickly see what has changed and who has changed it. All datasets on qri are automatically described at the time of ingest using a flexible schema that makes data naturally inter-operate. Qri comes with tools to turn all datasets on the network into a JSON API with a single command. Finally, all changes in qri are tracked & synced.
Packages
Qri is comprised of many specialized packages. Below you will find a summary of each package.
| Package | Go Docs | Go Report Card | Description |
|---|---|---|---|
actions |
|
functions that call to the repo to carry out tasks | |
api |
|
user accessible layer, primarily made for communication with our frontend webapp | |
cmd |
|
our command line interface | |
config |
|
user configuration details, includes peer's profile | |
lib |
|
takes arguments from the cmd and api layer and forms proper requests to call to the action layer | |
p2p |
|
the peer to peer communication layer of qri | |
repo |
|
the repository: saving, removing, and storing datasets, profiles, and the config | |
dataset |
 |
the blueprint for a dataset, the atoms that make up qri | |
registry |
 |
the blueprint for a registry: the service that allows profiles to be unique and datasets to be searchable | |
starlib |
 |
the starlark standard library available for qri transform scripts | |
qfs |
 |
"qri file sytem" is Qri's file system abstraction for getting & storing data from different sources | |
ioes |
 |
package to handle in, out, and error streams: gives us better control of where we send output and errors | |
jsonschema |
 |
used to describe the structure of a dataset, so we can validate datasets and determine dataset interop |
Outside Libraries
The following packages are not under Qri, but are important dependencies, so we display their latest versions for convenience.
| Package | Version |
|---|---|
ipfs |
 |
Building From Source
To build qri you'll need the go programming language on your machine. We require at least go version 1.12 to build qri.
Building then depends upon your operating system:
Mac OSX
Having go installed is enough, proceed to building.
Windows
Unfortunately, we do not have distributable single exe releases of qri yet, but those are planned. For now, you will need to build qri from source.
To start, make sure that you have enabled Developer Mode. A library that we depend on needs it enabled in order to properly handle symlinks. If not done, you'll likely get the error message "A required privilege is not held by the client".
You should not need to Run As Administrator to build or run qri. We do not recommend using administrator to run qri.
Shell
For your shell, we recommend using msys2. Other shells, such as cmd, Powershell, or cygwin may also be usable, but msys2 makes it easy to install our required dependencies. IPFS also recommends msys2, and qri is built on top of IPFS.
Dependencies
Building depends upon having git and make installed. If using msys2, you can easily install these by using the package manager "pacman". In a shell, type:
pacman -S git make
Assuming you've also installed go using the official Windows installer linked above, you will also need to add go to your PATH by modifying your environment variable.
Environment variables
To set Environment variables, open Windows Settings, and search for "Environment variables". This will open System Properties -> Advanced; on this dialog click "Environment Variables...". Here you can edit PATH by adding a new entry. The value of PATH is multiple locations on your file system, separated by a semi-colon (;) character. Using the list based editor will insert these semi-colons for you.
Due to how msys2 treats the PATH variable, you also need to add a new environment variable MSYS2_PATH_TYPE, with the value inherit, using the same procedure.
Alternatively, you can assign environment variables by modifying .bash_profile in msys2. Lookup bash documentation for more information on this approach.
You will also need to set your GOPATH using these same methods. Read the next section for more information.
Go workspaces
If you are new to developing go, you should know that go prefers to work a bit differently than most languages. Please read this guide How to Write Go Code, so that you understand how to create a workspace. Once you have your workspace decided upon, set your GOPATH and PATH variables setup as described in that guide.
Let's say, for example, that your workspace lives at "c:\Users\me\go", and go was installed at "c:\Go". Then you should have your GOPATH be equal to "c:\Users\me\go" and your PATH something like "...;c:\Go;$GOPATH\bin".
Once these steps are complete, proceed to building.
Linux
On a Raspberry PI, you'll need to increase your swap file size in order to build. Normal desktop and server linux OSes should be fine to proceed to building.
One symptom of having not enough swap space is the go install command producing an error message ending with:
link: signal: killed
To increase your swapfile size, first turn off the swapfile:
sudo dphys-swapfile swapoff
Then edit /etc/dphys-swapfile as root and set CONF_SWAPSIZE to 1024.
Finally turn on the swapfile again:
sudo dphys-swapfile swapon
Otherwise linux machines with reduced memory will have other ways to increase their swap file sizes. Check documentation for your particular machine.
Building
In your shell, navigate to your go workspace, and get the source code for this repository.
$ cd $GOPATH
$ mkdir -p src/github.com/qri-io
$ cd src/github.com/qri-io
$ git clone https://github.com/qri-io/qri
Now enter this main source repo, make sure go modules are enabled, and finally build:
$ cd qri
$ export GO111MODULE=on
$ make build
The make build command will have a lot of output. That's good! Its means it's working :)
It'll take a minute, but once everything's finished a new binary qri will appear in the $GOPATH/bin directory. You should be able to run:
$ qri help
and see help output.
Documentation
¶
Overview ¶
Qri is a distributed dataset version control tool. Bigger than a spreadsheet, smaller than a database, datasets are all around us. Use Qri to browse, download, create, fork, and publish datasets on a peer-to-peer network that works both on and offline.
more info at: https://qri.io
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package actions provides canonical business logic that operates on Repos to get higher-order functionality.
|
Package actions provides canonical business logic that operates on Repos to get higher-order functionality. |
|
Package api implements a JSON-API for interacting with a qri node
|
Package api implements a JSON-API for interacting with a qri node |
|
Package base defines business that operates on local data.
|
Package base defines business that operates on local data. |
|
fill
Package fill assigns arbitrary values to struct fields using reflection.
|
Package fill assigns arbitrary values to struct fields using reflection. |
|
Package cmd defines the CLI interface.
|
Package cmd defines the CLI interface. |
|
generate
Package generate is a command that creates a bash completion file for qri
|
Package generate is a command that creates a bash completion file for qri |
|
Package config encapsulates qri configuration options & details.
|
Package config encapsulates qri configuration options & details. |
|
migrate
Package migrate defines migrations for qri configuration files
|
Package migrate defines migrations for qri configuration files |
|
test
Package test contains predefined set of PeerInfo for testing.
|
Package test contains predefined set of PeerInfo for testing. |
|
Docs is a tool for generating markdown documentation of the qri command line interface (CLI)
|
Docs is a tool for generating markdown documentation of the qri command line interface (CLI) |
|
Package fsi defines qri file system integration: representing a dataset as files in a directory on a user's computer.
|
Package fsi defines qri file system integration: representing a dataset as files in a directory on a user's computer. |
|
Package lib implements core qri business logic.
|
Package lib implements core qri business logic. |
|
test
Package test defines utilities for testing the lib package, including caches of expensive processes like cryptographic key generation and ipfs repo creation
|
Package test defines utilities for testing the lib package, including caches of expensive processes like cryptographic key generation and ipfs repo creation |
|
Package p2p implements qri peer-to-peer communication.
|
Package p2p implements qri peer-to-peer communication. |
|
test
Package p2ptest defines utilities for qri peer-2-peer testing
|
Package p2ptest defines utilities for qri peer-2-peer testing |
|
Package registry defines primitives for keeping centralized repositories of qri types (peers, datasets, etc).
|
Package registry defines primitives for keeping centralized repositories of qri types (peers, datasets, etc). |
|
regclient
Package regclient defines a client for interacting with a registry server
|
Package regclient defines a client for interacting with a registry server |
|
regserver
Package regserver is a wrapper around the handlers package, turning it into a proper http server
|
Package regserver is a wrapper around the handlers package, turning it into a proper http server |
|
regserver/handlers
Package handlers creates HTTP handler functions for registry interface implementations
|
Package handlers creates HTTP handler functions for registry interface implementations |
|
regserver/mock
Package mock provides a mock registry server for testing purposes it mocks the behaviour of a registry server with in-memory storage
|
Package mock provides a mock registry server for testing purposes it mocks the behaviour of a registry server with in-memory storage |
|
Package remote implements syncronization between qri instances
|
Package remote implements syncronization between qri instances |
|
Package repo represents a repository of qri information Analogous to a git repository, repo expects a rigid structure filled with rich types specific to qri.
|
Package repo represents a repository of qri information Analogous to a git repository, repo expects a rigid structure filled with rich types specific to qri. |
|
fs
Package fsrepo is a file-system implementation of repo
|
Package fsrepo is a file-system implementation of repo |
|
gen
Package gen contains routines that perform expensive cryptographic operations.
|
Package gen contains routines that perform expensive cryptographic operations. |
|
profile
Package profile defines a qri peer profile
|
Package profile defines a qri peer profile |
|
search
Package search creates a local search index of a repo using the bleve search engine
|
Package search creates a local search index of a repo using the bleve search engine |
|
test
Package test contains a set of tests to ensure a repo implementation conforms to expected behaviors, calling RunRepoTests on a given repo implementation should pass all checks in order to properly work with Qri.
|
Package test contains a set of tests to ensure a repo implementation conforms to expected behaviors, calling RunRepoTests on a given repo implementation should pass all checks in order to properly work with Qri. |
|
Package rev defines structure and syntax for specifying revisions of a dataset history.
|
Package rev defines structure and syntax for specifying revisions of a dataset history. |
|
Package startf implements dataset transformations using the starlark programming dialect For more info on starlark check github.com/google/starlark
|
Package startf implements dataset transformations using the starlark programming dialect For more info on starlark check github.com/google/starlark |
|
ds
Package ds exposes the qri dataset document model into starlark Package ds defines the qri dataset object within starlark outline: ds ds defines the qri dataset object within starlark.
|
Package ds exposes the qri dataset document model into starlark Package ds defines the qri dataset object within starlark outline: ds ds defines the qri dataset object within starlark. |
|
Package update defines the update service
|
Package update defines the update service |
|
cron
Package cron schedules dataset and shell script updates
|
Package cron schedules dataset and shell script updates |