README
¶
rio
Repeatable I/O.
Rio is tooling for repeatable, reproducible, filesystem I/O and transport. It's both a command line tool, and usable as a series of libraries.
Rio is part of the Timeless Stack -- it handles packing, unpacking, syncing, copying, and hashing of Filesets.
You may also want to check out Repeatr, which is the container executor in the Timeless Stack (it uses Rio under the hood to provide snapshotting and decentralized sync of container filesystem images); and Reach, which provides pipelining tooling for managing both Rio and Repeatr with less copy-pasting of hashes.
What?
example of packing
Okay, by example:
rio pack tar /tmp/something/ --target=file://output.tgz
That just created a tar pack of the files you aimed it at.
It also gave you a hash on stdout. That's a content-addressable ID of the pack you just created.
example of unpacking
You can use that ID later to unpack things:
rio unpack tar:879UrF8j7E[...]udF57KpF8 \
/tmp/unpackhere/ --source=file://output.tgz
It's long. That's because it's a cryptographic hash.
Why is this neat?
You can put that packed data on any server, anywhere, and fetch it again by hash. The hash makes it immutable, and reproducible, even if you don't control the storage.
example of packing to remote storage and using content-addressing
rio pack tar /tmp/something/ --target=ca+https://mybucket.s3.amazonaws.com/assets/
Same drill except... we just:
- uploaded to a cloud storage host
- without a specific name -- the hash is used to organize storage (note the "ca+" segment of the url).
- and still got the same hash.
example of unpacking from remote storage while using content-addressing
Now, hand that hash to someone else:
rio unpack tar:879UrF8j7E[...]udF57KpF8 \
/tmp/unpackhere/ --source=ca+https:///mybucket.s3.amazonaws.com/assets/
They get your files. Boom. Huge amounts of data. Just one handle to copypaste: that hash.
Building
Rio is built in Golang and uses git submodules to track libraries by hash.
To build Rio, first get the submodules, then set up GOPATH, then use go:
git submodule update --init
GOPATH=$PWD/.gopath go install ./...
You may find the gof script makes this more convenient.
Directories
¶
| Path | Synopsis |
|---|---|
|
Provides helper functions for checking if we have some functional sets of capabilities.
|
Provides helper functions for checking if we have some functional sets of capabilities. |
|
cmd
|
|
|
Helpers for loading contextual config.
|
Helpers for loading contextual config. |
|
osfs
Devmodes in most linux follows the gnu_dev_major / gnu_dev_minor library functions.
|
Devmodes in most linux follows the gnu_dev_major / gnu_dev_minor library functions. |
|
lib
|
|
|
guid
`guid.New` generates a string that functions as a unique identifier.
|
`guid.New` generates a string that functions as a unique identifier. |
|
transmat
|
|
|
git
The git transmat can unpack filesystems from the Git version control system.
|
The git transmat can unpack filesystems from the Git version control system. |
|
mixins/log
Helper functions for emitting structured logs to the rio.Monitor.
|
Helper functions for emitting structured logs to the rio.Monitor. |
|
tar
The tar transmat packs filesystems into the widely-recognized "tar" format, and can use any k/v-styled warehouse for storage.
|
The tar transmat packs filesystems into the widely-recognized "tar" format, and can use any k/v-styled warehouse for storage. |
|
zip
Package ziptrans packs filesystems into the ZIP archive format.
|
Package ziptrans packs filesystems into the ZIP archive format. |
|
impl/git
The git warehouse is a repository style warehouse.
|
The git warehouse is a repository style warehouse. |