README
¶
DVID 
Status: In production use at Janelia. See wiki page for outside lab use of DVID.
See the DVID Wiki for more information including installation and examples of use.
DVID is a distributed, versioned, image-oriented dataservice written to support Janelia Farm Research Center's brain imaging, analysis and visualization efforts. It's goal is to provide:
- Easily extensible data types that allow tailoring of access speeds, storage space, and APIs.
- The ability to use a variety of storage systems by either creating a data type for that system or using a storage engine, currently limited to ordered key/value databases.
- A framework for thinking of distribution and versioning of data similar to distributed version control systems like git.
- A stable science-driven API that can be implemented either by native DVID data types and storage engines or by proxying to other connectomics services like Google BrainMaps, BOSS, etc.
DVID aspires to be a "github for large image-oriented data" because each DVID server can manage multiple repositories, each of which contains an image-oriented repo with related data like an image volume, labels, annotations, and skeletons. The goal is to provide scientists with a github-like web client + server that can push/pull data to a collaborator's DVID server.
Although DVID is easily extensible by adding custom data types, each of which fulfill a minimal interface (e.g., HTTP request handling), DVID's initial focus is on efficiently handling data essential for Janelia's connectomics research:
- image and 64-bit label 3d volumes, including multiscale support
- 2d images in XY, XZ, YZ, and arbitrary orientation
- multiscale 2d images in XY, XZ, and YZ, similar to quadtrees
- sparse volumes, corresponding to each unique label in a volume, that can be merged or split
- point annotations (e.g., synapse elements) that can be quickly accessed via subvolumes or labels
- label graphs
- regions of interest represented via a coarse subdivision of space using block indices
- 2d and 3d image and label data using Google BrainMaps API and other cloud-based services
Each of the above is handled by built-in data types via a Level 2 REST HTTP API implemented by Go language packages within the datatype directory. When dealing with novel data, we typically use the generic keyvalue datatype and store JSON-encoded or binary data until we understand the desired access patterns and API. When we outgrow the keyvalue type's GET, POST, and DELETE operations, we create a custom datatype package with a specialized HTTP API.
DVID allows you to assign different storage systems to data instances within a single repo, which allows great flexibility in optimizing storage for particular use cases. For example, easily compressed label data can be store in fast, expensive SSDs while larger, immutable grayscale image data can be stored in petabyte-scale read-optimized systems.
DVID is written in Go and supports different storage backends, a REST HTTP API, and command-line access (likely minimized in near future). Some components written in C, e.g., storage engines like Leveldb and fast codecs like lz4, are embedded or linked as a library.
DVID has been tested on MacOS X, Linux (Fedora 16, CentOS 6, Ubuntu), and Windows 10+ Bash Shell. It comes out-of-the-box with an embedded leveldb for storage although you can configure other storage backends.
If you just need nd-array access, consider DICED, which provides a simple python numpy interface to the main image/label datatypes in DVID. Because of the limited datatypes within DICED, petabyte-scale distributed computing support for mutations will be available sooner in DICED than for the full range of datatypes in DVID.
Command-line and HTTP API documentation can be found in help constants within packages or by visiting the /api/help HTTP endpoint on a running DVID server.
Documentation
¶
Overview ¶
DVID is a *distributed, versioned, image-oriented datastore* written in Go that supports different storage backends, a Level 2 REST HTTP API, command-line access, and a FUSE frontend to at least one of its data types. It has been tested on both MacOS X and Linux (Fedora 16, CentOS 6) but not on Windows.
The starting point for DVID documentation is the README.md file in the DVID repo: https://github.com/janelia-flyem/dvid#dvid-
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
Package datastore provides versioning and persisting supported data types using one of the supported storage engines.
|
Package datastore provides versioning and persisting supported data types using one of the supported storage engines. |
|
Package datatype provides interfaces for arbitrary datatypes supported in DVID.
|
Package datatype provides interfaces for arbitrary datatypes supported in DVID. |
|
annotation
Package annotation supports point annotation management and queries.
|
Package annotation supports point annotation management and queries. |
|
common/downres
Package downres provides a system for computing multi-scale 3d arrays given mutations.
|
Package downres provides a system for computing multi-scale 3d arrays given mutations. |
|
common/labels
Package labels supports label-based data types like labelblk, labelvol, labelsurf, labelsz, etc.
|
Package labels supports label-based data types like labelblk, labelvol, labelsurf, labelsz, etc. |
|
common/proto
Package proto is a generated protocol buffer package.
|
Package proto is a generated protocol buffer package. |
|
googlevoxels
Package googlevoxels implements DVID support for multi-scale tiles and volumes in XY, XZ, and YZ orientation using the Google BrainMaps API.
|
Package googlevoxels implements DVID support for multi-scale tiles and volumes in XY, XZ, and YZ orientation using the Google BrainMaps API. |
|
imageblk
Package imageblk implements DVID support for image blocks of various formats (uint8, uint16, rgba8).
|
Package imageblk implements DVID support for image blocks of various formats (uint8, uint16, rgba8). |
|
imagetile
Package imagetile implements DVID support for imagetiles in XY, XZ, and YZ orientation.
|
Package imagetile implements DVID support for imagetiles in XY, XZ, and YZ orientation. |
|
keyvalue
Package keyvalue is a generated protocol buffer package.
|
Package keyvalue is a generated protocol buffer package. |
|
labelarray
Package labelarray handles both volumes of label data as well as indexing to quickly find and generate sparse volumes of any particular label.
|
Package labelarray handles both volumes of label data as well as indexing to quickly find and generate sparse volumes of any particular label. |
|
labelblk
Package labelblk supports only label volumes.
|
Package labelblk supports only label volumes. |
|
labelgraph
Package labelgraph implements DVID support for data using the underlying graph storage engine.
|
Package labelgraph implements DVID support for data using the underlying graph storage engine. |
|
labelmap
Package labelmap handles both volumes of label data as well as indexing to quickly find and generate sparse volumes of any particular label.
|
Package labelmap handles both volumes of label data as well as indexing to quickly find and generate sparse volumes of any particular label. |
|
labelsz
Package labelsz supports ranking labels by # annotations of each type.
|
Package labelsz supports ranking labels by # annotations of each type. |
|
labelvol
Package labelvol supports label-specific sparse volumes.
|
Package labelvol supports label-specific sparse volumes. |
|
multichan16
Package multichan16 tailors the voxels data type for 16-bit fluorescent images with multiple channels that can be read from V3D Raw format.
|
Package multichan16 tailors the voxels data type for 16-bit fluorescent images with multiple channels that can be read from V3D Raw format. |
|
roi
Package roi implements DVID support for Region-Of-Interest operations.
|
Package roi implements DVID support for Region-Of-Interest operations. |
|
tarsupervoxels
Package tarsupervoxels implements DVID support for data blobs associated with supervoxels.
|
Package tarsupervoxels implements DVID support for data blobs associated with supervoxels. |
|
Package dvid provides types, constants, and functions that have no other dependencies and can be used by all packages within DVID.
|
Package dvid provides types, constants, and functions that have no other dependencies and can be used by all packages within DVID. |
|
Package server configures and launches http/rpc server and storage engines specific to the type of DVID platform: local (e.g., running on MacBook Pro), clustered, or using cloud-based services like Google Cloud.
|
Package server configures and launches http/rpc server and storage engines specific to the type of DVID platform: local (e.g., running on MacBook Pro), clustered, or using cloud-based services like Google Cloud. |
|
Package storage provides a unified interface to a number of storage engines.
|
Package storage provides a unified interface to a number of storage engines. |
|
swift
Package swift adds Openstack Swift support to DVID.
|
Package swift adds Openstack Swift support to DVID. |