labelarray

package
v0.8.9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 3, 2017 License: BSD-3-Clause Imports: 25 Imported by: 0

Documentation

Overview

Package labelarray tailors the voxels data type for 64-bit labels and allows loading of NRGBA images (e.g., Raveler superpixel PNG images) that implicitly use slice Z as part of the label index.

Index

Constants

View Source 
const (
	Version  = "0.1"
	RepoURL  = "github.com/janelia-flyem/dvid/datatype/labelarray"
	TypeName = "labelarray"
)
View Source 
const (
	DownsizeBlockEvent  = "LABELS64_DOWNSIZE_ADD"
	DownsizeCommitEvent = "LABELS64_DOWNSIZE_COMMIT"

	IngestBlockEvent = "LABELS64_BLOCK_INGEST"
	MutateBlockEvent = "LABELS64_BLOCK_MUTATE"
	DeleteBlockEvent = "LABELS64_BLOCK_DELETE"
)
View Source 
const HelpMessage = `` /* 26511-byte string literal not displayed */
View Source 
const KVWriteSize = 500

KVWriteSize is the # of key-value pairs we will write as one atomic batch write.

Variables

View Source 
var (
	DefaultBlockSize int32   = 64
	DefaultRes       float32 = imageblk.DefaultRes
	DefaultUnits             = imageblk.DefaultUnits
)

Functions

func DecodeBlockTKey 

func DecodeBlockTKey(tk storage.TKey) (*dvid.IndexZYX, error)

DecodeBlockTKey returns a spatial index from a label block key. TODO: Extend this when necessary to allow any form of spatial indexing like CZYX.

func DecodeLabelIndexTKey 

func DecodeLabelIndexTKey(tk storage.TKey) (label uint64, err error)

DecodeLabelIndexTKey parses a TKey and returns the corresponding label.

func EncodeFormat 

func EncodeFormat() dvid.DataValues

func GetBinaryData 

func GetBinaryData(compression string, in io.ReadCloser, estsize int64) ([]byte, error)

GetBinaryData returns label data from a potentially compressed ("lz4", "gzip") reader.

func NewBlockTKey 

func NewBlockTKey(idx dvid.Index) storage.TKey

NewBlockTKey returns a TKey for a label block, which is a slice suitable for lexicographical ordering on zyx coordinates.

func NewBlockTKeyByCoord 

func NewBlockTKeyByCoord(izyx dvid.IZYXString) storage.TKey

NewBlockTKeyByCoord returns a TKey for a block coord in string format.

func NewLabelIndexTKey 

func NewLabelIndexTKey(label uint64) storage.TKey

NewLabelIndexTKey returns a TKey corresponding to a label. Value will hold block coords that contain the label.

func RavelerSuperpixelBytes 

func RavelerSuperpixelBytes(slice, superpixel32 uint32) []byte

RavelerSuperpixelBytes returns an encoded slice for Raveler (slice, superpixel) tuple. TODO -- Move Raveler-specific functions out of DVID.

func ZeroBytes 

func ZeroBytes() []byte

ZeroBytes returns a slice of bytes that represents the zero label.

Types

type Data 

type Data struct {
	*imageblk.Data
	datastore.Updater

	// The maximum label id found in each version of this instance.
	// Can be unset if no new label was added at that version, in which case
	// you must traverse DAG to find max label of parent.
	MaxLabel map[dvid.VersionID]uint64

	// The maximum label for this instance in the entire repo.  This allows us to do
	// conflict-free merges without any relabeling.  Note that relabeling (rebasing)
	// is required if we move data between repos, e.g., when pushing remote nodes,
	// since we have no control over which labels were created remotely and there
	// could be conflicts between the local and remote repos.  When mutations only
	// occur within a single repo, however, this atomic label allows us to prevent
	// conflict across all versions within this repo.
	MaxRepoLabel uint64
	// contains filtered or unexported fields
}

Data of labelarray type is an extended form of imageblk Data

func GetByDataUUID 

func GetByDataUUID(dataUUID dvid.UUID) (*Data, error)

GetByDataUUID returns a pointer to labelarray data given a data UUID.

func GetByUUIDName 

func GetByUUIDName(uuid dvid.UUID, name dvid.InstanceName) (*Data, error)

GetByUUIDName returns a pointer to labelarray data given a UUID and data name.

func GetByVersionName 

func GetByVersionName(v dvid.VersionID, name dvid.InstanceName) (*Data, error)

GetByVersionName returns a pointer to labelarray data given a version and data name.

func NewData 

func NewData(uuid dvid.UUID, id dvid.InstanceID, name dvid.InstanceName, c dvid.Config) (*Data, error)

NewData returns a pointer to labelarray data.

func (*Data) BlockSize 

func (d *Data) BlockSize() dvid.Point

func (*Data) CopyPropertiesFrom 

func (d *Data) CopyPropertiesFrom(src datastore.DataService, fs storage.FilterSpec) error

CopyPropertiesFrom copies the data instance-specific properties from a given data instance into the receiver's properties.

func (*Data) CreateComposite 

func (d *Data) CreateComposite(request datastore.Request, reply *datastore.Response) error

CreateComposite creates a new rgba8 image by combining hash of labels + the grayscale

func (*Data) CreateCompositeChunk 

func (d *Data) CreateCompositeChunk(chunk *storage.Chunk) error

CreateCompositeChunk processes each chunk of labels and grayscale data, saving the composited result into an rgba8. Only some multiple of the # of CPU cores can be used for chunk handling before it waits for chunk processing to abate via the buffered server.HandlerToken channel.

func (*Data) DeleteBlocks 

func (d *Data) DeleteBlocks(ctx *datastore.VersionedCtx, start dvid.ChunkPoint3d, span int) error

func (*Data) DoRPC 

func (d *Data) DoRPC(req datastore.Request, reply *datastore.Response) error

DoRPC acts as a switchboard for RPC commands.

func (*Data) Equals 

func (d *Data) Equals(d2 *Data) bool

func (*Data) Extents 

func (d *Data) Extents() *dvid.Extents

func (*Data) FoundSparseVol 

func (d *Data) FoundSparseVol(ctx *datastore.VersionedCtx, label uint64, bounds dvid.Bounds) (bool, error)

FoundSparseVol returns true if a sparse volume is found for the given label within the given bounds.

func (*Data) GetBlocks 

func (d *Data) GetBlocks(v dvid.VersionID, start dvid.ChunkPoint3d, span int) ([]byte, error)

GetBlocks returns a slice of bytes corresponding to all the blocks along a span in X

func (*Data) GetImage 

func (d *Data) GetImage(v dvid.VersionID, vox *Labels, roiname dvid.InstanceName) (*dvid.Image, error)

GetImage retrieves a 2d image from a version node given a geometry of labels.

func (*Data) GetLabelAtPoint 

func (d *Data) GetLabelAtPoint(v dvid.VersionID, pt dvid.Point) (uint64, error)

GetLabelAtPoint returns the 64-bit unsigned int label for a given point.

func (*Data) GetLabelBlock 

func (d *Data) GetLabelBlock(v dvid.VersionID, blockCoord dvid.ChunkPoint3d) ([]byte, error)

GetLabelBlock returns a block of labels corresponding to the block coordinate.

func (*Data) GetLabelBytesAtPoint 

func (d *Data) GetLabelBytesAtPoint(v dvid.VersionID, pt dvid.Point) ([]byte, error)

GetLabelBytesAtPoint returns the 8 byte slice corresponding to a 64-bit label at a point.

func (*Data) GetLabelMeta 

func (d *Data) GetLabelMeta(ctx *datastore.VersionedCtx, lbls labels.Set, bounds dvid.Bounds) (*Meta, error)

GetLabelMeta returns a sorted list of ZYX blocks that contain the given labels. If block bounds are set, the number of voxels is unknown and set to zero.

func (*Data) GetLabels 

func (d *Data) GetLabels(v dvid.VersionID, vox *Labels, r *imageblk.ROI) error

GetLabels copies labels from the storage engine to Labels, a requested subvolume or 2d image.

func (*Data) GetLegacyRLE 

func (d *Data) GetLegacyRLE(ctx *datastore.VersionedCtx, label uint64, bounds dvid.Bounds, old bool) ([]byte, error)

GetLegacyRLE returns an encoded sparse volume given a label and an output format. If the old flag is true, the RLEs are computed using label volume instead of directly from the compressed label Blocks.

func (*Data) GetMappedLabelMeta 

func (d *Data) GetMappedLabelMeta(ctx *datastore.VersionedCtx, label uint64, bounds dvid.Bounds) (meta *Meta, lbls labels.Set, err error)

GetMappedLabelMeta returns a sorted list of ZYX blocks that contain the given label, including all labels that could be undergoing merge into that label. If block bounds are set, the number of voxels is unknown and set to zero.

func (*Data) GetSparseCoarseVol 

func (d *Data) GetSparseCoarseVol(ctx *datastore.VersionedCtx, label uint64, bounds dvid.Bounds) ([]byte, error)

GetSparseCoarseVol returns an encoded sparse volume given a label. The encoding has the following format where integers are little endian: 

		byte     Set to 0
		uint8    Number of dimensions
		uint8    Dimension of run (typically 0 = X)
		byte     Reserved (to be used later)
		uint32    # Blocks [TODO.  0 for now]
		uint32    # Spans
		Repeating unit of:
    		int32   Block coordinate of run start (dimension 0)
    		int32   Block coordinate of run start (dimension 1)
    		int32   Block coordinate of run start (dimension 2)
    		int32   Length of run

func (*Data) GetSyncSubs 

func (d *Data) GetSyncSubs(synced dvid.Data) (datastore.SyncSubs, error)

GetSyncSubs implements the datastore.Syncer interface

func (*Data) GetVolume 

func (d *Data) GetVolume(v dvid.VersionID, vox *Labels, roiname dvid.InstanceName) ([]byte, error)

GetVolume retrieves a n-d volume from a version node given a geometry of labels.

func (*Data) GobDecode 

func (d *Data) GobDecode(b []byte) error

func (*Data) GobEncode 

func (d *Data) GobEncode() ([]byte, error)

func (*Data) InitDataHandlers 

func (d *Data) InitDataHandlers() error

InitDataHandlers launches goroutines to handle each labelarray instance's syncs.

func (*Data) LoadImages 

func (d *Data) LoadImages(v dvid.VersionID, offset dvid.Point, filenames []string) error

LoadImages bulk loads images using different techniques if it is a multidimensional file like HDF5 or a sequence of PNG/JPG/TIF images.

func (*Data) LoadMutable 

func (d *Data) LoadMutable(root dvid.VersionID, storedVersion, expectedVersion uint64) (bool, error)

LoadMutable loads mutable properties of label volumes like the maximum labels for each version. Note that we load these max labels from key-value pairs rather than data instance properties persistence, because in the case of a crash, the actually stored repo data structure may be out-of-date compared to the guaranteed up-to-date key-value pairs for max labels.

func (*Data) MarshalJSON 

func (d *Data) MarshalJSON() ([]byte, error)

func (*Data) MergeLabels 

func (d *Data) MergeLabels(v dvid.VersionID, op labels.MergeOp) error

MergeLabels handles merging of any number of labels throughout the various label data structures. It assumes that the merges aren't cascading, e.g., there is no attempt to merge label 3 into 4 and also 4 into 5. The caller should have flattened the merges. TODO: Provide some indication that subset of labels are under evolution, returning 

an "unavailable" status or 203 for non-authoritative response.  This might not be
feasible for clustered DVID front-ends due to coordination issues.

EVENTS

labels.MergeStartEvent occurs at very start of merge and transmits labels.DeltaMergeStart struct.

labels.MergeBlockEvent occurs for every block of a merged label and transmits labels.DeltaMerge struct.

labels.MergeEndEvent occurs at end of merge and transmits labels.DeltaMergeEnd struct.

func (*Data) NewLabel 

func (d *Data) NewLabel(v dvid.VersionID) (uint64, error)

NewLabel returns a new label for the given version.

func (*Data) NewLabels 

func (d *Data) NewLabels(geom dvid.Geometry, img interface{}) (*Labels, error)

NewLabels returns labelarray Labels, a representation of externally usable subvolume or slice data, given some geometry and optional image data. If img is passed in, the function will initialize Voxels with data from the image. Otherwise, it will allocate a zero buffer of appropriate size.

func (*Data) PushData 

func (d *Data) PushData(p *datastore.PushSession) error

PushData pushes labelarray data to a remote DVID.

func (*Data) PutBlocks 

func (d *Data) PutBlocks(v dvid.VersionID, mutID uint64, start dvid.ChunkPoint3d, span int, data io.ReadCloser, mutate bool) error

PutBlocks stores blocks of data in a span along X

func (*Data) PutLabelMeta 

func (d *Data) PutLabelMeta(ctx *datastore.VersionedCtx, label uint64, meta *Meta) error

func (*Data) PutLabels 

func (d *Data) PutLabels(v dvid.VersionID, subvol *dvid.Subvolume, data []byte, roiname dvid.InstanceName, mutate bool) error

PutLabels persists voxels from a subvolume into the storage engine. This involves transforming a supplied volume of uint64 with known geometry into many labels.Block that tiles the subvolume. Messages are sent to subscribers for ingest or mutate events.

func (*Data) ReadChunk 

func (d *Data) ReadChunk(chunk *storage.Chunk) error

ReadChunk reads a chunk of data as part of a mapped operation. Only some multiple of the # of CPU cores can be used for chunk handling before it waits for chunk processing to abate via the buffered server.HandlerToken channel.

func (*Data) SendBlocks 

func (d *Data) SendBlocks(ctx *datastore.VersionedCtx, w http.ResponseWriter, subvol *dvid.Subvolume, compression string) error

GetBlocks returns a slice of bytes corresponding to all the blocks along a span in X

func (*Data) ServeHTTP 

func (d *Data) ServeHTTP(uuid dvid.UUID, ctx *datastore.VersionedCtx, w http.ResponseWriter, r *http.Request)

ServeHTTP handles all incoming HTTP requests for this data.

func (*Data) SetResolution 

func (d *Data) SetResolution(uuid dvid.UUID, jsonBytes []byte) error

SetResolution loads JSON data giving Resolution.

func (*Data) Shutdown 

func (d *Data) Shutdown()

Shutdown terminates blocks until syncs are done then terminates background goroutines processing data.

func (*Data) SplitCoarseLabels 

func (d *Data) SplitCoarseLabels(v dvid.VersionID, fromLabel, splitLabel uint64, r io.ReadCloser) (toLabel uint64, err error)

SplitCoarseLabels splits a portion of a label's voxels into a given split label or, if the given split label is 0, a new label, which is returned. The input is a binary sparse volume defined by block coordinates and should be the smaller portion of a labeled region-to-be-split.

EVENTS

labels.SplitStartEvent occurs at very start of split and transmits labels.DeltaSplitStart struct.

labels.SplitBlockEvent occurs for every block of a split label and transmits labels.DeltaSplit struct.

labels.SplitEndEvent occurs at end of split and transmits labels.DeltaSplitEnd struct.

func (*Data) SplitLabels 

func (d *Data) SplitLabels(v dvid.VersionID, fromLabel, splitLabel uint64, r io.ReadCloser) (toLabel uint64, err error)

SplitLabels splits a portion of a label's voxels into a given split label or, if the given split label is 0, a new label, which is returned. The input is a binary sparse volume and should preferably be the smaller portion of a labeled region. In other words, the caller should chose to submit for relabeling the smaller portion of any split. It is assumed that the given split voxels are within the fromLabel set of voxels and will generate unspecified behavior if this is not the case.

EVENTS

labels.SplitStartEvent occurs at very start of split and transmits labels.DeltaSplitStart struct.

labels.SplitBlockEvent occurs for every block of a split label and transmits labels.DeltaSplit struct.

labels.SplitEndEvent occurs at end of split and transmits labels.DeltaSplitEnd struct.

func (*Data) WriteLegacyRLE 

func (d *Data) WriteLegacyRLE(ctx *datastore.VersionedCtx, label uint64, b dvid.Bounds, compression string, format SparseVolFormat, w io.Writer) (found bool, err error)

func (*Data) WriteStreamingRLE 

func (d *Data) WriteStreamingRLE(ctx *datastore.VersionedCtx, label uint64, bounds dvid.Bounds, compression string, w io.Writer) (bool, error)

WriteStreamingRLE does a streaming write of an encoded sparse volume given a label. It returns a bool whether the label was found in the given bounds and any error.

type IngestedBlock 

type IngestedBlock struct {
	Index *dvid.IndexZYX
	Data  *labels.Block
	MutID uint64
}

IngestedBlock encodes a 3d block coordinate and block data. It is the unit of delta for a IngestBlockEvent.

type Labels 

type Labels struct {
	*imageblk.Voxels
}

Labels are voxels that have uint64 labels.

func (*Labels) ComputeTransform 

func (v *Labels) ComputeTransform(tkey storage.TKey, blockSize dvid.Point) (blockBeg, dataBeg, dataEnd dvid.Point, err error)

ComputeTransform determines the block coordinate and beginning + ending voxel points for the data corresponding to the given Block.

func (*Labels) Interpolable 

func (l *Labels) Interpolable() bool

func (*Labels) String 

func (l *Labels) String() string

type Meta 

type Meta struct {
	Voxels uint64         // Total # of voxels in label.
	Blocks dvid.IZYXSlice // Sorted block coordinates occupied by label.

}

Meta gives a high-level overview of all voxels in a label including the # voxels, block index.

func (Meta) MarshalBinary 

func (m Meta) MarshalBinary() ([]byte, error)

MarshalBinary implements the encoding.BinaryMarshaler interface

func (*Meta) UnmarshalBinary 

func (m *Meta) UnmarshalBinary(b []byte) error

UnmarshalBinary implements the encoding.BinaryUnmarshaler interface.

type MetaCache 

type MetaCache struct {
	// contains filtered or unexported fields
}

MetaCache is a label Meta cache based on time since last access.

func MakeMetaCache 

func MakeMetaCache(size uint16) *MetaCache

func (MetaCache) AddLabelMeta 

func (m MetaCache) AddLabelMeta(label uint64, meta *Meta)

AddLabelMeta adds a label's Meta to the cache, possibly evicting older entries.

func (MetaCache) GetLabelMeta 

func (m MetaCache) GetLabelMeta(label uint64) *Meta

GetLabelMeta returns a label's Meta if in the cache, else returns nil.

type MutatedBlock 

type MutatedBlock struct {
	Index *dvid.IndexZYX
	Prev  *labels.Block
	Data  *labels.Block
	MutID uint64
}

MutatedBlock encodes a 3d block coordinate and previous and updated block data. It is the unit of delta for a MutateBlockEvent.

type SparseVolFormat 

type SparseVolFormat uint8

SparseVolFormat indicates the type of encoding used for sparse volume representation. 

const (
	// Legacy RLE encoding with header that gives # spans.
	FormatLegacyRLE SparseVolFormat = iota

	// Streaming RLE encoding
	FormatStreamingRLE

	// Streaming set of binary Blocks
	FormatBinaryBlocks

	// Legacy RLE encoding computed using old system.
	FormatLegacySlowRLE
)

type Type 

type Type struct {
	imageblk.Type
}

Type uses imageblk data type by composition.

func (*Type) Help 

func (dtype *Type) Help() string

func (*Type) NewDataService 

func (dtype *Type) NewDataService(uuid dvid.UUID, id dvid.InstanceID, name dvid.InstanceName, c dvid.Config) (datastore.DataService, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.