labels

package
v0.8.12 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 25, 2017 License: BSD-3-Clause Imports: 10 Imported by: 0

Documentation

Overview

Package labels supports label-based data types like labelblk, labelvol, labelsurf, labelsz, etc. Basic 64-bit label data and deltas are kept here so all label-based data types can use them without cyclic package dependencies, especially when writing code to synchronize across data instances.

Index

Constants

View Source 
const (
	IngestBlockEvent    = imageblk.IngestBlockEvent
	MutateBlockEvent    = imageblk.MutateBlockEvent
	DeleteBlockEvent    = imageblk.DeleteBlockEvent
	SparsevolStartEvent = "SPARSEVOL_START"
	SparsevolModEvent   = "SPARSEVOL_MOD"
	SparsevolEndEvent   = "SPARSEVOL_END"
	ChangeSizeEvent     = "LABEL_SIZE_CHANGE"
	MergeStartEvent     = "MERGE_START"
	MergeBlockEvent     = "MERGE_BLOCK"
	MergeEndEvent       = "MERGE_END"
	SplitStartEvent     = "SPLIT_START"
	SplitLabelEvent     = "SPLIT_LABEL"
	SplitEndEvent       = "SPLIT_END"
)

Label change event identifiers

View Source 
const DefaultBlockSize = DefaultSubBlocksPerBlock * SubBlockSize
View Source 
const DefaultSubBlocksPerBlock = 8
View Source 
const (
	// MaxAllowedLabel is the largest label that should be allowed by DVID if we want
	// to take into account the maximum integer size within Javascript (due to its
	// underlying use of a double float for numbers, leading to max int = 2^53 - 1).
	// This would circumvent the need to use strings within JSON (e.g., the Google
	// solution) to represent integer labels that could exceed the max javascript
	// number.  It would require adding a value check on each label voxel of a
	// mutation request, which might be too much of a hit to handle an edge case.
	MaxAllowedLabel = 9007199254740991
)
View Source 
const SubBlockSize = 8

Variables

This section is empty.

Functions

func Compress added in v0.8.7 

func Compress(b []byte, volsize dvid.Point) ([]byte, error)

Compress takes an input 3D label array of a given volume dimensions and compresses it using the segmentation format defined in Neuroglancer. VolSize must be tileable by 8x8x8 block size.

func Decompress added in v0.8.7 

func Decompress(b []byte, volsize dvid.Point) ([]byte, error)

Decompress takes an input compressed array of 64-bit labels of a given volume dimensions and decompresses it. VolSize must be tileable by 8x8x8 block size. The output data is a slice of bytes that represents packed uint64.

func DecompressUint64 added in v0.8.7 

func DecompressUint64(input CompressedSegData, volsize dvid.Point) ([]uint64, error)

DecompressUint64 takes an input compressed array of 64-bit labels of a given volume dimensions and decompresses it. VolSize must be tileable by 8x8x8 block size. The output data is an array of 64 bit numbers.

func MergeStart added in v0.8.0 

func MergeStart(iv dvid.InstanceVersion, op MergeOp) error

MergeStart handles label map caches during an active merge operation. Note that if there are multiple synced label instances, the InstanceVersion will always be the labelblk instance. Multiple merges into a single label are allowed, but chained merges are not. For example, you can merge label 1, 2, and 3 into 4, then later merge 6 into 4. However, you cannot concurrently merge label 4 into some other label because there can be a race condition between 3 -> 4 and 4 -> X.

func MergeStop added in v0.8.0 

func MergeStop(iv dvid.InstanceVersion, op MergeOp)

MergeStop marks the end of a merge operation.

func NewVolSize added in v0.8.7 

func NewVolSize(x int32, y int32, z int32) dvid.Point3d

func SplitStart added in v0.8.0 

func SplitStart(iv dvid.InstanceVersion, op DeltaSplitStart) error

SplitStart checks current label map to see if the split conflicts.

func SplitStop added in v0.8.0 

func SplitStop(iv dvid.InstanceVersion, op DeltaSplitEnd)

SplitStop marks the end of a split operation.

func WriteRLEs added in v0.8.7 

func WriteRLEs(lbls Set, w io.Writer, pbCh chan *PositionedBlock, bounds dvid.Bounds, errCh chan error)

WriteRLEs, like WriteBinaryBlock, writes a compact serialization of a binarized Block to the supplied Writer. In this case, the serialization uses little-endian encoded integers and RLEs with the repeating units of the following format: 

int32   Coordinate of run start (dimension 0)
int32   Coordinate of run start (dimension 1)
int32   Coordinate of run start (dimension 2)
int32   Length of run in X direction

The offset is the DVID space offset to the first voxel in the Block.

Types

type Block added in v0.8.7 

type Block struct {
	Labels    []uint64 // shares memory with data
	Numbers   []uint16 // # of labels per sub-block
	SubBlocks []SubBlock
	// contains filtered or unexported fields
}

Block is the unit of storage for compressed DVID labels. It is inspired by the Neuroglancer compression scheme and makes the following changes: (1) a global label list with block-level indices into the list (23 bits vs 64 bits in original Neuroglancer scheme) and a 9 bit count of the # of voxels in the block with that label, (2) the number of bits for encoding values is not required to be a power of two. A global LUT allows easy sharing of labels between blocks, and block-level storage can be more efficient due to the smaller LUT (at the cost of an indirection) and better encoded value packing (at the cost of byte alignment). In both cases memory is gained for increased computation.

func MakeBlock added in v0.8.7 

func MakeBlock(uint64array []byte, blockSize dvid.Point3d) (*Block, error)

MakeBlock returns a compressed label Block given a packed little-endian uint64 label array. It is the inverse of MakeLabelVolume(). There is no sharing of underlying memory between the returned Block and the given byte slice.

func MakeSolidBlock added in v0.8.7 

func MakeSolidBlock(label uint64, blockSize dvid.Point3d) *Block

MakeSolidBlock returns a Block that represents a single label of the given block size.

func SubvolumeToBlock added in v0.8.7 

func SubvolumeToBlock(subvol *dvid.Subvolume, uint64array []byte, index dvid.IndexZYX, blockSize dvid.Point3d) (*Block, error)

SubvolumeToBlock converts a portion of the given label array into a compressed Block. It accepts a packed little-endian uint64 label array and a description of its subvolume, i.e., its extents in dvid space, and returns a compressed Block for the given chunk when tiling dvid space with the given chunk size.

func (Block) MakeLabelVolume added in v0.8.7 

func (b Block) MakeLabelVolume() (uint64array []byte, size dvid.Point3d, err error)

MakeLabelVolume returns a byte slice with packed little-endian uint64 labels in ZYX order, i.e., a uint64 for each voxel where consecutive values are in the (x,y,z) order: (0,0,0), (1,0,0), (2,0,0) ... (0,1,0) There is no sharing of memory between the returned byte slice and the Block data.

func (Block) MarshalBinary added in v0.8.7 

func (b Block) MarshalBinary() ([]byte, error)

MarshalBinary implements the encoding.BinaryMarshaler interface. Note that for efficiency, the returned byte slice will share memory with the receiver Block. Blocks cover nx * ny * nz voxels. This implementation allows any choice of nx, ny, and nz with two restrictions: (1) nx, ny, and nz must be a multiple of 8 greater than 16, and (2) the total number of labels cannot exceed the capacity of a uint32.

Internally, labels are stored in 8x8x8 sub-blocks. There are gx * gy * gz sub-blocks where gx = nx / 8; gy = ny / 8; gz = nz / 8.

The byte layout will be the following if there are N labels in the Block: 

     3 * uint32            values of gx, gy, and gz
     uint32                # of labels (N)
     N * uint64            packed labels in little-endian format

     ----- Data below is only included if N > 1, otherwise it is a solid block.

     gx*gy*gz * uint16     # of labels for sub-blocks.  Ns[i] = # labels for a sub-block

     Labels within gz * gy * gx sub-blocks with each sub-block data in the following format:

			Ns[i] * uint32    Ns packed indices into the N labels.
         values            512 * ceil(log2(Ns[i])) bits, padded to 32-bit words.
                           At most we use 9 bits per voxel for up to the 512 labels in sub-block.
                           A value gives the sub-block index which points to the index into
                           the N labels.  If Ns[i] <= 1, there are no values.  If Ns[i] = 0,
                           the 8x8x8 voxels are set to label 0.  If Ns[i] = 1, all voxels
                           are the given label index.

func (*Block) UnmarshalBinary added in v0.8.7 

func (b *Block) UnmarshalBinary(data []byte) (err error)

UnmarshalBinary implements the encoding.BinaryUnmarshaler interface. Note that for efficiency, the receive Block will share memory with the given byte slice.

func (Block) WriteGoogleCompression added in v0.8.7 

func (b Block) WriteGoogleCompression(w io.Writer) error

GoogleCompression writes label compression compliant with the Google Neuroglancer specification: https://goo.gl/IyQbzL

type CompressedSegData added in v0.8.7 

type CompressedSegData []uint32

func CompressUint64 added in v0.8.7 

func CompressUint64(input []uint64, volsize dvid.Point) (CompressedSegData, error)

CompressUint64 takes an input 3D label array of a given volume dimensions and compresses it using the segmentation format defined in Neuroglancer. VolSize must be tileable by 8x8x8 block size. The output data is an array of 32 bit numbers. TODO: reuse table encoding to improve compression.

type Counts 

type Counts struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

Counts is a thread-safe type for counting label references.

func (*Counts) Decr 

func (c *Counts) Decr(label uint64)

Decr decrements the count for a label.

func (*Counts) Empty 

func (c *Counts) Empty() bool

Empty returns true if there are no counts.

func (*Counts) Incr 

func (c *Counts) Incr(label uint64)

Incr increments the count for a label.

func (*Counts) Value 

func (c *Counts) Value(label uint64) int

Value returns the count for a label.

type DeleteBlock 

type DeleteBlock imageblk.Block

DeleteBlock encapsulates data necessary to delete blocks of labels.

type DeltaDeleteSize 

type DeltaDeleteSize struct {
	Label    uint64
	OldSize  uint64
	OldKnown bool // true if OldSize is valid, otherwise delete all size k/v for this label.
}

DeltaDeleteSize gives info to delete a label's size.

type DeltaMerge 

type DeltaMerge struct {
	MergeOp
	Blocks       dvid.IZYXSlice               // not nil if labelarray used.
	BlockMap     map[dvid.IZYXString]struct{} // not nil if labelblk used, to be deprecated.
	TargetVoxels uint64
	MergedVoxels uint64
}

DeltaMerge describes the labels and blocks affected by a merge operation. It is sent during a MergeBlockEvent.

type DeltaMergeEnd 

type DeltaMergeEnd struct {
	MergeOp
}

DeltaMergeEnd is the data sent during a MergeEndEvent.

type DeltaMergeStart 

type DeltaMergeStart struct {
	MergeOp
}

DeltaMergeStart is the data sent during a MergeStartEvent.

type DeltaModSize 

type DeltaModSize struct {
	Label      uint64
	SizeChange int64 // Adds to old label size
}

DeltaModSize gives info to modify an existing label size without knowing the old size.

type DeltaNewSize 

type DeltaNewSize struct {
	Label uint64
	Size  uint64
}

DeltaNewSize is a new label being introduced.

type DeltaReplaceSize 

type DeltaReplaceSize struct {
	Label   uint64
	OldSize uint64
	NewSize uint64
}

DeltaReplaceSize gives info to precisely remove an old label size and add the updated size.

type DeltaSparsevol 

type DeltaSparsevol struct {
	Label uint64
	Mods  dvid.BlockRLEs
}

DeltaSparsevol describes a change to an existing label.

type DeltaSplit 

type DeltaSplit struct {
	OldLabel     uint64
	NewLabel     uint64
	Split        dvid.BlockRLEs
	SortedBlocks dvid.IZYXSlice
	SplitVoxels  uint64
}

DeltaSplit describes the voxels modified during a split operation. The Split field may be null if this is a coarse split only defined by block indices.

type DeltaSplitEnd 

type DeltaSplitEnd struct {
	OldLabel uint64
	NewLabel uint64
}

DeltaSplitEnd is the data sent during a SplitEndEvent.

type DeltaSplitStart 

type DeltaSplitStart struct {
	OldLabel uint64
	NewLabel uint64
}

DeltaSplitStart is the data sent during a SplitStartEvent.

type Mapping 

type Mapping struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

Mapping is a thread-safe, mapping of labels to labels in both forward and backward direction. Mutation of a Mapping instance can only be done through labels.MergeCache.

func LabelMap added in v0.8.0 

func LabelMap(iv dvid.InstanceVersion) *Mapping

LabelMap returns a label mapping for a version of a data instance. If no label mapping is available, a nil is returned.

func (*Mapping) ConstituentLabels added in v0.8.0 

func (m *Mapping) ConstituentLabels(final uint64) Set

ConstituentLabels returns a set of labels that will be mapped to the given label. The set will always include the given label.

func (*Mapping) FinalLabel 

func (m *Mapping) FinalLabel(start uint64) (uint64, bool)

FinalLabel follows mappings from a start label until a final mapped label is reached.

func (*Mapping) Get 

func (m *Mapping) Get(label uint64) (uint64, bool)

Get returns the mapping or false if no mapping exists.

type MergeOp 

type MergeOp struct {
	Target uint64
	Merged Set
}

MergeOp represents the merging of a set of labels into a target label.

func (MergeOp) String added in v0.8.5 

func (op MergeOp) String() string

type MergeTuple 

type MergeTuple []uint64

MergeTuple represents a merge of labels. Its first element is the destination label and all later elements in the slice are labels to be merged. It's an easy JSON representation as a list of labels.

func (MergeTuple) Op 

func (t MergeTuple) Op() (MergeOp, error)

Op converts a MergeTuple into a MergeOp.

type PositionedBlock added in v0.8.7 

type PositionedBlock struct {
	Block
	Coord dvid.ChunkPoint3d
}

PositionedBlock is a Block that also knows its position in DVID space via a chunk coordinate.

func (PositionedBlock) OffsetDVID added in v0.8.7 

func (pb PositionedBlock) OffsetDVID() dvid.Point3d

OffsetDVID returns the DVID voxel coordinate corresponding to the first voxel of the Block, i.e., the lowest (x,y,z).

func (*PositionedBlock) WriteBinaryBlock added in v0.8.7 

func (b *PositionedBlock) WriteBinaryBlock(label uint64, w io.Writer) error

WriteBinaryBlock writes the binary version of a Block to the supplied Writer, where the serialized data represents just the label voxels. By definition, a binary block has at most two labels (0 = background, 1 = given label) and encoding is a bit per voxel. The binary format is related to the Google and internal DVID label block compression but is simplified, the DVID space offset of the block is included, and the sub-block data are arranged to allow streaming.

Internally, the mask is stored in 8x8x8 sub-blocks. There are gx * gy * gz sub-blocks where gx = nx / 8; gy = ny / 8; gz = nz / 8.

The byte layout will be the following: 

3 * uint32      values of gx, gy, and gz
3 * int32       offset of first voxel of Block in DVID space (x, y, z)
uint64          foreground label

Stream of gx * gy * gz sub-blocks with the following data:

byte            0 = background ONLY  (no more data for this sub-block)
                1 = foreground ONLY  (no more data for this sub-block)
                2 = both background and foreground so mask data required.
mask            64 byte bitmask where each voxel is 0 (background) or 1 (foreground)

type Set 

type Set map[uint64]struct{}

Set is a set of labels.

func NewSet added in v0.8.7 

func NewSet(lbls ...uint64) Set

Merge returns a set made of the given labels.

func (Set) Copy added in v0.8.5 

func (s Set) Copy() Set

Copy returns a duplicate of the Set.

func (Set) Exists added in v0.8.5 

func (s Set) Exists(i uint64) bool

Exists returns true if the given uint64 is present in the Set.

func (Set) Merge added in v0.8.7 

func (s Set) Merge(s2 Set)

Merge adds the elements in the given set to the receiver.

func (Set) String 

func (s Set) String() string

type SubBlock added in v0.8.7 

type SubBlock struct {
	Index  []uint32
	Values []byte
}

SubBlock is the basic unit of compressed DVID labels, set to 8x8x8 to simplify Go coding and lock in the number of bits required for maximum # of labels, etc. The slices share memory with the containing Block data buffer and are a referencing convenience.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.