data

package
v0.56.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 30, 2024 License: AGPL-3.0 Imports: 30 Imported by: 7

Documentation

Overview

Package data contains the code involved with Viam's Data Management Platform for automatically collecting component readings from robots.

Index

Constants

View Source 
const (
	// InProgressCaptureFileExt defines the file extension for Viam data capture files
	// which are currently being written to.
	InProgressCaptureFileExt = ".prog"
	// CompletedCaptureFileExt defines the file extension for Viam data capture files
	// which are no longer being written to.
	CompletedCaptureFileExt = ".capture"

	// GetImages is used for getting simultaneous images from different imagers.
	GetImages = "GetImages"
)
View Source 
const (
	// ExtDefault is the default file extension.
	ExtDefault = ""
	// ExtDat is the file extension for tabular data.
	ExtDat = ".dat"
	// ExtPcd is the file extension for pcd files.
	ExtPcd = ".pcd"
	// ExtJpeg is the file extension for jpeg files.
	ExtJpeg = ".jpeg"
	// ExtPng is the file extension for png files.
	ExtPng = ".png"
)
View Source 
const FromDMString = "fromDataManagement"

FromDMString is used to access the 'fromDataManagement' value from a request's Extra struct.

Variables

View Source 
var ErrNoCaptureToStore = status.Error(codes.FailedPrecondition, "no capture from filter module")

ErrNoCaptureToStore is returned when a modular filter resource filters the capture coming from the base resource.

View Source 
var FromDMExtraMap = map[string]interface{}{FromDMString: true}

FromDMExtraMap is a map with 'fromDataManagement' set to true.

Functions

func CaptureFilePathWithReplacedReservedChars added in v0.39.0 

func CaptureFilePathWithReplacedReservedChars(filepath string) string

CaptureFilePathWithReplacedReservedChars returns the filepath with substitutions for reserved characters.

func DumpRegisteredCollectors added in v0.45.0 

func DumpRegisteredCollectors() map[MethodMetadata]CollectorConstructor

DumpRegisteredCollectors returns all registered collectores this is only intended for services/datamanager/builtin/builtin_test.go.

func FailedToReadErr 

func FailedToReadErr(component, method string, err error) error

FailedToReadErr is the error describing when a Capturer was unable to get the reading of a method.

func FormatBytesI64 added in v0.44.0 

func FormatBytesI64(b int64) string

FormatBytesI64 formats an int64 representing bytes as an easily human parsable string.

func FormatBytesU64 added in v0.44.0 

func FormatBytesU64(b uint64) string

FormatBytesU64 formats an uint64 representing bytes as an easily human parsable string.

func GetDurationFromHz added in v0.39.0 

func GetDurationFromHz(captureFrequencyHz float32) time.Duration

GetDurationFromHz gets time.Duration from hz.

func InvalidInterfaceErr 

func InvalidInterfaceErr(api resource.API) error

InvalidInterfaceErr is the error describing when an interface not conforming to the expected resource.API was passed into a CollectorConstructor.

func IsBinary added in v0.55.0 

func IsBinary(item *v1.SensorData) bool

IsBinary returns true when the *v1.SensorData is of type binary.

func IsDataCaptureFile added in v0.39.0 

func IsDataCaptureFile(f *os.File) bool

IsDataCaptureFile returns whether or not f is a data capture file.

func MimeTypeToCameraFormat added in v0.55.0 

func MimeTypeToCameraFormat(mt MimeType) camerapb.Format

MimeTypeToCameraFormat converts a data.MimeType into a camerapb.Format.

func RegisterCollector 

func RegisterCollector(method MethodMetadata, c CollectorConstructor)

RegisterCollector registers a Collector to its corresponding MethodMetadata.

func SensorDataFromCaptureFile added in v0.39.0 

func SensorDataFromCaptureFile(f *CaptureFile) ([]*v1.SensorData, error)

SensorDataFromCaptureFile returns all readings in f.

func SensorDataFromCaptureFilePath added in v0.39.0 

func SensorDataFromCaptureFilePath(filePath string) ([]*v1.SensorData, error)

SensorDataFromCaptureFilePath returns all readings in the file at filePath. NOTE: (Nick S) At time of writing this is only used in tests.

Types

type Annotations added in v0.55.0 

type Annotations struct {
	BoundingBoxes   []BoundingBox
	Classifications []Classification
}

Annotations represents ML classifications.

func (Annotations) Empty added in v0.55.0 

func (mt Annotations) Empty() bool

Empty returns true when Annotations are empty.

func (Annotations) ToProto added in v0.55.0 

func (mt Annotations) ToProto() *dataPB.Annotations

ToProto converts Annotations to *dataPB.Annotations.

type Binary added in v0.55.0 

type Binary struct {
	// Payload contains the binary payload
	Payload []byte
	// MimeType  descibes the payload's MimeType
	MimeType MimeType
	// Annotations provide metadata about the Payload
	Annotations Annotations
}

Binary represents an element of a binary capture result response.

type BoundingBox added in v0.55.0 

type BoundingBox struct {
	Label          string
	Confidence     *float64
	XMinNormalized float64
	YMinNormalized float64
	XMaxNormalized float64
	YMaxNormalized float64
}

BoundingBox represents a labeled bounding box with an optional confidence interval between 0 and 1.

type CaptureBuffer added in v0.39.0 

type CaptureBuffer struct {
	Directory string
	MetaData  *v1.DataCaptureMetadata
	// contains filtered or unexported fields
}

CaptureBuffer is a persistent queue of SensorData backed by a series of *data.CaptureFile.

func NewCaptureBuffer added in v0.39.0 

func NewCaptureBuffer(dir string, md *v1.DataCaptureMetadata, maxCaptureFileSize int64) *CaptureBuffer

NewCaptureBuffer returns a new Buffer.

func (*CaptureBuffer) Flush added in v0.39.0 

func (b *CaptureBuffer) Flush() error

Flush flushes all buffered data to disk and marks any in progress file as complete.

func (*CaptureBuffer) Path added in v0.39.0 

func (b *CaptureBuffer) Path() string

Path returns the path to the directory containing the backing data capture files.

func (*CaptureBuffer) WriteBinary added in v0.55.0 

func (b *CaptureBuffer) WriteBinary(items []*v1.SensorData) error

WriteBinary writes the items to their own file. Files that are still being written to are indicated with the extension '.prog'. Files that have finished being written to are indicated by '.capture'.

func (*CaptureBuffer) WriteTabular added in v0.55.0 

func (b *CaptureBuffer) WriteTabular(item *v1.SensorData) error

WriteTabular writes Tabular data to disk in maxCaptureFileSize sized files. Files that are still being written to are indicated with the extension '.prog'. Files that have finished being written to are indicated by '.capture'.

type CaptureBufferedWriter added in v0.39.0 

type CaptureBufferedWriter interface {
	WriteBinary(items []*v1.SensorData) error
	WriteTabular(items *v1.SensorData) error
	Flush() error
	Path() string
}

CaptureBufferedWriter is a buffered, persistent queue of SensorData.

type CaptureFile added in v0.39.0 

type CaptureFile struct {
	// contains filtered or unexported fields
}

CaptureFile is the data structure containing data captured by collectors. It is backed by a file on disk containing length delimited protobuf messages, where the first message is the CaptureMetadata for the file, and ensuing messages contain the captured data.

func NewCaptureFile added in v0.39.0 

func NewCaptureFile(dir string, md *v1.DataCaptureMetadata) (*CaptureFile, error)

NewCaptureFile creates a new *CaptureFile with the specified md in the specified directory.

func ReadCaptureFile added in v0.39.0 

func ReadCaptureFile(f *os.File) (*CaptureFile, error)

ReadCaptureFile creates a File struct from a passed os.File previously constructed using NewFile.

func (*CaptureFile) Close added in v0.39.0 

func (f *CaptureFile) Close() error

Close closes the file.

func (*CaptureFile) Delete added in v0.39.0 

func (f *CaptureFile) Delete() error

Delete deletes the file.

func (*CaptureFile) Flush added in v0.39.0 

func (f *CaptureFile) Flush() error

Flush flushes any buffered writes to disk.

func (*CaptureFile) GetPath added in v0.39.0 

func (f *CaptureFile) GetPath() string

GetPath returns the path of the underlying os.File.

func (*CaptureFile) ReadMetadata added in v0.39.0 

func (f *CaptureFile) ReadMetadata() *v1.DataCaptureMetadata

ReadMetadata reads and returns the metadata in f.

func (*CaptureFile) ReadNext added in v0.39.0 

func (f *CaptureFile) ReadNext() (*v1.SensorData, error)

ReadNext returns the next SensorData reading.

func (*CaptureFile) Reset added in v0.39.0 

func (f *CaptureFile) Reset()

Reset resets the read pointer of f.

func (*CaptureFile) Size added in v0.39.0 

func (f *CaptureFile) Size() int64

Size returns the size of the file.

func (*CaptureFile) WriteNext added in v0.39.0 

func (f *CaptureFile) WriteNext(data *v1.SensorData) error

WriteNext writes the next SensorData reading.

type CaptureFunc 

type CaptureFunc func(ctx context.Context, params map[string]*anypb.Any) (CaptureResult, error)

CaptureFunc allows the creation of simple Capturers with anonymous functions.

type CaptureResult added in v0.55.0 

type CaptureResult struct {
	// Type represents the type of result (binary or tabular)
	Type CaptureType
	// Timestamps contain the time the data was requested and received
	Timestamps
	// TabularData contains the tabular data payload when Type == CaptureResultTypeTabular
	TabularData TabularData
	// Binaries contains binary data responses when Type == CaptureResultTypeBinary
	Binaries []Binary
}

CaptureResult is the result of a capture function.

func NewBinaryCaptureResult added in v0.55.0 

func NewBinaryCaptureResult(ts Timestamps, binaries []Binary) CaptureResult

NewBinaryCaptureResult returns a binary capture result.

func NewTabularCaptureResult added in v0.55.0 

func NewTabularCaptureResult(ts Timestamps, i interface{}) (CaptureResult, error)

NewTabularCaptureResult returns a tabular result.

func NewTabularCaptureResultReadings added in v0.55.0 

func NewTabularCaptureResultReadings(ts Timestamps, readings map[string]interface{}) (CaptureResult, error)

NewTabularCaptureResultReadings returns a tabular readings result.

func (*CaptureResult) ToProto added in v0.55.0 

func (cr *CaptureResult) ToProto() []*datasyncPB.SensorData

ToProto converts a CaptureResult into a []*datasyncPB.SensorData{}.

func (*CaptureResult) Validate added in v0.55.0 

func (cr *CaptureResult) Validate() error

Validate returns an error if the *CaptureResult is invalid.

type CaptureType added in v0.55.0 

type CaptureType int

CaptureType represents captured tabular or binary data. 

const (
	// CaptureTypeUnspecified represents that the data type of the captured data was not specified.
	CaptureTypeUnspecified CaptureType = iota
	// CaptureTypeTabular represents that the data type of the captured data is tabular.
	CaptureTypeTabular
	// CaptureTypeBinary represents that the data type of the captured data is binary.
	CaptureTypeBinary
)

func BuildCaptureMetadata added in v0.39.0 

func BuildCaptureMetadata(
	api resource.API,
	name string,
	method string,
	additionalParams map[string]string,
	methodParams map[string]*anypb.Any,
	tags []string,
) (*v1.DataCaptureMetadata, CaptureType)

BuildCaptureMetadata builds a DataCaptureMetadata object and returns error if additionalParams fails to convert to anypb map.

func MethodToCaptureType added in v0.55.0 

func MethodToCaptureType(methodName string) CaptureType

MethodToCaptureType returns the DataType of the method.

func (CaptureType) ToProto added in v0.55.0 

func (dt CaptureType) ToProto() datasyncPB.DataType

ToProto converts a CaptureType into a v1.DataType.

type Classification added in v0.55.0 

type Classification struct {
	Label      string
	Confidence *float64
}

Classification represents a labeled classification with an optional confidence interval between 0 and 1.

type Collector 

type Collector interface {
	Close()
	Collect()
	Flush()
}

Collector collects data to some target.

func NewCollector 

func NewCollector(captureFunc CaptureFunc, params CollectorParams) (Collector, error)

NewCollector returns a new Collector with the passed capturer and configuration options. It calls capturer at the specified Interval, and appends the resulting reading to target.

type CollectorConstructor 

type CollectorConstructor func(resource interface{}, params CollectorParams) (Collector, error)

CollectorConstructor contains a function for constructing an instance of a Collector.

func CollectorLookup 

func CollectorLookup(method MethodMetadata) CollectorConstructor

CollectorLookup looks up a Collector by the given MethodMetadata. nil is returned if there is None.

type CollectorParams 

type CollectorParams struct {
	BufferSize      int
	Clock           clock.Clock
	ComponentName   string
	ComponentType   string
	DataType        CaptureType
	Interval        time.Duration
	Logger          logging.Logger
	MethodName      string
	MethodParams    map[string]*anypb.Any
	MongoCollection *mongo.Collection
	QueueSize       int
	Target          CaptureBufferedWriter
}

CollectorParams contain the parameters needed to construct a Collector.

func (CollectorParams) Validate 

func (p CollectorParams) Validate() error

Validate validates that p contains all required parameters.

type FromDMContextKey added in v0.8.0 

type FromDMContextKey struct{}

FromDMContextKey is used to check whether the context is from data management. Deprecated: use a camera.Extra with camera.NewContext instead.

type MethodMetadata 

type MethodMetadata struct {
	API        resource.API
	MethodName string
}

MethodMetadata contains the metadata identifying a component method that we are going to capture and collect.

func (MethodMetadata) String added in v0.21.0 

func (m MethodMetadata) String() string

type MimeType added in v0.55.0 

type MimeType int

MimeType represents the mime type of the sensor data. 

const (
	// MimeTypeUnspecified means that the mime type was not specified.
	MimeTypeUnspecified MimeType = iota
	// MimeTypeImageJpeg means that the mime type is jpeg.
	MimeTypeImageJpeg
	// MimeTypeImagePng means that the mime type is png.
	MimeTypeImagePng
	// MimeTypeApplicationPcd means that the mime type is pcd.
	MimeTypeApplicationPcd
)

This follows the mime types supported in https://github.com/viamrobotics/api/blob/d7436a969cbc03bf7e84bb456b6dbfeb51f664d7/proto/viam/app/datasync/v1/data_sync.proto#L69

func CameraFormatToMimeType added in v0.55.0 

func CameraFormatToMimeType(f camerapb.Format) MimeType

CameraFormatToMimeType converts a camera camerapb.Format into a MimeType.

func MimeTypeFromProto added in v0.55.0 

func MimeTypeFromProto(mt datasyncPB.MimeType) MimeType

MimeTypeFromProto converts a datasyncPB.MimeType to a data.MimeType.

func (MimeType) ToProto added in v0.55.0 

func (mt MimeType) ToProto() datasyncPB.MimeType

ToProto converts MimeType to datasyncPB.

type TabularData added in v0.50.0 

type TabularData struct {
	Payload *structpb.Struct
}

TabularData contains a tabular data payload.

type TabularDataBson added in v0.55.0 

type TabularDataBson struct {
	TimeRequested time.Time `bson:"time_requested"`
	TimeReceived  time.Time `bson:"time_received"`
	ComponentName string    `bson:"component_name"`
	ComponentType string    `bson:"component_type"`
	MethodName    string    `bson:"method_name"`
	Data          bson.M    `bson:"data"`
}

TabularDataBson is a denormalized sensor reading that can be encoded into BSON.

type Timestamps added in v0.55.0 

type Timestamps struct {
	// TimeRequested represents the time the request for the data was started
	TimeRequested time.Time
	// TimeReceived represents the time the response for the request for the data
	// was received
	TimeReceived time.Time
}

Timestamps are the timestamps the data was captured.

Source Files

View all Source files

Directories

Path Synopsis
Package data contains the code for automatically collecting readings from robots.
 Package data contains the code for automatically collecting readings from robots.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.