data

package
v0.52.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 26, 2024 License: AGPL-3.0 Imports: 29 Imported by: 7

Documentation

Overview

Package data contains the code involved with Viam's Data Management Platform for automatically collecting component readings from robots.

Index

Constants

View Source 
const (
	// InProgressCaptureFileExt defines the file extension for Viam data capture files
	// which are currently being written to.
	InProgressCaptureFileExt = ".prog"
	// CompletedCaptureFileExt defines the file extension for Viam data capture files
	// which are no longer being written to.
	CompletedCaptureFileExt = ".capture"

	// GetImages is used for getting simultaneous images from different imagers.
	GetImages = "GetImages"
)
View Source 
const FromDMString = "fromDataManagement"

FromDMString is used to access the 'fromDataManagement' value from a request's Extra struct.

Variables

View Source 
var ErrNoCaptureToStore = status.Error(codes.FailedPrecondition, "no capture from filter module")

ErrNoCaptureToStore is returned when a modular filter resource filters the capture coming from the base resource.

View Source 
var FromDMExtraMap = map[string]interface{}{FromDMString: true}

FromDMExtraMap is a map with 'fromDataManagement' set to true.

Functions

func BuildCaptureMetadata added in v0.39.0 

func BuildCaptureMetadata(
	compAPI resource.API,
	compName string,
	method string,
	additionalParams map[string]string,
	methodParams map[string]*anypb.Any,
	tags []string,
) *v1.DataCaptureMetadata

BuildCaptureMetadata builds a DataCaptureMetadata object and returns error if additionalParams fails to convert to anypb map.

func CaptureFilePathWithReplacedReservedChars added in v0.39.0 

func CaptureFilePathWithReplacedReservedChars(filepath string) string

CaptureFilePathWithReplacedReservedChars returns the filepath with substitutions for reserved characters.

func DumpRegisteredCollectors added in v0.45.0 

func DumpRegisteredCollectors() map[MethodMetadata]CollectorConstructor

DumpRegisteredCollectors returns all registered collectores this is only intended for services/datamanager/builtin/builtin_test.go.

func FailedToReadErr 

func FailedToReadErr(component, method string, err error) error

FailedToReadErr is the error describing when a Capturer was unable to get the reading of a method.

func FormatBytesI64 added in v0.44.0 

func FormatBytesI64(b int64) string

FormatBytesI64 formats an int64 representing bytes as an easily human parsable string.

func FormatBytesU64 added in v0.44.0 

func FormatBytesU64(b uint64) string

FormatBytesU64 formats an uint64 representing bytes as an easily human parsable string.

func GetDurationFromHz added in v0.39.0 

func GetDurationFromHz(captureFrequencyHz float32) time.Duration

GetDurationFromHz gets time.Duration from hz.

func GetExtraFromContext added in v0.8.0 

func GetExtraFromContext(ctx context.Context) (*structpb.Struct, error)

GetExtraFromContext sets the extra struct with "fromDataManagement": true if the flag is true in the context. Deprecated: Use camera.FromContext instead.

func GetFileExt added in v0.39.0 

func GetFileExt(dataType v1.DataType, methodName string, parameters map[string]string) string

GetFileExt gets the file extension for a capture file.

func InvalidInterfaceErr 

func InvalidInterfaceErr(api resource.API) error

InvalidInterfaceErr is the error describing when an interface not conforming to the expected resource.API was passed into a CollectorConstructor.

func IsDataCaptureFile added in v0.39.0 

func IsDataCaptureFile(f *os.File) bool

IsDataCaptureFile returns whether or not f is a data capture file.

func RegisterCollector 

func RegisterCollector(method MethodMetadata, c CollectorConstructor)

RegisterCollector registers a Collector to its corresponding MethodMetadata.

func SensorDataFromCaptureFile added in v0.39.0 

func SensorDataFromCaptureFile(f *CaptureFile) ([]*v1.SensorData, error)

SensorDataFromCaptureFile returns all readings in f.

func SensorDataFromCaptureFilePath added in v0.39.0 

func SensorDataFromCaptureFilePath(filePath string) ([]*v1.SensorData, error)

SensorDataFromCaptureFilePath returns all readings in the file at filePath. NOTE: (Nick S) At time of writing this is only used in tests.

Types

type CaptureBuffer added in v0.39.0 

type CaptureBuffer struct {
	Directory string
	MetaData  *v1.DataCaptureMetadata
	// contains filtered or unexported fields
}

CaptureBuffer is a persistent queue of SensorData backed by a series of *data.CaptureFile.

func NewCaptureBuffer added in v0.39.0 

func NewCaptureBuffer(dir string, md *v1.DataCaptureMetadata, maxCaptureFileSize int64) *CaptureBuffer

NewCaptureBuffer returns a new Buffer.

func (*CaptureBuffer) Flush added in v0.39.0 

func (b *CaptureBuffer) Flush() error

Flush flushes all buffered data to disk and marks any in progress file as complete.

func (*CaptureBuffer) Path added in v0.39.0 

func (b *CaptureBuffer) Path() string

Path returns the path to the directory containing the backing data capture files.

func (*CaptureBuffer) Write added in v0.39.0 

func (b *CaptureBuffer) Write(item *v1.SensorData) error

Write writes item onto b. Binary sensor data is written to its own file. Tabular data is written to disk in maxCaptureFileSize sized files. Files that are still being written to are indicated with the extension InProgressFileExt. Files that have finished being written to are indicated by FileExt.

type CaptureBufferedWriter added in v0.39.0 

type CaptureBufferedWriter interface {
	Write(item *v1.SensorData) error
	Flush() error
	Path() string
}

CaptureBufferedWriter is a buffered, persistent queue of SensorData.

type CaptureFile added in v0.39.0 

type CaptureFile struct {
	// contains filtered or unexported fields
}

CaptureFile is the data structure containing data captured by collectors. It is backed by a file on disk containing length delimited protobuf messages, where the first message is the CaptureMetadata for the file, and ensuing messages contain the captured data.

func NewCaptureFile added in v0.39.0 

func NewCaptureFile(dir string, md *v1.DataCaptureMetadata) (*CaptureFile, error)

NewCaptureFile creates a new *CaptureFile with the specified md in the specified directory.

func ReadCaptureFile added in v0.39.0 

func ReadCaptureFile(f *os.File) (*CaptureFile, error)

ReadCaptureFile creates a File struct from a passed os.File previously constructed using NewFile.

func (*CaptureFile) Close added in v0.39.0 

func (f *CaptureFile) Close() error

Close closes the file.

func (*CaptureFile) Delete added in v0.39.0 

func (f *CaptureFile) Delete() error

Delete deletes the file.

func (*CaptureFile) Flush added in v0.39.0 

func (f *CaptureFile) Flush() error

Flush flushes any buffered writes to disk.

func (*CaptureFile) GetPath added in v0.39.0 

func (f *CaptureFile) GetPath() string

GetPath returns the path of the underlying os.File.

func (*CaptureFile) ReadMetadata added in v0.39.0 

func (f *CaptureFile) ReadMetadata() *v1.DataCaptureMetadata

ReadMetadata reads and returns the metadata in f.

func (*CaptureFile) ReadNext added in v0.39.0 

func (f *CaptureFile) ReadNext() (*v1.SensorData, error)

ReadNext returns the next SensorData reading.

func (*CaptureFile) Reset added in v0.39.0 

func (f *CaptureFile) Reset()

Reset resets the read pointer of f.

func (*CaptureFile) Size added in v0.39.0 

func (f *CaptureFile) Size() int64

Size returns the size of the file.

func (*CaptureFile) WriteNext added in v0.39.0 

func (f *CaptureFile) WriteNext(data *v1.SensorData) error

WriteNext writes the next SensorData reading.

type CaptureFunc 

type CaptureFunc func(ctx context.Context, params map[string]*anypb.Any) (interface{}, error)

CaptureFunc allows the creation of simple Capturers with anonymous functions.

type Collector 

type Collector interface {
	Close()
	Collect()
	Flush()
}

Collector collects data to some target.

func NewCollector 

func NewCollector(captureFunc CaptureFunc, params CollectorParams) (Collector, error)

NewCollector returns a new Collector with the passed capturer and configuration options. It calls capturer at the specified Interval, and appends the resulting reading to target.

type CollectorConstructor 

type CollectorConstructor func(resource interface{}, params CollectorParams) (Collector, error)

CollectorConstructor contains a function for constructing an instance of a Collector.

func CollectorLookup 

func CollectorLookup(method MethodMetadata) CollectorConstructor

CollectorLookup looks up a Collector by the given MethodMetadata. nil is returned if there is None.

type CollectorParams 

type CollectorParams struct {
	MongoCollection *mongo.Collection
	ComponentName   string
	ComponentType   string
	MethodName      string
	Interval        time.Duration
	MethodParams    map[string]*anypb.Any
	Target          CaptureBufferedWriter
	QueueSize       int
	BufferSize      int
	Logger          logging.Logger
	Clock           clock.Clock
}

CollectorParams contain the parameters needed to construct a Collector.

func (CollectorParams) Validate 

func (p CollectorParams) Validate() error

Validate validates that p contains all required parameters.

type FromDMContextKey added in v0.8.0 

type FromDMContextKey struct{}

FromDMContextKey is used to check whether the context is from data management. Deprecated: use a camera.Extra with camera.NewContext instead.

type MethodMetadata 

type MethodMetadata struct {
	API        resource.API
	MethodName string
}

MethodMetadata contains the metadata identifying a component method that we are going to capture and collect.

func (MethodMetadata) String added in v0.21.0 

func (m MethodMetadata) String() string

type TabularData added in v0.50.0 

type TabularData struct {
	TimeRequested time.Time `bson:"time_requested"`
	TimeReceived  time.Time `bson:"time_received"`
	ComponentName string    `bson:"component_name"`
	ComponentType string    `bson:"component_type"`
	MethodName    string    `bson:"method_name"`
	Data          bson.M    `bson:"data"`
}

TabularData is a denormalized sensor reading.

Source Files

View all Source files

Directories

Path Synopsis
Package data contains the code for automatically collecting readings from robots.
 Package data contains the code for automatically collecting readings from robots.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.