vision

Documentation

Overview

Package vision is the service that allows you to access various computer vision algorithms (like detection, segmentation, tracking, etc) that usually only require a camera or image input.

Index

• Constants
• Variables
• func FindFirstName(r robot.Robot) string
• func ImageToFloatBuffer(img image.Image) []float32
• func ImageToUInt8Buffer(img image.Image) []byte
• func Named(name string) resource.Name
• func NewServer(s subtype.Service) pb.VisionServiceServer
• func NewTFLiteClassifier(ctx context.Context, conf *VisModelConfig, logger golog.Logger) (classification.Classifier, *inf.TFLiteStruct, error)
• func NewTFLiteDetector(ctx context.Context, cfg *VisModelConfig, logger golog.Logger) (objectdetection.Detector, *inf.TFLiteStruct, error)
• func WrapWithReconfigurable(s interface{}) (resource.Reconfigurable, error)
• type Attributes
• type Service
  • func FirstFromRobot(r robot.Robot) (Service, error)
  • func FromRobot(r robot.Robot, name string) (Service, error)
  • func New(ctx context.Context, r robot.Robot, config config.Service, logger golog.Logger) (Service, error)
  • func NewClientFromConn(ctx context.Context, conn rpc.ClientConn, name string, logger golog.Logger) Service
• type TFLiteClassifierConfig
• type TFLiteDetectorConfig
• type VisModelConfig
• type VisModelType
• type VisOperation

Constants

const (
	TFLiteDetector   = VisModelType("tflite_detector")
	TFDetector       = VisModelType("tf_detector")
	ColorDetector    = VisModelType("color_detector")
	TFLiteClassifier = VisModelType("tflite_classifier")
	TFClassifier     = VisModelType("tf_classifier")
	RCSegmenter      = VisModelType("radius_clustering_segmenter")
	ObjectSegmenter  = VisModelType("object_segmenter")
)

The set of allowed vision model types.

const (
	VisDetection      = VisOperation("detection")
	VisClassification = VisOperation("classification")
	VisSegmentation   = VisOperation("segmentation")
)

The set of operations supported by the vision model types.

const RadiusClusteringSegmenter = "radius_clustering"

RadiusClusteringSegmenter is the name of a segmenter that finds well separated objects on a flat plane.

const SubtypeName = resource.SubtypeName("vision")

SubtypeName is the name of the type of service.

Variables

var Subtype = resource.NewSubtype(
	resource.ResourceNamespaceRDK,
	resource.ResourceTypeService,
	SubtypeName,
)

Subtype is a constant that identifies the vision service resource subtype.

Functions

func FindFirstName

func FindFirstName(r robot.Robot) string

FindFirstName returns name of first vision service found.

func ImageToFloatBuffer

func ImageToFloatBuffer(img image.Image) []float32

ImageToFloatBuffer reads an image into a byte slice (buffer) the most common sense way. Left to right like a book; R, then G, then B. No funny stuff. Assumes values between -1 and 1.

func ImageToUInt8Buffer

func ImageToUInt8Buffer(img image.Image) []byte

ImageToUInt8Buffer reads an image into a byte slice in the most common sense way. Left to right like a book; R, then G, then B. No funny stuff. Assumes values should be between 0-255.

func Named

func Named(name string) resource.Name

Named is a helper for getting the named vision's typed resource name. RSDK-347 Implements vision's Named.

func NewServer

func NewServer(s subtype.Service) pb.VisionServiceServer

NewServer constructs a vision gRPC service server.

func NewTFLiteClassifier

func NewTFLiteClassifier(ctx context.Context, conf *VisModelConfig,
	logger golog.Logger,
) (classification.Classifier, *inf.TFLiteStruct, error)

NewTFLiteClassifier creates an RDK classifier given a VisModelConfig. In other words, this function returns a function from image-->[]classifier.Classifications. It does this by making calls to an inference package and wrapping the result.

func NewTFLiteDetector

func NewTFLiteDetector(ctx context.Context, cfg *VisModelConfig, logger golog.Logger) (objectdetection.Detector, *inf.TFLiteStruct, error)

NewTFLiteDetector creates an RDK detector given a DetectorConfig. In other words, this function returns a function from image-->[]objectdetection.Detection. It does this by making calls to an inference package and wrapping the result.

func WrapWithReconfigurable

func WrapWithReconfigurable(s interface{}) (resource.Reconfigurable, error)

WrapWithReconfigurable wraps a vision service as a Reconfigurable.

Types

type Attributes

type Attributes struct {
	ModelRegistry []VisModelConfig `json:"register_models"`
}

Attributes contains a list of the user-provided details necessary to register a new vision service.

type Service

type Service interface {
	// detector methods
	GetDetectorNames(ctx context.Context) ([]string, error)
	AddDetector(ctx context.Context, cfg VisModelConfig) error
	RemoveDetector(ctx context.Context, detectorName string) error
	GetDetectionsFromCamera(ctx context.Context, cameraName, detectorName string) ([]objdet.Detection, error)
	GetDetections(ctx context.Context, img image.Image, detectorName string) ([]objdet.Detection, error)
	// classifier methods
	GetClassifierNames(ctx context.Context) ([]string, error)
	AddClassifier(ctx context.Context, cfg VisModelConfig) error
	RemoveClassifier(ctx context.Context, classifierName string) error
	GetClassificationsFromCamera(ctx context.Context, cameraName, classifierName string, n int) (classification.Classifications, error)
	GetClassifications(ctx context.Context, img image.Image, classifierName string, n int) (classification.Classifications, error)
	// segmenter methods
	GetSegmenterNames(ctx context.Context) ([]string, error)
	GetSegmenterParameters(ctx context.Context, segmenterName string) ([]utils.TypedName, error)
	GetObjectPointClouds(ctx context.Context, cameraName, segmenterName string, params config.AttributeMap) ([]*viz.Object, error)
}

A Service that implements various computer vision algorithms like detection and segmentation.

func FirstFromRobot

func FirstFromRobot(r robot.Robot) (Service, error)

FirstFromRobot returns the first vision service in this robot.

func FromRobot

func FromRobot(r robot.Robot, name string) (Service, error)

FromRobot is a helper for getting the named vision service from the given Robot.

func New

func New(ctx context.Context, r robot.Robot, config config.Service, logger golog.Logger) (Service, error)

New registers new detectors from the config and returns a new object detection service for the given robot.

func NewClientFromConn

func NewClientFromConn(ctx context.Context, conn rpc.ClientConn, name string, logger golog.Logger) Service

NewClientFromConn constructs a new Client from connection passed in.

type TFLiteClassifierConfig

type TFLiteClassifierConfig struct {
	// this should come from the attributes part of the detector config
	ModelPath  string  `json:"model_path"`
	NumThreads int     `json:"num_threads"`
	LabelPath  *string `json:"label_path"`
}

TFLiteClassifierConfig specifies the fields necessary for creating a TFLite classifier.

type TFLiteDetectorConfig

type TFLiteDetectorConfig struct {
	// this should come from the attributes part of the detector config
	ModelPath  string  `json:"model_path"`
	NumThreads int     `json:"num_threads"`
	LabelPath  *string `json:"label_path"`
	ServiceURL *string `json:"service_url"`
}

TFLiteDetectorConfig specifies the fields necessary for creating a TFLite detector.

type VisModelConfig

type VisModelConfig struct {
	Name       string              `json:"name"`
	Type       string              `json:"type"`
	Parameters config.AttributeMap `json:"parameters"`
}

VisModelConfig specifies the name of the detector, the type of detector, and the necessary parameters needed to build the detector.

type VisModelType

type VisModelType string

VisModelType defines what vision models are known by the vision service.

type VisOperation

type VisOperation string

VisOperation defines what types of operations are allowed by the vision service.