server

README

MLModelScope Agent Sever Commands

Note that evaluations currently only run on datasets known by DLDataset

Running Evaluations

One can run evaluations across different frameworks and models or on a single framework and model.

Running Evaluations on all Frameworks / Models

evaluate.go is a wrapper tool exists to make it easier to run evaluations across frameworks and models. One can specify the frameworks, models, and batch sizes to use within the file and then run evaluate.go.

• [ ]: TODO: allow one to specify the frameworks, models, and batch sizes from the command line

Running Evaluations on a single Framework / Model

Example Usage

./tensorflow_agent dataset --debug --verbose --publish=true --fail_on_error=true --gpu=true --batch_size=320 --model_name=BVLC-Reference-CaffeNet --model_version=1.0 --database_name=tx2_carml_model_trace --database_address=minsky1-1.csl.illinois.edu --publish_predictions=false --num_file_parts=8 --trace_level=FULL_TRACE

Command line options

Available Models

agent info models

Checking Divergence

• [ ]: TODO

To compare a single prediction's divergence you use

agent database divergence --database_address=$DATABASE_ADDR --database_name=carml --source=$SOURCE_ID --target=$TARGET_ID

Analysing / Summarizing Results

• [ ]: TODO

agent evaluation --help

Documentation

Index

• Variables
• func GetOutboundIP() net.IP
• func NewRootCommand(frameworkRegisterFunc FrameworkRegisterFunction, ...) (*cobra.Command, error)
• func RunRootE(c *cobra.Command, framework dlframework.FrameworkManifest, args []string) (<-chan bool, error)
• func SetupFlags(c *cobra.Command)
• type FrameworkRegisterFunction

Variables

var (

	//	evaluationTable            *evaluation.EvaluationCollection
	//	modelAccuracyTable         *evaluation.ModelAccuracyCollection
	//	performanceTable           *evaluation.PerformanceCollection
	//	inputPredictionsTable      *evaluation.InputPredictionCollection
	DefaultChannelBuffer = 100000
)

var (
	DefaultRunOptions = &robustly.RunOptions{
		RateLimit:  1,
		Timeout:    time.Second,
		PrintStack: true,
		RetryDelay: 0 * time.Nanosecond,
	}
)

Functions

func GetOutboundIP

func GetOutboundIP() net.IP

Get preferred outbound ip of this machine

func NewRootCommand

func NewRootCommand(frameworkRegisterFunc FrameworkRegisterFunction, framework0 dlframework.FrameworkManifest) (*cobra.Command, error)

represents the base command when called without any subcommands

func RunRootE

func RunRootE(c *cobra.Command, framework dlframework.FrameworkManifest, args []string) (<-chan bool, error)

func SetupFlags

func SetupFlags(c *cobra.Command)

Types

type FrameworkRegisterFunction

type FrameworkRegisterFunction func()