storage

Documentation

Index

• Constants
• func ClearDirectory(dirpath string) (err error)
• func InferDatasetProperties(sourcePath string) (id, name, description string, err error)
• func InferDatasetSchema(sourcePath string) (schemaIn, schemaOut *sch.Schema, err error)
• func ScanReadme(r io.Reader) (name, description string)
• type Context
  • func (context Context) GetAllTaskPaths(id string) (paths TaskPaths, err error)
  • func (context Context) GetDatasetPath(id string, subdir string) (path string, err error)
  • func (context Context) GetModulePath(id string, moduleType string, subdir string) (path string, err error)
  • func (context Context) GetProcessPath(id string, subdir string) (path string, err error)
  • func (context Context) GetSchedulingInputPath(subdir string) (path string, err error)
  • func (context Context) GetTaskPath(id string, subdir string) (path string, err error)
• type TaskPaths

Constants

const DefaultFilePerm = os.FileMode(0755)

DefaultFilePerm is the default file mode to be used when creating directories.

Functions

func ClearDirectory

func ClearDirectory(dirpath string) (err error)

ClearDirectory removes all files in a directory.

func InferDatasetProperties

func InferDatasetProperties(sourcePath string) (id, name, description string, err error)

InferDatasetProperties takes a dataset available on the file system and tries to infer its basic properties such as id, name and description.

func InferDatasetSchema

func InferDatasetSchema(sourcePath string) (schemaIn, schemaOut *sch.Schema, err error)

InferDatasetSchema tries to infer the schema of a dataset.

func ScanReadme

func ScanReadme(r io.Reader) (name, description string)

ScanReadme parser a readme file from the given reader and extracts the name and description.

Types

type Context

type Context struct {
	WorkingDir string
}

Context contains the storage model context with all information needed to access the working directory.

func (Context) GetAllTaskPaths

func (context Context) GetAllTaskPaths(id string) (paths TaskPaths, err error)

GetAllTaskPaths constructs and returns all paths during storage execution.

func (Context) GetDatasetPath

func (context Context) GetDatasetPath(id string, subdir string) (path string, err error)

GetDatasetPath constructs the path for a given dataset, ensures it exists and returns the path string.

func (Context) GetModulePath

func (context Context) GetModulePath(id string, moduleType string, subdir string) (path string, err error)

GetModulePath constructs the path for a given module, ensures it exists and returns the path string.

func (Context) GetProcessPath

func (context Context) GetProcessPath(id string, subdir string) (path string, err error)

GetProcessPath constructs the path for a given process, ensures it exists and returns the path string.

func (Context) GetSchedulingInputPath

func (context Context) GetSchedulingInputPath(subdir string) (path string, err error)

GetSchedulingInputPath returns the path which holds temp data

func (Context) GetTaskPath

func (context Context) GetTaskPath(id string, subdir string) (path string, err error)

GetTaskPath constructs the path for a given dataset, ensures it exists and returns the path string.

type TaskPaths

type TaskPaths struct {
	Parameters  string
	Logs        string
	Metadata    string
	Debug       string
	Predictions string
	Evaluations string
	Config      string
}

TaskPaths is used to store all relevant storage paths used during task execution.