types

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 30, 2024 License: Apache-2.0 Imports: 20 Imported by: 0

README

types

Table of Contents

  1. Description
  2. Structure and Organisation
  3. Class Diagram
  4. Functionality
  5. Data Types
  6. Testing
  7. Proposed Functionality/Requirements
  8. References

Specification

Description

types package defines and keeps data structures and interfaces that are used across the whole DMS component by different packages.

Structure and Organisation

Here is quick overview of the contents of this pacakge:

  • README: Current file which is aimed towards developers who wish to use and modify the package functionality.

  • capability: This file contains data structures to describe machine capability.

  • comparison: This file contains constans and types used for capability comparison.

  • constants: This file contains constants that are used across different packages.

  • deployment: This file contains data structure related to job deployment.

  • elk_stats: This file contains data structure to be sent to elasticsearch collector.

  • encryption: This file contains data structure related to encryption in DMS.

  • execution: This file contains data structure related to executor functionality.

  • firecracker: This file contains data structure related to firecracker.

  • machine: This file contains data structure related to the machine - resources, peer details, services etc.

  • network: This file contains data structure related to networking functionality of DMS

  • network_config: This file defines message types, network types (libp2p, NATS) with configurations, and libp2p specific configurations (DHT, keys, peers, scheduling etc).

  • onboarding: This file contains data structure related to compute provider onboarding.

  • resource: This file contains data structures of GPU and execution resources.

  • spec_config: This file defines a SpecConfig struct for configuration data with type, parameters, normalization, validation, and type checking functionalities.

  • storage: This file contains data structures related to storage.

  • telemetry: This file defines structs related to telemetry configuration and methods to load configuration from environment variables.

  • types: This file defines a base model for entities in the application with auto-generated UUIDs, timestamps, and soft delete functionality using GORM hooks.

Class Diagram
Source

types class diagram

Rendered from source file
!$rootUrlGitlab = "https://gitlab.com/nunet/device-management-service/-/raw/main"
!$packageRelativePath = "/types"
!$packageUrlGitlab = $rootUrlGitlab + $packageRelativePath
 
!include $packageUrlGitlab/specs/class_diagram.puml
Functionality

types package holds interfaces and methods that are used by multiple packages. The functionality of these interfaces/methods are typically implemented in other packages.

Here are some methods defined in types package:

NewExecutionResult

  • signature: NewExecutionResult(code int) *ExecutionResult

  • input: exit code

  • output: types.ExecutionResult

NewExecutionResult creates a new ExecutionResult object.

NewFailedExecutionResult

  • signature: NewFailedExecutionResult(err error) *ExecutionResult

  • input: error

  • output: types.ExecutionResult

NewFailedExecutionResult creates a new ExecutionResult object for a failed execution. It sets the error message from the provided error and sets the exit code to -1.

Config interface TBD
type Config interface {
	GetNetworkConfig() *SpecConfig
}

GetNetworkConfig will return the network configuration parameters.

NewSpecConfig

  • signature: NewSpecConfig(t string) *SpecConfig

  • input: type for the configuration object

  • output: types.SpecConfig

NewSpecConfig creates new SpecConfig with the given type and an empty params map.

WithParam

  • signature: (s *SpecConfig) WithParam(key string, value interface{}) *SpecConfig

  • input #1 : key

  • input #2 : value associated with the key

  • output: types.SpecConfig

WithParam adds a new key-value pair to the spec parameters and returns the updated SpecConfig object.

Normalize

  • signature: (s *SpecConfig) Normalize()

  • input: None

  • output: None

Normalize ensures that the spec config is in a valid state by trimming whitespace from the Type field and initializing the Params map if empty.

Validate

  • signature: (s *SpecConfig) Validate() error

  • input: None

  • output: error

Validate checks if the spec config is valid. It returns an error if the SpecConfig is nil or if the Type field is missing (blank). Otherwise, it returns no error indicating a valid configuration.

IsType

  • signature: (s *SpecConfig) IsType(t string) bool

  • input: type

  • output: bool

IsType checks if the SpecConfig matches the provided type, ignoring case sensitivity. It returns true if there's a match, otherwise false.

IsEmpty

  • signature: (s *SpecConfig) IsEmpty() bool

  • input: None

  • output: bool

IsEmpty checks if the SpecConfig is empty, meaning it's either nil or has an empty Type and no parameters. It returns true if empty, otherwise false.

GetID

  • signature: (m types.BaseDBModel) GetID() string

  • input: None

  • output: identifier of the entity

GetID returns the identifier of the entity.

BeforeCreate

  • signature: (m *Model) BeforeCreate(tx *gorm.DB) error

  • input: gorm database object to be created

  • output: bool

BeforeCreate sets the ID and CreatedAt fields before creating a new entity.

BeforeUpdate

  • signature: (m *Model) BeforeUpdate(tx *gorm.DB) error

  • input: gorm database object to be updated

  • output: bool

BeforeUpdate returns true if the spec config is empty.

LoadConfigFromEnv

  • signature: LoadConfigFromEnv() (*TelemetryConfig, error)

  • input: none

  • output: types.TelemetryConfig

  • output (error): error message

LoadConfigFromEnv loads the telemetry configuration from environment variables. This includes observabilitty level and collector configuration. It returns the final configuration as types.TelemetryConfig object.

parseObservabilityLevel

  • signature: parseObservabilityLevel(levelStr string) int

  • input: observability level

  • output: int value corresponding to the input observability level

parseObservabilityLevel returns the integer representation of the provided observability level string. When the input string does not match the defined observability levels, default observability level INFO is considered and its integer value is returned.

Data Types
Deployment
  • types.DeploymentRequest
type DeploymentRequest struct {
	RequesterWalletAddress string    `json:"address_user"` // service provider wallet address
	MaxNtx                 int       `json:"max_ntx"`
	Blockchain             string    `json:"blockchain"`
	TxHash                 string    `json:"tx_hash"`
	ServiceType            string    `json:"service_type"`
	Timestamp              time.Time `json:"timestamp"`
	MetadataHash           string    `json:"metadata_hash"`
	WithdrawHash           string    `json:"withdraw_hash"`
	RefundHash             string    `json:"refund_hash"`
	Distribute_50Hash      string    `json:"distribute_50_hash"`
	Distribute_75Hash      string    `json:"distribute_75_hash"`
	Params                 struct {
		ImageID   string `json:"image_id"`
		ModelURL  string `json:"model_url"`
		ResumeJob struct {
			Resume       bool   `json:"resume"`
			ProgressFile string `json:"progress_file"` // file path
		} `json:"resume_job"`
		Packages        []string `json:"packages"`
		RemoteNodeID    string   `json:"node_id"`          // NodeID of compute provider (machine to deploy the job on)
		RemotePublicKey string   `json:"public_key"`       // Public key of compute provider
		LocalNodeID     string   `json:"local_node_id"`    // NodeID of service provider (machine triggering the job)
		LocalPublicKey  string   `json:"local_public_key"` // Public key of service provider
		MachineType     string   `json:"machine_type"`
	} `json:"params"`
	Constraints struct {
		Complexity string `json:"complexity"`
		CPU        int    `json:"cpu"`
		RAM        int    `json:"ram"`
		Vram       int    `json:"vram"`
		Power      int    `json:"power"`
		Time       int    `json:"time"`
	} `json:"constraints"`
	TraceInfo struct {
		TraceID     string `json:"trace_id"`
		SpanID      string `json:"span_id"`
		TraceFlags  string `json:"trace_flags"`
		TraceStates string `json:"trace_state"`
	} `json:"traceinfo"`
}
  • types.DeploymentResponse
type DeploymentResponse struct {
	Success bool   `json:"success"`
	Content string `json:"content"`
}
  • types.DeploymentUpdate
type DeploymentUpdate struct {
	MsgType string `json:"msg_type"`
	Msg     string `json:"msg"`
}
  • types.DeploymentRequestFlat
type DeploymentRequestFlat struct {
	types.BaseDBModel
	DeploymentRequest string `json:"deployment_request"`
	// represents job status from services table; goal is to keep then in sync (both tables are on different DMSes).
	JobStatus string `json:"job_status"`
}
  • types.BlockchainTxStatus
type BlockchainTxStatus struct {
	TransactionType   string `json:"transaction_type"` // No need of this param maybe be deprecated in future
	TransactionStatus string `json:"transaction_status"`
	TxHash            string `json:"tx_hash"`
}
ELK
  • types.NewDeviceOnboarded
// NewDeviceOnboarded defines the schema of the data to be sent to stats db when a new device gets onboarded
type NewDeviceOnboarded struct {
	PeerID        string
	CPU           float32
	RAM           float32
	Network       float32
	DedicatedTime float32
	Timestamp     float32
}
  • types.DeviceStatusChange
// DeviceStatusChange defines the schema of the data to be sent to stats db when a device status gets changed
type DeviceStatusChange struct {
	PeerID    string
	Status    string
	Timestamp float32
}
  • types.DeviceResourceChange
// DeviceResourceChange defines the schema of the data to be sent to stats db when a device resource gets changed
type DeviceResourceChange struct {
	PeerID                   string
	ChangedAttributeAndValue struct {
		CPU           float32
		RAM           float32
		Network       float32
		DedicatedTime float32
	}
	Timestamp float32
}
  • types.DeviceResourceConfig
// DeviceResourceConfig defines the schema of the data to be sent to stats db when a device resource config gets changed
type DeviceResourceConfig struct {
	PeerID                   string
	ChangedAttributeAndValue struct {
		CPU           float32
		RAM           float32
		Network       float32
		DedicatedTime float32
	}
	Timestamp float32
}
  • types.NewService

// NewService defines the schema of the data to be sent to stats db when a new service gets registered in the platform type NewService struct { ServiceID string ServiceName string ServiceDescription string Timestamp float32 }

  • types.ServiceCall
// ServiceCall defines the schema of the data to be sent to stats db when a host machine accepts a deployement request
type ServiceCall struct {
	CallID              float32
	PeerIDOfServiceHost string
	ServiceID           string
	CPUUsed             float32
	MaxRAM              float32
	MemoryUsed          float32
	NetworkBwUsed       float32
	TimeTaken           float32
	Status              string
	AmountOfNtx         int32
	Timestamp           float32
}
  • types.ServiceStatus
// ServiceStatus defines the schema of update the status of service to stats db of the job being executed on host machine
type ServiceStatus struct {
	CallID              float32
	PeerIDOfServiceHost string
	ServiceID           string
	Status              string
	Timestamp           float32
}
  • types.ServiceRemove
// ServiceRemove defines the schema of the data to be sent to stats db when a new service gets removed from the platform
type ServiceRemove struct {
	ServiceID string
	Timestamp float32
}
  • types.NtxPayment
// NtxPayment defines the schema of the data to be sent to stats db when a payment is made to device for the completion of service.
type NtxPayment struct {
	CallID            float32
	ServiceID         string
	AmountOfNtx       int32
	PeerID            string
	SuccessFailStatus string
	Timestamp         float32
}
Executor
  • types.Executor: This defines the type of executor that is used to execute the job.
type Executor struct {
	ExecutorType ExecutorType `json:"executor_type"`
}

type ExecutorType string
  • types.ExecutionRequest: This is the input that executor receives to initiate a job execution.
type ExecutionRequest struct {
	// ID of the job to execute
	JobID string

	// ID of the execution
	ExecutionID string

	// Engine spec for the execution
	EngineSpec types.SpecConfig

	// Resources for the execution
	Resources types.Resources

	// Input volumes for the execution
	Inputs []storage.StorageVolume

	// Output volumes for the results
	Outputs []storage.StorageVolume

	// Directory to store the results
	ResultsDir string
}
  • types.ExecutionResult: This contains the result of the job execution.
type ExecutionResult struct {
	// STDOUT of the execution
	STDOUT string `json:"stdout"`

	// STDERR of the execution
	STDERR string `json:"stderr"`

	// Exit code of the execution
	ExitCode int `json:"exit_code"`

	// Error message if the execution failed
	ErrorMsg string `json:"error_msg"`
}
  • types.SpecConfig TBD: This allows arbitrary configuration/parameters as needed during implementation of specific executor.
// SpecConfig represents a configuration for a spec
// A SpecConfig can be used to define an engine spec, a storage volume, etc.
type SpecConfig struct {
	// Type of the spec (e.g. docker, firecracker, storage, etc.)
	Type string `json:"type"`

	// Params of the spec
	// This allows passing arbitrary parameters to the spec implementation
	Params map[string]interface{} `json:"params,omitempty"`
}
  • types.LogStreamRequest: This is the input provided when a request to stream logs of an execution is made.
type LogStreamRequest struct {
	JobID       string // ID of the job
	ExecutionID string // ID of the execution
	Tail        bool   // Tail the logs
	Follow      bool   // Follow the logs
}
Firecracker
  • types.BootSource: This contains configuration parameters for booting a Firecracker VM.
type BootSource struct {
	KernelImagePath string `json:"kernel_image_path"`
	BootArgs        string `json:"boot_args"`
}
  • types.Drives: This contains properties of a virtual drive for Firecracker VM.
type Drives struct {
	DriveID      string `json:"drive_id"`
	PathOnHost   string `json:"path_on_host"`
	IsRootDevice bool   `json:"is_root_device"`
	IsReadOnly   bool   `json:"is_read_only"`
}
  • types.MachineConfig: This defines the configuration parameters of the machine to be used while creating a new Firecracker VM.
type MachineConfig struct {
	VCPUCount  int `json:"vcpu_count"`
	MemSizeMib int `json:"mem_size_mib"`
}
  • types.NetworkInterfaces: This defines the network configuration parameters.
type NetworkInterfaces struct {
	IfaceID     string `json:"iface_id"`
	GuestMac    string `json:"guest_mac"`
	HostDevName string `json:"host_dev_name"`
}
  • types.MMDSConfig: This contains a list of the network configuration parameters defined by NetworkInterfaces struct.
type MMDSConfig struct {
	NetworkInterface []string `json:"network_interfaces"`
}
  • types.MMDSMsg TBD: This contains the latest metadata of the machine.
type MMDSMsg struct {
	Latest struct {
		Metadata struct {
			types.MMDSMetadata
		} `json:"meta-data"`
	} `json:"latest"`
}
  • types.MMDSMetadata TBD: This contains the metadata of the machine.
type MMDSMetadata struct {
	NodeId string `json:"node_id"`
	PKey   string `json:"pkey"`
}
  • types.Actions TBD: This contains the type of action to be performed on the Firecracker VM.
type Actions struct {
	ActionType string `json:"action_type"`
}
  • types.VirtualMachine: This contains the configuration parameters of Firecracker virtual machine.
type VirtualMachine struct {
	types.BaseDBModel
	SocketFile string `json:"socket_file"`
	BootSource string `json:"boot_source"`
	Filesystem string `json:"filesystem"`
	VCPUCount  int    `json:"vcpu_count"`
	MemSizeMib int    `json:"mem_size_mib"`
	TapDevice  string `json:"tap_device"`
	State      string `json:"state"`
}
Machine
  • types.IP
type IP []any
  • types.PeerInfo TBD: This contains parameters of the peer node.
type PeerInfo struct {
	types.BaseDBModel
	NodeID    string `json:"nodeID,omitempty"`
	Key       string `json:"key,omitempty"`
	Mid       string `json:"mid,omitempty"`
	PublicKey string `json:"public_key,omitempty"`
	Address   string `json:"_address,omitempty"`
}
  • types.Machine: This contains the configuration parameters of the machine.
type Machine struct {
	types.BaseDBModel
	NodeId               string
	PeerInfo             int
	IpAddr               string
	AvailableResources   int
	FreeResources        int
	TokenomicsAddress    string
	TokenomicsBlockchain string
}
  • types.FreeResources: This contains the resources currently available for a job.
// FreeResources are the resources free to be used by new services,
// plugins and any other processes started by DMS. It's basically
// the subtraction between AvailableResources and the amount of resources
// already used by DMS and its processes (mostly services)
type FreeResources struct {
	BaseDBModel
	TotCpuHz          int     `json:"tot_cpu_hz"`
	PriceCpu          float64 `json:"price_cpu"`
	Ram               int     `json:"ram"`
	PriceRam          float64 `json:"price_ram"`
	Vcpu              int     `json:"vcpu"`
	Disk              float64 `json:"disk"`
	PriceDisk         float64 `json:"price_disk"`
	NTXPricePerMinute float64 `json:"ntx_price"`
}
  • types.AvailableResources: This contains the resources onboarded to Nunet by the user.
// AvailableResources are the amount of resources onboarded which
// can be used by NuNet
type AvailableResources struct {
	types.BaseDBModel
	TotCpuHz          int
	CpuNo             int
	CpuHz             float64
	PriceCpu          float64
	Ram               int
	PriceRam          float64
	Vcpu              int
	Disk              float64
	PriceDisk         float64
	NTXPricePerMinute float64
}
  • types.Services TBD: This contains the details of the services running on the machine.
type Services struct {
	types.BaseDBModel
	TxHash               string
	TransactionType      string // transaction type can be running, done, withdraw, refund and distribute
	JobStatus            string // whether job is running or exited; one of these 'running', 'finished without errors', 'finished with errors'
	JobDuration          int64  // job duration in minutes
	EstimatedJobDuration int64  // job duration in minutes
	ServiceName          string
	ContainerID          string
	ResourceRequirements int
	ImageID              string
	LogURL               string
	LastLogFetch         time.Time
	ServiceProviderAddr  string
	ComputeProviderAddr  string
	MetadataHash         string
	WithdrawHash         string
	RefundHash           string // saving hashes for call the `/request-reward` endpoint by SPD
	Distribute_50Hash    string
	Distribute_75Hash    string
	SignatureDatum       string
	MessageHashDatum     string
	Datum                string
	SignatureAction      string // saving signatures for removing redundancy of calling Oracle
	MessageHashAction    string
	Action               string
	// TODO: Add ContainerType field

}
  • types.ServiceResourceRequirements: This contains the resource requirements for a service.
type ServiceResourceRequirements struct {
	types.BaseDBModel
	CPU  int
	RAM  int
	VCPU int
	HDD  int
}
  • types.ContainerImages: This contains parameters of a container image.
type ContainerImages struct {
	gorm.Model
	ImageID   string
	ImageName string
	Digest    string
}
  • types.Libp2pInfo: This contains parameters of Libp2p node.
type Libp2pInfo struct {
	BaseDBModel
	PrivateKey []byte `json:"private_key"`
	PublicKey  []byte `json:"public_key"`
	ServerMode bool   `json:"server_mode"`
	Available  bool   `json:"available"`
}
  • types.MachineUUID: This defines the unique identifier for the machine.
type MachineUUID struct {
	BaseDBModel
	UUID string `json:"uuid"`
}
  • types.Gpu: This contains the GPU parameters of the machine.
type Gpu struct {
	Name     string `json:"name"`
	TotVram  uint64 `json:"tot_vram"`
	FreeVram uint64 `json:"free_vram"`
}
  • types.resources: This defines the resource parameters of the machine.
type resources struct {
	TotCpuHz  float64
	PriceCpu  float64
	Ram       int
	PriceRam  float64
	Vcpu      int
	Disk      float64
	PriceDisk float64
}
  • types.PeerData: This contains the details of the peer node.
type PeerData struct {
	PeerID               string        `json:"peer_id"`
	IsAvailable          bool          `json:"is_available"`
	HasGpu               bool          `json:"has_gpu"`
	GpuInfo              []Gpu         `json:"gpu_info"`
	TokenomicsAddress    string        `json:"tokenomics_addrs"`
	TokenomicsBlockchain string        `json:"tokenomics_blockchain"`
	AvailableResources   FreeResources `json:"available_resources"`
	Services             []Services    `json:"services"`
	Timestamp            int64         `json:"timestamp,omitempty"`
}
  • types.Connection: TBD
type Connection struct {
	types.BaseDBModel
	PeerID     string `json:"peer_id"`
	Multiaddrs string `json:"multiaddrs"`
}
  • types.PingResult: The contains the details of the ping result.
type PingResult struct {
	RTT     time.Duration
	Success bool
	Error   error
}
  • types.Machines: TBD
type Machines map[string]PeerData
  • types.KadDHTMachineUpdate: This contains machine info for KAD-DHT.
// machine info for KAD-DHT
type KadDHTMachineUpdate struct {
	Data      []byte `json:"data"`
	Signature []byte `json:"signature"`
}
  • types.ElasticToken: TBD
type ElasticToken struct {
	types.BaseDBModel
	NodeId      string
	Token       string
	ChannelName string
}
Onboarding
  • types.BlockchainAddressPrivKey
// BlockchainAddressPrivKey holds Ethereum/Cardano wallet address and private key from which the
// address is derived.
type BlockchainAddressPrivKey struct {
	Address    string `json:"address,omitempty"`
	PrivateKey string `json:"private_key,omitempty"`
	Mnemonic   string `json:"mnemonic,omitempty"`
}
  • types.CapacityForNunet
// CapacityForNunet is a struct required in request body for the onboarding
type CapacityForNunet struct {
	Memory            int64   `json:"memory,omitempty"`
	CPU               int64   `json:"cpu,omitempty"`
	NTXPricePerMinute float64 `json:"ntx_price,omitempty"`
	Channel           string  `json:"channel,omitempty"`
	PaymentAddress    string  `json:"payment_addr,omitempty"`
	ServerMode        bool    `json:"server_mode,omitempty,"`
	IsAvailable       bool    `json:"is_available"`
}
  • types.Provisioned
// Provisioned struct holds data about how much total resource
// host machine is equipped with
type Provisioned struct {
	CPU      float64 `json:"cpu,omitempty"`
	Memory   uint64  `json:"memory,omitempty"`
	NumCores uint64  `json:"total_cores,omitempty"`
}
  • types.Metadata
// Metadata - machine metadata of onboarding parameters
type Metadata struct {
	Name            string `json:"name,omitempty"`
	UpdateTimestamp int64  `json:"update_timestamp,omitempty"`
	Resource        struct {
		MemoryMax int64 `json:"memory_max,omitempty"`
		TotalCore int64 `json:"total_core,omitempty"`
		CPUMax    int64 `json:"cpu_max,omitempty"`
	} `json:"resource,omitempty"`
	Available struct {
		CPU    int64 `json:"cpu,omitempty"`
		Memory int64 `json:"memory,omitempty"`
	} `json:"available,omitempty"`
	Reserved struct {
		CPU    int64 `json:"cpu,omitempty"`
		Memory int64 `json:"memory,omitempty"`
	} `json:"reserved,omitempty"`
	Network           string  `json:"network,omitempty"`
	PublicKey         string  `json:"public_key,omitempty"`
	NodeID            string  `json:"node_id,omitempty"`
	GpuInfo           []Gpu   `json:"gpu_info,omitempty"`
	Dashboard         string  `json:"dashboard,omitempty"`
	NTXPricePerMinute float64 `json:"ntx_price,omitempty"`
}
  • types.OnboardingStatus
type OnboardingStatus struct {
	Onboarded    bool   `json:"onboarded"`
	Error        error  `json:"error"`
	MachineUUID  string `json:"machine_uuid"`
	MetadataPath string `json:"metadata_path"`
	DatabasePath string `json:"database_path"`
}
  • types.LogBinAuth: This stores the authorisation token for LogBin.
type LogBinAuth struct {
	types.BaseDBModel
	PeerID      string `json:"peer_id"`
	MachineUUID string `json:"machine_uuid"`
	Token       string `json:"token"`
}
Resource
  • types.Resources: resources defined for the machine.
type Resources struct {
    CPU      float64
    NumCores uint64
    GPU      []types.GPU `gorm:"foreignKey:ResourceID"`
    RAM      uint64
    Disk     uint64
}
  • types.AvailableResources: resources onboarded to Nunet.
type AvailableResources struct {
    types.BaseDBModel
    Resources
}
  • types.FreeResources: resources currently available for new jobs.
type FreeResources struct {
    types.BaseDBModel
    Resources
}
  • types.RequiredResources: resources required by the jobs running on the machine.
type RequiredResources struct {
    types.BaseDBModel
    Resources
}

types.GPUVendor: GPU vendors available on the machine.

type GPUVendor string

const (
	GPUVendorNvidia  GPUVendor = "NVIDIA"
	GPUVendorAMDATI  GPUVendor = "AMD/ATI"
	GPUVendorIntel   GPUVendor = "Intel"
	GPUVendorUnknown GPUVendor = "Unknown"
	None             GPUVendor = "None"
)
  • types.GPU: GPU details.
type GPU struct {
	// Index is the self-reported index of the device in the system
	Index int
	// Vendor is the maker of the GPU, e.g. NVidia, AMD, Intel
	Vendor types.GPUVendor
	// PCIAddress is the PCI address of the device, in the format AAAA:BB:CC.C
	// Used to discover the correct device rendering cards
	PCIAddress string
	// Model represents the GPU model name, e.g., "Tesla T4", "A100"
	Model string `json:"model" description:"GPU model, e.g., Tesla T4, A100"`
	// TotalVRAM is the total amount of VRAM on the device
	TotalVRAM uint64
	// UsedVRAM is the amount of VRAM currently in use
	UsedVRAM uint64
	// FreeVRAM is the amount of VRAM currently free
	FreeVRAM uint64

	// Gorm fields
	ResourceID uint `gorm:"foreignKey:ID"`
}
  • types.GPUList: A slice of GPU.
type GPUList []types.GPU
  • types.CPUInfo: CPU information of the machine.
type CPUInfo struct {
    NumCores   uint64
    MHzPerCore float64
    Compute    float64
}
  • types.SpecInfo: detailed specifications of the machine.
type SpecInfo struct {
	CPUs    []types.CPU
	GPUs    []types.GPU
	RAMs    []types.RAM
	Disks   []types.Disk
	Network NetworkInfo
}
  • types.CPU: CPU details.
type CPU struct {
	// Model represents the CPU model, e.g., "Intel Core i7-9700K", "AMD Ryzen 9 5900X"
	Model string

	// Vendor represents the CPU manufacturer, e.g., "Intel", "AMD"
	Vendor string

	// ClockSpeedHz represents the CPU clock speed in Hz
	ClockSpeedHz uint64

	// Cores represents the number of physical CPU cores
	Cores int

	// Threads represents the number of logical CPU threads (including hyperthreading)
	Threads int

	// Architecture represents the CPU architecture, e.g., "x86", "x86_64", "arm64"
	Architecture string

	// Cache size in bytes
	CacheSize uint64
}
  • types.RAM: RAM details.
type RAM struct {
	// Size in bytes
	Size uint64

	// Clock speed in Hz
	ClockSpeedHz uint64

	// Type represents the RAM type, e.g., "DDR4", "DDR5", "LPDDR4"
	Type string
}
  • types.Disk: Disk details.
type Disk struct {
	// Model represents the disk model, e.g., "Samsung 970 EVO Plus", "Western Digital Blue SN550"
	Model string

	// Vendor represents the disk manufacturer, e.g., "Samsung", "Western Digital"
	Vendor string

	// Size in bytes
	Size uint64

	// Type represents the disk type, e.g., "SSD", "HDD", "NVMe"
	Type string

	// Interface represents the disk interface, e.g., "SATA", "PCIe", "M.2"
	Interface string

	// Read speed in bytes per second
	ReadSpeed uint64
	// Write speed in bytes per second
	WriteSpeed uint64
}
  • types.NetworkInfo: Network details.
type NetworkInfo struct {
	// Bandwidth in bits per second (b/s)
	Bandwidth uint64

	// NetworkType represents the network type, e.g., "Ethernet", "Wi-Fi", "Cellular"
	NetworkType string
}
  • types.Resource: resources resources required to execute a task
Spec_config
  • types.SpecConfig: This allows arbitrary configuration to be defined as needed.
// SpecConfig represents a configuration for a spec
// A SpecConfig can be used to define an engine spec, a storage volume, etc.
type SpecConfig struct {
	// Type of the spec (e.g. docker, firecracker, storage, etc.)
	Type string `json:"type"`
	// Params of the spec
	Params map[string]interface{} `json:"params,omitempty"`
}
Storage
  • types.StorageVolume: This contains the parameters related to the storage volume that is created by the DMS on the local machine.
// StorageVolume represents a prepared storage volume that can be mounted to an execution
type StorageVolume struct {
	// Type of the volume (e.g. bind)
	Type string `json:"type"`
	// Source path of the volume on the host
	Source string `json:"source"`
	// Target path of the volume in the execution
	Target string `json:"target"`
	// ReadOnly flag to mount the volume as read-only
	ReadOnly bool `json:"readonly"`
}
Telemetry Config
  • types.CollectorConfig: This contains the parameters for a collector.
type CollectorConfig struct {
	CollectorType     string
	CollectorEndpoint string
}
  • types.TelemetryConfig: This defines the telemetry parameters such as obervability level, collector configurations etc.
type TelemetryConfig struct {
	ServiceName        string
	GlobalEndpoint     string
	ObservabilityLevel int
	CollectorConfigs   map[string]CollectorConfig
}

ObservabilityLevel is an enum that defines the level of observability. Currently logging is done at these observability levels.

    TRACE ObservabilityLevel = 1
	DEBUG ObservabilityLevel = 2
	INFO  ObservabilityLevel = 3
	WARN  ObservabilityLevel = 4
	ERROR ObservabilityLevel = 5
	FATAL ObservabilityLevel = 6
Types
  • types.BaseDBModel
// BaseDBModel is a base model for all entities. It'll be mainly used for database
// records.
type BaseDBModel struct {
	ID        string `gorm:"type:uuid"`
	CreatedAt time.Time
	UpdatedAt time.Time
	DeletedAt gorm.DeletedAt `gorm:"index"`
}
Testing

Test are defined in other packages where functionality is implemented.

Proposed Functionality / Requirements
List of issues

All issues that are related to the implementation of types package can be found below. These include any proposals for modifications to the package or new data structures needed to cover the requirements of other packages.

proposed Encryption interfaces

These are placeholder interface definitions which will be developed in the future.

Encryptor must support encryption of files/directories

type Encryptor interface {
	Encrypt([]byte) ([]byte, error)
}

Decryptor must support decryption of files/directories

type Decryptor interface {
	Decrypt([]byte) ([]byte, error)
}
proposed Network types and methods

This section contains the proposed data types and methods related to network functionality.

  • types.NetworkSpec
// NetworkSpec is a stub. Please expand based on requirements.
type NetworkSpec struct {
}
  • types.NetConfig
// NetConfig is a stub. Please expand it or completely change it based on requirements.
type NetConfig struct {
	NetworkSpec SpecConfig `json:"network_spec"` // Network specification
}

types.NetConfig struct will implement a GetNetworkConfig method which returns network configuration parameters.

func (nc *NetConfig) GetNetworkConfig() *SpecConfig {
	return &nc.NetworkSpec
}
  • types.NetworkStats
// NetworkStats should contain all network info the user is interested in.
// for now there's only peerID and listening address but reachability, local and remote addr etc...
// can be added when necessary.
type NetworkStats struct {
	ID         string `json:"id"`
	ListenAddr string `json:"listen_addr"`
}
  • types.MessageInfo
// MessageInfo is a stub. Please expand it or completely change it based on requirements.
type MessageInfo struct {
	Info string `json:"info"` // Message information
}
proposed Network configuration data type
  • types.MessageEnvelope
type MessageType string

type MessageEnvelope struct {
	Type MessageType
	Data []byte
}
  • types.NetworkConfig
type NetworkType string

type NetworkConfig struct {
	Type NetworkType

	// libp2p
	types.Libp2pConfig

	// nats
	NATSUrl string
}
  • types.Libp2pConfig

// Libp2pConfig holds the libp2p configuration type Libp2pConfig struct { DHTPrefix string //crypto is Go-libp2p package that implements various cryptographic utilities PrivateKey crypto.PrivKey BootstrapPeers []multiaddr.Multiaddr Rendezvous string Server bool Scheduler *bt.Scheduler CustomNamespace string ListenAddress []string PeerCountDiscoveryLimit int PrivateNetwork types.PrivateNetworkConfig GracePeriodMs int GossipMaxMessageSize int }

  • types.PrivateNetworkConfig
type PrivateNetworkConfig struct {
	// WithSwarmKey if true, DMS will try to fetch the key from
	// `<config_path>/swarm.key`.
	WithSwarmKey bool

	// ACL defines the access control list for the private network.
	ACL []multiaddr.Multiaddr
}
References

Documentation

Index

Constants

View Source 
const (
	StorageVolumeTypeBind = "bind"

	StorageProviderS3   = "s3"
	StorageProviderIPFS = "ipfs"
)
View Source 
const (
	NetP2P = "p2p"
)

Variables

This section is empty.

Functions

func ConvertBytesToGB 

func ConvertBytesToGB(bytes float64) float64

ConvertBytesToGB converts bytes to gigabytes

func ConvertGBToBytes 

func ConvertGBToBytes(gb float64) float64

ConvertGBToBytes converts gigabytes to bytes

func ConvertHzToGHz 

func ConvertHzToGHz(hz float64) float64

ConvertHzToGHz converts hertz to gigahertz

func ConvertMibToBytes 

func ConvertMibToBytes(mib float64) float64

ConvertMibToBytes converts mebibytes to bytes

func ConvertNumericToFloat64 

func ConvertNumericToFloat64(n any) (float64, bool)

func ConvertTypedSliceToUntypedSlice 

func ConvertTypedSliceToUntypedSlice(typedSlice interface{}) []interface{}

func IsSameShallowType 

func IsSameShallowType(a, b interface{}) bool

func IsStrictlyContained 

func IsStrictlyContained(leftSlice, rightSlice []interface{}) bool

IsStrictlyContained checks if all elements of rightSlice are contained in leftSlice

func IsStrictlyContainedInt 

func IsStrictlyContainedInt(leftSlice, rightSlice []int) bool

IsStrictlyContainedInt checks if all elements of rightSlice are contained in leftSlice

func NewHealthCheck 

func NewHealthCheck(mf HealthCheckManifest, fn func(HealthCheckManifest) error) (func() error, error)

func SliceContainsOneValue 

func SliceContainsOneValue(slice []Comparison, value Comparison) bool

Types

type Account 

type Account struct {
	Address    string `json:"address,omitempty"`
	PrivateKey string `json:"private_key,omitempty"`
	Mnemonic   string `json:"mnemonic,omitempty"`
}

Account represents a blockchain account with an address, private key, and mnemonic.

type BaseDBModel 

type BaseDBModel struct {
	ID        string `gorm:"type:uuid"`
	CreatedAt time.Time
	UpdatedAt time.Time
	DeletedAt gorm.DeletedAt `gorm:"index"`
}

BaseDBModel is a base model for all entities. It'll be mainly used for database records.

func (*BaseDBModel) BeforeCreate 

func (m *BaseDBModel) BeforeCreate(_ *gorm.DB) error

BeforeCreate sets the ID and CreatedAt fields before creating a new entity. This is a GORM hook and should not be called directly. We can move this to generic repository create methods.

func (*BaseDBModel) BeforeUpdate 

func (m *BaseDBModel) BeforeUpdate(_ *gorm.DB) error

BeforeUpdate sets the UpdatedAt field before updating an entity. This is a GORM hook and should not be called directly. We can move this to generic repository create methods.

func (BaseDBModel) GetID 

func (m BaseDBModel) GetID() string

GetID returns the ID of the entity.

type Behavior 

type Behavior struct {
	DynamicTemplate string
	Static          string
}

type BootSource 

type BootSource struct {
	KernelImagePath string `json:"kernel_image_path"`
	BootArgs        string `json:"boot_args"`
}

type CPU 

type CPU struct {
	// ClockSpeed represents the CPU clock speed in Hz
	ClockSpeed float64 `json:"clock_speed" description:"CPU clock speed in Hz"`

	// Cores represents the number of physical CPU cores
	Cores float32 `json:"cores" description:"Number of physical CPU cores"`

	// TODO: capture the below fields if required
	// Model represents the CPU model, e.g., "Intel Core i7-9700K", "AMD Ryzen 9 5900X"
	Model string `json:"model,omitempty" description:"CPU model, e.g., Intel Core i7-9700K, AMD Ryzen 9 5900X"`

	// Vendor represents the CPU manufacturer, e.g., "Intel", "AMD"
	Vendor string `json:"vendor,omitempty" description:"CPU manufacturer, e.g., Intel, AMD"`

	// Threads represents the number of logical CPU threads (including hyperthreading)
	Threads int `json:"threads,omitempty" description:"Number of logical CPU threads (including hyperthreading)"`

	// Architecture represents the CPU architecture, e.g., "x86", "x86_64", "arm64"
	Architecture string `json:"architecture,omitempty" description:"CPU architecture, e.g., x86, x86_64, arm64"`

	// Cache size in bytes
	CacheSize uint64 `json:"cache_size,omitempty" description:"CPU cache size in bytes"`
}

CPU represents the CPU information

func (*CPU) Add 

func (c *CPU) Add(other CPU) error

Add adds the other CPU to the current CPU

func (*CPU) ClockSpeedInGHz 

func (c *CPU) ClockSpeedInGHz() float64

ClockSpeedInGHz returns the clock speed in GHz

func (*CPU) Compare 

func (c *CPU) Compare(other CPU) (Comparison, error)

Compare compares the CPU with the other CPU

func (*CPU) Compute 

func (c *CPU) Compute() float64

Compute returns the total compute power of the CPU in Hz

func (*CPU) ComputeInGHz 

func (c *CPU) ComputeInGHz() float64

ComputeInGHz returns the total compute power of the CPU in GHz

func (*CPU) Subtract 

func (c *CPU) Subtract(other CPU) error

Subtract subtracts the other CPU from the current CPU

type Calculable 

type Calculable[T any] interface {
	Add(other T) error
	Subtract(other T) error
}

type CommittedResources 

type CommittedResources struct {
	BaseDBModel
	Resources
	AllocationID string `json:"allocationID"`
}

CommittedResources represents the committed resources of the machine

type Comparable 

type Comparable[T any] interface {
	Compare(other T) (Comparison, error)
}

Comparable public Comparable interface to be enforced on types that can be compared

type Comparator 

type Comparator func(l, r interface{}, preference ...Preference) Comparison

type Comparison 

type Comparison string

Comparison is a type for comparison results 

const (
	// Worse means left object is 'worse' than right object
	Worse Comparison = "Worse"
	// Better means left object is 'better' than right object
	Better Comparison = "Better"
	// Equal means objects on the left and right are 'equally good'
	Equal Comparison = "Equal"
	// None means comparison could not be performed
	None Comparison = "None"
)

func LiteralComparator 

func LiteralComparator[T ~string](l, r T, _ ...Preference) Comparison

func NumericComparator 

func NumericComparator[T float64 | float32 | int | int32 | int64 | uint64](l, r T, _ ...Preference) Comparison

func (Comparison) And 

func (c Comparison) And(cmp Comparison) Comparison

And returns the result of AND operation of two Comparison values it respects the following table of truth: | AND | Better | Worse | Equal | None | | ------ | ------ |--------|--------|--------| | Better | Better | Worse | Better | None | | Worse | Worse | Worse | Worse | None | | Equal | Better | Worse | Equal | None | | None | None | None | None | None |

type ComplexComparison 

type ComplexComparison map[string]Comparison

ComplexComparison is a map of string to Comparison

func ComplexCompare 

func ComplexCompare(l, r interface{}) ComplexComparison

ComplexCompare helper function to return a complex comparison of two complex types this uses reflection, could become a performance bottleneck with generics, we wouldn't need this function and could use the ComplexCompare method directly

func (*ComplexComparison) Result 

func (c *ComplexComparison) Result() Comparison

Result returns the result of AND operation of all Comparison values in the ComplexComparison

type Config 

type Config interface {
	GetNetworkConfig() *SpecConfig
}

type Connectivity 

type Connectivity struct {
	Ports []int `json:"ports" description:"Ports that need to be open for the job to run"`
	VPN   bool  `json:"vpn" description:"Whether VPN is required"`
}

Connectivity represents the network configuration

func (*Connectivity) Add 

func (c *Connectivity) Add(other Connectivity) error

func (*Connectivity) Compare 

func (c *Connectivity) Compare(other Connectivity) (Comparison, error)

func (*Connectivity) Subtract 

func (c *Connectivity) Subtract(other Connectivity) error

type Decryptor 

type Decryptor interface {
	Decrypt([]byte) ([]byte, error)
}

Decryptor (DRAFT)

Warning: this is just a draft. And it might be moved to an Encryption package

TODO: it must support decryption of files/directories, otherwise we have to create another interface for the usecase

type DeviceResourceChange 

type DeviceResourceChange struct {
	PeerID                   string
	ChangedAttributeAndValue struct {
		CPU           float32
		RAM           float32
		Network       float32
		DedicatedTime float32
	}
	Timestamp float32
}

DeviceResourceChange defines the schema of the data to be sent to stats db when a device resource gets changed

type DeviceResourceConfig 

type DeviceResourceConfig struct {
	PeerID                   string
	ChangedAttributeAndValue struct {
		CPU           float32
		RAM           float32
		Network       float32
		DedicatedTime float32
	}
	Timestamp float32
}

DeviceResourceConfig defines the schema of the data to be sent to stats db when a device resource config gets changed

type DeviceStatusChange 

type DeviceStatusChange struct {
	PeerID    string
	Status    string
	Timestamp float32
}

DeviceStatusChange defines the schema of the data to be sent to stats db when a device status gets changed

type Disk 

type Disk struct {
	// Size in bytes
	// TODO: uint64!!!
	Size float64 `json:"size" description:"Size of the disk in bytes"`

	// TODO: capture the below fields if required
	// Model represents the disk model, e.g., "Samsung 970 EVO Plus", "Western Digital Blue SN550"
	Model string `json:"model,omitempty" description:"Disk model, e.g., Samsung 970 EVO Plus, Western Digital Blue SN550"`

	// Vendor represents the disk manufacturer, e.g., "Samsung", "Western Digital"
	Vendor string `json:"vendor,omitempty" description:"Disk manufacturer, e.g., Samsung, Western Digital"`

	// Type represents the disk type, e.g., "SSD", "HDD", "NVMe"
	Type string `json:"type,omitempty" description:"Disk type, e.g., SSD, HDD, NVMe"`

	// Interface represents the disk interface, e.g., "SATA", "PCIe", "M.2"
	Interface string `json:"interface,omitempty" description:"Disk interface, e.g., SATA, PCIe, M.2"`

	// Read speed in bytes per second
	ReadSpeed uint64 `json:"read_speed,omitempty" description:"Read speed in bytes per second"`

	// Write speed in bytes per second
	WriteSpeed uint64 `json:"write_speed,omitempty" description:"Write speed in bytes per second"`
}

Disk represents the disk information

func (*Disk) Add 

func (d *Disk) Add(other Disk) error

Add adds the other Disk to the current Disk

func (*Disk) Compare 

func (d *Disk) Compare(other Disk) (Comparison, error)

Compare compares the Disk with the other Disk

func (*Disk) SizeInGB 

func (d *Disk) SizeInGB() float64

SizeInGB returns the size in gigabytes

func (*Disk) Subtract 

func (d *Disk) Subtract(other Disk) error

Subtract subtracts the other Disk from the current Disk

type Drives 

type Drives struct {
	DriveID      string `json:"drive_id"`
	PathOnHost   string `json:"path_on_host"`
	IsRootDevice bool   `json:"is_root_device"`
	IsReadOnly   bool   `json:"is_read_only"`
}

type EncryptionType 

type EncryptionType int
const (
	EncryptionTypeNull EncryptionType = iota
)

type Encryptor 

type Encryptor interface {
	Encrypt([]byte) ([]byte, error)
}

Encryptor (DRAFT)

Warning: this is just a draft. And it might be moved to an Encryption package

TODO: it must support encryption of files/directories, otherwise we have to create another interface for the usecase

type ExecutionListItem 

type ExecutionListItem struct {
	ExecutionID string // ID of the execution
	Running     bool
}

ExecutionListItem is the result of the current executions.

type ExecutionRequest 

type ExecutionRequest struct {
	JobID               string                   // ID of the job to execute
	ExecutionID         string                   // ID of the execution
	EngineSpec          *SpecConfig              // Engine spec for the execution
	Resources           *Resources               // Resources for the execution
	Inputs              []*StorageVolumeExecutor // Input volumes for the execution
	Outputs             []*StorageVolumeExecutor // Output volumes for the results
	ResultsDir          string                   // Directory to store the results
	PersistLogsDuration time.Duration            // Duration to persist logs on disk
	ProvisionScripts    map[string][]byte        // (named) Scripts to run when initiating the execution
	PortsToBind         []PortsToBind            // List of ports to bind
	GatewayIP           string                   // Gateway IP to use as dns resolver
}

ExecutionRequest is the request object for executing a job

type ExecutionResult 

type ExecutionResult struct {
	STDOUT   string `json:"stdout"`    // STDOUT of the execution
	STDERR   string `json:"stderr"`    // STDERR of the execution
	ExitCode int    `json:"exit_code"` // Exit code of the execution
	ErrorMsg string `json:"error_msg"` // Error message if the execution failed
}

ExecutionResult is the result of an execution

func NewExecutionResult 

func NewExecutionResult(code int) *ExecutionResult

NewExecutionResult creates a new ExecutionResult object

func NewFailedExecutionResult 

func NewFailedExecutionResult(err error) *ExecutionResult

NewFailedExecutionResult creates a new ExecutionResult object for a failed execution It sets the error message from the provided error and sets the exit code to -1

type ExecutionStatus 

type ExecutionStatus string

ExecutionStatus is the status of an execution 

const (
	ExecutionStatusPending ExecutionStatus = "pending"
	ExecutionStatusRunning ExecutionStatus = "running"
	ExecutionStatusPaused  ExecutionStatus = "paused"
	ExecutionStatusFailed  ExecutionStatus = "failed"
	ExecutionStatusSuccess ExecutionStatus = "success"
)

type Executor 

type Executor struct {
	ExecutorType ExecutorType `json:"executor_type"`
}

Executor is the executor type

func (*Executor) Compare 

func (e *Executor) Compare(other Executor) (Comparison, error)

Compare compares two Executor objects

func (*Executor) Equal 

func (e *Executor) Equal(executor Executor) bool

Equal checks if two Executor objects are equal

type ExecutorType 

type ExecutorType string

ExecutorType is the type of the executor 

const (
	ExecutorTypeDocker      ExecutorType = "docker"
	ExecutorTypeFirecracker ExecutorType = "firecracker"
	ExecutorTypeWasm        ExecutorType = "wasm"

	ExecutionStatusCodeSuccess = 0
)

func (ExecutorType) Compare 

func (e ExecutorType) Compare(other ExecutorType) (Comparison, error)

Compare compares two ExecutorType objects

func (ExecutorType) String 

func (e ExecutorType) String() string

String returns the string representation of the ExecutorType

type Executors 

type Executors []Executor

Executors is a list of Executor objects

func (*Executors) Add 

func (e *Executors) Add(other Executors) error

Add adds the Executor object to another Executor object

func (*Executors) Compare 

func (e *Executors) Compare(other Executors) (Comparison, error)

Compare compares two Executors objects

func (*Executors) Contains 

func (e *Executors) Contains(executor Executor) bool

Contains checks if an Executor object is in the list of Executors

func (*Executors) Subtract 

func (e *Executors) Subtract(other Executors) error

Subtract subtracts the Executor object from another Executor object

type FreeResources 

type FreeResources struct {
	BaseDBModel
	Resources
}

FreeResources represents the free resources of the machine

type GPU 

type GPU struct {
	// Index is the self-reported index of the device in the system
	Index int `json:"index" description:"GPU index in the system"`
	// Vendor is the maker of the GPU, e.g. NVidia, AMD, Intel
	Vendor GPUVendor `json:"vendor" description:"GPU vendor, e.g., NVidia, AMD, Intel"`
	// PCIAddress is the PCI address of the device, in the format AAAA:BB:CC.C
	// Used to discover the correct device rendering cards
	PCIAddress string `json:"pci_address" description:"PCI address of the device, in the format AAAA:BB:CC.C"`
	// Model represents the GPU model name, e.g., "Tesla T4", "A100"
	Model string `json:"model" description:"GPU model, e.g., Tesla T4, A100"`
	// VRAM is the total amount of VRAM on the device
	// TODO: uint64!!
	VRAM float64 `json:"vram" description:"Total amount of VRAM on the device"`
	// UUID is the unique identifier of the device
	UUID string `json:"uuid" description:"Unique identifier of the device"`

	// Gorm fields
	// Team, is this the right way to do this? What is the best practice we're following?
	ResourceID string `json:"-" gorm:"foreignKey:ID"`
}

GPU represents the GPU information

func (*GPU) Add 

func (g *GPU) Add(other GPU) error

Add adds the other GPU to the current GPU

func (*GPU) Compare 

func (g *GPU) Compare(other GPU) (Comparison, error)

Compare compares the GPU with the other GPU

func (*GPU) Equal 

func (g *GPU) Equal(other GPU) bool

Equal checks if the two GPUs are equal

func (*GPU) Subtract 

func (g *GPU) Subtract(other GPU) error

Subtract subtracts the other GPU from the current GPU

func (*GPU) VRAMInGB 

func (g *GPU) VRAMInGB() float64

VRAMInGB returns the VRAM in gigabytes

type GPUConnector 

type GPUConnector interface {
	GetGPUs() (GPUs, error)
	GetGPUUsage(uuid string) (float64, error)
	Shutdown() error
}

GPUConnector vendor-specific adapter that interacts with actual GPU hardware by using vendor-specific libraries

type GPUManager 

type GPUManager interface {
	GetGPUs() (GPUs, error)
	GetGPUUsage(uuid ...string) (GPUs, error)
	Shutdown() error
}

GPUManager defines the interface for managing GPU resources.

type GPUVendor 

type GPUVendor string
const (
	GPUVendorNvidia  GPUVendor = "NVIDIA"
	GPUVendorAMDATI  GPUVendor = "AMD/ATI"
	GPUVendorIntel   GPUVendor = "Intel"
	GPUVendorUnknown GPUVendor = "Unknown"
	GPUVendorNone    GPUVendor = "None"
)

func ParseGPUVendor 

func ParseGPUVendor(vendor string) GPUVendor

ParseGPUVendor parses the GPU vendor string and returns the corresponding GPUVendor enum

func (GPUVendor) Compare 

func (g GPUVendor) Compare(other GPUVendor) (Comparison, error)

type GPUs 

type GPUs []GPU

func (GPUs) Add 

func (gpus GPUs) Add(other GPUs) error

func (GPUs) Compare 

func (gpus GPUs) Compare(other GPUs) (Comparison, error)

func (GPUs) Copy 

func (gpus GPUs) Copy() GPUs

Copy returns a safe copy of the GPUs

func (GPUs) Equal 

func (gpus GPUs) Equal(other GPUs) bool

func (GPUs) GetWithIndex 

func (gpus GPUs) GetWithIndex(index int) (GPU, error)

GetWithIndex returns the GPU with the specified index

func (GPUs) MaxFreeVRAMGPU 

func (gpus GPUs) MaxFreeVRAMGPU() (GPU, error)

MaxFreeVRAMGPU returns the GPU with the maximum free VRAM from the list of GPUs

func (GPUs) Subtract 

func (gpus GPUs) Subtract(other GPUs) error

type GeoIPLocator 

type GeoIPLocator interface {
	Country(ipAddress net.IP) (*geoip2.Country, error)
	City(ipAddress net.IP) (*geoip2.City, error)
}

GeoIPLocator returns the info about an IP.

type HardwareCapability 

type HardwareCapability struct {
	Executors    Executors         `json:"executor" description:"Executor type required for the job (docker, vm, wasm, or others)"`
	JobTypes     JobTypes          `` /* 148-byte string literal not displayed */
	Resources    Resources         `json:"resources" description:"Resources required for the job"`
	Libraries    Libraries         `json:"libraries" description:"Libraries required for the job"`
	Localities   Localities        `json:"locality" description:"Preferred localities of the machine for execution"`
	Connectivity Connectivity      `json:"connectivity" description:"Network configuration required"`
	Price        PricesInformation `json:"price" description:"Pricing information"`
	Time         TimeInformation   `json:"time" description:"Time constraints"`
	KYCs         KYCs
}

HardwareCapability represents the hardware capability of the machine

func (*HardwareCapability) Add 

func (c *HardwareCapability) Add(other HardwareCapability) error

Add adds the resources of the given HardwareCapability to the current HardwareCapability

func (*HardwareCapability) Compare 

func (c *HardwareCapability) Compare(other HardwareCapability) (Comparison, error)

Compare compares two HardwareCapability objects

func (*HardwareCapability) Subtract 

func (c *HardwareCapability) Subtract(cap HardwareCapability) error

Subtract subtracts the resources of the given HardwareCapability from the current HardwareCapability

type HardwareManager 

type HardwareManager interface {
	GetMachineResources() (MachineResources, error)
	GetUsage() (Resources, error)
	GetFreeResources() (Resources, error)
}

HardwareManager defines the interface for managing machine resources.

type HealthCheckManifest 

type HealthCheckManifest struct {
	Type     string              `json:"type"`     // type of healthcheck
	Exec     []string            `json:"exec"`     // command to execute
	Endpoint string              `json:"endpoint"` // endpoint to check
	Response HealthCheckResponse `json:"response"` // expected response
	Interval time.Duration       `json:"interval"` // interval between healthchecks
}

type HealthCheckResponse 

type HealthCheckResponse struct {
	Type  string `json:"type"`  // type of response
	Value string `json:"value"` // value of response
}

type JobType 

type JobType string

JobType represents the type of the job 

const (
	Batch       JobType = "batch"
	SingleRun   JobType = "single_run"
	Recurring   JobType = "recurring"
	LongRunning JobType = "long_running"
)

func (JobType) Compare 

func (j JobType) Compare(other JobType) (Comparison, error)

type JobTypes 

type JobTypes []JobType

JobTypes a slice of JobType

func (*JobTypes) Add 

func (j *JobTypes) Add(other JobTypes) error

func (*JobTypes) Compare 

func (j *JobTypes) Compare(other JobTypes) (Comparison, error)

func (*JobTypes) Contains 

func (j *JobTypes) Contains(jobType JobType) bool

func (*JobTypes) Subtract 

func (j *JobTypes) Subtract(other JobTypes) error

type KYC 

type KYC struct {
	Type string `json:"type" description:"Type of KYC"`
	Data string `json:"data" description:"Data required for KYC"`
}

KYC represents the KYC data

func (*KYC) Compare 

func (k *KYC) Compare(other KYC) (Comparison, error)

func (*KYC) Equal 

func (k *KYC) Equal(kyc KYC) bool

type KYCs 

type KYCs []KYC

func (*KYCs) Add 

func (k *KYCs) Add(other KYCs) error

func (*KYCs) Compare 

func (k *KYCs) Compare(other KYCs) (Comparison, error)

func (*KYCs) Contains 

func (k *KYCs) Contains(kyc KYC) bool

func (*KYCs) Subtract 

func (k *KYCs) Subtract(other KYCs) error

type Libp2pConfig 

type Libp2pConfig struct {
	DHTPrefix               string
	PrivateKey              crypto.PrivKey
	BootstrapPeers          []multiaddr.Multiaddr
	Rendezvous              string
	Server                  bool
	Scheduler               *bt.Scheduler
	CustomNamespace         string
	ListenAddress           []string
	PeerCountDiscoveryLimit int
	GracePeriodMs           int
	GossipMaxMessageSize    int
	BootstrapMaxSleep       int // in minutes
	Memory                  int // in MB
	FileDescriptors         int
}

Libp2pConfig holds the libp2p configuration

type Libraries 

type Libraries []Library

func (*Libraries) Add 

func (l *Libraries) Add(other Libraries) error

func (*Libraries) Compare 

func (l *Libraries) Compare(other Libraries) (Comparison, error)

func (*Libraries) Contains 

func (l *Libraries) Contains(library Library) bool

func (*Libraries) Subtract 

func (l *Libraries) Subtract(other Libraries) error

type Library 

type Library struct {
	Name       string `json:"name" description:"Name of the library"`
	Constraint string `json:"constraint" description:"Constraint of the library"`
	Version    string `json:"version" description:"Version of the library"`
}

Library represents the libraries

func (*Library) Compare 

func (lib *Library) Compare(other Library) (Comparison, error)

func (*Library) Equal 

func (lib *Library) Equal(library Library) bool

type Localities 

type Localities []Locality

func (*Localities) Add 

func (l *Localities) Add(other Localities) error

func (*Localities) Compare 

func (l *Localities) Compare(other Localities) (Comparison, error)

func (*Localities) Contains 

func (l *Localities) Contains(locality Locality) bool

func (*Localities) Subtract 

func (l *Localities) Subtract(other Localities) error

type Locality 

type Locality struct {
	Kind string `json:"kind" description:"Kind of the region (geographic, nunet-defined, etc)"`
	Name string `json:"name" description:"Name of the region"`
}

Locality represents the locality

func (*Locality) Compare 

func (loc *Locality) Compare(other Locality) (Comparison, error)

func (*Locality) Equal 

func (loc *Locality) Equal(locality Locality) bool

type LogStreamRequest 

type LogStreamRequest struct {
	JobID       string // ID of the job
	ExecutionID string // ID of the execution
	Tail        bool   // Tail the logs
	Follow      bool   // Follow the logs
}

LogStreamRequest is the request object for streaming logs from an execution

type MMDSConfig 

type MMDSConfig struct {
	NetworkInterface []string `json:"network_interfaces"`
}

type MMDSMetadata 

type MMDSMetadata struct {
	NodeID string `json:"node_id"`
	PKey   string `json:"pkey"`
}

type MMDSMsg 

type MMDSMsg struct {
	Latest struct {
		Metadata struct {
			MMDSMetadata
		} `json:"meta-data"`
	} `json:"latest"`
}

type MachineConfig 

type MachineConfig struct {
	VCPUCount  int `json:"vcpu_count"`
	MemSizeMib int `json:"mem_size_mib"`
}

type MachineResources 

type MachineResources struct {
	BaseDBModel
	Resources
}

MachineResources represents the total resources of the machine

type MessageEnvelope 

type MessageEnvelope struct {
	Type MessageType
	Data []byte
}

type MessageInfo 

type MessageInfo struct {
	Info string `json:"info"` // Message information
}

MessageInfo is a stub. Please expand it or completely change it based on requirements.

type MessageType 

type MessageType string

type NetConfig 

type NetConfig struct {
	NetworkSpec SpecConfig `json:"network_spec"` // Network specification
}

NetConfig is a stub. Please expand it or completely change it based on requirements.

func (*NetConfig) GetNetworkConfig 

func (nc *NetConfig) GetNetworkConfig() *SpecConfig

type NetworkConfig 

type NetworkConfig struct {
	Type NetworkType

	// libp2p
	Libp2pConfig

	// nats
	NATSUrl string
}

type NetworkInfo 

type NetworkInfo struct {
	// Bandwidth in bits per second (b/s)
	Bandwidth uint64

	// NetworkType represents the network type, e.g., "Ethernet", "Wi-Fi", "Cellular"
	NetworkType string
}

NetworkInfo represents the network information TODO: not yet used, but can be used to capture the network information

type NetworkInterfaces 

type NetworkInterfaces struct {
	IfaceID     string `json:"iface_id"`
	GuestMac    string `json:"guest_mac"`
	HostDevName string `json:"host_dev_name"`
}

type NetworkSpec 

type NetworkSpec interface{}

NetworkSpec defines the network specification

type NetworkStats 

type NetworkStats struct {
	ID         string `json:"id"`
	ListenAddr string `json:"listen_addr"`
}

NetworkStats should contain all network info the user is interested in. for now there's only peerID and listening address but reachability, local and remote addr etc... can be added when necessary.

type NetworkType 

type NetworkType string
const (
	Libp2pNetwork NetworkType = "libp2p"
	NATSNetwork   NetworkType = "nats"
)

type NewDeviceOnboarded 

type NewDeviceOnboarded struct {
	PeerID        string
	CPU           float32
	RAM           float32
	Network       float32
	DedicatedTime float32
	Timestamp     float32
}

NewDeviceOnboarded defines the schema of the data to be sent to stats db when a new device gets onboarded

type NewService 

type NewService struct {
	ServiceID          string
	ServiceName        string
	ServiceDescription string
	Timestamp          float32
}

NewService defines the schema of the data to be sent to stats db when a new service gets registered in the platform

type NtxPayment 

type NtxPayment struct {
	CallID            float32
	ServiceID         string
	AmountOfNtx       int32
	PeerID            string
	SuccessFailStatus string
	Timestamp         float32
}

NtxPayment defines the schema of the data to be sent to stats db when a payment is made to device for the completion of service.

type OnboardedResources 

type OnboardedResources struct {
	BaseDBModel
	Resources
}

OnboardedResources represents the onboarded resources of the machine

type OnboardingConfig 

type OnboardingConfig struct {
	BaseDBModel
	IsOnboarded bool `json:"is_onboarded"`

	// OnboardedResources - resources that are onboarded
	// this is a transient field and not stored in the database directly
	// it is populated using the ResourceManager
	OnboardedResources Resources `json:"onboarded_resources,omitempty" gorm:"-" clover:"-"`
}

OnboardingConfig - parameters to configure onboarding

type OnboardingManager 

type OnboardingManager interface {
	IsOnboarded() (bool, error)
	Info(ctx context.Context) (OnboardingConfig, error)
	Onboard(ctx context.Context, config OnboardingConfig) (OnboardingConfig, error)
	Offboard(ctx context.Context) error
}

OnboardingManager - interface for onboarding

type PingResult 

type PingResult struct {
	RTT     time.Duration
	Success bool
	Error   error
}

type PortsToBind 

type PortsToBind struct {
	IP           string
	HostPort     int
	ExecutorPort int
}

type Preference 

type Preference struct {
	TypeName                  string
	Strength                  PreferenceString
	DefaultComparatorOverride Comparator
}

type PreferenceString 

type PreferenceString string
const (
	Hard PreferenceString = "Hard"
	Soft PreferenceString = "Soft"
)

type PriceInformation 

type PriceInformation struct {
	Currency        string `json:"currency" description:"Currency used for pricing"`
	CurrencyPerHour int    `json:"currency_per_hour" description:"Price charged per hour"`
	TotalPerJob     int    `json:"total_per_job" description:"Maximum total price or budget of the job"`
	Preference      int    `json:"preference" description:"Pricing preference"`
}

PriceInformation represents the pricing information

func (*PriceInformation) Compare 

func (p *PriceInformation) Compare(other PriceInformation) (Comparison, error)

func (*PriceInformation) Equal 

func (p *PriceInformation) Equal(price PriceInformation) bool

type PricesInformation 

type PricesInformation []PriceInformation

func (*PricesInformation) Add 

func (ps *PricesInformation) Add(other PricesInformation) error

func (*PricesInformation) Compare 

func (ps *PricesInformation) Compare(other PricesInformation) (Comparison, error)

func (*PricesInformation) Contains 

func (ps *PricesInformation) Contains(price PriceInformation) bool

func (*PricesInformation) Subtract 

func (ps *PricesInformation) Subtract(other PricesInformation) error

type RAM 

type RAM struct {
	// Size in bytes
	// TODO: uint64!!!
	Size float64 `json:"size" description:"Size of the RAM in bytes"`

	// TODO: capture the below fields if required
	// Clock speed in Hz
	ClockSpeed uint64 `json:"clock_speed,omitempty" description:"Clock speed of the RAM in Hz"`

	// Type represents the RAM type, e.g., "DDR4", "DDR5", "LPDDR4"
	Type string `json:"type,omitempty" description:"RAM type, e.g., DDR4, DDR5, LPDDR4"`
}

RAM represents the RAM information

func (*RAM) Add 

func (r *RAM) Add(other RAM) error

Add adds the other RAM to the current RAM

func (*RAM) Compare 

func (r *RAM) Compare(other RAM) (Comparison, error)

Compare compares the RAM with the other RAM

func (*RAM) SizeInGB 

func (r *RAM) SizeInGB() float64

SizeInGB returns the size in gigabytes

func (*RAM) Subtract 

func (r *RAM) Subtract(other RAM) error

Subtract subtracts the other RAM from the current RAM

type RequestTracker 

type RequestTracker struct {
	BaseDBModel
	ServiceType string
	NodeID      string
	CallID      int64
	Status      string
	RequestID   string
	MaxTokens   int
}

RequestTracker defines the schema of the data to be saved in db for tracking the status of the deployement request

nolint:nolintlint // Deprecated: Should be removed soon since its use has run out when statsdb was deprecated.

type ResourceAllocation 

type ResourceAllocation struct {
	BaseDBModel
	AllocationID string `json:"allocation_id"`
	Resources
}

ResourceAllocation represents the allocation of resources for a job

type ResourceManager 

type ResourceManager interface {
	// CommitResources preallocates the resources required by the jobs
	CommitResources(context.Context, CommittedResources) error
	// UncommitResources releases the resources that were preallocated for the jobs
	UncommitResources(context.Context, string) error
	// AllocateResources allocates the resources required by a job
	AllocateResources(context.Context, ResourceAllocation) error
	// DeallocateResources deallocates the resources required by a job
	DeallocateResources(context.Context, string) error
	// GetTotalAllocation returns the total allocations for the jobs
	GetTotalAllocation() (Resources, error)
	// GetFreeResources returns the free resources in the allocation pool
	GetFreeResources(ctx context.Context) (FreeResources, error)
	// GetOnboardedResources returns the onboarded resources of the machine
	GetOnboardedResources(context.Context) (OnboardedResources, error)
	// UpdateOnboardedResources updates the onboarded resources of the machine in the database
	UpdateOnboardedResources(context.Context, Resources) error
}

ResourceManager is an interface that defines the methods to manage the resources of the machine

type Resources 

type Resources struct {
	CPU  CPU  `json:"cpu" gorm:"embedded;embeddedPrefix:cpu_"`
	GPUs GPUs `json:"gpus,omitempty" gorm:"foreignKey:ResourceID"`
	RAM  RAM  `json:"ram" gorm:"embedded;embeddedPrefix:ram_"`
	Disk Disk `json:"disk" gorm:"embedded;embeddedPrefix:disk_"`
}

Resources represents the resources of the machine

func (*Resources) Add 

func (r *Resources) Add(other Resources) error

Add returns the sum of the resources

func (*Resources) Compare 

func (r *Resources) Compare(other Resources) (Comparison, error)

Compare compares two Resources objects

func (*Resources) Equal 

func (r *Resources) Equal(other Resources) bool

Equal returns true if the resources are equal

func (*Resources) Subtract 

func (r *Resources) Subtract(other Resources) error

Subtract returns the difference of the resources

type ServiceCall 

type ServiceCall struct {
	CallID              float32
	PeerIDOfServiceHost string
	ServiceID           string
	CPUUsed             float32
	MaxRAM              float32
	MemoryUsed          float32
	NetworkBwUsed       float32
	TimeTaken           float32
	Status              string
	AmountOfNtx         int32
	Timestamp           float32
}

ServiceCall defines the schema of the data to be sent to stats db when a host machine accepts a deployement request

type ServiceRemove 

type ServiceRemove struct {
	ServiceID string
	Timestamp float32
}

ServiceRemove defines the schema of the data to be sent to stats db when a new service gets removed from the platform

type ServiceStatus 

type ServiceStatus struct {
	CallID              float32
	PeerIDOfServiceHost string
	ServiceID           string
	Status              string
	Timestamp           float32
}

ServiceStatus defines the schema of update the status of service to stats db of the job being executed on host machine

type SpecConfig 

type SpecConfig struct {
	// Type of the spec (e.g. docker, firecracker, storage, etc.)
	Type string `json:"type"`
	// Params of the spec
	Params map[string]interface{} `json:"params,omitempty"`
}

SpecConfig represents a configuration for a spec A SpecConfig can be used to define an engine spec, a storage volume, etc.

func NewSpecConfig 

func NewSpecConfig(t string) *SpecConfig

NewSpecConfig creates a new SpecConfig with the given type

func (*SpecConfig) IsEmpty 

func (s *SpecConfig) IsEmpty() bool

IsEmpty returns true if the spec config is empty

func (*SpecConfig) IsType 

func (s *SpecConfig) IsType(t string) bool

IsType returns true if the current SpecConfig is of the given type

func (*SpecConfig) Normalize 

func (s *SpecConfig) Normalize()

Normalize ensures that the spec config is in a valid state

func (*SpecConfig) Validate 

func (s *SpecConfig) Validate() error

Validate checks if the spec config is valid

func (*SpecConfig) WithParam 

func (s *SpecConfig) WithParam(key string, value interface{}) *SpecConfig

WithParam adds a new key-value pair to the spec params

type StorageVolume 

type StorageVolume struct {
	BaseDBModel
	// CID is the content identifier of the storage volume.
	//
	// Warning: CID must be updated ONLY when locking volume (aka when volume was
	// is set to read-only)
	//
	// Be aware: Before relying on data's CID, be aware that it might be encrypted (
	// EncryptionType might be checked first if needed)
	//
	// TODO-maybe [type]: maybe it should be of type cid.CID
	CID string

	// Path points to the root of a DIRECTORY where data may be stored.
	Path string

	// ReadOnly indicates whether the storage volume is read-only or not.
	ReadOnly bool

	// Private indicates whether the storage volume is private or not.
	// If it's private, it shouldn't be shared with other nodes and it shouldn't
	// be persisted after the job is finished.
	// Practical application: if private, peer maintaining it shouldn't publish
	// its CID as if it was available to be worked on by other jobs.
	Private bool

	// EncryptionType indicates the type of encryption used for the storage volume.
	// In case no encryption is used, the value will be EncryptionTypeNull
	EncryptionType EncryptionType
}

StorageVolume contains the location (FS path) of a directory where certain data may be stored and metadata about the volume itself + data (if any).

Be aware that StorageVolume might not contain any data within it, specially when it's just created using the VolumeController, there will be no data in it by default.

TODO-maybe [job]: should volumes be related to a job or []job?

TODO-maybe: most of the fields should be private, callers shouldn't be able to altere the state of the volume except when using certain methods

type StorageVolumeExecutor 

type StorageVolumeExecutor struct {
	// Type of the volume (e.g. bind)
	Type string `json:"type"`
	// Source path of the volume on the host
	Source string `json:"source"`
	// Target path of the volume in the execution
	Target string `json:"target"`
	// ReadOnly flag to mount the volume as read-only
	ReadOnly bool `json:"readonly"`
}

StorageVolumeExecutor represents a prepared storage volume that can be mounted to an execution

type TimeInformation 

type TimeInformation struct {
	Units      string `json:"units" description:"Time units"`
	MaxTime    int    `json:"max_time" description:"Maximum time that job should run"`
	Preference int    `json:"preference" description:"Time preference"`
}

TimeInformation represents the time constraints

func (*TimeInformation) Add 

func (t *TimeInformation) Add(other TimeInformation) error

func (*TimeInformation) Compare 

func (t *TimeInformation) Compare(other TimeInformation) (Comparison, error)

func (*TimeInformation) Subtract 

func (t *TimeInformation) Subtract(other TimeInformation) error

func (*TimeInformation) TotalTime 

func (t *TimeInformation) TotalTime() int

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.