llmtrain

Documentation

Index

• func SetDefaultJobParams(job *model.JobM)
• func ShouldSkipOnIdempotency(job *model.JobM, condType string) bool
• type Limiter
• type StateMachine
  • func NewStateMachine(initial string, watcher *Watcher, job *model.JobM) *StateMachine
  • func (sm *StateMachine) Download(ctx context.Context, event *fsm.Event) error
  • func (sm *StateMachine) Embedding(ctx context.Context, event *fsm.Event) error
  • func (sm *StateMachine) EnterState(ctx context.Context, event *fsm.Event) error
  • func (sm *StateMachine) Train(ctx context.Context, event *fsm.Event) error
• type Watcher
  • func (w *Watcher) Run()
  • func (w *Watcher) SetAggregateConfig(config *watcher.AggregateConfig)
  • func (w *Watcher) SetMaxWorkers(maxWorkers int64)
  • func (w *Watcher) Spec() string

Functions

func SetDefaultJobParams

func SetDefaultJobParams(job *model.JobM)

SetDefaultJobParams sets default parameters for the job if they are not already set.

func ShouldSkipOnIdempotency

func ShouldSkipOnIdempotency(job *model.JobM, condType string) bool

ShouldSkipOnIdempotency determines whether a job should skip execution based on idempotency conditions.

Types

type Limiter

type Limiter struct {
	Embedding ratelimit.Limiter
	Train     ratelimit.Limiter
}

Limiter holds rate limiters for different operations.

type StateMachine

type StateMachine struct {
	Watcher *Watcher
	Job     *model.JobM
	FSM     *fsm.FSM
}

StateMachine represents a finite state machine for managing daily estimation jobs.

func NewStateMachine

func NewStateMachine(initial string, watcher *Watcher, job *model.JobM) *StateMachine

NewStateMachine initializes a new StateMachine with the given initial state, watcher, and job. It configures the FSM with defined events and their corresponding state transitions, as well as callbacks for entering specific states.

func (*StateMachine) Download

func (sm *StateMachine) Download(ctx context.Context, event *fsm.Event) error

Download retrieves feedback data from VOC and saves it to TOS.

func (*StateMachine) Embedding

func (sm *StateMachine) Embedding(ctx context.Context, event *fsm.Event) error

Embedding embedding daily estimation data.

func (*StateMachine) EnterState

func (sm *StateMachine) EnterState(ctx context.Context, event *fsm.Event) error

EnterState handles the state transition of the state machine and updates the Job's status and conditions based on the event.

func (*StateMachine) Train

func (sm *StateMachine) Train(ctx context.Context, event *fsm.Event) error

type Watcher

type Watcher struct {
	//Metric          metrics.Metric
	Train *train.TrainManager
	Minio minio.IMinio
	Store store.IStore

	// Maximum number of concurrent workers.
	MaxWorkers int64
	// Rate limiters for operations.
	Limiter Limiter
}

Watcher monitors and processes daily estimation jobs.

func (*Watcher) Run

func (w *Watcher) Run()

Run executes the watcher logic to process jobs.

func (*Watcher) SetAggregateConfig

func (w *Watcher) SetAggregateConfig(config *watcher.AggregateConfig)

SetAggregateConfig configures the watcher with the provided aggregate configuration.

func (*Watcher) SetMaxWorkers

func (w *Watcher) SetMaxWorkers(maxWorkers int64)

SetMaxWorkers sets the maximum number of concurrent workers for the watcher.

func (*Watcher) Spec

func (w *Watcher) Spec() string

Spec returns the cron job specification for scheduling.