tasks

Documentation

Overview

Package tasks contains definition of task queue tasks used by the mapper.

Index

• Variables
• type FanOutShards
  • func (*FanOutShards) Descriptor() ([]byte, []int)
  • func (x *FanOutShards) GetJobId() int64
  • func (*FanOutShards) ProtoMessage()
  • func (x *FanOutShards) ProtoReflect() protoreflect.Message
  • func (x *FanOutShards) Reset()
  • func (x *FanOutShards) String() string
• type ProcessShard
  • func (*ProcessShard) Descriptor() ([]byte, []int)
  • func (x *ProcessShard) GetJobId() int64
  • func (x *ProcessShard) GetShardId() int64
  • func (x *ProcessShard) GetTaskNum() int64
  • func (*ProcessShard) ProtoMessage()
  • func (x *ProcessShard) ProtoReflect() protoreflect.Message
  • func (x *ProcessShard) Reset()
  • func (x *ProcessShard) String() string
• type RequestJobStateUpdate
  • func (*RequestJobStateUpdate) Descriptor() ([]byte, []int)
  • func (x *RequestJobStateUpdate) GetJobId() int64
  • func (x *RequestJobStateUpdate) GetShardId() int64
  • func (*RequestJobStateUpdate) ProtoMessage()
  • func (x *RequestJobStateUpdate) ProtoReflect() protoreflect.Message
  • func (x *RequestJobStateUpdate) Reset()
  • func (x *RequestJobStateUpdate) String() string
• type SplitAndLaunch
  • func (*SplitAndLaunch) Descriptor() ([]byte, []int)
  • func (x *SplitAndLaunch) GetJobId() int64
  • func (*SplitAndLaunch) ProtoMessage()
  • func (x *SplitAndLaunch) ProtoReflect() protoreflect.Message
  • func (x *SplitAndLaunch) Reset()
  • func (x *SplitAndLaunch) String() string
• type UpdateJobState
  • func (*UpdateJobState) Descriptor() ([]byte, []int)
  • func (x *UpdateJobState) GetJobId() int64
  • func (*UpdateJobState) ProtoMessage()
  • func (x *UpdateJobState) ProtoReflect() protoreflect.Message
  • func (x *UpdateJobState) Reset()
  • func (x *UpdateJobState) String() string

Variables

var File_go_chromium_org_luci_server_dsmapper_internal_tasks_tasks_proto protoreflect.FileDescriptor

Types

type FanOutShards

type FanOutShards struct {
	JobId int64 `protobuf:"varint,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
	// contains filtered or unexported fields
}

FanOutShards enqueues a bunch of ProcessShard named tasks (one per shard).

Enqueued transactionally by SplitAndLaunch after it has constructed shards.

func (*FanOutShards) Descriptor

func (*FanOutShards) Descriptor() ([]byte, []int)

Deprecated: Use FanOutShards.ProtoReflect.Descriptor instead.

func (*FanOutShards) GetJobId

func (x *FanOutShards) GetJobId() int64

func (*FanOutShards) ProtoMessage

func (*FanOutShards) ProtoMessage()

func (*FanOutShards) ProtoReflect

func (x *FanOutShards) ProtoReflect() protoreflect.Message

func (*FanOutShards) Reset

func (x *FanOutShards) Reset()

func (*FanOutShards) String

func (x *FanOutShards) String() string

type ProcessShard

type ProcessShard struct {
	JobId   int64 `protobuf:"varint,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
	ShardId int64 `protobuf:"varint,2,opt,name=shard_id,json=shardId,proto3" json:"shard_id,omitempty"`
	TaskNum int64 `protobuf:"varint,3,opt,name=task_num,json=taskNum,proto3" json:"task_num,omitempty"`
	// contains filtered or unexported fields
}

ProcessShard sequentially reads the entities belonging to a key range assigned to a shard and applies the mapper to them.

Upon reaching 1 min mark, relaunches itself, increasing task_num. Thus ProcessShard is actually a chain of tasks that runs as long as needed to completely process the shard.

func (*ProcessShard) Descriptor

func (*ProcessShard) Descriptor() ([]byte, []int)

Deprecated: Use ProcessShard.ProtoReflect.Descriptor instead.

func (*ProcessShard) GetJobId

func (x *ProcessShard) GetJobId() int64

func (*ProcessShard) GetShardId

func (x *ProcessShard) GetShardId() int64

func (*ProcessShard) GetTaskNum

func (x *ProcessShard) GetTaskNum() int64

func (*ProcessShard) ProtoMessage

func (*ProcessShard) ProtoMessage()

func (*ProcessShard) ProtoReflect

func (x *ProcessShard) ProtoReflect() protoreflect.Message

func (*ProcessShard) Reset

func (x *ProcessShard) Reset()

func (*ProcessShard) String

func (x *ProcessShard) String() string

type RequestJobStateUpdate

type RequestJobStateUpdate struct {
	JobId   int64 `protobuf:"varint,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
	ShardId int64 `protobuf:"varint,2,opt,name=shard_id,json=shardId,proto3" json:"shard_id,omitempty"` // mostly FYI
	// contains filtered or unexported fields
}

RequestJobStateUpdate is transactionally emitted by ProcessShard when shard's state changes.

It eventually (with some throttling) causes UpdateJobState to be emitted, which updates the job state based on states of the shards.

func (*RequestJobStateUpdate) Descriptor

func (*RequestJobStateUpdate) Descriptor() ([]byte, []int)

Deprecated: Use RequestJobStateUpdate.ProtoReflect.Descriptor instead.

func (*RequestJobStateUpdate) GetJobId

func (x *RequestJobStateUpdate) GetJobId() int64

func (*RequestJobStateUpdate) GetShardId

func (x *RequestJobStateUpdate) GetShardId() int64

func (*RequestJobStateUpdate) ProtoMessage

func (*RequestJobStateUpdate) ProtoMessage()

func (*RequestJobStateUpdate) ProtoReflect

func (x *RequestJobStateUpdate) ProtoReflect() protoreflect.Message

func (*RequestJobStateUpdate) Reset

func (x *RequestJobStateUpdate) Reset()

func (*RequestJobStateUpdate) String

func (x *RequestJobStateUpdate) String() string

type SplitAndLaunch

type SplitAndLaunch struct {
	JobId int64 `protobuf:"varint,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
	// contains filtered or unexported fields
}

SplitAndLaunch task splits the key range into shards and kicks off processing of each individual shard.

Enqueued transactionally when creating a new mapping job.

func (*SplitAndLaunch) Descriptor

func (*SplitAndLaunch) Descriptor() ([]byte, []int)

Deprecated: Use SplitAndLaunch.ProtoReflect.Descriptor instead.

func (*SplitAndLaunch) GetJobId

func (x *SplitAndLaunch) GetJobId() int64

func (*SplitAndLaunch) ProtoMessage

func (*SplitAndLaunch) ProtoMessage()

func (*SplitAndLaunch) ProtoReflect

func (x *SplitAndLaunch) ProtoReflect() protoreflect.Message

func (*SplitAndLaunch) Reset

func (x *SplitAndLaunch) Reset()

func (*SplitAndLaunch) String

func (x *SplitAndLaunch) String() string

type UpdateJobState

type UpdateJobState struct {
	JobId int64 `protobuf:"varint,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
	// contains filtered or unexported fields
}

UpdateJobState is emitted after one or more shards have changed their state.

It recalculates the job's state based on state of all its shards. Throttled to 0.5 QPS.

func (*UpdateJobState) Descriptor

func (*UpdateJobState) Descriptor() ([]byte, []int)

Deprecated: Use UpdateJobState.ProtoReflect.Descriptor instead.

func (*UpdateJobState) GetJobId

func (x *UpdateJobState) GetJobId() int64

func (*UpdateJobState) ProtoMessage

func (*UpdateJobState) ProtoMessage()

func (*UpdateJobState) ProtoReflect

func (x *UpdateJobState) ProtoReflect() protoreflect.Message

func (*UpdateJobState) Reset

func (x *UpdateJobState) Reset()

func (*UpdateJobState) String

func (x *UpdateJobState) String() string