tasks

package
v0.0.0-...-a0a3655 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 28, 2019 License: Apache-2.0 Imports: 3 Imported by: 0

Documentation

Overview

Package tasks contains definition of task queue tasks used by the mapper.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type FanOutShards 

type FanOutShards struct {
	JobId                int64    `protobuf:"varint,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
	XXX_NoUnkeyedLiteral struct{} `json:"-"`
	XXX_unrecognized     []byte   `json:"-"`
	XXX_sizecache        int32    `json:"-"`
}

FanOutShards enqueues a bunch of ProcessShard named tasks (one per shard).

Enqueued transactionally by SplitAndLaunch after it has constructed shards.

func (*FanOutShards) Descriptor 

func (*FanOutShards) Descriptor() ([]byte, []int)

func (*FanOutShards) GetJobId 

func (m *FanOutShards) GetJobId() int64

func (*FanOutShards) ProtoMessage 

func (*FanOutShards) ProtoMessage()

func (*FanOutShards) Reset 

func (m *FanOutShards) Reset()

func (*FanOutShards) String 

func (m *FanOutShards) String() string

func (*FanOutShards) XXX_DiscardUnknown 

func (m *FanOutShards) XXX_DiscardUnknown()

func (*FanOutShards) XXX_Marshal 

func (m *FanOutShards) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*FanOutShards) XXX_Merge 

func (m *FanOutShards) XXX_Merge(src proto.Message)

func (*FanOutShards) XXX_Size 

func (m *FanOutShards) XXX_Size() int

func (*FanOutShards) XXX_Unmarshal 

func (m *FanOutShards) XXX_Unmarshal(b []byte) error

type ProcessShard 

type ProcessShard struct {
	JobId                int64    `protobuf:"varint,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
	ShardId              int64    `protobuf:"varint,2,opt,name=shard_id,json=shardId,proto3" json:"shard_id,omitempty"`
	TaskNum              int64    `protobuf:"varint,3,opt,name=task_num,json=taskNum,proto3" json:"task_num,omitempty"`
	XXX_NoUnkeyedLiteral struct{} `json:"-"`
	XXX_unrecognized     []byte   `json:"-"`
	XXX_sizecache        int32    `json:"-"`
}

ProcessShard sequentially reads the entities belonging to a key range assigned to a shard and applies the mapper to them.

Upon reaching 1 min mark, relaunches itself, increasing task_num. Thus ProcessShard is actually a chain of tasks that runs as long as needed to completely process the shard.

func (*ProcessShard) Descriptor 

func (*ProcessShard) Descriptor() ([]byte, []int)

func (*ProcessShard) GetJobId 

func (m *ProcessShard) GetJobId() int64

func (*ProcessShard) GetShardId 

func (m *ProcessShard) GetShardId() int64

func (*ProcessShard) GetTaskNum 

func (m *ProcessShard) GetTaskNum() int64

func (*ProcessShard) ProtoMessage 

func (*ProcessShard) ProtoMessage()

func (*ProcessShard) Reset 

func (m *ProcessShard) Reset()

func (*ProcessShard) String 

func (m *ProcessShard) String() string

func (*ProcessShard) XXX_DiscardUnknown 

func (m *ProcessShard) XXX_DiscardUnknown()

func (*ProcessShard) XXX_Marshal 

func (m *ProcessShard) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*ProcessShard) XXX_Merge 

func (m *ProcessShard) XXX_Merge(src proto.Message)

func (*ProcessShard) XXX_Size 

func (m *ProcessShard) XXX_Size() int

func (*ProcessShard) XXX_Unmarshal 

func (m *ProcessShard) XXX_Unmarshal(b []byte) error

type RequestJobStateUpdate 

type RequestJobStateUpdate struct {
	JobId                int64    `protobuf:"varint,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
	ShardId              int64    `protobuf:"varint,2,opt,name=shard_id,json=shardId,proto3" json:"shard_id,omitempty"`
	XXX_NoUnkeyedLiteral struct{} `json:"-"`
	XXX_unrecognized     []byte   `json:"-"`
	XXX_sizecache        int32    `json:"-"`
}

RequestJobStateUpdate is transactionally emitted by ProcessShard when shard's state changes.

It eventually (with some throttling) causes UpdateJobState to be emitted, which updates the job state based on states of the shards.

func (*RequestJobStateUpdate) Descriptor 

func (*RequestJobStateUpdate) Descriptor() ([]byte, []int)

func (*RequestJobStateUpdate) GetJobId 

func (m *RequestJobStateUpdate) GetJobId() int64

func (*RequestJobStateUpdate) GetShardId 

func (m *RequestJobStateUpdate) GetShardId() int64

func (*RequestJobStateUpdate) ProtoMessage 

func (*RequestJobStateUpdate) ProtoMessage()

func (*RequestJobStateUpdate) Reset 

func (m *RequestJobStateUpdate) Reset()

func (*RequestJobStateUpdate) String 

func (m *RequestJobStateUpdate) String() string

func (*RequestJobStateUpdate) XXX_DiscardUnknown 

func (m *RequestJobStateUpdate) XXX_DiscardUnknown()

func (*RequestJobStateUpdate) XXX_Marshal 

func (m *RequestJobStateUpdate) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*RequestJobStateUpdate) XXX_Merge 

func (m *RequestJobStateUpdate) XXX_Merge(src proto.Message)

func (*RequestJobStateUpdate) XXX_Size 

func (m *RequestJobStateUpdate) XXX_Size() int

func (*RequestJobStateUpdate) XXX_Unmarshal 

func (m *RequestJobStateUpdate) XXX_Unmarshal(b []byte) error

type SplitAndLaunch 

type SplitAndLaunch struct {
	JobId                int64    `protobuf:"varint,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
	XXX_NoUnkeyedLiteral struct{} `json:"-"`
	XXX_unrecognized     []byte   `json:"-"`
	XXX_sizecache        int32    `json:"-"`
}

SplitAndLaunch task splits the key range into shards and kicks off processing of each individual shard.

Enqueued transactionally when creating a new mapping job.

func (*SplitAndLaunch) Descriptor 

func (*SplitAndLaunch) Descriptor() ([]byte, []int)

func (*SplitAndLaunch) GetJobId 

func (m *SplitAndLaunch) GetJobId() int64

func (*SplitAndLaunch) ProtoMessage 

func (*SplitAndLaunch) ProtoMessage()

func (*SplitAndLaunch) Reset 

func (m *SplitAndLaunch) Reset()

func (*SplitAndLaunch) String 

func (m *SplitAndLaunch) String() string

func (*SplitAndLaunch) XXX_DiscardUnknown 

func (m *SplitAndLaunch) XXX_DiscardUnknown()

func (*SplitAndLaunch) XXX_Marshal 

func (m *SplitAndLaunch) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*SplitAndLaunch) XXX_Merge 

func (m *SplitAndLaunch) XXX_Merge(src proto.Message)

func (*SplitAndLaunch) XXX_Size 

func (m *SplitAndLaunch) XXX_Size() int

func (*SplitAndLaunch) XXX_Unmarshal 

func (m *SplitAndLaunch) XXX_Unmarshal(b []byte) error

type UpdateJobState 

type UpdateJobState struct {
	JobId                int64    `protobuf:"varint,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
	XXX_NoUnkeyedLiteral struct{} `json:"-"`
	XXX_unrecognized     []byte   `json:"-"`
	XXX_sizecache        int32    `json:"-"`
}

UpdateJobState is emitted after one or more shards have changed their state.

It recalculates the job's state based on state of all its shards. Throttled to 0.5 QPS.

func (*UpdateJobState) Descriptor 

func (*UpdateJobState) Descriptor() ([]byte, []int)

func (*UpdateJobState) GetJobId 

func (m *UpdateJobState) GetJobId() int64

func (*UpdateJobState) ProtoMessage 

func (*UpdateJobState) ProtoMessage()

func (*UpdateJobState) Reset 

func (m *UpdateJobState) Reset()

func (*UpdateJobState) String 

func (m *UpdateJobState) String() string

func (*UpdateJobState) XXX_DiscardUnknown 

func (m *UpdateJobState) XXX_DiscardUnknown()

func (*UpdateJobState) XXX_Marshal 

func (m *UpdateJobState) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*UpdateJobState) XXX_Merge 

func (m *UpdateJobState) XXX_Merge(src proto.Message)

func (*UpdateJobState) XXX_Size 

func (m *UpdateJobState) XXX_Size() int

func (*UpdateJobState) XXX_Unmarshal 

func (m *UpdateJobState) XXX_Unmarshal(b []byte) error

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.