datanode

package
v0.0.0-...-34e0b2d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 9, 2024 License: Apache-2.0 Imports: 52 Imported by: 0

README

Data Node

DataNode is the component to write insert and delete messages into persistent blob storage, for example MinIO or S3.

Dependency

  • KV store: a kv store that persists messages into blob storage.
  • Message stream: receive messages and publish imformation
  • Root Coordinator: get the latest unique IDs.
  • Data Coordinator: get the flush information and which message stream to subscribe.

Documentation

Overview

Package datanode implements data persistence logic.

Data node persists insert logs into persistent storage like minIO/S3.

Package datanode implements data persistence logic.

Data node persists insert logs into persistent storage like minIO/S3.

Index

Constants

View Source 
const (
	// ConnectEtcdMaxRetryTime is used to limit the max retry time for connection etcd
	ConnectEtcdMaxRetryTime = 100
)

Variables

View Source 
var Params *paramtable.ComponentParam = paramtable.Get()

Params from config.yaml

Functions

This section is empty.

Types

type DataNode 

type DataNode struct {
	Role string
	// contains filtered or unexported fields
}

DataNode communicates with outside services and unioun all services in datanode package.

DataNode implements `types.Component`, `types.DataNode` interfaces. 

`etcdCli`   is a connection of etcd
`rootCoord` is a grpc client of root coordinator.
`dataCoord` is a grpc client of data service.
`stateCode` is current statement of this data node, indicating whether it's healthy.

`clearSignal` is a signal channel for releasing the flowgraph resources.
`segmentCache` stores all flushing and flushed segments.

func NewDataNode 

func NewDataNode(ctx context.Context, factory dependency.Factory) *DataNode

NewDataNode will return a DataNode with abnormal state.

func (*DataNode) CheckChannelOperationProgress 

func (node *DataNode) CheckChannelOperationProgress(ctx context.Context, req *datapb.ChannelWatchInfo) (*datapb.ChannelOperationProgressResponse, error)

func (*DataNode) CompactionV2 

func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPlan) (*commonpb.Status, error)

CompactionV2 handles compaction request from DataCoord returns status as long as compaction task enqueued or invalid

func (*DataNode) DropCompactionPlan 

func (node *DataNode) DropCompactionPlan(ctx context.Context, req *datapb.DropCompactionPlanRequest) (*commonpb.Status, error)

func (*DataNode) DropImport 

func (node *DataNode) DropImport(ctx context.Context, req *datapb.DropImportRequest) (*commonpb.Status, error)

func (*DataNode) FlushChannels 

func (node *DataNode) FlushChannels(ctx context.Context, req *datapb.FlushChannelsRequest) (*commonpb.Status, error)

func (*DataNode) FlushSegments 

func (node *DataNode) FlushSegments(ctx context.Context, req *datapb.FlushSegmentsRequest) (*commonpb.Status, error)

func (*DataNode) GetAddress 

func (node *DataNode) GetAddress() string

func (*DataNode) GetCompactionState 

func (node *DataNode) GetCompactionState(ctx context.Context, req *datapb.CompactionStateRequest) (*datapb.CompactionStateResponse, error)

GetCompactionState called by DataCoord return status of all compaction plans

func (*DataNode) GetComponentStates 

func (node *DataNode) GetComponentStates(ctx context.Context, req *milvuspb.GetComponentStatesRequest) (*milvuspb.ComponentStates, error)

GetComponentStates will return current state of DataNode

func (*DataNode) GetMetrics 

func (node *DataNode) GetMetrics(ctx context.Context, req *milvuspb.GetMetricsRequest) (*milvuspb.GetMetricsResponse, error)

GetMetrics return datanode metrics

func (*DataNode) GetNodeID 

func (node *DataNode) GetNodeID() int64

func (*DataNode) GetSession 

func (node *DataNode) GetSession() *sessionutil.Session

GetSession to fix data race

func (*DataNode) GetStateCode 

func (node *DataNode) GetStateCode() commonpb.StateCode

GetStateCode return datanode's state code

func (*DataNode) GetStatisticsChannel 

func (node *DataNode) GetStatisticsChannel(ctx context.Context, req *internalpb.GetStatisticsChannelRequest) (*milvuspb.StringResponse, error)

GetStatisticsChannel currently do nothing

func (*DataNode) GetTimeTickChannel 

func (node *DataNode) GetTimeTickChannel(ctx context.Context, req *internalpb.GetTimeTickChannelRequest) (*milvuspb.StringResponse, error)

GetTimeTickChannel currently do nothing

func (*DataNode) ImportV2 

func (node *DataNode) ImportV2(ctx context.Context, req *datapb.ImportRequest) (*commonpb.Status, error)

func (*DataNode) Init 

func (node *DataNode) Init() error

func (*DataNode) NotifyChannelOperation 

func (node *DataNode) NotifyChannelOperation(ctx context.Context, req *datapb.ChannelOperationsRequest) (*commonpb.Status, error)

func (*DataNode) PreImport 

func (node *DataNode) PreImport(ctx context.Context, req *datapb.PreImportRequest) (*commonpb.Status, error)

func (*DataNode) QueryImport 

func (node *DataNode) QueryImport(ctx context.Context, req *datapb.QueryImportRequest) (*datapb.QueryImportResponse, error)

func (*DataNode) QueryPreImport 

func (node *DataNode) QueryPreImport(ctx context.Context, req *datapb.QueryPreImportRequest) (*datapb.QueryPreImportResponse, error)

func (*DataNode) QuerySlot 

func (node *DataNode) QuerySlot(ctx context.Context, req *datapb.QuerySlotRequest) (*datapb.QuerySlotResponse, error)

func (*DataNode) ReadyToFlush 

func (node *DataNode) ReadyToFlush() error

ReadyToFlush tells whether DataNode is ready for flushing

func (*DataNode) Register 

func (node *DataNode) Register() error

Register register datanode to etcd

func (*DataNode) ResendSegmentStats 

func (node *DataNode) ResendSegmentStats(ctx context.Context, req *datapb.ResendSegmentStatsRequest) (*datapb.ResendSegmentStatsResponse, error)

ResendSegmentStats . ResendSegmentStats resend un-flushed segment stats back upstream to DataCoord by resending DataNode time tick message. It returns a list of segments to be sent. Deprecated in 2.3.2, reversed it just for compatibility during rolling back

func (*DataNode) SetAddress 

func (node *DataNode) SetAddress(address string)

func (*DataNode) SetDataCoordClient 

func (node *DataNode) SetDataCoordClient(ds types.DataCoordClient) error

SetDataCoordClient sets data service's grpc client, error is returned if repeatedly set.

func (*DataNode) SetEtcdClient 

func (node *DataNode) SetEtcdClient(etcdCli *clientv3.Client)

SetEtcdClient sets etcd client for DataNode

func (*DataNode) SetRootCoordClient 

func (node *DataNode) SetRootCoordClient(rc types.RootCoordClient) error

SetRootCoordClient sets RootCoord's grpc client, error is returned if repeatedly set.

func (*DataNode) SetSession 

func (node *DataNode) SetSession(session *sessionutil.Session)

SetSession to fix data race

func (*DataNode) ShowConfigurations 

func (node *DataNode) ShowConfigurations(ctx context.Context, req *internalpb.ShowConfigurationsRequest) (*internalpb.ShowConfigurationsResponse, error)

ShowConfigurations returns the configurations of DataNode matching req.Pattern

func (*DataNode) Start 

func (node *DataNode) Start() error

Start will update DataNode state to HEALTHY

func (*DataNode) Stop 

func (node *DataNode) Stop() error

Stop will release DataNode resources and shutdown datanode

func (*DataNode) SyncSegments 

func (node *DataNode) SyncSegments(ctx context.Context, req *datapb.SyncSegmentsRequest) (*commonpb.Status, error)

SyncSegments called by DataCoord, sync the compacted segments' meta between DC and DN

func (*DataNode) UpdateStateCode 

func (node *DataNode) UpdateStateCode(code commonpb.StateCode)

UpdateStateCode updates datanode's state code

func (*DataNode) WatchDmChannels 

func (node *DataNode) WatchDmChannels(ctx context.Context, in *datapb.WatchDmChannelsRequest) (*commonpb.Status, error)

WatchDmChannels is not in use

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.