v1

package
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 23, 2024 License: Apache-2.0 Imports: 10 Imported by: 2

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var File_pkg_apis_proto_source_v1_source_proto protoreflect.FileDescriptor
View Source 
var Source_ServiceDesc = grpc.ServiceDesc{
	ServiceName: "source.v1.Source",
	HandlerType: (*SourceServer)(nil),
	Methods: []grpc.MethodDesc{
		{
			MethodName: "AckFn",
			Handler:    _Source_AckFn_Handler,
		},
		{
			MethodName: "PendingFn",
			Handler:    _Source_PendingFn_Handler,
		},
		{
			MethodName: "PartitionsFn",
			Handler:    _Source_PartitionsFn_Handler,
		},
		{
			MethodName: "IsReady",
			Handler:    _Source_IsReady_Handler,
		},
	},
	Streams: []grpc.StreamDesc{
		{
			StreamName:    "ReadFn",
			Handler:       _Source_ReadFn_Handler,
			ServerStreams: true,
		},
	},
	Metadata: "pkg/apis/proto/source/v1/source.proto",
}

Source_ServiceDesc is the grpc.ServiceDesc for Source service. It's only intended for direct use with grpc.RegisterService, and not to be introspected or modified (even as a copy)

Functions

func RegisterSourceServer 

func RegisterSourceServer(s grpc.ServiceRegistrar, srv SourceServer)

Types

type AckRequest 

type AckRequest struct {

	// Required field holding the request. The list will be ordered and will have the same order as the original Read response.
	Request *AckRequest_Request `protobuf:"bytes,1,opt,name=request,proto3" json:"request,omitempty"`
	// contains filtered or unexported fields
}

AckRequest is the request for acknowledging datum. It takes a list of offsets to be acknowledged.

func (*AckRequest) Descriptor deprecated 

func (*AckRequest) Descriptor() ([]byte, []int)

Deprecated: Use AckRequest.ProtoReflect.Descriptor instead.

func (*AckRequest) GetRequest 

func (x *AckRequest) GetRequest() *AckRequest_Request

func (*AckRequest) ProtoMessage 

func (*AckRequest) ProtoMessage()

func (*AckRequest) ProtoReflect 

func (x *AckRequest) ProtoReflect() protoreflect.Message

func (*AckRequest) Reset 

func (x *AckRequest) Reset()

func (*AckRequest) String 

func (x *AckRequest) String() string

type AckRequest_Request 

type AckRequest_Request struct {

	// Required field holding a list of offsets to be acknowledged.
	// The offsets must be strictly corresponding to the previously read batch,
	// meaning the offsets must be in the same order as the datum responses in the ReadResponse.
	// By enforcing ordering, we can save deserialization effort on the server side, assuming the server keeps a local copy of the raw/un-serialized offsets.
	Offsets []*Offset `protobuf:"bytes,1,rep,name=offsets,proto3" json:"offsets,omitempty"`
	// contains filtered or unexported fields
}

func (*AckRequest_Request) Descriptor deprecated 

func (*AckRequest_Request) Descriptor() ([]byte, []int)

Deprecated: Use AckRequest_Request.ProtoReflect.Descriptor instead.

func (*AckRequest_Request) GetOffsets 

func (x *AckRequest_Request) GetOffsets() []*Offset

func (*AckRequest_Request) ProtoMessage 

func (*AckRequest_Request) ProtoMessage()

func (*AckRequest_Request) ProtoReflect 

func (x *AckRequest_Request) ProtoReflect() protoreflect.Message

func (*AckRequest_Request) Reset 

func (x *AckRequest_Request) Reset()

func (*AckRequest_Request) String 

func (x *AckRequest_Request) String() string

type AckResponse 

type AckResponse struct {

	// Required field holding the result.
	Result *AckResponse_Result `protobuf:"bytes,1,opt,name=result,proto3" json:"result,omitempty"`
	// contains filtered or unexported fields
}

AckResponse is the response for acknowledging datum. It contains one empty field confirming the batch of offsets that have been successfully acknowledged. The contract between client and server is that the server will only return the AckResponse if the ack request is successful. If the server hangs during the ack request, the client can decide to timeout and error out the data forwarder. The reason why we define such contract is that we always expect the server to be able to process the ack request. Client is expected to send the AckRequest to the server with offsets that are strictly corresponding to the previously read batch. If the client sends the AckRequest with offsets that are not, it is considered as a client error and the server will not return the AckResponse.

func (*AckResponse) Descriptor deprecated 

func (*AckResponse) Descriptor() ([]byte, []int)

Deprecated: Use AckResponse.ProtoReflect.Descriptor instead.

func (*AckResponse) GetResult 

func (x *AckResponse) GetResult() *AckResponse_Result

func (*AckResponse) ProtoMessage 

func (*AckResponse) ProtoMessage()

func (*AckResponse) ProtoReflect 

func (x *AckResponse) ProtoReflect() protoreflect.Message

func (*AckResponse) Reset 

func (x *AckResponse) Reset()

func (*AckResponse) String 

func (x *AckResponse) String() string

type AckResponse_Result 

type AckResponse_Result struct {

	// Required field indicating the ack request is successful.
	Success *emptypb.Empty `protobuf:"bytes,1,opt,name=success,proto3" json:"success,omitempty"`
	// contains filtered or unexported fields
}

func (*AckResponse_Result) Descriptor deprecated 

func (*AckResponse_Result) Descriptor() ([]byte, []int)

Deprecated: Use AckResponse_Result.ProtoReflect.Descriptor instead.

func (*AckResponse_Result) GetSuccess 

func (x *AckResponse_Result) GetSuccess() *emptypb.Empty

func (*AckResponse_Result) ProtoMessage 

func (*AckResponse_Result) ProtoMessage()

func (*AckResponse_Result) ProtoReflect 

func (x *AckResponse_Result) ProtoReflect() protoreflect.Message

func (*AckResponse_Result) Reset 

func (x *AckResponse_Result) Reset()

func (*AckResponse_Result) String 

func (x *AckResponse_Result) String() string

type Offset 

type Offset struct {

	// offset is the offset of the datum. This field is required.
	// We define Offset as a byte array because different input data sources can have different representations for Offset.
	// The only way to generalize it is to define it as a byte array,
	// Such that we can let the UDSource to de-serialize the offset using its own interpretation logics.
	Offset []byte `protobuf:"bytes,1,opt,name=offset,proto3" json:"offset,omitempty"`
	// Optional partition_id indicates which partition of the source the datum belongs to.
	// It is useful for sources that have multiple partitions. e.g. Kafka.
	// If the partition_id is not specified, it is assumed that the source has a single partition.
	PartitionId int32 `protobuf:"varint,2,opt,name=partition_id,json=partitionId,proto3" json:"partition_id,omitempty"`
	// contains filtered or unexported fields
}

Offset is the offset of the datum.

func (*Offset) Descriptor deprecated 

func (*Offset) Descriptor() ([]byte, []int)

Deprecated: Use Offset.ProtoReflect.Descriptor instead.

func (*Offset) GetOffset 

func (x *Offset) GetOffset() []byte

func (*Offset) GetPartitionId 

func (x *Offset) GetPartitionId() int32

func (*Offset) ProtoMessage 

func (*Offset) ProtoMessage()

func (*Offset) ProtoReflect 

func (x *Offset) ProtoReflect() protoreflect.Message

func (*Offset) Reset 

func (x *Offset) Reset()

func (*Offset) String 

func (x *Offset) String() string

type PartitionsResponse added in v0.6.0 

type PartitionsResponse struct {

	// Required field holding the result.
	Result *PartitionsResponse_Result `protobuf:"bytes,1,opt,name=result,proto3" json:"result,omitempty"`
	// contains filtered or unexported fields
}

PartitionsResponse is the response for the partitions request.

func (*PartitionsResponse) Descriptor deprecated added in v0.6.0 

func (*PartitionsResponse) Descriptor() ([]byte, []int)

Deprecated: Use PartitionsResponse.ProtoReflect.Descriptor instead.

func (*PartitionsResponse) GetResult added in v0.6.0 

func (x *PartitionsResponse) GetResult() *PartitionsResponse_Result

func (*PartitionsResponse) ProtoMessage added in v0.6.0 

func (*PartitionsResponse) ProtoMessage()

func (*PartitionsResponse) ProtoReflect added in v0.6.0 

func (x *PartitionsResponse) ProtoReflect() protoreflect.Message

func (*PartitionsResponse) Reset added in v0.6.0 

func (x *PartitionsResponse) Reset()

func (*PartitionsResponse) String added in v0.6.0 

func (x *PartitionsResponse) String() string

type PartitionsResponse_Result added in v0.6.0 

type PartitionsResponse_Result struct {

	// Required field holding the list of partitions.
	Partitions []int32 `protobuf:"varint,1,rep,packed,name=partitions,proto3" json:"partitions,omitempty"`
	// contains filtered or unexported fields
}

func (*PartitionsResponse_Result) Descriptor deprecated added in v0.6.0 

func (*PartitionsResponse_Result) Descriptor() ([]byte, []int)

Deprecated: Use PartitionsResponse_Result.ProtoReflect.Descriptor instead.

func (*PartitionsResponse_Result) GetPartitions added in v0.6.0 

func (x *PartitionsResponse_Result) GetPartitions() []int32

func (*PartitionsResponse_Result) ProtoMessage added in v0.6.0 

func (*PartitionsResponse_Result) ProtoMessage()

func (*PartitionsResponse_Result) ProtoReflect added in v0.6.0 

func (x *PartitionsResponse_Result) ProtoReflect() protoreflect.Message

func (*PartitionsResponse_Result) Reset added in v0.6.0 

func (x *PartitionsResponse_Result) Reset()

func (*PartitionsResponse_Result) String added in v0.6.0 

func (x *PartitionsResponse_Result) String() string

type PendingResponse 

type PendingResponse struct {

	// Required field holding the result.
	Result *PendingResponse_Result `protobuf:"bytes,1,opt,name=result,proto3" json:"result,omitempty"`
	// contains filtered or unexported fields
}

PendingResponse is the response for the pending request.

func (*PendingResponse) Descriptor deprecated 

func (*PendingResponse) Descriptor() ([]byte, []int)

Deprecated: Use PendingResponse.ProtoReflect.Descriptor instead.

func (*PendingResponse) GetResult 

func (x *PendingResponse) GetResult() *PendingResponse_Result

func (*PendingResponse) ProtoMessage 

func (*PendingResponse) ProtoMessage()

func (*PendingResponse) ProtoReflect 

func (x *PendingResponse) ProtoReflect() protoreflect.Message

func (*PendingResponse) Reset 

func (x *PendingResponse) Reset()

func (*PendingResponse) String 

func (x *PendingResponse) String() string

type PendingResponse_Result 

type PendingResponse_Result struct {

	// Required field holding the number of pending records at the user defined source.
	// A negative count indicates that the pending information is not available.
	Count int64 `protobuf:"varint,1,opt,name=count,proto3" json:"count,omitempty"`
	// contains filtered or unexported fields
}

func (*PendingResponse_Result) Descriptor deprecated 

func (*PendingResponse_Result) Descriptor() ([]byte, []int)

Deprecated: Use PendingResponse_Result.ProtoReflect.Descriptor instead.

func (*PendingResponse_Result) GetCount 

func (x *PendingResponse_Result) GetCount() int64

func (*PendingResponse_Result) ProtoMessage 

func (*PendingResponse_Result) ProtoMessage()

func (*PendingResponse_Result) ProtoReflect 

func (x *PendingResponse_Result) ProtoReflect() protoreflect.Message

func (*PendingResponse_Result) Reset 

func (x *PendingResponse_Result) Reset()

func (*PendingResponse_Result) String 

func (x *PendingResponse_Result) String() string

type ReadRequest 

type ReadRequest struct {

	// Required field indicating the request.
	Request *ReadRequest_Request `protobuf:"bytes,1,opt,name=request,proto3" json:"request,omitempty"`
	// contains filtered or unexported fields
}

ReadRequest is the request for reading datum stream from user defined source.

func (*ReadRequest) Descriptor deprecated 

func (*ReadRequest) Descriptor() ([]byte, []int)

Deprecated: Use ReadRequest.ProtoReflect.Descriptor instead.

func (*ReadRequest) GetRequest 

func (x *ReadRequest) GetRequest() *ReadRequest_Request

func (*ReadRequest) ProtoMessage 

func (*ReadRequest) ProtoMessage()

func (*ReadRequest) ProtoReflect 

func (x *ReadRequest) ProtoReflect() protoreflect.Message

func (*ReadRequest) Reset 

func (x *ReadRequest) Reset()

func (*ReadRequest) String 

func (x *ReadRequest) String() string

type ReadRequest_Request 

type ReadRequest_Request struct {

	// Required field indicating the number of records to read.
	NumRecords uint64 `protobuf:"varint,1,opt,name=num_records,json=numRecords,proto3" json:"num_records,omitempty"`
	// Required field indicating the request timeout in milliseconds.
	// uint32 can represent 2^32 milliseconds, which is about 49 days.
	// We don't use uint64 because time.Duration takes int64 as nano seconds. Using uint64 for milli will cause overflow.
	TimeoutInMs uint32 `protobuf:"varint,2,opt,name=timeout_in_ms,json=timeoutInMs,proto3" json:"timeout_in_ms,omitempty"`
	// contains filtered or unexported fields
}

func (*ReadRequest_Request) Descriptor deprecated 

func (*ReadRequest_Request) Descriptor() ([]byte, []int)

Deprecated: Use ReadRequest_Request.ProtoReflect.Descriptor instead.

func (*ReadRequest_Request) GetNumRecords 

func (x *ReadRequest_Request) GetNumRecords() uint64

func (*ReadRequest_Request) GetTimeoutInMs 

func (x *ReadRequest_Request) GetTimeoutInMs() uint32

func (*ReadRequest_Request) ProtoMessage 

func (*ReadRequest_Request) ProtoMessage()

func (*ReadRequest_Request) ProtoReflect 

func (x *ReadRequest_Request) ProtoReflect() protoreflect.Message

func (*ReadRequest_Request) Reset 

func (x *ReadRequest_Request) Reset()

func (*ReadRequest_Request) String 

func (x *ReadRequest_Request) String() string

type ReadResponse 

type ReadResponse struct {

	// Required field holding the result.
	Result *ReadResponse_Result `protobuf:"bytes,1,opt,name=result,proto3" json:"result,omitempty"`
	// contains filtered or unexported fields
}

ReadResponse is the response for reading datum stream from user defined source.

func (*ReadResponse) Descriptor deprecated 

func (*ReadResponse) Descriptor() ([]byte, []int)

Deprecated: Use ReadResponse.ProtoReflect.Descriptor instead.

func (*ReadResponse) GetResult 

func (x *ReadResponse) GetResult() *ReadResponse_Result

func (*ReadResponse) ProtoMessage 

func (*ReadResponse) ProtoMessage()

func (*ReadResponse) ProtoReflect 

func (x *ReadResponse) ProtoReflect() protoreflect.Message

func (*ReadResponse) Reset 

func (x *ReadResponse) Reset()

func (*ReadResponse) String 

func (x *ReadResponse) String() string

type ReadResponse_Result 

type ReadResponse_Result struct {

	// Required field holding the payload of the datum.
	Payload []byte `protobuf:"bytes,1,opt,name=payload,proto3" json:"payload,omitempty"`
	// Required field indicating the offset information of the datum.
	Offset *Offset `protobuf:"bytes,2,opt,name=offset,proto3" json:"offset,omitempty"`
	// Required field representing the time associated with each datum. It is used for watermarking.
	EventTime *timestamppb.Timestamp `protobuf:"bytes,3,opt,name=event_time,json=eventTime,proto3" json:"event_time,omitempty"`
	// Optional list of keys associated with the datum.
	// Key is the "key" attribute in (key,value) as in the map-reduce paradigm.
	// We add this optional field to support the use case where the user defined source can provide keys for the datum.
	// e.g. Kafka and Redis Stream message usually include information about the keys.
	Keys []string `protobuf:"bytes,4,rep,name=keys,proto3" json:"keys,omitempty"`
	// Optional list of headers associated with the datum.
	// Headers are the metadata associated with the datum.
	// e.g. Kafka and Redis Stream message usually include information about the headers.
	Headers map[string]string `` /* 155-byte string literal not displayed */
	// contains filtered or unexported fields
}

func (*ReadResponse_Result) Descriptor deprecated 

func (*ReadResponse_Result) Descriptor() ([]byte, []int)

Deprecated: Use ReadResponse_Result.ProtoReflect.Descriptor instead.

func (*ReadResponse_Result) GetEventTime 

func (x *ReadResponse_Result) GetEventTime() *timestamppb.Timestamp

func (*ReadResponse_Result) GetHeaders added in v0.7.0 

func (x *ReadResponse_Result) GetHeaders() map[string]string

func (*ReadResponse_Result) GetKeys 

func (x *ReadResponse_Result) GetKeys() []string

func (*ReadResponse_Result) GetOffset 

func (x *ReadResponse_Result) GetOffset() *Offset

func (*ReadResponse_Result) GetPayload 

func (x *ReadResponse_Result) GetPayload() []byte

func (*ReadResponse_Result) ProtoMessage 

func (*ReadResponse_Result) ProtoMessage()

func (*ReadResponse_Result) ProtoReflect 

func (x *ReadResponse_Result) ProtoReflect() protoreflect.Message

func (*ReadResponse_Result) Reset 

func (x *ReadResponse_Result) Reset()

func (*ReadResponse_Result) String 

func (x *ReadResponse_Result) String() string

type ReadyResponse 

type ReadyResponse struct {

	// Required field holding the health check result.
	Ready bool `protobuf:"varint,1,opt,name=ready,proto3" json:"ready,omitempty"`
	// contains filtered or unexported fields
}

ReadyResponse is the health check result for user defined source.

func (*ReadyResponse) Descriptor deprecated 

func (*ReadyResponse) Descriptor() ([]byte, []int)

Deprecated: Use ReadyResponse.ProtoReflect.Descriptor instead.

func (*ReadyResponse) GetReady 

func (x *ReadyResponse) GetReady() bool

func (*ReadyResponse) ProtoMessage 

func (*ReadyResponse) ProtoMessage()

func (*ReadyResponse) ProtoReflect 

func (x *ReadyResponse) ProtoReflect() protoreflect.Message

func (*ReadyResponse) Reset 

func (x *ReadyResponse) Reset()

func (*ReadyResponse) String 

func (x *ReadyResponse) String() string

type SourceClient 

type SourceClient interface {
	// Read returns a stream of datum responses.
	// The size of the returned ReadResponse is less than or equal to the num_records specified in ReadRequest.
	// If the request timeout is reached on server side, the returned ReadResponse will contain all the datum that have been read (which could be an empty list).
	ReadFn(ctx context.Context, in *ReadRequest, opts ...grpc.CallOption) (Source_ReadFnClient, error)
	// AckFn acknowledges a list of datum offsets.
	// When AckFn is called, it implicitly indicates that the datum stream has been processed by the source vertex.
	// The caller (numa) expects the AckFn to be successful, and it does not expect any errors.
	// If there are some irrecoverable errors when the callee (UDSource) is processing the AckFn request,
	// then it is best to crash because there are no other retry mechanisms possible.
	AckFn(ctx context.Context, in *AckRequest, opts ...grpc.CallOption) (*AckResponse, error)
	// PendingFn returns the number of pending records at the user defined source.
	PendingFn(ctx context.Context, in *emptypb.Empty, opts ...grpc.CallOption) (*PendingResponse, error)
	// PartitionsFn returns the list of partitions for the user defined source.
	PartitionsFn(ctx context.Context, in *emptypb.Empty, opts ...grpc.CallOption) (*PartitionsResponse, error)
	// IsReady is the heartbeat endpoint for user defined source gRPC.
	IsReady(ctx context.Context, in *emptypb.Empty, opts ...grpc.CallOption) (*ReadyResponse, error)
}

SourceClient is the client API for Source service.

For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.

func NewSourceClient 

func NewSourceClient(cc grpc.ClientConnInterface) SourceClient

type SourceServer 

type SourceServer interface {
	// Read returns a stream of datum responses.
	// The size of the returned ReadResponse is less than or equal to the num_records specified in ReadRequest.
	// If the request timeout is reached on server side, the returned ReadResponse will contain all the datum that have been read (which could be an empty list).
	ReadFn(*ReadRequest, Source_ReadFnServer) error
	// AckFn acknowledges a list of datum offsets.
	// When AckFn is called, it implicitly indicates that the datum stream has been processed by the source vertex.
	// The caller (numa) expects the AckFn to be successful, and it does not expect any errors.
	// If there are some irrecoverable errors when the callee (UDSource) is processing the AckFn request,
	// then it is best to crash because there are no other retry mechanisms possible.
	AckFn(context.Context, *AckRequest) (*AckResponse, error)
	// PendingFn returns the number of pending records at the user defined source.
	PendingFn(context.Context, *emptypb.Empty) (*PendingResponse, error)
	// PartitionsFn returns the list of partitions for the user defined source.
	PartitionsFn(context.Context, *emptypb.Empty) (*PartitionsResponse, error)
	// IsReady is the heartbeat endpoint for user defined source gRPC.
	IsReady(context.Context, *emptypb.Empty) (*ReadyResponse, error)
	// contains filtered or unexported methods
}

SourceServer is the server API for Source service. All implementations must embed UnimplementedSourceServer for forward compatibility

type Source_ReadFnClient 

type Source_ReadFnClient interface {
	Recv() (*ReadResponse, error)
	grpc.ClientStream
}

type Source_ReadFnServer 

type Source_ReadFnServer interface {
	Send(*ReadResponse) error
	grpc.ServerStream
}

type UnimplementedSourceServer 

type UnimplementedSourceServer struct {
}

UnimplementedSourceServer must be embedded to have forward compatible implementations.

func (UnimplementedSourceServer) AckFn 

func (UnimplementedSourceServer) AckFn(context.Context, *AckRequest) (*AckResponse, error)

func (UnimplementedSourceServer) IsReady 

func (UnimplementedSourceServer) IsReady(context.Context, *emptypb.Empty) (*ReadyResponse, error)

func (UnimplementedSourceServer) PartitionsFn added in v0.6.0 

func (UnimplementedSourceServer) PartitionsFn(context.Context, *emptypb.Empty) (*PartitionsResponse, error)

func (UnimplementedSourceServer) PendingFn 

func (UnimplementedSourceServer) PendingFn(context.Context, *emptypb.Empty) (*PendingResponse, error)

func (UnimplementedSourceServer) ReadFn 

func (UnimplementedSourceServer) ReadFn(*ReadRequest, Source_ReadFnServer) error

type UnsafeSourceServer 

type UnsafeSourceServer interface {
	// contains filtered or unexported methods
}

UnsafeSourceServer may be embedded to opt out of forward compatibility for this service. Use of this interface is not recommended, as added methods to SourceServer will result in compilation errors.

Source Files

View all Source files

Directories

Path Synopsis
Package sourcemock is a generated GoMock package.
 Package sourcemock is a generated GoMock package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.