arrow_record

Documentation

Overview

Package arrow_record contains the consumer and producer for OTLP Arrow protocol.

Index

• Variables
• func NewConsoleObserver(maxRows, maxPrints int) observer.ProducerObserver
• type Config
• type Consumer
  • func NewConsumer(opts ...Option) *Consumer
  • func (c *Consumer) Close() error
  • func (c *Consumer) Consume(bar *colarspb.BatchArrowRecords) (ibes []*record_message.RecordMessage, retErr error)
  • func (c *Consumer) LogsFrom(bar *colarspb.BatchArrowRecords) ([]plog.Logs, error)
  • func (c *Consumer) MetricsFrom(bar *colarspb.BatchArrowRecords) ([]pmetric.Metrics, error)
  • func (c *Consumer) TracesFrom(bar *colarspb.BatchArrowRecords) ([]ptrace.Traces, error)
• type ConsumerAPI
• type Option
  • func WithMemoryLimit(bytes uint64) Option
  • func WithMeterProvider(p metric.MeterProvider) Option
  • func WithMeterProviderAlt(p metric.MeterProvider) Option
  • func WithTracesConfig(tcfg *arrow.Config) Option
• type Producer
  • func NewProducer() *Producer
  • func NewProducerWithOptions(options ...cfg.Option) *Producer
  • func (p *Producer) BatchArrowRecordsFromLogs(ls plog.Logs) (*colarspb.BatchArrowRecords, error)
  • func (p *Producer) BatchArrowRecordsFromMetrics(metrics pmetric.Metrics) (*colarspb.BatchArrowRecords, error)
  • func (p *Producer) BatchArrowRecordsFromTraces(ts ptrace.Traces) (*colarspb.BatchArrowRecords, error)
  • func (p *Producer) Close() error
  • func (p *Producer) GetAndResetStats() pstats.ProducerStats
  • func (p *Producer) LogsBuilder() *logsarrow.LogsBuilder
  • func (p *Producer) LogsRecordBuilderExt() *builder.RecordBuilderExt
  • func (p *Producer) MetricsBuilder() *metricsarrow.MetricsBuilder
  • func (p *Producer) MetricsRecordBuilderExt() *builder.RecordBuilderExt
  • func (p *Producer) Produce(rms []*record_message.RecordMessage) (*colarspb.BatchArrowRecords, error)
  • func (p *Producer) RecordSizeStats() map[string]*pstats.RecordSizeStats
  • func (p *Producer) SetObserver(observer observer.ProducerObserver)
  • func (p *Producer) ShowStats()
  • func (p *Producer) TracesBuilder() *tracesarrow.TracesBuilder
  • func (p *Producer) TracesRecordBuilderExt() *builder.RecordBuilderExt
• type ProducerAPI

Variables

var ErrConsumerMemoryLimit error = common.LimitError{}

ErrConsumerMemoryLimit is used by calling code to check errors.Is(err, ErrConsumerMemoryLimit). It is never returned.

Functions

func NewConsoleObserver

func NewConsoleObserver(maxRows, maxPrints int) observer.ProducerObserver

Types

type Config

type Config struct {
	// contains filtered or unexported fields
}

type Consumer

type Consumer struct {

	// Config embeds the configurable parameters.
	Config
	// contains filtered or unexported fields
}

Consumer is a BatchArrowRecords consumer.

func NewConsumer

func NewConsumer(opts ...Option) *Consumer

NewConsumer creates a new BatchArrowRecords consumer, i.e. a decoder consuming BatchArrowRecords and returning the corresponding OTLP representation (pmetric,Metrics, plog.Logs, ptrace.Traces).

func (*Consumer) Close

func (c *Consumer) Close() error

Close closes the consumer and all its ipc readers.

func (*Consumer) Consume

func (c *Consumer) Consume(bar *colarspb.BatchArrowRecords) (ibes []*record_message.RecordMessage, retErr error)

Consume takes a BatchArrowRecords protobuf message and returns an array of RecordMessage. Note: the records wrapped in the RecordMessage must be released after use by the caller.

func (*Consumer) LogsFrom

func (c *Consumer) LogsFrom(bar *colarspb.BatchArrowRecords) ([]plog.Logs, error)

LogsFrom produces an array of plog.Logs from a BatchArrowRecords message.

func (*Consumer) MetricsFrom

func (c *Consumer) MetricsFrom(bar *colarspb.BatchArrowRecords) ([]pmetric.Metrics, error)

MetricsFrom produces an array of pmetric.Metrics from a BatchArrowRecords message.

func (*Consumer) TracesFrom

func (c *Consumer) TracesFrom(bar *colarspb.BatchArrowRecords) ([]ptrace.Traces, error)

TracesFrom produces an array of ptrace.Traces from a BatchArrowRecords message.

type ConsumerAPI

type ConsumerAPI interface {
	LogsFrom(*colarspb.BatchArrowRecords) ([]plog.Logs, error)
	TracesFrom(*colarspb.BatchArrowRecords) ([]ptrace.Traces, error)
	MetricsFrom(*colarspb.BatchArrowRecords) ([]pmetric.Metrics, error)
	Close() error
}

ConsumerAPI is the interface of a Consumer considering all signals. This is useful for mock testing.

type Option

type Option func(*Config)

func WithMemoryLimit

func WithMemoryLimit(bytes uint64) Option

WithMemoryLimit configures the Arrow limited memory allocator.

func WithMeterProvider

func WithMeterProvider(p metric.MeterProvider) Option

WithMeterProvider configures an OTel metrics provider. If none is configured, the global meter provider will be used.

func WithMeterProviderAlt

func WithMeterProviderAlt(p metric.MeterProvider) Option

WithMeterProviderAlt configures an OTel metrics provider. If none is configured, the global meter provider will be used. This is an alternative to WithMeterProvider, and will eventually be the only option to remove dependency on configtelemetry.Level.

func WithTracesConfig

func WithTracesConfig(tcfg *arrow.Config) Option

WithTracesConfig configures trace-specific Arrow encoding options.

type Producer

type Producer struct {
	// contains filtered or unexported fields
}

Producer is a BatchArrowRecords producer.

func NewProducer

func NewProducer() *Producer

NewProducer creates a new BatchArrowRecords producer.

The method close MUST be called when the producer is not used anymore to release the memory and avoid memory leaks.

func NewProducerWithOptions

func NewProducerWithOptions(options ...cfg.Option) *Producer

NewProducerWithOptions creates a new BatchArrowRecords producer with a set of options.

The method close MUST be called when the producer is not used anymore to release the memory and avoid memory leaks.

func (*Producer) BatchArrowRecordsFromLogs

func (p *Producer) BatchArrowRecordsFromLogs(ls plog.Logs) (*colarspb.BatchArrowRecords, error)

BatchArrowRecordsFromLogs produces a BatchArrowRecords message from a plog.Logs messages.

func (*Producer) BatchArrowRecordsFromMetrics

func (p *Producer) BatchArrowRecordsFromMetrics(metrics pmetric.Metrics) (*colarspb.BatchArrowRecords, error)

BatchArrowRecordsFromMetrics produces a BatchArrowRecords message from a pmetric.Metrics messages.

func (*Producer) BatchArrowRecordsFromTraces

func (p *Producer) BatchArrowRecordsFromTraces(ts ptrace.Traces) (*colarspb.BatchArrowRecords, error)

BatchArrowRecordsFromTraces produces a BatchArrowRecords message from a ptrace.Traces messages.

func (*Producer) Close

func (p *Producer) Close() error

Close closes all stream producers.

func (*Producer) GetAndResetStats

func (p *Producer) GetAndResetStats() pstats.ProducerStats

GetAndResetStats returns the stats and resets them.

func (*Producer) LogsBuilder

func (p *Producer) LogsBuilder() *logsarrow.LogsBuilder

func (*Producer) LogsRecordBuilderExt

func (p *Producer) LogsRecordBuilderExt() *builder.RecordBuilderExt

LogsRecordBuilderExt returns the record builder used to encode logs.

func (*Producer) MetricsBuilder

func (p *Producer) MetricsBuilder() *metricsarrow.MetricsBuilder

func (*Producer) MetricsRecordBuilderExt

func (p *Producer) MetricsRecordBuilderExt() *builder.RecordBuilderExt

MetricsRecordBuilderExt returns the record builder used to encode metrics.

func (*Producer) Produce

func (p *Producer) Produce(rms []*record_message.RecordMessage) (*colarspb.BatchArrowRecords, error)

Produce takes a slice of RecordMessage and returns the corresponding BatchArrowRecords protobuf message.

func (*Producer) RecordSizeStats

func (p *Producer) RecordSizeStats() map[string]*pstats.RecordSizeStats

RecordSizeStats returns statistics per record payload type.

func (*Producer) SetObserver

func (p *Producer) SetObserver(observer observer.ProducerObserver)

SetObserver adds an observer to the producer.

func (*Producer) ShowStats

func (p *Producer) ShowStats()

ShowStats prints the stats to the console.

func (*Producer) TracesBuilder

func (p *Producer) TracesBuilder() *tracesarrow.TracesBuilder

func (*Producer) TracesRecordBuilderExt

func (p *Producer) TracesRecordBuilderExt() *builder.RecordBuilderExt

TracesRecordBuilderExt returns the record builder used to encode traces.

type ProducerAPI

type ProducerAPI interface {
	BatchArrowRecordsFromTraces(ptrace.Traces) (*colarspb.BatchArrowRecords, error)
	BatchArrowRecordsFromLogs(plog.Logs) (*colarspb.BatchArrowRecords, error)
	BatchArrowRecordsFromMetrics(pmetric.Metrics) (*colarspb.BatchArrowRecords, error)
	Close() error
}

ProducerAPI is the interface of a Producer considering all signals. This is useful for mock testing.