ingest

Documentation

Overview

Package ingest provides the datatypes and functions helping with ingesting data into Axiom.

Usage:

import "github.com/axiomhq/axiom-go/axiom/ingest"

Index

• Constants
• type Failure
• type Option
  • func AddCSVField(field ...string) Option
  • func SetCSVDelimiter(delim string) Option
  • func SetCSVFields(fields ...string) Option
  • func SetEventLabel(key string, value any) Option
  • func SetEventLabels(labels map[string]any) Option
  • func SetTimestampField(field string) Option
  • func SetTimestampFormat(format string) Option
• type Options
• type Status
  • func (s *Status) Add(other *Status)

Constants

const TimestampField = "_time"

TimestampField is the default field the server will look for a timestamp to use as the ingestion time. If not present, the server will set the ingestion time to the current server time.

Types

type Failure

type Failure struct {
	// Timestamp of the event that failed to ingest.
	Timestamp time.Time `json:"timestamp"`
	// Error that made the event fail to ingest.
	Error string `json:"error"`
}

Failure describes the ingestion failure of a single event.

type Option

type Option func(*Options)

An Option applies optional parameters to an ingest operation.

func AddCSVField

func AddCSVField(field ...string) Option

AddCSVField adds one or more fields to be ingested with every CSV event.

func SetCSVDelimiter

func SetCSVDelimiter(delim string) Option

SetCSVDelimiter specifies the delimiter that separates CSV fields. Only valid when the content to be ingested is CSV formatted.

func SetCSVFields

func SetCSVFields(fields ...string) Option

SetCSVFields sets the fields to be ingested with every CSV event.

func SetEventLabel

func SetEventLabel(key string, value any) Option

SetEventLabel adds a label to apply to all events. This option can be called multiple times to add multiple labels. If a label with the same key already exists, it will be overwritten.

func SetEventLabels

func SetEventLabels(labels map[string]any) Option

SetEventLabels sets the labels to apply to all events. It will overwrite any existing labels.

func SetTimestampField

func SetTimestampField(field string) Option

SetTimestampField specifies the field Axiom will use to extract the events time from. Defaults to TimestampField

func SetTimestampFormat

func SetTimestampFormat(format string) Option

SetTimestampFormat specifies the format of the timestamp field. The reference time is "Mon Jan 2 15:04:05 -0700 MST 2006", as specified in https://pkg.go.dev/time/?tab=doc#Parse.

type Options

type Options struct {
	// TimestampField defines a custom field to extract the ingestion timestamp
	// from. Defaults to [TimestampField].
	TimestampField string `url:"timestamp-field,omitempty"`
	// TimestampFormat defines a custom format for the [Options.TimestampField].
	// The reference time is "Mon Jan 2 15:04:05 -0700 MST 2006", as specified
	// in https://pkg.go.dev/time/?tab=doc#Parse.
	TimestampFormat string `url:"timestamp-format,omitempty"`
	// CSVDelimiter is the delimiter that separates CSV fields. Only valid when
	// the content to be ingested is CSV formatted.
	CSVDelimiter string `url:"csv-delimiter,omitempty"`
	// EventLabels are a key-value pairs that will be added to all events. Their
	// purpose is to allow for labeling events without alterting the original
	// event data. This is especially useful when ingesting events from a
	// third-party source that you do not have control over.
	EventLabels map[string]any `url:"-"`
	// Fields is a list of fields to be ingested with every event. This is only
	// valid for CSV content and also completely optional. It comes in handy
	// when the CSV content does not have a header row.
	CSVFields []string `url:"-"`
}

Options specifies the optional parameters for ingestion.

type Status

type Status struct {
	// Ingested is the amount of events that have been ingested.
	Ingested uint64 `json:"ingested"`
	// Failed is the amount of events that failed to ingest.
	Failed uint64 `json:"failed"`
	// Failures are the ingestion failures, if any.
	Failures []*Failure `json:"failures"`
	// ProcessedBytes is the number of bytes processed.
	ProcessedBytes uint64 `json:"processedBytes"`
	// BlocksCreated is the amount of blocks created.
	//
	// Deprecated: BlocksCreated is deprecated and will be removed in a future
	// release.
	BlocksCreated uint32 `json:"blocksCreated"`
	// WALLength is the length of the Write-Ahead Log.
	//
	// Deprecated: WALLength is deprecated and will be removed in a future
	// release.
	WALLength uint32 `json:"walLength"`
	// TraceID is the ID of the trace that was generated by the server for this
	// statuses ingest request.
	TraceID string `json:"-"`
}

Status is the status of an event ingestion operation.

func (*Status) Add

func (s *Status) Add(other *Status)

Add adds the status of another ingestion operation to the current status. The trace ID is ignored.