reporting

README

reporting

The autoscaler-agent reports multiple types of data (billing data, scaling events) in multiple ways (HTTP, S3, Azure Blob), so reporting is the abstraction allowing us to deduplicate code between them.

Documentation

Index

• func WrapSerialize[E any](wrapper func([]byte) ([]byte, SimplifiableError), ...) func([]E) ([]byte, SimplifiableError)
• type AzureAuthSharedKey
• type AzureBlobStorageClientConfig
• type AzureClient
  • func NewAzureBlobStorageClient(cfg AzureBlobStorageClientConfig, generateKey func() string) (*AzureClient, error)
  • func NewAzureBlobStorageClientWithBaseClient(client *azblob.Client, cfg AzureBlobStorageClientConfig, ...) *AzureClient
  • func (c AzureClient) NewRequest() ClientRequest
• type AzureError
  • func (e AzureError) Error() string
  • func (e AzureError) Simplified() string
  • func (e AzureError) Unwrap() error
• type BaseClient
• type BaseClientConfig
• type Client
• type ClientRequest
• type EventSink
  • func NewEventSink[E any](logger *zap.Logger, metrics *EventSinkMetrics, clients ...Client[E]) *EventSink[E]
  • func (s *EventSink[E]) Enqueue(event E)
• type EventSinkMetrics
  • func NewEventSinkMetrics(prefix string) *EventSinkMetrics
  • func (m *EventSinkMetrics) MustRegister(reg *prometheus.Registry)
• type HTTPClient
  • func NewHTTPClient(client *http.Client, cfg HTTPClientConfig) HTTPClient
  • func (c HTTPClient) NewRequest() ClientRequest
• type HTTPClientConfig
• type S3Client
  • func NewS3Client(ctx context.Context, cfg S3ClientConfig, generateKey func() string) (*S3Client, error)
  • func (c *S3Client) NewRequest() ClientRequest
• type S3ClientConfig
• type S3Error
  • func (e S3Error) Error() string
  • func (e S3Error) Simplified() string
  • func (e S3Error) Unwrap() error
• type SimplifiableError
  • func GZIPCompress(payload []byte) ([]byte, SimplifiableError)
  • func JSONLinesMarshalBatch[E any](events []E) ([]byte, SimplifiableError)
  • func JSONMarshalBatch[V any](value V) ([]byte, SimplifiableError)

Functions

func WrapSerialize

func WrapSerialize[E any](
	wrapper func([]byte) ([]byte, SimplifiableError),
	base func([]E) ([]byte, SimplifiableError),
) func([]E) ([]byte, SimplifiableError)

WrapSerialize is a combinator that takes an existing function valid for Client.SerializeBatch and produces a new function that applies the 'wrapper' function to the output before returning it.

This can be used, for example, to provide a SerializeBatch implementation that gzips the data after encoding it as JSON, e.g., by:

WrapSerialize(GZIPCompress, JSONLinesMarshalBatch)

Types

type AzureAuthSharedKey

type AzureAuthSharedKey struct {
	AccountName string `json:"accountName"`
	AccountKey  string `json:"accountKey"`
}

type AzureBlobStorageClientConfig

type AzureBlobStorageClientConfig struct {
	// In Azure a Container is close to a bucket in AWS S3
	Container string `json:"container"`
	// Example Endpoint: "https://MYSTORAGEACCOUNT.blob.core.windows.net/"
	Endpoint string `json:"endpoint"`
}

type AzureClient

type AzureClient struct {
	// contains filtered or unexported fields
}

func NewAzureBlobStorageClient

func NewAzureBlobStorageClient(
	cfg AzureBlobStorageClientConfig,
	generateKey func() string,
) (*AzureClient, error)

func NewAzureBlobStorageClientWithBaseClient

func NewAzureBlobStorageClientWithBaseClient(
	client *azblob.Client,
	cfg AzureBlobStorageClientConfig,
	generateKey func() string,
) *AzureClient

func (AzureClient) NewRequest

func (c AzureClient) NewRequest() ClientRequest

NewRequest implements BaseClient

type AzureError

type AzureError struct {
	Err error
}

func (AzureError) Error

func (e AzureError) Error() string

func (AzureError) Simplified

func (e AzureError) Simplified() string

func (AzureError) Unwrap

func (e AzureError) Unwrap() error

type BaseClient

type BaseClient interface {
	NewRequest() ClientRequest
}

BaseClient is the shared lower-level interface to send the processed data somewhere.

It's split into the client itself, intended to be used as a kind of persistent object, and a separate ClientRequest object, intended to be used only for the lifetime of a single request.

See S3Client, AzureBlobClient, and HTTPClient.

type BaseClientConfig

type BaseClientConfig struct {
	PushEverySeconds          uint `json:"pushEverySeconds"`
	PushRequestTimeoutSeconds uint `json:"pushRequestTimeoutSeconds"`
	MaxBatchSize              uint `json:"maxBatchSize"`
}

type Client

type Client[E any] struct {
	Name           string
	Base           BaseClient
	BaseConfig     BaseClientConfig
	SerializeBatch func(events []E) ([]byte, SimplifiableError)
}

type ClientRequest

type ClientRequest interface {
	LogFields() zap.Field
	Send(ctx context.Context, payload []byte) SimplifiableError
}

ClientRequest is the abstract interface for a single request to send a batch of processed data.

This exists as a separate interface because there are some request-scoped values that we'd like to include in the call to LogFields().

type EventSink

type EventSink[E any] struct {
	// contains filtered or unexported fields
}

func NewEventSink

func NewEventSink[E any](logger *zap.Logger, metrics *EventSinkMetrics, clients ...Client[E]) *EventSink[E]

func (*EventSink[E]) Enqueue

func (s *EventSink[E]) Enqueue(event E)

Enqueue submits the event to the internal client sending queues, returning without blocking.

type EventSinkMetrics

type EventSinkMetrics struct {
	// contains filtered or unexported fields
}

func NewEventSinkMetrics

func NewEventSinkMetrics(prefix string) *EventSinkMetrics

func (*EventSinkMetrics) MustRegister

func (m *EventSinkMetrics) MustRegister(reg *prometheus.Registry)

type HTTPClient

type HTTPClient struct {
	// contains filtered or unexported fields
}

func NewHTTPClient

func NewHTTPClient(client *http.Client, cfg HTTPClientConfig) HTTPClient

func (HTTPClient) NewRequest

func (c HTTPClient) NewRequest() ClientRequest

NewRequest implements BaseClient

type HTTPClientConfig

type HTTPClientConfig struct {
	URL    string `json:"url"`
	Method string `json:"method"`
}

type S3Client

type S3Client struct {
	// contains filtered or unexported fields
}

S3Client is a BaseClient for S3

func NewS3Client

func NewS3Client(
	ctx context.Context,
	cfg S3ClientConfig,
	generateKey func() string,
) (*S3Client, error)

func (*S3Client) NewRequest

func (c *S3Client) NewRequest() ClientRequest

NewRequest implements BaseClient

type S3ClientConfig

type S3ClientConfig struct {
	Bucket   string `json:"bucket"`
	Region   string `json:"region"`
	Endpoint string `json:"endpoint"`
}

type S3Error

type S3Error struct {
	Err error
}

func (S3Error) Error

func (e S3Error) Error() string

func (S3Error) Simplified

func (e S3Error) Simplified() string

func (S3Error) Unwrap

func (e S3Error) Unwrap() error

type SimplifiableError

type SimplifiableError interface {
	error

	Simplified() string
}

SimplifiableError is an extension of the standard 'error' interface that provides a safe-for-metrics string representing the error.

func GZIPCompress

func GZIPCompress(payload []byte) ([]byte, SimplifiableError)

GZIPCompress is a helper function to compress a byte string with gzip

func JSONLinesMarshalBatch

func JSONLinesMarshalBatch[E any](events []E) ([]byte, SimplifiableError)

JSONLinesMarshalBatch is a function to implement Client.SerializeBatch by serializing each event in the batch onto a separate JSON line.

See also: JSONMarshalBatch

func JSONMarshalBatch

func JSONMarshalBatch[V any](value V) ([]byte, SimplifiableError)

JSONMarshalBatch is a helper function to trivially build a function satisfying Client.SerializeBatch.

This function can't *directly* be used, because it takes any type as input, but a small wrapper function typically will suffice.

Why not take a list directly? Sometimes there's a small amount of wrapping we'd like to do, e.g. packaging it as a struct instead of directly an array.

See also: JSONLinesMarshalBatch, which *can* be used directly.