googlecloudstorageexporter

README

Google Cloud Storage Exporter

This exporter allows you to export logs, metrics, and traces to Google Cloud Storage. Telemetry is exported in OpenTelemetry Protocol JSON format.

Minimum Agent Versions

• Introduced: v1.72.0

Supported Pipelines

• Logs
• Metrics
• Traces

How It Works

1. The exporter marshals any telemetry sent to it into OTLP Json Format. It will apply compression if configured to do so.
2. Exports telemetry to Google Cloud Storage. See Object Path for more information on the expected object path format.

Notes

1. The exporter will create a bucket if it doesn't exist. Bucket names in GCS are globally unique, so if any organization contains a bucket with the given name, it will fail to create and will likely give 403 Forbidden codes back when it tries to write. You can manually create the bucket in GCS to ensure the name is not taken. More info
2. You can authenticate to Google Cloud using the provided credentials, credentials_file, or by using Application Default Credentials.
3. Your authentication credentials must have the Storage Admin permission to create buckets, folders, and objects.

Configuration

Field	Type	Default	Required	Description
bucket_name	string		true	The name of the bucket to store objects in.
project_id	string		false	The ID of the Google Cloud project the bucket belongs to. The exporter will read this value from the credentials if it is not configured.
bucket_location	string		false	The location of the bucket. Uses GCS default location if not set. Can only be set during bucket creation.
bucket_storage_class	string		false	The storage class of the bucket. Uses GCS default storage class if not set. Can only be set during bucket creation.
folder_name	string		false	An optional folder to put the objects in.
object_prefix	string		false	An optional prefix to prepend to the object file name.
credentials	string		false	Optional credentials to provide authentication to Google Cloud. Mutually exclusive with credentials_file.
credentials_file	string		false	Optional file path to credentials to provide authentication to Google Cloud. Mutually exclusive with credentials.
partition	string	minute	false	Time granularity of object name. Valid values are hour or minute.
compression	string	none	false	The type of compression applied to the data before sending it to storage. Valid values are none and gzip.
timeout	string		false	See doc for details.
sending_queue	map		false	See doc for details.
retry_on_failure	map		false	See doc for details.

Object Path

Object paths will be in the form:

{folder_name}/year=XXXX/month=XX/day=XX/hour=XX/minute=XX

Example Configurations

Minimal Configuration

This configuration only specifies the bucket name. The exporter will use the default partition of minute and will not apply compression. It will use the Google Cloud default location and storage class for the bucket, will not add a folder name or object prefix, and will authenticate to google cloud using application default credentials instead of credentials or credentials_file. The project ID is read from the credentials.

googlecloudstorage:
  bucket_name: "my-bucket-name"

Example Object Names:

year=2021/month=01/day=01/hour=01/minute=00/metrics_{random_id}.json
year=2021/month=01/day=01/hour=01/minute=00/logs_{random_id}.json
year=2021/month=01/day=01/hour=01/minute=00/traces_{random_id}.json

Hour Partition Configuration with Compression

This shows specifying a partition of hour and that the minute=XX portion of the blob path will be omitted. It also adds compression, reducing the object size significantly.

googlecloudstorage:
  bucket_name: "my-bucket-name"
  partition: "hour"
  compression: "gzip"

Example Object Names:

year=2021/month=01/day=01/hour=01/metrics_{random_id}.json.gz
year=2021/month=01/day=01/hour=01/logs_{random_id}.json.gz
year=2021/month=01/day=01/hour=01/traces_{random_id}.json.gz

Full Configuration

This configuration shows all fields filled out.

googlecloudstorage:
  project_id: "my-project-id-18352"
  bucket_name: "my-bucket-name"
  bucket_location: "US-EAST1"
  bucket_storage_class: "NEARLINE"
  credentials_file: "/path/to/googlecloud/credentials/file"
  folder_name: "my-folder-name"
  object_prefix: "object-prefix_"
  partition: "hour"
  compression: "gzip"

Example Blob Names:

my-folder-name/year=2021/month=01/day=01/hour=01/object-prefix_metrics_{random_id}.json.gz
my-folder-name/year=2021/month=01/day=01/hour=01/object-prefix_logs_{random_id}.json.gz
my-folder-name/year=2021/month=01/day=01/hour=01/object-prefix_traces_{random_id}.json.gz

Documentation

Overview

Package googlecloudstorageexporter stores OpenTelemetry data as a Google Cloud Storage exporter.

Index

• func NewFactory() exporter.Factory
• type Config
  • func (c *Config) Validate() error

Functions

func NewFactory

func NewFactory() exporter.Factory

NewFactory creates a factory for Google Cloud Storage Exporter

Types

type Config

type Config struct {
	// ProjectID is the ID of the Google Cloud project the bucket belongs to.
	ProjectID string `mapstructure:"project_id"`

	// BucketName is the name of the bucket to store objects in.
	BucketName string `mapstructure:"bucket_name"`

	// BucketLocation is the location of the bucket.
	BucketLocation string `mapstructure:"bucket_location"`

	// BucketStorageClass is the storage class of the bucket.
	BucketStorageClass string `mapstructure:"bucket_storage_class"`

	// FolderName is the name of the folder to store objects under.
	FolderName string `mapstructure:"folder_name"`

	// ObjectPrefix is the prefix to add to the object name.
	ObjectPrefix string `mapstructure:"object_prefix"`

	// Credentials and CredentialsFile are mutually exclusive and provide authentication to Google Cloud Storage.
	Credentials     string `mapstructure:"credentials"`
	CredentialsFile string `mapstructure:"credentials_file"`

	// Partition is the time granularity of the object.
	// Valid values are "hour" or "minute". Default: minute
	Partition partitionType `mapstructure:"partition"`

	// Compression is the type of compression to use.
	// Valid values are "none" or "gzip". Default: none
	Compression compressionType `mapstructure:"compression"`

	// TimeoutConfig configures timeout settings for exporter operations.
	exporterhelper.TimeoutConfig `mapstructure:",squash"` // squash ensures fields are correctly decoded in embedded struct.

	// QueueConfig defines the queuing behavior for the exporter.
	exporterhelper.QueueConfig `mapstructure:"sending_queue"`

	// BackOffConfig defines the retry behavior for failed operations.
	configretry.BackOffConfig `mapstructure:"retry_on_failure"`
}

Config is the configuration for the googlecloudstorage exporter

func (*Config) Validate

func (c *Config) Validate() error

Validate validates the config.