cloudwatchwriter2

README

cloudwatchwriter2 aka slog-cloudwatch

A robust log/slog handler or zerolog writer for AWS CloudWatch using Go SDK v2. It can be used for writing logs from any kind of logging library as it implements io.Writer interface. Each call of the Write method contain a byte slice payload with valid JSON, longer payloads must not be written in more than 1 block/call.

This library assumes that you have IAM credentials to allow you to talk to AWS CloudWatch Logs. The specific permissions that are required are:

• CreateLogGroup,
• CreateLogStream,
• DescribeLogStreams,
• PutLogEvents.

If these permissions aren't assigned to the user who's IAM credentials you're using then this package will not work. There are two exceptions to that:

• if the log group already exists, then you don't need permission to CreateLogGroup;
• if the log stream already exists, then you don't need permission to CreateLogStream.

The official AWS SDK client must be constructed and passed to the cloudwatchwriter2:

aws_region := os.Getenv("AWS_REGION")
aws_key := os.Getenv("AWS_KEY")
aws_secret := os.Getenv("AWS_SECRET")
aws_session := os.Getenv("AWS_SESSION")

options := cloudwatchlogs.Options{
    Region:      aws_region,
    Credentials: aws.NewCredentialsCache(credentials.NewStaticCredentialsProvider(aws_key, aws_secret, aws_session)),
}
client := cloudwatchlogs.New(options)

Make sure to close the writer to flush the queue, you can defer the Close() call in main. The Close() function blocks until all the logs have been processed with a maximum timeout of 2 seconds.

Usage with log/slog

Use the high-performance built-in JSON handler from Go library to directly write data.

w, err := cww.NewWithClient(client, 500*time.Millisecond, logGroupName, logStreamName)
h := slog.NewJSONHandler(w, &slog.HandlerOptions{AddSource: true})
slog.SetDefault(slog.New(h))

slog.Info("log", "from", "slog", "i", 42)

See the example for more info.

Usage with Zerolog

logger := zerolog.New(cloudWatchWriter).With().Timestamp().Logger()

logger.Info().Str("from", "zerolog").Msgf("Log %d", 42)

See the example for more info.

Write to CloudWatch and the console

What I personally prefer is to write to both CloudWatch and the console, e.g.

log.Logger := zerolog.New(zerolog.MultiLevelWriter(consoleWriter, cloudWatchWriter)).With().Timestamp().Logger()

Changing the default settings

Batch interval

The logs are sent in batches because AWS has a maximum of 5 PutLogEvents requests per second per log stream. The default value of the batch period is 5 seconds, which means it will send the a batch of logs at least once every 5 seconds. Batches of logs will be sent earlier if the size of the collected logs exceeds 1MB (another AWS restriction). To change the batch frequency, you can set the time interval between batches to a smaller or larger value, e.g. 1 second:

err := cloudWatchWriter.SetBatchInterval(time.Second)

If you set it below 200 milliseconds it will return an error.

The batch interval is not guaranteed as two things can alter how often the batches get delivered:

• as soon as 1MB of logs or 10k logs have accumulated, they are sent (due to AWS restrictions on batch size);
• we have to send the batches in sequence (an AWS restriction) so a long running request to CloudWatch can delay the next batch.

LICENSE

MIT

Acknowledgements

The original library was written by mac07 (https://github.com/mec07/cloudwatchwriter), I upgraded it to SDK v2. The work is based on logrus implementation (https://github.com/kdar/logrus-cloudwatchlogs) and a gist (https://gist.github.com/asdine/f821abe6189a04250ae61b77a3048bd9). Thanks all!

In 2025, I have decided to rewrite the whole queueing functionality as it turned out the original code had some issues with performance. The maximum bandwidth was 1000 events per second. I cleaned up the code, added statistics, simplified and updated dependencies. Finally, added some information on how to use the library with log/slog.

Documentation

Index

• Variables
• type CloudWatchLogsClient
• type CloudWatchWriter
  • func NewWithClient(client CloudWatchLogsClient, batchInterval time.Duration, ...) (*CloudWatchWriter, error)
  • func NewWithClientContext(ctx context.Context, client CloudWatchLogsClient, batchInterval time.Duration, ...) (*CloudWatchWriter, error)
  • func (c *CloudWatchWriter) Close()
  • func (c *CloudWatchWriter) Flush()
  • func (c *CloudWatchWriter) Write(log []byte) (int, error)
• type Handler
  • func NewHandler(config HandlerConfig) (*Handler, error)
  • func (h *Handler) Close()
• type HandlerConfig
• type LastErr
• type Stats

Variables

var ErrBatchIntervalTooSmall = errors.New("batch interval is too small")

ErrBatchIntervalTooSmall is returned when the batch interval is too small.

var ErrFullOrClosed = errors.New("cannot create new event: channel full or closed")

ErrFullOrClosed is returned when the payloads channel is full or closed via close().

Types

type CloudWatchLogsClient

type CloudWatchLogsClient interface {
	DescribeLogStreams(context.Context, *cloudwatchlogs.DescribeLogStreamsInput, ...func(*cloudwatchlogs.Options)) (*cloudwatchlogs.DescribeLogStreamsOutput, error)
	CreateLogGroup(context.Context, *cloudwatchlogs.CreateLogGroupInput, ...func(*cloudwatchlogs.Options)) (*cloudwatchlogs.CreateLogGroupOutput, error)
	CreateLogStream(context.Context, *cloudwatchlogs.CreateLogStreamInput, ...func(*cloudwatchlogs.Options)) (*cloudwatchlogs.CreateLogStreamOutput, error)
	PutLogEvents(context.Context, *cloudwatchlogs.PutLogEventsInput, ...func(*cloudwatchlogs.Options)) (*cloudwatchlogs.PutLogEventsOutput, error)
}

CloudWatchLogsClient represents the AWS cloudwatchlogs client that we need to talk to CloudWatch

type CloudWatchWriter

type CloudWatchWriter struct {
	Stats Stats
	// contains filtered or unexported fields
}

CloudWatchWriter can be inserted into zerolog to send logs to CloudWatch.

func NewWithClient

func NewWithClient(client CloudWatchLogsClient, batchInterval time.Duration, logGroupName, logStreamName string) (*CloudWatchWriter, error)

NewWithClient does the same as NewWithClientContext but uses context.Background() as the context.

func NewWithClientContext

func NewWithClientContext(ctx context.Context, client CloudWatchLogsClient, batchInterval time.Duration, logGroupName, logStreamName string) (*CloudWatchWriter, error)

NewWithClientContext creates a new CloudWatchWriter with the given client, batch interval, log group name and log stream name. Use Close method to properly close the writer. The writer will not accept any new events after Close is called. The writer will flush the buffer and close the payloads channel when Close is called. Use context cancellation to stop the writer and Close to properly close it.

func (*CloudWatchWriter) Close

func (c *CloudWatchWriter) Close()

Close will flush the buffer, close the channel and wait until all payloads are sent, not longer than 2 seconds. It is safe to call close multiple times. After close is called the client will not accept any new events, all attemtps to send new events will return ErrFullOrClosed.

func (*CloudWatchWriter) Flush

func (c *CloudWatchWriter) Flush()

Flush will cause the logger to flush the current buffer. It does not block, there is no guarantee that the buffer will be flushed immediately. Use Close in order to properly close during application termination.

func (*CloudWatchWriter) Write

func (c *CloudWatchWriter) Write(log []byte) (int, error)

type Handler

type Handler struct {
	*slog.JSONHandler
	// contains filtered or unexported fields
}

Handler is a slog.Handler that sends logs to AWS CloudWatch.

func NewHandler

func NewHandler(config HandlerConfig) (*Handler, error)

NewHandler creates a new log/slog handler.

func (*Handler) Close

func (h *Handler) Close()

Flush flushes all pending payloads to the CloudWatch client. See CloudWatchWriter.Close for more information.

type HandlerConfig

type HandlerConfig struct {
	// Level is the logging level for this output.
	Level slog.Leveler

	// AddSource is a flag to add source to the log record.
	AddSource bool

	// AWSRegion is the AWS region.
	AWSRegion string

	// AWSKey is the AWS access key.
	AWSKey string

	// AWSSecret is the AWS secret key.
	AWSSecret string

	// AWSSession is an optional AWS session token.
	AWSSession string

	// AWSLogGroup is the AWS CloudWatch log group.
	AWSLogGroup string

	// AWSLogStream is the AWS CloudWatch log stream.
	AWSLogStream string
}

HandlerConfig is the configuration for the Cloudwatch handler.

type LastErr

type LastErr struct {
	// contains filtered or unexported fields
}

type Stats

type Stats struct {
	// Total number of events queued for sending
	QueuedEventCount atomic.Uint64

	// Total number of events sent (EventsEnqueued >= SentEventCount)
	SentEventCount atomic.Uint64

	// Total number of requests sent
	BatchCount atomic.Uint64

	// Total number of HTTP retries
	RetryCount atomic.Uint64
}