logzio

README

Logzio Golang API client

Sends logs to logz.io over HTTP. It is a low level lib that can to be integrated with other logging libs.

Prerequisites

go 1.x

Installation

$ go get -u github.com/logzio/logzio-go

Logzio golang api client offers two queue implementations that you can use:

Disk queue

Logzio go client uses goleveldb and goqueue as a persistent storage. Every 5 seconds logs are sent to logz.io (if any are available)

In memory queue

You can see the logzio go client queue implementation in inMemoryQueue.go file. The in memory queue is initialized with 500k log count limit and 20mb capacity by default. You can use the SetinMemoryCapacity() and SetlogCountLimit() functions to override default settings.

Quick Start

Disk queue

package main

import (
  "fmt"
  "github.com/logzio/logzio-go"
  "os"
  "time"
)

func main() {
  l, err := logzio.New(
  		"fake-token",
	  logzio.SetDebug(os.Stderr),
	  logzio.SetUrl("http://localhost:12345"),
	  logzio.SetDrainDuration(time.Minute*10),
	  logzio.SetTempDirectory("myQueue"),
	  logzio.SetDrainDiskThreshold(99),
  	) // token is required
  if err != nil {
    panic(err)
  }
  msg := fmt.Sprintf("{ \"%s\": \"%s\"}", "message", time.Now().UnixNano())

  err = l.Send([]byte(msg))
  if err != nil {
     panic(err)
  }

  l.Stop() //logs are buffered on disk. Stop will drain the buffer
}

In memory queue

package main

import (
  "fmt"
  "github.com/logzio/logzio-go"
  "os"
  "time"
)

func main() {
  l, err := logzio.New(
  		"fake-token",
	  logzio.SetDebug(os.Stderr),
	  logzio.SetUrl("http://localhost:12345"),
	  logzio.SetInMemoryQueue(true),
	  logzio.SetinMemoryCapacity(24000000),
	  logzio.SetlogCountLimit(6000000),
  	) // token is required
  if err != nil {
    panic(err)
  }
  msg := fmt.Sprintf("{ \"%s\": \"%s\"}", "message", time.Now().UnixNano())

  err = l.Send([]byte(msg))
  if err != nil {
     panic(err)
  }

  l.Stop() 
}

Usage

• Set url mode: logzio.New(token, SetUrl(ts.URL))

• Set drain duration (flush logs on disk): logzio.New(token, SetDrainDuration(time.Hour))

• Set debug mode: logzio.New(token, SetDebug(os.Stderr))

• Set queue dir: logzio.New(token, SetSetTempDirectory(os.Stderr))

• Set the sender to check if it crosses the maximum allowed disk usage: logzio.New(token, SetCheckDiskSpace(true))

• Set disk queue threshold, once the threshold is crossed the sender will not enqueue the received logs: logzio.New(token, SetDrainDiskThreshold(99))

• Set the sender to Use in memory queue: logzio.New(token, SetInMemoryQueue(true))

• Set the sender to Use in memory queue with log count limit and capacity: logzio.New(token, SetInMemoryQueue(true), SetinMemoryCapacity(500), SetlogCountLimit(6000000), )

Data compression

All bulks are compressed with gzip by default to disable compressing initialize the client with SetCompress(false):

  logzio.New(token,
	  SetCompress(false),
  )

Tests

$ go test -v

Contributing

All PRs are welcome

Authors

• Douglas Chimento - dougEfresh
• Ido Halevi - idohalevi

Maintainers

• Yotam Loewenbach - yotamloe

License

This project is licensed under the Apache License - see the LICENSE file for details

Acknowledgments

• logzio-java-sender

Changelog

• v1.0.5
  • Change token query parameter to optional for generic use
  • Changed logging levels
• v1.0.4
  • Update gopsutil version (v3.21.6 -> v3.22.3)
• v1.0.3
  • Adjust buffer clearance
  • Changed logging format
• v1.0.2
  • Update dependencies
• v1.0.1
  • Add gzip compression
  • Add option for in Memory queue

Documentation

Index

• type Item
• type LogzioSender
  • func New(token string, options ...SenderOptionFunc) (*LogzioSender, error)
  • func (l *LogzioSender) CloseIdleConnections()
  • func (l *LogzioSender) Drain()
  • func (l *LogzioSender) Send(payload []byte) error
  • func (l *LogzioSender) Stop()
  • func (l *LogzioSender) Sync() error
  • func (l *LogzioSender) Write(p []byte) (n int, err error)
• type Sender
• type SenderOptionFunc
  • func SetCheckDiskSpace(check bool) SenderOptionFunc
  • func SetCompress(b bool) SenderOptionFunc
  • func SetDebug(debug io.Writer) SenderOptionFunc
  • func SetDrainDiskThreshold(th int) SenderOptionFunc
  • func SetDrainDuration(duration time.Duration) SenderOptionFunc
  • func SetHttpClient(client *http.Client) SenderOptionFunc
  • func SetInMemoryQueue(b bool) SenderOptionFunc
  • func SetTempDirectory(dir string) SenderOptionFunc
  • func SetUrl(url string) SenderOptionFunc
  • func SetinMemoryCapacity(size uint64) SenderOptionFunc
  • func SetlogCountLimit(limit int) SenderOptionFunc

Types

type Item

type Item = queue.Item

type LogzioSender

type LogzioSender struct {
	// contains filtered or unexported fields
}

LogzioSender instance of the

func New

func New(token string, options ...SenderOptionFunc) (*LogzioSender, error)

New creates a new Logzio sender with a token and options

func (*LogzioSender) CloseIdleConnections

func (l *LogzioSender) CloseIdleConnections()

CloseIdleConnections to close all remaining open connections

func (*LogzioSender) Drain

func (l *LogzioSender) Drain()

Drain - Send remaining logs

func (*LogzioSender) Send

func (l *LogzioSender) Send(payload []byte) error

Send the payload to logz.io

func (*LogzioSender) Stop

func (l *LogzioSender) Stop()

Stop will close the LevelDB queue and do a final drain

func (*LogzioSender) Sync

func (l *LogzioSender) Sync() error

Sync drains the queue

func (*LogzioSender) Write

func (l *LogzioSender) Write(p []byte) (n int, err error)

type Sender

type Sender LogzioSender

Sender Alias to LogzioSender

type SenderOptionFunc

type SenderOptionFunc func(*LogzioSender) error

SenderOptionFunc options for logz

func SetCheckDiskSpace

func SetCheckDiskSpace(check bool) SenderOptionFunc

SetCheckDiskSpace to check if it crosses the maximum allowed disk usage

func SetCompress

func SetCompress(b bool) SenderOptionFunc

func SetDebug

func SetDebug(debug io.Writer) SenderOptionFunc

SetDebug mode and send logs to this writer

func SetDrainDiskThreshold

func SetDrainDiskThreshold(th int) SenderOptionFunc

SetDrainDiskThreshold to change the maximum used disk space

func SetDrainDuration

func SetDrainDuration(duration time.Duration) SenderOptionFunc

SetDrainDuration to change the interval between drains

func SetHttpClient

func SetHttpClient(client *http.Client) SenderOptionFunc

SetHttpClient to change the default http client

func SetInMemoryQueue

func SetInMemoryQueue(b bool) SenderOptionFunc

SetInMemoryQueue to change the default disk queue

func SetTempDirectory

func SetTempDirectory(dir string) SenderOptionFunc

SetTempDirectory Use this temporary dir

func SetUrl

func SetUrl(url string) SenderOptionFunc

SetUrl set the url which maybe different from the defaultUrl

func SetinMemoryCapacity

func SetinMemoryCapacity(size uint64) SenderOptionFunc

SetinMemoryCapacity to change the default capacity

func SetlogCountLimit

func SetlogCountLimit(limit int) SenderOptionFunc

SetlogCountLimit to change the default limit