pandora

README

Pandora Golang API client

This is based on https://github.com/dougEfresh/logzio-go. Thanks to @dougEfresh

Sends logs to pandora 2.0 over HTTP. It is a low level lib that can to be integrated with other logging libs.

Prerequisites

go 1.x

Installation

$ go get -u github.com/lvheyang/pandora-go

Quick Start

package main

import (
	"github.com/lvheyang/pandora-go"
	"os"
	"time"
)

func main() {
	l, err := pandora.New(
		"fake-token",
		pandora.SetDebug(os.Stderr),
		pandora.SetUrl("http://pandora-express-rc.qiniu.io/api/v1/data"),
		pandora.SetDrainDuration(time.Second*10),
		pandora.SetTempDirectory("myQueue"),
		pandora.SetDrainDiskThreshold(99)) // token is required
	if err != nil {
		panic(err)
	}
	msg := pandora.PandoraReqBody{
		Raw:         "{\"a\":\"lllllll\"}", //必填，原始日志
		SourceType:  "json", //必填，来源类型
		Repo:        "default", //必填，仓库
		Host:        "", //选填，Host地址
		Origin:      "", //选填，来源
		Timestamp:   0,  //选填，事件时间
		CollectTime: 0,  //选填，收集时间
	}

	err = l.SendData(msg)
	if err != nil {
		panic(err)
	}

	l.Stop() //logs are buffered on disk. Stop will drain the buffer
}

Usage

• Set url mode: pandora.New(token, SetUrl(ts.URL))

• Set drain duration (flush logs on disk): pandora.New(token, SetDrainDuration(time.Hour))

• Set debug mode: pandora.New(token, SetDebug(os.Stderr))

• Set queue dir: pandora.New(token, SetSetTempDirectory(os.Stderr))

• Set the sender to check if it crosses the maximum allowed disk usage: pandora.New(token, SetCheckDiskSpace(true))

• Set disk queue threshold, once the threshold is crossed the sender will not enqueue the received logs: pandora.New(token, SetDrainDiskThreshold(99))

Disk queue

Pandora go client uses goleveldb and goqueue as a persistent storage. Every 5 seconds logs are sent to logz.io (if any are available)

Tests

$ go test -v

See travis.yaml for running benchmark tests

Contributing

All PRs are welcome

Authors

• Douglas Chimento - [dougEfresh][me]
• Ido Halevi - idohalevi

License

This project is licensed under the Apache License - see the LICENSE file for details

Documentation

Index

• type PandoraReqBody
• type PandoraSender
  • func New(token string, options ...SenderOptionFunc) (*PandoraSender, error)
  • func (l *PandoraSender) CloseIdleConnections()
  • func (l *PandoraSender) Drain()
  • func (l *PandoraSender) Send(payload []byte) error
  • func (l *PandoraSender) SendData(body PandoraReqBody) error
  • func (l *PandoraSender) Stop()
  • func (l *PandoraSender) Sync() error
  • func (l *PandoraSender) Write(p []byte) (n int, err error)
• type Sender
• type SenderOptionFunc
  • func SetCheckDiskSpace(check bool) SenderOptionFunc
  • func SetDebug(debug io.Writer) SenderOptionFunc
  • func SetDrainDiskThreshold(th int) SenderOptionFunc
  • func SetDrainDuration(duration time.Duration) SenderOptionFunc
  • func SetTempDirectory(dir string) SenderOptionFunc
  • func SetUrl(url string) SenderOptionFunc

Types

type PandoraReqBody

type PandoraReqBody struct {
	Raw         string `json:"raw"`
	SourceType  string `json:"sourcetype"`
	Repo        string `json:"repo"`
	Host        string `json:"host"`
	Origin      string `json:"origin"`
	Timestamp   int64  `json:"timestamp"`
	CollectTime int64  `json:"collectTime"`
}

type PandoraSender

type PandoraSender struct {
	// contains filtered or unexported fields
}

PandoraSender instance of the

func New

func New(token string, options ...SenderOptionFunc) (*PandoraSender, error)

New creates a new Pandora sender with a token and options

func (*PandoraSender) CloseIdleConnections

func (l *PandoraSender) CloseIdleConnections()

CloseIdleConnections to close all remaining open connections

func (*PandoraSender) Drain

func (l *PandoraSender) Drain()

Drain - Send remaining logs

func (*PandoraSender) Send

func (l *PandoraSender) Send(payload []byte) error

Send the payload to Pandora

func (*PandoraSender) SendData

func (l *PandoraSender) SendData(body PandoraReqBody) error

Send reqbody to Pandora

func (*PandoraSender) Stop

func (l *PandoraSender) Stop()

Stop will close the LevelDB queue and do a final drain

func (*PandoraSender) Sync

func (l *PandoraSender) Sync() error

Sync drains the queue

func (*PandoraSender) Write

func (l *PandoraSender) Write(p []byte) (n int, err error)

type Sender

type Sender PandoraSender

Sender Alias to PandoraSender

type SenderOptionFunc

type SenderOptionFunc func(*PandoraSender) error

SenderOptionFunc options for sender

func SetCheckDiskSpace

func SetCheckDiskSpace(check bool) SenderOptionFunc

SetCheckDiskSpace to check if it crosses the maximum allowed disk usage

func SetDebug

func SetDebug(debug io.Writer) SenderOptionFunc

SetDebug mode and send logs to this writer

func SetDrainDiskThreshold

func SetDrainDiskThreshold(th int) SenderOptionFunc

SetDrainDiskThreshold to change the maximum used disk space

func SetDrainDuration

func SetDrainDuration(duration time.Duration) SenderOptionFunc

SetDrainDuration to change the interval between drains

func SetTempDirectory

func SetTempDirectory(dir string) SenderOptionFunc

SetTempDirectory Use this temporary dir

func SetUrl

func SetUrl(url string) SenderOptionFunc

SetUrl set the url which maybe different from the defaultUrl