logstreamer

README

logstreamer

logstreamer is a tool that allows the streaming of logs to an arbitrary amount of users.

Motivation

Sometimes, when working on project with other people, not everyone has access to your server machine(s). It becomes annoying to have to SSH into the machine and show / provide them with the log outputs for debugging purposes. Sometimes it's annoying to have to SSH into the machine for logs in general.

logstreamer is a quick and dirty solution to this. Given an input stream, it allows multiple observers to 'observe' the stream (either using websockets, cURL, etc.). Additionally, logstreamer can remember the last X amount of log lines, so if something bad happens, an observer will hopefully be able to see it restroactively.

In some cases, your development machine may be a tiny instance (t2.micro, smallest Digital Ocean droplet), with very limited disk space. Letting logs persist to disk can often fill up disk space quickly. logstreamer can help with this by just receiving the logs of stdin. Since it's a development machine, logs generally don't matter unless something goes wrong, or someone is watching.

Building

go get github.com/mfycheng/logstreamer

Using

logstreamer comes with a tool logstream that offers multiple options for streaming logs. Currently, it only offers websocket access. An example use of logstream is:

application | go run cmd/logstream/main.go

or, if installed to your $PATH / $GOPATH:

application | logstream

TODO

There are still things to do, such as:

• Configurable options for the server
• Output log to file (with possible rotation)

Documentation

Overview

Package logstreamer provides a mechanism to allow multiple observers to observer log output when backpressure is not acceptable.

Index

• func StreamToWriter(stream *BufferedLogStream, w io.Writer, addTimestamps bool) (err error)
• type BufferedLogStream
  • func NewBufferedLogStream(maxEntries int) *BufferedLogStream
  • func (b *BufferedLogStream) NewObserver() StreamObserver
  • func (b *BufferedLogStream) WriteLine(line string) error
• type LogEntry
• type StreamObserver
  • func (s *StreamObserver) Chan() <-chan LogEntry
  • func (s *StreamObserver) Close() error

Functions

func StreamToWriter

func StreamToWriter(stream *BufferedLogStream, w io.Writer, addTimestamps bool) (err error)

StreamToWriter creates a new observer for a provided stream, and writes the results to the provided writer. If skipped entries are detected, a message is written to the writer indicating so.

Types

type BufferedLogStream

type BufferedLogStream struct {
	// contains filtered or unexported fields
}

BufferedLogStream is a buffered stream of logs. It allows observers to view X amount of entries in the past upon creation.

func NewBufferedLogStream

func NewBufferedLogStream(maxEntries int) *BufferedLogStream

NewBufferedLogStream creates a new BufferedLogStream, with maxEntries as the buffer size.

func (*BufferedLogStream) NewObserver

func (b *BufferedLogStream) NewObserver() StreamObserver

NewObserver creates a new StreamObserver, pre-populated with whatever is currently in the buffer. Callers should call Close() when finished.

func (*BufferedLogStream) WriteLine

func (b *BufferedLogStream) WriteLine(line string) error

WriteLine writes a line to the stream. The line is timestamped based on insertion time, not observed time.

type LogEntry

type LogEntry struct {
	Number    int64
	Timestamp time.Time
	Line      string
}

LogEntry represents a log entry. The log number indicates the log number since the start of the stream. If an observer observes a gap in log numbers, then they have lost log entries. This is caused by the observer not being able to process logs as fast as they're being generated.

type StreamObserver

type StreamObserver struct {
	// contains filtered or unexported fields
}

StreamObserver is an observer to a stream. Log entries are sent through the streams channel. Users of StreamObserver should call Close() when finished with the observer.

func (*StreamObserver) Chan

func (s *StreamObserver) Chan() <-chan LogEntry

Chan returns a receive-only channel of LogEntries.

func (*StreamObserver) Close

func (s *StreamObserver) Close() error

Close closes the stream observer, freeing any resources required by the associated stream.