gwr

package module
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 29, 2016 License: MIT Imports: 12 Imported by: 0

README

GWR: Get / Watch / Report -ing of operational data

GWR provides on demand access to operational data:

  • define your data sources
  • poll and/or watch them over HTTP or Redis Protocol

Status: beta

GWR is currently in beta devolopment:

  • basic support for get and watch are done, with a couple simple sources; these interfaces are not likely to change before 1.0
  • reporting is not yet started, and is the major blocker before 1.0

Using

GWR exposes a dual HTTP and RESP (Redis Protocol) interface. Integrators may specify the port, the example below uses 4040.

The following examples are against a running instance of example_server/.

HTTP

Example for http:

$ curl localhost:4040/meta/nouns
- /meta/nouns formats: <no value>
- /request_log formats: <no value>
- /response_log formats: <no value>

$ curl -X WATCH localhost:4040/request_log&
$ curl -X WATCH localhost:4040/response_log&

$ curl localhost:8080/foo
404 page not found                                         # this is the normal curl output
GET /foo                                                   # this comes from the first watch-curl
404 19 text/plain; charset=utf-8                           # this comes from the first watch-curl

Resp

$ redis-cli -p 4040 ls                                     # this is a convenience alias for "get /meta/nouns"
1) - /meta/nouns formats: <no value>
2) - /request_log formats: <no value>
3) - /response_log formats: <no value>

$ redis-cli -p 4040 monitor /request_log text /response_log text&
OK

$ curl localhost:8080/bar
404 page not found                                         # this is the curl output
/request_log> GET /bar                                     # this is from redis-cli
/response_log> 404 19 text/plain; charset=utf-8            # so is this, ordering not guaranteed

Integration

To add gwr to a program, all you need to do is call:

gwrProto.ListenAndServe(":4040", nil)

This hosts dual protocol HTTP and RESP server on port 4040.

Defining data sources

To define a data source, the easiest way is to implement the gwr.GenericDataSource interface.

TODO: example

For now see example_server/req_logger.go and example_server/res_logger.go

Running the example server

Should work by:

$ go run example_server/*.go

The example server hosts a dummy 404-ing web server on port 8080 and exposes a request and response log GWR noun. The HTTP and Resp usage examples above are against it.

Documentation

Overview

Package gwr provides on demand operational data sources in Go. A typical use is adding in-depth debug tracing to your program that only turns on when there are any active consumer(s).

Basics

A gwr data source is a named subset data that is Get-able and/or Watch-able. The gwr library can then build reporting on top of Watch-able sources, or by falling back to polling Get-able sources.

For example a request log source would be naturally Watch-able for future requests as they come in. An implementation could go further and add a "last-10" buffer to also become Get-able.

Integrating

To bootstrap gwr and start its server listening on port 4040: 

gwr.Configure(&gwr.Config{ListenAddr: ":4040"})

GWR also adds a handler to the default http server; so if you already have a default http server like: 

log.Fatal(http.ListenAndServe(":8080", nil))

Then gwr will already be accessible at "/gwr/..." on port 8080; you should still call gwr.Configure: 

gwr.Configure(nil)
Example (Httpserver_accesslog) 
package main

import (
	"bufio"
	"fmt"
	"io"
	"log"
	"net"
	"net/http"
	"net/http/httptest"
	"os"
	"text/template"

	gwr "github.com/uber-go/gwr"
	"github.com/uber-go/gwr/source"
)

type accessLogger struct {
	handler http.Handler
	watcher source.GenericDataWatcher
}

func logged(handler http.Handler) *accessLogger {
	if handler == nil {
		handler = http.DefaultServeMux
	}
	return &accessLogger{
		handler: handler,
	}
}

type accessEntry struct {
	Method      string `json:"method"`
	Path        string `json:"path"`
	Query       string `json:"query"`
	Code        int    `json:"code"`
	Bytes       int    `json:"bytes"`
	ContentType string `json:"content_type"`
}

var accessLogTextTemplate = template.Must(template.New("req_logger_text").Parse(`
{{ define "item" }}{{ .Method }} {{ .Path }}{{ if .Query }}?{{ .Query }}{{ end }} {{ .Code }} {{ .Bytes }} {{ .ContentType }}{{ end }}
`))

func (al *accessLogger) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	if !al.watcher.Active() {
		al.handler.ServeHTTP(w, r)
		return
	}

	rec := httptest.NewRecorder()
	al.handler.ServeHTTP(rec, r)
	bytes := rec.Body.Len()

	// finishing work first...
	hdr := w.Header()
	for key, vals := range rec.HeaderMap {
		hdr[key] = vals
	}
	w.WriteHeader(rec.Code)
	if _, err := rec.Body.WriteTo(w); err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
	}

	// ...then emitting the entry may afford slightly less overhead; most
	// overhead, like marshalling, should be deferred by the gwr library,
	// watcher.HandleItem is supposed to be fast enough to not need a channel
	// indirection within each source.
	al.watcher.HandleItem(accessEntry{
		Method:      r.Method,
		Path:        r.URL.Path,
		Query:       r.URL.RawQuery,
		Code:        rec.Code,
		Bytes:       bytes,
		ContentType: rec.HeaderMap.Get("Content-Type"),
	})
}

func (al *accessLogger) Name() string {
	return "/access_log"
}

func (al *accessLogger) TextTemplate() *template.Template {
	return accessLogTextTemplate
}

func (al *accessLogger) SetWatcher(watcher source.GenericDataWatcher) {
	al.watcher = watcher
}

// TODO: this has become more test than example; maybe just make it a test?

func main() {
	// Uses :0 for no conflict in test.
	if err := gwr.Configure(&gwr.Config{ListenAddr: ":0"}); err != nil {
		log.Fatal(err)
	}
	defer gwr.DefaultServer().Stop()
	gwrAddr := gwr.DefaultServer().Addr()

	// a handler so we get more than just 404s
	http.HandleFunc("/foo", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "text/plain; charset=utf-8")
		if _, err := io.WriteString(w, "Ok ;-)\n"); err != nil {
			panic(err.Error())
		}
	})

	// wraps the default serve mux in an access logging gwr source...
	loggedHTTPHandler := logged(nil)

	// ...which we then register with gwr
	gwr.AddGenericDataSource(loggedHTTPHandler)

	// Again note the :0 pattern complicates things more than normal; this is
	// just the default http server
	ln, err := net.Listen("tcp", ":0")
	if err != nil {
		log.Fatal(err)
	}
	svcAddr := ln.Addr()
	go http.Serve(ln, loggedHTTPHandler)

	// make two requests, one should get a 200, the other a 404; we should see
	// only their output since no watchers are going yet
	fmt.Println("start")
	httpGetStdout("http://%s/foo?num=1", svcAddr)
	httpGetStdout("http://%s/bar?num=2", svcAddr)

	// start a json watch on the access log; NOTE we don't care about any copy
	// error because normal termination here is closed-mid-read; TODO we could
	// tighten this to log fatal any non "closed" error
	jsonLines := newHTTPGetChan("JSON", "http://%s/access_log?format=json&watch=1", gwrAddr)
	fmt.Println("\nwatching json")

	// make two requests, now with json watcher
	httpGetStdout("http://%s/foo?num=3", svcAddr)
	httpGetStdout("http://%s/bar?num=4", svcAddr)
	jsonLines.printOne()
	jsonLines.printOne()

	// start a text watch on the access log
	textLines := newHTTPGetChan("TEXT", "http://%s/access_log?format=text&watch=1", gwrAddr)
	fmt.Println("\nwatching text & json")

	// make two requests, now with both watchers
	httpGetStdout("http://%s/foo?num=5", svcAddr)
	httpGetStdout("http://%s/bar?num=6", svcAddr)
	jsonLines.printOne()
	jsonLines.printOne()
	textLines.printOne()
	textLines.printOne()

	// shutdown the json watch
	jsonLines.close()
	fmt.Println("\njust text")

	// make two requests; we should still see the body copies, but only get the text watch data
	httpGetStdout("http://%s/foo?num=7", svcAddr)
	httpGetStdout("http://%s/bar?num=8", svcAddr)
	textLines.printOne()
	textLines.printOne()

	// shutdown the json watch
	textLines.close()
	fmt.Println("\nno watchers")

	// make two requests; we should still see the body copies, but get no watch data for them
	httpGetStdout("http://%s/foo?num=9", svcAddr)
	httpGetStdout("http://%s/bar?num=10", svcAddr)

}

// test brevity conveniences

func httpGetStdout(format string, args ...interface{}) {
	url := fmt.Sprintf(format, args...)
	resp, err := http.Get(url)
	if err != nil {
		log.Fatal(err)
	}
	if _, err := io.Copy(os.Stdout, resp.Body); err != nil {
		log.Fatal(err)
	}
	if err := resp.Body.Close(); err != nil {
		log.Fatal(err)
	}
}

type httpGetChan struct {
	tag string
	c   chan string
	r   *http.Response
}

func newHTTPGetChan(tag string, format string, args ...interface{}) *httpGetChan {
	url := fmt.Sprintf(format, args...)
	resp, err := http.Get(url)
	if err != nil {
		log.Fatal(err)
	}
	hc := &httpGetChan{
		tag: tag,
		c:   make(chan string),
		r:   resp,
	}
	go hc.scanLines()
	return hc
}

func (hc *httpGetChan) printOne() {
	if _, err := fmt.Printf("%s: %s\n", hc.tag, <-hc.c); err != nil {
		log.Fatal(err)
	}
}

func (hc *httpGetChan) close() {
	if err := hc.r.Body.Close(); err != nil {
		log.Fatal(err)
	}
	hc.printOne()
}

func (hc *httpGetChan) scanLines() {
	s := bufio.NewScanner(hc.r.Body)
	for s.Scan() {
		hc.c <- s.Text()
	}
	hc.c <- "CLOSE"
	close(hc.c)
	// NOTE: s.Err() intentionally not checking since this is a "just"
	// function; in particular, we expect to get a closed-during-read error
}

Output:

start
Ok ;-)
404 page not found

watching json
Ok ;-)
404 page not found
JSON: {"method":"GET","path":"/foo","query":"num=3","code":200,"bytes":7,"content_type":"text/plain; charset=utf-8"}
JSON: {"method":"GET","path":"/bar","query":"num=4","code":404,"bytes":19,"content_type":"text/plain; charset=utf-8"}

watching text & json
Ok ;-)
404 page not found
JSON: {"method":"GET","path":"/foo","query":"num=5","code":200,"bytes":7,"content_type":"text/plain; charset=utf-8"}
JSON: {"method":"GET","path":"/bar","query":"num=6","code":404,"bytes":19,"content_type":"text/plain; charset=utf-8"}
TEXT: GET /foo?num=5 200 7 text/plain; charset=utf-8
TEXT: GET /bar?num=6 404 19 text/plain; charset=utf-8
JSON: CLOSE

just text
Ok ;-)
404 page not found
TEXT: GET /foo?num=7 200 7 text/plain; charset=utf-8
TEXT: GET /bar?num=8 404 19 text/plain; charset=utf-8
TEXT: CLOSE

no watchers
Ok ;-)
404 page not found

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// ErrAlreadyConfigured is returned by gwr.Configure when called more than
	// once.
	ErrAlreadyConfigured = errors.New("gwr already configured")

	// ErrAlreadyStarted is returned by ConfiguredServer.Start if the server is
	// already listening.
	ErrAlreadyStarted = errors.New("gwr server already started")
)
View Source 
var DefaultDataSources *source.DataSources

DefaultDataSources is default data sources registry which data sources are added to by the module-level Add* functions. It is used by all of the protocol servers if no data sources are provided.

Functions

func AddDataSource 

func AddDataSource(ds source.DataSource) error

AddDataSource adds a data source to the default data sources registry. It returns an error if there's already a data source defined with the same name.

func AddGenericDataSource 

func AddGenericDataSource(gds source.GenericDataSource) error

AddGenericDataSource adds a generic data source to the default data sources registry. It returns an error if there's already a data source defined with the same name.

func Configure 

func Configure(config *Config) error

Configure sets up the gwr library and starts any resources (like a listening server) if enabled. - if nil config is passed, it's a convenience for &gwr.Config{} - if called more than once, ErrAlreadyConfigured is returned - otherwise any ConfiguredServer.Start error is returned.

func Enabled 

func Enabled() bool

Enabled returns true if the gwr library is configured and enabled.

func ListenAndServe 

func ListenAndServe(hostPort string, dss *source.DataSources) error

ListenAndServe starts an "auto" protocol server that will respond to HTTP or RESP on the given hostPort.

func ListenAndServeHTTP 

func ListenAndServeHTTP(hostPort string, dss *source.DataSources) error

ListenAndServeHTTP starts an http protocol gwr server.

func ListenAndServeResp 

func ListenAndServeResp(hostPort string, dss *source.DataSources) error

ListenAndServeResp starts a resp protocol gwr server.

func NewServer 

func NewServer(dss *source.DataSources) stacked.Server

NewServer creates an "auto" protocol server that will respond to HTTP or RESP requests.

Types

type Config 

type Config struct {
	// Enabled controls whether GWR is enabled or not, it defaults true.
	// Currently this only controls whether ConfiguredServer starting.
	Enabled *bool `yaml:"enabled"`

	// ListenAddr controls what address ConfiguredServer will listen on.  It is
	// superceded by the $GWR_LISTEN environment variable.
	//
	// If no listen address is set, then GWR does not start its own listening
	// server; however GWR can still be accessed under "/gwr/..." from any
	// default http servers.
	ListenAddr string `yaml:"listen"`
}

Config defines configuration for GWR. For now this only defines server configuration; however once we have reporting support we'll add something ReportingCofiguration here.

type ConfiguredServer 

type ConfiguredServer struct {
	// contains filtered or unexported fields
}

ConfiguredServer manages the lifecycle of a configured GWR server, as created by gwr.NewServer.

func DefaultServer 

func DefaultServer() *ConfiguredServer

DefaultServer returns the configured gwr server, or nil if Configure hasn't been called yet.

func NewConfiguredServer 

func NewConfiguredServer(cfg Config) *ConfiguredServer

NewConfiguredServer creates a new ConfiguredServer for a given config.

func (*ConfiguredServer) Addr 

func (srv *ConfiguredServer) Addr() net.Addr

Addr returns the current listening address, if any.

func (*ConfiguredServer) Enabled 

func (srv *ConfiguredServer) Enabled() bool

Enabled return true if the server is enabled.

func (*ConfiguredServer) ListenAddr 

func (srv *ConfiguredServer) ListenAddr() string

ListenAddr returns the configured listen address string.

func (*ConfiguredServer) Start 

func (srv *ConfiguredServer) Start() error

Start starts the server by creating the listener and a server goroutine to accept connections.

  • if not enabled, or if no listen address is configured, noops and returns nil
  • if already listening, returns ErrAlreadyStarted
  • otherwise any net.Listen error is returned.

func (*ConfiguredServer) StartOn 

func (srv *ConfiguredServer) StartOn(laddr string) error

StartOn starts the server on a given listening address. If the start succeeds, it also updates the configured listening address for later reference. It has all the same error cases as ConfiguredServer.Start.

func (*ConfiguredServer) Stop 

func (srv *ConfiguredServer) Stop() error

Stop closes the current listener and shuts down the server goroutine started by Start (if any).

Source Files

View all Source files

Directories

Path Synopsis
marshaled
meta
protocol
resp
test
Package report implements reporting support around watchable source.DataSource.
 Package report implements reporting support around watchable source.DataSource.
tap
Package tap provides a simple item emitter source and a tracing source.
 Package tap provides a simple item emitter source and a tracing source.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.