gcputils

README

GCP Utils

Little Go helpers for Google Cloud Platform.

This is where I experiment with Go things specifically for Google Cloud.

Logging

Why another lib? I got annoyed with trying to get Zap and others to work properly with Stack Driver (tip: they don't). On top of that, the various Google services handle logging differently which also adds to the annoying level.

So I thought I'd just make one that works with any Google Cloud services.

Here's what I found:

• If you are using GCE and writing to stdout/stderr, you probably won't/can't get it right, even if you use the Docker gcplogs logging driver. You will most likely spend way too much time trying to figure this out and you WON'T be able to do it. So don't bother.
• For GCE you must use the logging API so go grab a client and start logging that way. 
• If you are using Cloud Run, you CAN write to stdout/stderr and you CAN get it right. You just need to log in a specific JSON format. 
• If you are using Cloud Run and you write a log with severity ERROR or higher, it will automatically create an incident in error reporting. Awesome! But this is ONLY in Cloud Run. And you must include the stack trace in the message field. Yes, that's right, append a line break and your stack trace to the "message" field in order to get error reporting to pick it up.
• If you want to use Error Reporting on GCE, you have to call it explicitly.

The logging stuff in here handles all those cases.

While I was at it, I thought I'd try to make it more stdlib'ish. Here's how to use it.

This is a Loggable for treeder/gotils so you really just have to have gotils use this:

On startup:

gotils.SetLoggable(gcputils.NewLogger())

Then usage is just gotils usage:

// basic log message
gotils.L(ctx).Info().Println("hi")
// add fields
l := gotils.With("abc", 123)
// then anytime you write a log, those structured fields will be output in the proper format for Google Cloud, or human
// readable when developing locally. 
gotils.L(ctx).Error().Printf("some error: %v", err)
// To keep stack traces and have those logged to google cloud logging, just use this whenever you return an error:
return gotils.C(ctx).Errorf("some error: %w", err)

Documentation

Index

• func CredentialsAndProjectIDFromEnv(gKeyEnvVarName, projectIDEnvVarName string) ([]option.ClientOption, string, error)
• func CredentialsOptionsFromEnv(envKey string) ([]option.ClientOption, error)
• func Decrypt(ctx context.Context, kmsClient *kms.KeyManagementClient, ...) ([]byte, error)
• func Encrypt(ctx context.Context, kmsClient *kms.KeyManagementClient, ...) ([]byte, error)
• func Err(err error) error
• func Errorf(format string, v ...interface{}) error
• func GetEnvVar(name, def string) string
• func InitLogging(ctx context.Context, projectID string, opts []option.ClientOption) (io.Closer, error)
• func Print(v ...interface{})
• func Printf(format string, v ...interface{})
• func Println(v ...interface{})
• func SetComponent(s string)
• func WithTrace(ctx context.Context, r *http.Request) context.Context
• type Entry
  • func (e Entry) String() string
• type Fielder
• type GoogleJSON
  • func AccountAndCredentialsFromEnv(envKey string) (*GoogleJSON, []option.ClientOption, error)
• type Leveler
• type Line
  • func Debug() Line
  • func Error() Line
  • func F(key string, value interface{}) Line
  • func Info() Line
  • func NewLogger() Line
  • func P(sev string) Line
  • func With(key string, value interface{}) Line
• type Printer

Functions

func CredentialsAndProjectIDFromEnv

func CredentialsAndProjectIDFromEnv(gKeyEnvVarName, projectIDEnvVarName string) ([]option.ClientOption, string, error)

CredentialsAndProjectIDFromEnv returns ClientOption's with credentials filled in and the project ID ProjectID returns the project ID by looking in several places in the following order of preference: * env var passed in via envVarName * set in GCE metadata with matching name to envVarName (user defined) * instance metadata * in credentials key/json

gKeyEnvVarName is required only if not running on GCP compute projectIDEnvVarName is optional

func CredentialsOptionsFromEnv

func CredentialsOptionsFromEnv(envKey string) ([]option.ClientOption, error)

CredentialsOptionsFromEnv this will check an environment var with key you provide, which should contain your JSON credentials base64 encoded. Can passed returned value directly into clients. Run `base64 -w 0 account.json` to create this value. This also supports running on GCP, just don't set this environment variable or metadata on GCP. This will not error if it doesn't exist, so you can use this locally and let Google automatically get credentials when running on GCP.

func Decrypt

func Decrypt(ctx context.Context, kmsClient *kms.KeyManagementClient, projectID, region, keyRingName, keyName string, data []byte) ([]byte, error)

Decrypt using Google KMS to encrypt data keyName is just the last part of the default keyring we're using

func Encrypt

func Encrypt(ctx context.Context, kmsClient *kms.KeyManagementClient, projectID, region, keyRingName, keyName string, data []byte) ([]byte, error)

Encrypt using Google KMS to encrypt data keyName is just the last part of the default keyring we're using

func Err

func Err(err error) error

Err will log an error (if it hasn't already been logged) and return the same error

func Errorf

func Errorf(format string, v ...interface{}) error

Errorf will log an error (if it hasn't already been logged) and return an error as if fmt.Errorf was called

func GetEnvVar

func GetEnvVar(name, def string) string

GetEnvVar def is default, leave blank to fatal if not found checks in env, then GCE metadata

func InitLogging

func InitLogging(ctx context.Context, projectID string, opts []option.ClientOption) (io.Closer, error)

InitLogging you must call this to initialize the logging and error reporting clients. Not required if using cloud run. Call defer x.Close() on the returned closer to ensure logs get flushed.

func Print

func Print(v ...interface{})

Print take a wild guess

func Printf

func Printf(format string, v ...interface{})

Printf take a wild guess

func Println

func Println(v ...interface{})

Println take a wild guess

func SetComponent

func SetComponent(s string)

SetComponent Stackdriver Log Viewer allows filtering and display of this as `jsonPayload.component`.

func WithTrace

func WithTrace(ctx context.Context, r *http.Request) context.Context

WithTrace adds tracing info which Cloud Logging uses to correlate logs related to a particular request.WithTrace This is for use in conjuction with gotils contextual errors, whereas the other function with the same name is used for logging.

Types

type Entry

type Entry struct {
	Message  string `json:"message"`
	Severity string `json:"severity,omitempty"`
	Trace    string `json:"logging.googleapis.com/trace,omitempty"`

	// Stackdriver Log Viewer allows filtering and display of this as `jsonPayload.component`.
	Component string `json:"component,omitempty"`

	Fields map[string]interface{}
}

Entry defines a log entry.

func (Entry) String

func (e Entry) String() string

String renders an entry structure to the JSON format expected by Stackdriver.

type Fielder

type Fielder interface {
	// F adds structured key/value pairs which will show up nicely in Cloud Logging.
	// Typically use this on the same line as your Printx()
	F(string, interface{}) Line
	// With clones (unlike F), then adds structured key/value pairs which will show up nicely in Cloud Logging.
	// Use this one if you plan on passing this along to other functions or setting global fields.
	With(string, interface{}) Line

	WithTrace(r *http.Request) Line
}

Fielder methods for adding structured fields

type GoogleJSON

type GoogleJSON struct {
	ProjectID   string `json:"project_id"`
	Type        string `json:"type"`
	ClientEmail string `json:"client_email"`
}

GoogleJSON is the struct you get when you create a new service account

func AccountAndCredentialsFromEnv

func AccountAndCredentialsFromEnv(envKey string) (*GoogleJSON, []option.ClientOption, error)

AccountAndCredentialsFromEnv this will check an environment var with key you provide, which should contain your JSON credentials base64 encoded. Can passed returned value directly into clients. Run `base64 -w 0 account.json` to create this value. This also supports running on GCP, just don't set this environment variable or metadata on GCP. This will not error if it doesn't exist, so you can use this locally and let Google automatically get credentials when running on GCP.

type Leveler

type Leveler interface {
	// Debug returns a new logger with Debug severity
	Debug() Line
	// Info returns a new logger with INFO severity
	Info() Line
	// Error returns a new logger with ERROR severity
	Error() Line
}

Leveler methods to set levels on loggers

type Line

type Line interface {
	Fielder
	Printer
	Leveler
	Logf(ctx context.Context, severity, format string, a ...interface{})
	Log(ctx context.Context, severity string, a ...interface{})
}

Line is the main interface returned from most functions

func Debug

func Debug() Line

Debug returns a new logger with DEBUG severity

func Error

func Error() Line

Error returns a new logger with ERROR severity

func F

func F(key string, value interface{}) Line

F see line.F()

func Info

func Info() Line

Info returns a new logger with INFO severity

func NewLogger

func NewLogger() Line

func P

func P(sev string) Line

P returns a new logger with the provided severity

func With

func With(key string, value interface{}) Line

With returns a new logger with the fields passed in

type Printer

type Printer interface {
	Print(v ...interface{})
	Println(v ...interface{})
	Printf(format string, v ...interface{})
}

Printer common interface