cloud

README

Google Cloud Client Libraries for Go

Go packages for Google Cloud Platform services.

import "cloud.google.com/go"

To install the packages on your system, do not clone the repo. Instead use

$ go get -u cloud.google.com/go/...

NOTE: Some of these packages are under development, and may occasionally make backwards-incompatible changes.

NOTE: Github repo is a mirror of https://code.googlesource.com/gocloud.

• News
• Supported APIs
• Go Versions Supported
• Authorization
• Cloud Datastore
• Cloud Storage
• Cloud Pub/Sub
• BigQuery
• Stackdriver Logging
• Cloud Spanner

News

7 August 2018

As of November 1, the code in the repo will no longer support Go versions 1.8 and earlier. No one other than AppEngine users should be on those old versions, and AppEngine Standard and Flex will stop supporting new deployments with those versions on that date.

Changes have been moved to CHANGES.

Supported APIs

Google API	Status	Package
Asset	alpha	cloud.google.com/go/asset/v1beta
BigQuery	stable	cloud.google.com/go/bigquery
Bigtable	stable	cloud.google.com/go/bigtable
Cloudtasks	beta	cloud.google.com/go/cloudtasks/apiv2beta3
Container	stable	cloud.google.com/go/container/apiv1
ContainerAnalysis	beta	cloud.google.com/go/containeranalysis/apiv1beta1
Dataproc	stable	cloud.google.com/go/dataproc/apiv1
Datastore	stable	cloud.google.com/go/datastore
Debugger	alpha	cloud.google.com/go/debugger/apiv2
Dialogflow	alpha	cloud.google.com/go/dialogflow/apiv2
Data Loss Prevention	alpha	cloud.google.com/go/dlp/apiv2
ErrorReporting	alpha	cloud.google.com/go/errorreporting
Firestore	stable	cloud.google.com/go/firestore
IAM	stable	cloud.google.com/go/iam
KMS	stable	cloud.google.com/go/kms
Natural Language	stable	cloud.google.com/go/language/apiv1
Logging	stable	cloud.google.com/go/logging
Monitoring	alpha	cloud.google.com/go/monitoring/apiv3
OS Login	alpha	cloud.google.com/compute/docs/oslogin/rest
Pub/Sub	stable	cloud.google.com/go/pubsub
Memorystore	alpha	cloud.google.com/go/redis/apiv1
Spanner	stable	cloud.google.com/go/spanner
Speech	stable	cloud.google.com/go/speech/apiv1
Storage	stable	cloud.google.com/go/storage
Text To Speech	alpha	cloud.google.com/go/texttospeech/apiv1
Trace	alpha	cloud.google.com/go/trace/apiv2
Translation	stable	cloud.google.com/go/translate
Video Intelligence	alpha	cloud.google.com/go/videointelligence/apiv1beta1
Vision	stable	cloud.google.com/go/vision/apiv1

Alpha status: the API is still being actively developed. As a result, it might change in backward-incompatible ways and is not recommended for production use.

Beta status: the API is largely complete, but still has outstanding features and bugs to be addressed. There may be minor backwards-incompatible changes where necessary.

Stable status: the API is mature and ready for production use. We will continue addressing bugs and feature requests.

Documentation and examples are available at https://godoc.org/cloud.google.com/go

Visit or join the google-api-go-announce group for updates on these packages.

Go Versions Supported

We support the two most recent major versions of Go. If Google App Engine uses an older version, we support that as well.

Authorization

By default, each API will use Google Application Default Credentials for authorization credentials used in calling the API endpoints. This will allow your application to run in many environments without requiring explicit configuration.

client, err := storage.NewClient(ctx)

To authorize using a JSON key file, pass option.WithCredentialsFile to the NewClient function of the desired package. For example:

client, err := storage.NewClient(ctx, option.WithCredentialsFile("path/to/keyfile.json"))

You can exert more control over authorization by using the golang.org/x/oauth2 package to create an oauth2.TokenSource. Then pass option.WithTokenSource to the NewClient function: snip:# (auth-ts)

tokenSource := ...
client, err := storage.NewClient(ctx, option.WithTokenSource(tokenSource))

Cloud Datastore

• About Cloud Datastore
• Activating the API for your project
• API documentation
• Go client documentation
• Complete sample program

Example Usage

First create a datastore.Client to use throughout your application:

client, err := datastore.NewClient(ctx, "my-project-id")
if err != nil {
	log.Fatal(err)
}

Then use that client to interact with the API:

type Post struct {
	Title       string
	Body        string `datastore:",noindex"`
	PublishedAt time.Time
}
keys := []*datastore.Key{
	datastore.NameKey("Post", "post1", nil),
	datastore.NameKey("Post", "post2", nil),
}
posts := []*Post{
	{Title: "Post 1", Body: "...", PublishedAt: time.Now()},
	{Title: "Post 2", Body: "...", PublishedAt: time.Now()},
}
if _, err := client.PutMulti(ctx, keys, posts); err != nil {
	log.Fatal(err)
}

Cloud Storage

• About Cloud Storage
• API documentation
• Go client documentation
• Complete sample programs

Example Usage

First create a storage.Client to use throughout your application:

client, err := storage.NewClient(ctx)
if err != nil {
	log.Fatal(err)
}

// Read the object1 from bucket.
rc, err := client.Bucket("bucket").Object("object1").NewReader(ctx)
if err != nil {
	log.Fatal(err)
}
defer rc.Close()
body, err := ioutil.ReadAll(rc)
if err != nil {
	log.Fatal(err)
}

Cloud Pub/Sub

• About Cloud Pubsub
• API documentation
• Go client documentation
• Complete sample programs

Example Usage

First create a pubsub.Client to use throughout your application:

client, err := pubsub.NewClient(ctx, "project-id")
if err != nil {
	log.Fatal(err)
}

Then use the client to publish and subscribe:

// Publish "hello world" on topic1.
topic := client.Topic("topic1")
res := topic.Publish(ctx, &pubsub.Message{
	Data: []byte("hello world"),
})
// The publish happens asynchronously.
// Later, you can get the result from res:
...
msgID, err := res.Get(ctx)
if err != nil {
	log.Fatal(err)
}

// Use a callback to receive messages via subscription1.
sub := client.Subscription("subscription1")
err = sub.Receive(ctx, func(ctx context.Context, m *pubsub.Message) {
	fmt.Println(m.Data)
	m.Ack() // Acknowledge that we've consumed the message.
})
if err != nil {
	log.Println(err)
}

BigQuery

• About BigQuery
• API documentation
• Go client documentation
• Complete sample programs

Example Usage

First create a bigquery.Client to use throughout your application: snip:# (bq-1)

c, err := bigquery.NewClient(ctx, "my-project-ID")
if err != nil {
	// TODO: Handle error.
}

Then use that client to interact with the API: snip:# (bq-2)

// Construct a query.
q := c.Query(`
    SELECT year, SUM(number)
    FROM [bigquery-public-data:usa_names.usa_1910_2013]
    WHERE name = "William"
    GROUP BY year
    ORDER BY year
`)
// Execute the query.
it, err := q.Read(ctx)
if err != nil {
	// TODO: Handle error.
}
// Iterate through the results.
for {
	var values []bigquery.Value
	err := it.Next(&values)
	if err == iterator.Done {
		break
	}
	if err != nil {
		// TODO: Handle error.
	}
	fmt.Println(values)
}

Stackdriver Logging

• About Stackdriver Logging
• API documentation
• Go client documentation
• Complete sample programs

Example Usage

First create a logging.Client to use throughout your application: snip:# (logging-1)

ctx := context.Background()
client, err := logging.NewClient(ctx, "my-project")
if err != nil {
	// TODO: Handle error.
}

Usually, you'll want to add log entries to a buffer to be periodically flushed (automatically and asynchronously) to the Stackdriver Logging service. snip:# (logging-2)

logger := client.Logger("my-log")
logger.Log(logging.Entry{Payload: "something happened!"})

Close your client before your program exits, to flush any buffered log entries. snip:# (logging-3)

err = client.Close()
if err != nil {
	// TODO: Handle error.
}

Cloud Spanner

• About Cloud Spanner
• API documentation
• Go client documentation

Example Usage

First create a spanner.Client to use throughout your application:

client, err := spanner.NewClient(ctx, "projects/P/instances/I/databases/D")
if err != nil {
	log.Fatal(err)
}

// Simple Reads And Writes
_, err = client.Apply(ctx, []*spanner.Mutation{
	spanner.Insert("Users",
		[]string{"name", "email"},
		[]interface{}{"alice", "a@example.com"})})
if err != nil {
	log.Fatal(err)
}
row, err := client.Single().ReadRow(ctx, "Users",
	spanner.Key{"alice"}, []string{"email"})
if err != nil {
	log.Fatal(err)
}

Contributing

Contributions are welcome. Please, see the CONTRIBUTING document for details. We're using Gerrit for our code reviews. Please don't open pull requests against this repo, new pull requests will be automatically closed.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See Contributor Code of Conduct for more information.

Documentation

Overview

Package cloud is the root of the packages used to access Google Cloud Services. See https://godoc.org/cloud.google.com/go for a full list of sub-packages.

Client Options

All clients in sub-packages are configurable via client options. These options are described here: https://godoc.org/google.golang.org/api/option.

Authentication and Authorization

All the clients in sub-packages support authentication via Google Application Default Credentials (see https://cloud.google.com/docs/authentication/production), or by providing a JSON key file for a Service Account. See the authentication examples in this package for details.

Timeouts and Cancellation

By default, all requests in sub-packages will run indefinitely, retrying on transient errors when correctness allows. To set timeouts or arrange for cancellation, use contexts. See the examples for details.

Do not attempt to control the initial connection (dialing) of a service by setting a timeout on the context passed to NewClient. Dialing is non-blocking, so timeouts would be ineffective and would only interfere with credential refreshing, which uses the same context.

Connection Pooling

Connection pooling differs in clients based on their transport. Cloud clients either rely on HTTP or gRPC transports to communicate with Google Cloud.

Cloud clients that use HTTP (bigquery, compute, storage, and translate) rely on the underlying HTTP transport to cache connections for later re-use. These are cached to the default http.MaxIdleConns and http.MaxIdleConnsPerHost settings in http.DefaultTransport.

For gRPC clients (all others in this repo), connection pooling is configurable. Users of cloud client libraries may specify option.WithGRPCConnectionPool(n) as a client option to NewClient calls. This configures the underlying gRPC connections to be pooled and addressed in a round robin fashion.

Using the Libraries with Docker

Minimal docker images like Alpine lack CA certificates. This causes RPCs to appear to hang, because gRPC retries indefinitely. See https://github.com/googleapis/google-cloud-go/issues/928 for more information.

Debugging

To see gRPC logs, set the environment variable GRPC_GO_LOG_SEVERITY_LEVEL. See https://godoc.org/google.golang.org/grpc/grpclog for more information.

For HTTP logging, set the GODEBUG environment variable to "http2debug=1" or "http2debug=2".

Example (ApplicationDefaultCredentials)

Google Application Default Credentials is the recommended way to authorize and authenticate clients.

For information on how to create and obtain Application Default Credentials, see https://developers.google.com/identity/protocols/application-default-credentials.

package main

import (
	"context"

	"cloud.google.com/go/datastore"
)

func main() {
	client, err := datastore.NewClient(context.Background(), "project-id")
	if err != nil {
		// TODO: handle error.
	}
	_ = client // Use the client.
}

Example (Cancellation)

To arrange for an RPC to be canceled, use context.WithCancel.

package main

import (
	"context"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	// Do not cancel the context passed to NewClient: dialing happens asynchronously,
	// and the context is used to refresh credentials in the background.
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: handle error.
	}
	cctx, cancel := context.WithCancel(ctx)
	defer cancel() // Always call cancel.

	// TODO: Make the cancel function available to whatever might want to cancel the
	// call--perhaps a GUI button.
	if err := client.Dataset("new-dataset").Create(cctx, nil); err != nil {
		// TODO: handle error.
	}
}

Example (CredentialsFile)

You can use a file with credentials to authenticate and authorize, such as a JSON key file associated with a Google service account. Service Account keys can be created and downloaded from https://console.developers.google.com/permissions/serviceaccounts.

This example uses the Datastore client, but the same steps apply to the other client libraries underneath this package.

package main

import (
	"context"

	"cloud.google.com/go/datastore"
	"google.golang.org/api/option"
)

func main() {
	client, err := datastore.NewClient(context.Background(),
		"project-id", option.WithCredentialsFile("/path/to/service-account-key.json"))
	if err != nil {
		// TODO: handle error.
	}
	_ = client // Use the client.
}

Example (CredentialsFromJSON)

In some cases (for instance, you don't want to store secrets on disk), you can create credentials from in-memory JSON and use the WithCredentials option.

The google package in this example is at golang.org/x/oauth2/google.

This example uses the PubSub client, but the same steps apply to the other client libraries underneath this package.

package main

import (
	"context"

	"cloud.google.com/go/pubsub"
	"golang.org/x/oauth2/google"
	"google.golang.org/api/option"
)

func main() {
	ctx := context.Background()
	creds, err := google.CredentialsFromJSON(ctx, []byte("JSON creds"), pubsub.ScopePubSub)
	if err != nil {
		// TODO: handle error.
	}
	client, err := pubsub.NewClient(ctx, "project-id", option.WithCredentials(creds))
	if err != nil {
		// TODO: handle error.
	}
	_ = client // Use the client.
}

Example (Timeout)

To set a timeout for an RPC, use context.WithTimeout.

package main

import (
	"context"
	"time"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	// Do not set a timeout on the context passed to NewClient: dialing happens
	// asynchronously, and the context is used to refresh credentials in the
	// background.
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: handle error.
	}
	// Time out if it takes more than 10 seconds to create a dataset.
	tctx, cancel := context.WithTimeout(ctx, 10*time.Second)
	defer cancel() // Always call cancel.

	if err := client.Dataset("new-dataset").Create(tctx, nil); err != nil {
		// TODO: handle error.
	}
}