cloud

README

Google Cloud for Go

import "cloud.google.com/go"

Go packages for Google Cloud Platform services.

NOTE: These packages are under development, and may occasionally make backwards-incompatible changes.

NOTE: Github repo is a mirror of https://code.googlesource.com/gocloud.

News

November 17, 2016

Change to BigQuery: values from INTEGER columns will now be returned as int64, not int. This will avoid errors arising from large values on 32-bit systems.

November 8, 2016

New datastore feature: datastore now encodes your nested Go structs as Entity values, instead of a flattened list of the embedded struct's fields. This means that you may now have twice-nested slices, eg.

type State struct {
  Cities  []struct{
    Populations []int
  }
}

See the announcement for more details.

November 8, 2016

Breaking changes to datastore: contexts no longer hold namespaces; instead you must set a key's namespace explicitly. Also, key functions have been changed and renamed.

• The WithNamespace function has been removed. To specify a namespace in a Query, use the Query.Namespace method:

  q := datastore.NewQuery("Kind").Namespace("ns")
  

• All the fields of Key are exported. That means you can construct any Key with a struct literal:

  k := &Key{Kind: "Kind",  ID: 37, Namespace: "ns"}
  

• As a result of the above, the Key methods Kind, ID, d.Name, Parent, SetParent and Namespace have been removed.

• NewIncompleteKey has been removed, replaced by IncompleteKey. Replace

  NewIncompleteKey(ctx, kind, parent)
  

  with

  IncompleteKey(kind, parent)
  

  and if you do use namespaces, make sure you set the namespace on the returned key.

• NewKey has been removed, replaced by NameKey and IDKey. Replace

  NewKey(ctx, kind, name, 0, parent)
  NewKey(ctx, kind, "", id, parent)
  

  with

  NameKey(kind, name, parent)
  IDKey(kind, id, parent)
  

  and if you do use namespaces, make sure you set the namespace on the returned key.

• The Done variable has been removed. Replace datastore.Done with iterator.Done, from the package google.golang.org/api/iterator.

• The Client.Close method will have a return type of error. It will return the result of closing the underlying gRPC connection.

See the announcement for more details.

October 27, 2016

Breaking change to bigquery: NewGCSReference is now a function, not a method on Client.

New bigquery feature: Table.LoaderFrom now accepts a ReaderSource, enabling loading data into a table from a file or any io.Reader.

October 21, 2016

Breaking change to pubsub: removed pubsub.Done.

Use iterator.Done instead, where iterator is the package google.golang.org/api/iterator.

October 19, 2016

Breaking changes to cloud.google.com/go/bigquery:

• Client.Table and Client.OpenTable have been removed. Replace

  client.OpenTable("project", "dataset", "table")
  

  with

  client.DatasetInProject("project", "dataset").Table("table")
  

• Client.CreateTable has been removed. Replace

  client.CreateTable(ctx, "project", "dataset", "table")
  

  with

  client.DatasetInProject("project", "dataset").Table("table").Create(ctx)
  

• Dataset.ListTables have been replaced with Dataset.Tables. Replace

  tables, err := ds.ListTables(ctx)
  

  with

  it := ds.Tables(ctx)
  for {
      table, err := it.Next()
      if err == iterator.Done {
          break
      }
      if err != nil {
          // TODO: Handle error.
      }
      // TODO: use table.
  }
  

• Client.Read has been replaced with Job.Read, Table.Read and Query.Read. Replace

  it, err := client.Read(ctx, job)
  

  with

  it, err := job.Read(ctx)
  

  and similarly for reading from tables or queries.

• The iterator returned from the Read methods is now named RowIterator. Its behavior is closer to the other iterators in these libraries. It no longer supports the Schema method; see the next item. Replace

  for it.Next(ctx) {
      var vals ValueList
      if err := it.Get(&vals); err != nil {
          // TODO: Handle error.
      }
      // TODO: use vals.
  }
  if err := it.Err(); err != nil {
      // TODO: Handle error.
  }
  

  with

  for {
      var vals ValueList
      err := it.Next(&vals)
      if err == iterator.Done {
          break
      }
      if err != nil {
          // TODO: Handle error.
      }
      // TODO: use vals.
  }
  

  Instead of the RecordsPerRequest(n) option, write

  it.PageInfo().MaxSize = n
  

  Instead of the StartIndex(i) option, write

  it.StartIndex = i
  

• ValueLoader.Load now takes a Schema in addition to a slice of Values. Replace

  func (vl *myValueLoader) Load(v []bigquery.Value)
  

  with

  func (vl *myValueLoader) Load(v []bigquery.Value, s bigquery.Schema)
  

• Table.Patch is replace by Table.Update. Replace

  p := table.Patch()
  p.Description("new description")
  metadata, err := p.Apply(ctx)
  

  with

  metadata, err := table.Update(ctx, bigquery.TableMetadataToUpdate{
      Description: "new description",
  })
  

• Client.Copy is replaced by separate methods for each of its four functions. All options have been replaced by struct fields.

  • To load data from Google Cloud Storage into a table, use Table.LoaderFrom.

    Replace

    client.Copy(ctx, table, gcsRef)
    

    with

    table.LoaderFrom(gcsRef).Run(ctx)
    

    Instead of passing options to Copy, set fields on the Loader:

    loader := table.LoaderFrom(gcsRef)
    loader.WriteDisposition = bigquery.WriteTruncate
    

  • To extract data from a table into Google Cloud Storage, use Table.ExtractorTo. Set fields on the returned Extractor instead of passing options.

    Replace

    client.Copy(ctx, gcsRef, table)
    

    with

    table.ExtractorTo(gcsRef).Run(ctx)
    

  • To copy data into a table from one or more other tables, use Table.CopierFrom. Set fields on the returned Copier instead of passing options.

    Replace

    client.Copy(ctx, dstTable, srcTable)
    

    with

    dst.Table.CopierFrom(srcTable).Run(ctx)
    

  • To start a query job, create a Query and call its Run method. Set fields on the query instead of passing options.

    Replace

    client.Copy(ctx, table, query)
    

    with

    query.Run(ctx)
    

• Table.NewUploader has been renamed to Table.Uploader. Instead of options, configure an Uploader by setting its fields. Replace

  u := table.NewUploader(bigquery.UploadIgnoreUnknownValues())
  

  with

  u := table.NewUploader(bigquery.UploadIgnoreUnknownValues())
  u.IgnoreUnknownValues = true
  

Older news

Supported APIs

Google API	Status	Package
Datastore	beta	cloud.google.com/go/datastore
Storage	beta	cloud.google.com/go/storage
Pub/Sub	experimental	cloud.google.com/go/pubsub
Bigtable	beta	cloud.google.com/go/bigtable
BigQuery	experimental	cloud.google.com/go/bigquery
Logging	experimental	cloud.google.com/go/logging
Vision	experimental	cloud.google.com/go/vision
Language	experimental	cloud.google.com/go/language/apiv1beta1
Speech	experimental	cloud.google.com/go/speech/apiv1beta

Experimental status: the API is still being actively developed. As a result, it might change in backward-incompatible ways and is not recommended for production use.

Beta status: the API is largely complete, but still has outstanding features and bugs to be addressed. There may be minor backwards-incompatible changes where necessary.

Stable status: the API is mature and ready for production use. We will continue addressing bugs and feature requests.

Documentation and examples are available at https://godoc.org/cloud.google.com/go

Visit or join the google-api-go-announce group for updates on these packages.

Go Versions Supported

We support the two most recent major versions of Go. If Google App Engine uses an older version, we support that as well. You can see which versions are currently supported by looking at the lines following go: in .travis.yml.

Authorization

By default, each API will use Google Application Default Credentials for authorization credentials used in calling the API endpoints. This will allow your application to run in many environments without requiring explicit configuration.

To authorize using a JSON key file, pass option.WithServiceAccountFile to the NewClient function of the desired package. For example:

client, err := storage.NewClient(ctx, option.WithServiceAccountFile("path/to/keyfile.json"))

You can exert more control over authorization by using the golang.org/x/oauth2 package to create an oauth2.TokenSource. Then pass option.WithTokenSource to the NewClient function:

tokenSource := ...
client, err := storage.NewClient(ctx, option.WithTokenSource(tokenSource))

Cloud Datastore

• About Cloud Datastore
• Activating the API for your project
• API documentation
• Go client documentation
• Complete sample program

Example Usage

First create a datastore.Client to use throughout your application:

client, err := datastore.NewClient(ctx, "my-project-id")
if err != nil {
	log.Fatal(err)
}

Then use that client to interact with the API:

type Post struct {
	Title       string
	Body        string `datastore:",noindex"`
	PublishedAt time.Time
}
keys := []*datastore.Key{
	datastore.NewKey(ctx, "Post", "post1", 0, nil),
	datastore.NewKey(ctx, "Post", "post2", 0, nil),
}
posts := []*Post{
	{Title: "Post 1", Body: "...", PublishedAt: time.Now()},
	{Title: "Post 2", Body: "...", PublishedAt: time.Now()},
}
if _, err := client.PutMulti(ctx, keys, posts); err != nil {
	log.Fatal(err)
}

Cloud Storage

• About Cloud Storage
• API documentation
• Go client documentation
• Complete sample programs

Example Usage

First create a storage.Client to use throughout your application:

client, err := storage.NewClient(ctx)
if err != nil {
	log.Fatal(err)
}

// Read the object1 from bucket.
rc, err := client.Bucket("bucket").Object("object1").NewReader(ctx)
if err != nil {
	log.Fatal(err)
}
defer rc.Close()
body, err := ioutil.ReadAll(rc)
if err != nil {
	log.Fatal(err)
}

Cloud Pub/Sub

• About Cloud Pubsub
• API documentation
• Go client documentation
• Complete sample programs

Example Usage

First create a pubsub.Client to use throughout your application:

client, err := pubsub.NewClient(ctx, "project-id")
if err != nil {
	log.Fatal(err)
}

// Publish "hello world" on topic1.
topic := client.Topic("topic1")
msgIDs, err := topic.Publish(ctx, &pubsub.Message{
	Data: []byte("hello world"),
})
if err != nil {
	log.Fatal(err)
}

// Create an iterator to pull messages via subscription1.
it, err := client.Subscription("subscription1").Pull(ctx)
if err != nil {
	log.Println(err)
}
defer it.Stop()

// Consume N messages from the iterator.
for i := 0; i < N; i++ {
	msg, err := it.Next()
	if err == iterator.Done {
		break
	}
	if err != nil {
		log.Fatalf("Failed to retrieve message: %v", err)
	}

	fmt.Printf("Message %d: %s\n", i, msg.Data)
	msg.Done(true) // Acknowledge that we've consumed the message.
}

Cloud BigQuery

• About Cloud BigQuery
• API documentation
• Go client documentation
• Complete sample programs

Example Usage

First create a bigquery.Client to use throughout your application:

c, err := bigquery.NewClient(ctx, "my-project-ID")
if err != nil {
    // TODO: Handle error.
}

Then use that client to interact with the API:

// Construct a query.
q := c.Query(`
    SELECT year, SUM(number)
    FROM [bigquery-public-data:usa_names.usa_1910_2013]
    WHERE name = "William"
    GROUP BY year
    ORDER BY year
`)
// Execute the query.
it, err := q.Read(ctx)
if err != nil {
    // TODO: Handle error.
}
// Iterate through the results.
for {
    var values bigquery.ValueList
    err := it.Next(&values)
    if err == iterator.Done {
        break
    }
    if err != nil {
        // TODO: Handle error.
    }
    fmt.Println(values)
}

Stackdriver Logging

• About Stackdriver Logging
• API documentation
• Go client documentation
• Complete sample programs

Example Usage

First create a logging.Client to use throughout your application:

ctx := context.Background()
client, err := logging.NewClient(ctx, "my-project")
if err != nil {
    // TODO: Handle error.
}

Usually, you'll want to add log entries to a buffer to be periodically flushed (automatically and asynchronously) to the Stackdriver Logging service.

logger := client.Logger("my-log")
logger.Log(logging.Entry{Payload: "something happened!"})

Close your client before your program exits, to flush any buffered log entries.

err = client.Close()
if err != nil {
    // TODO: Handle error.
}

Contributing

Contributions are welcome. Please, see the CONTRIBUTING document for details. We're using Gerrit for our code reviews. Please don't open pull requests against this repo, new pull requests will be automatically closed.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See Contributor Code of Conduct for more information.

Documentation

Overview

Package cloud is the root of the packages used to access Google Cloud Services. See https://godoc.org/cloud.google.com/go for a full list of sub-packages.

This package documents how to authorize and authenticate the sub packages.

Example (ApplicationDefaultCredentials)

package main

import (
	"cloud.google.com/go/datastore"
	"golang.org/x/net/context"
)

func main() {
	ctx := context.Background()
	// Use Google Application Default Credentials to authorize and authenticate the client.
	// More information about Application Default Credentials and how to enable is at
	// https://developers.google.com/identity/protocols/application-default-credentials.
	//
	// This is the recommended way of authorizing and authenticating.
	//
	// Note: The example uses the datastore client, but the same steps apply to
	// the other client libraries underneath this package.
	client, err := datastore.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: handle error.
	}
	// Use the client.
	_ = client
}

Example (ServiceAccountFile)

package main

import (
	"cloud.google.com/go/datastore"
	"golang.org/x/net/context"
	"google.golang.org/api/option"
)

func main() {
	// Warning: The better way to use service accounts is to set GOOGLE_APPLICATION_CREDENTIALS
	// and use the Application Default Credentials.
	ctx := context.Background()
	// Use a JSON key file associated with a Google service account to
	// authenticate and authorize.
	// Go to https://console.developers.google.com/permissions/serviceaccounts to create
	// and download a service account key for your project.
	//
	// Note: The example uses the datastore client, but the same steps apply to
	// the other client libraries underneath this package.
	client, err := datastore.NewClient(ctx,
		"project-id",
		option.WithServiceAccountFile("/path/to/service-account-key.json"))
	if err != nil {
		// TODO: handle error.
	}
	// Use the client.
	_ = client
}