aetools

README

aetools

import "ronoaldo.gopkg.net/aetools"

The aetools package help you test and analyse Google App Engine Applications by providing a simple API to export datastore endities as JSON files as well as load them back into the Datastore.

bigquerysync

import "ronoaldo.gopkg.net/aetools/bigquerysync"

The aetools/bigquerysync package provides Datastore to Bigquery synchronization functions, allowing you to sync your data from Datastore to Bigquery, using the recomended aproach of a non-conciliated data table and a conciliated table as described in this document.

aeremote

The aetools/aeremote command is a simple CLI to interact with the Google Cloud Datastore, currently via the App Engine Remote API.

bundle

The aetools/bundle package contains a ready-to-use Google App Engine webapp providing handlers to create tables and sync the Datastore directly to Bigquery, using the aetools and aetools/bigquerysync packages.

vmproxy

The aetools/vmproxy package contains a simple utility to allow on-demand, low-traffic handlers to dispatch requests to a reusable compute engine instance.

Documentation

Overview

Package aetools helps writting, testing and analysing Google App Engine applications.

The aetools package implements a simple API to export the entity data from Datastore as a JSON stream, as well as load a JSON stream back into the Datastore. This can be used as a simple way to express state into a unit test, to backup a development environment state that can be shared with team members, or to make quick batch changes to data offline, like setting up configuration entities via Remote API.

The goal is to provide both an API and a set of executable tools that uses that API, allowing for maximum flexibility.

Load and Dump Data Format

The functions Load, LoadJSON, Dump and DumpJSON operate using JSON data that represents datastore entities. Each entity is mapped to a JSON Object, where each entity property name is an Object atribute, and each property value is the corresponding Object atribute value.

The property value is encoded using a JSON primitive, when possible. When the primitives are not sufficient to represent the property value, a JSON Object with the attributes "type" and "value" is used. The "type" attribute is a Datastore type, and value is a json-primitive serialization of that value. For instance, Blobs are encoded as a base64 JSON string, and time.Time values are encoded using the time.RFC3339 layout, also as strings.

Datastore Keys are aways encoded as a JSON Array that represents the Key Path, including ancestors, but without the application ID. This is done to allow the entity key to be more readable and to be application independent. Currently, they don't support namespaces.

Multiple properties are represented as a JSON Array of values described above. Unindexed properties are aways JSON objects with the "indexed" attribute set to false.

This format is intended to make use of the JSON types as much as possible, so an entity can be easily represented as a text file, suitable for read or SCM checkin.

The exported data format can also be used as an alternative way to export from Datastore, and then load the results right into other service, such as Google BigQuery or MongoDB.

The Web Bundle

The package aetools/bundle contains a sample webapp to help you manage and stream datastore entities into BigQuery. The bundle uses the aetools/bigquerysync functions to infer an usefull schema from datastore statistics, and sync your entity data into BigQuery.

The Remote API CLI

The command aetools/remote_api is a Remote API client that exposes the Load and Dump functions to make backup and restore of development environment state quick and easy. This tool can also help setting up Q.A. or Production apps, but should be used with care.

Index

• Constants
• Variables
• func Dump(c context.Context, w io.Writer, o *Options) error
• func DumpEntity(c context.Context, w io.Writer, keyString string, o *Options) error
• func DumpJSON(c context.Context, o *Options) (string, error)
• func EncodeEntities(entities []Entity, w io.Writer) error
• func KeyPath(k *datastore.Key) string
• func Load(c context.Context, r io.Reader, o *Options) error
• func LoadJSON(c context.Context, s string, o *Options) error
• type DumpOptions
• type Entity
  • func DecodeEntities(c context.Context, r io.Reader) ([]Entity, error)
  • func (e *Entity) Add(p datastore.Property)
  • func (e *Entity) Get(name string) interface{}
  • func (e *Entity) GetBool(name string) bool
  • func (e *Entity) GetFloat(name string) float64
  • func (e *Entity) GetInt(name string) int64
  • func (e *Entity) GetString(name string) string
  • func (e *Entity) Load(p []datastore.Property) error
  • func (e *Entity) Map() (map[string]interface{}, error)
  • func (e *Entity) MarshalJSON() ([]byte, error)
  • func (e *Entity) Save() ([]datastore.Property, error)
• type Options

Constants

const (
	// DateTimeFormat is used to store and load time.Time objects
	DateTimeFormat = time.RFC3339
)

Variables

var (
	// ErrInvalidRootElement is returned when the root element is not a valid JSON Array.
	ErrInvalidRootElement = errors.New("aetools: root object is not an array")
	// ErrInvalidElementType is retunred when the element is not a JSON Object.
	ErrInvalidElementType = errors.New("aetools: element is not a JSON object")
	// ErrInvalidPropertiesElement is returned when the field to be decoded is not valid.
	ErrInvalidPropertiesElement = errors.New("aetools: element's properties field is invalid")
	// ErrNoKeyElement is returned for an entity with missing key information.
	ErrNoKeyElement = errors.New("aetools: element's key field is not present")
	// ErrInvalidKeyElement is returned when the key is not properly encoded.
	ErrInvalidKeyElement = errors.New("aetools: element's key field is invalid")
)

var (
	// LoadSync is an aetools.Options that enforces data to sync
	// after it get loaded into the datastore.
	// This Options will cause a datastore.Get to happen
	// for each entity loaded.
	LoadSync = &Options{
		GetAfterPut: true,
	}
)

Functions

func Dump

func Dump(c context.Context, w io.Writer, o *Options) error

Dump exports entities from the context c using the specified Options o and writing the generated JSON representations to the io.Writer w. You can configure how the dump will run by using the Options parameter. If there is an error generating the output, or writting to the writer, it is returned. This method may return an error after writting bytes to w: the output is not buffered.

func DumpEntity

func DumpEntity(c context.Context, w io.Writer, keyString string, o *Options) error

DumpEntity export a single entity from the context

func DumpJSON

func DumpJSON(c context.Context, o *Options) (string, error)

DumpJSON is a convenient wrapper that captures the generated JSON from Dump in memory, and return it as a string. If Dump returns an error, an empty string and the error are returned.

func EncodeEntities

func EncodeEntities(entities []Entity, w io.Writer) error

EncodeEntities serializes the parameter into a JSON string.

func KeyPath

func KeyPath(k *datastore.Key) string

KeyPath returns a representation of a Key as a string with each value in the key path separated by a coma. The key representation has all ancestors, but has no information about namespaces or ancestors.

func Load

func Load(c context.Context, r io.Reader, o *Options) error

Load reads the JSON representation of entities from the io.Reader "r", and stores them in the Datastore using the given context.Context. The Options parameter allows you to configure how the dump will work. If there is any parsing erros, improper format, or datastore failures during the process, that error is returned and processing stops. The error may be returned after some entities were loaded: there is no parsing cache.

func LoadJSON

func LoadJSON(c context.Context, s string, o *Options) error

LoadJSON is a convenient wrapper to call Load using a JSON string in memory, wrapped by a strings.Reader. The error result from Load, if any, is returned.

Types

type DumpOptions

type DumpOptions struct {
	Options
}

DumpOptions is deprecated. Use Options instead.

type Entity

type Entity struct {
	Key        *datastore.Key
	Properties datastore.PropertyList
}

Entity is a small wrapper around datastore.PropertyList to also hold a *datastore.Key.

func DecodeEntities

func DecodeEntities(c context.Context, r io.Reader) ([]Entity, error)

DecodeEntities deserielizes the parameter from a JSON string

func (*Entity) Add

func (e *Entity) Add(p datastore.Property)

Add append p to the Properties attribute.

func (*Entity) Get

func (e *Entity) Get(name string) interface{}

Get returns the property value of the given name. If no property with that name exists, returns nil. It also returns nil if the property Value attribute is nil.

func (*Entity) GetBool

func (e *Entity) GetBool(name string) bool

GetBool returns the string value of the named property, and returns the zero value (false) if the property is not found, if its value is nil, or if its type is not bool.

func (*Entity) GetFloat

func (e *Entity) GetFloat(name string) float64

GetFloat returns the string value of the named property, and returns the zero value (0.0) if the property is not found, if its value is nil or if its type is not float32 or float64.

func (*Entity) GetInt

func (e *Entity) GetInt(name string) int64

GetInt returns the int value of the named property, and returns the zero value (0) if the property is not found, if its value is nil or if its type is not int, int32 or int64.

func (*Entity) GetString

func (e *Entity) GetString(name string) string

GetString returns the string value of the named property, and returns the zero value ("") if the property is not found, if its value is nil or if its type is not string.

func (*Entity) Load

func (e *Entity) Load(p []datastore.Property) error

Load decodes all properties into Entity.Propertioes object, implementing the datastore.PropertyLoadSaver interface.

func (*Entity) Map

func (e *Entity) Map() (map[string]interface{}, error)

Map converts the entity data into a JSON compatible map. Usefull to manually encode the entity using different marshallers than the JSON built-in.

func (*Entity) MarshalJSON

func (e *Entity) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.Marshaller interface by dumping the entity key and properties.

func (*Entity) Save

func (e *Entity) Save() ([]datastore.Property, error)

Save encodes all properties from Entity.Properties object, implmenting the datastore.PropetyLoadSaver interface.

type Options

type Options struct {
	// GetAfterPut indicates if we must force the Datastore to load
	// entities to be visible for non-ancestor queries, by issuing a
	// Get by key.
	// Not used when loading.
	GetAfterPut bool

	// The size for batch operations when loading/dumping
	BatchSize int

	// Kind is used to specify the kind when dumping.
	// Not used when loading.
	Kind string

	// PrettyPrint is used to specify if the dump should beaultify the output.
	// Not used when loading.
	PrettyPrint bool
}

Options allows callees to specify parameters to the Load function.