dbprotos

README

Protobuf Schema for NoSQL databases

Define your NoSQL store (e.g. Datastore) schema in a Protobuf.

Status: Work in Progress.

Why?

Typically structs for data representations of key value storages (Google Datastore, MongoDB) rely on the data model being expressed in the code. In Go, these are usually struct field tags.

Protocol buffers provide a way of expressing datastructures that is centred around the data itself, not the implementation. The ideas is: if you're changing a database proto file, it may have significant consequences. That's why this project proposes that all database enties, including their indexes are defined in .proto files.

Example

syntax = "proto3";

message SomeProto {
  option (dbp.entity).datastore.kind = "MySomeProto";

  string UnusedField = 1;
  string SomeSingleString = 2 [
    (dbp.datastore).name = "single_string",
    (dbp.index) = { single: True, composite: [ {id: "first"}, {id: "second" } ]
  ];
  repeated string SomeMultiString = 4 [
    (dbp.datastore) = {name: "multi_string"},
    (dbp.index) = { single: True }
  ];
  google.protobuf.Timestamp SomeSingleTime = 5 [
    (dbp.datastore) = {name: "single_time"},
    (dbp.index) = { single: True, composite: [ {id: "first"} ] }
  ];
  repeated int32 SomeMultiInt = 6 [
    (dbp.datastore) = {name: "multi_int"},
  ];
  repeated google.protobuf.Timestamp SomeMultiTimes = 7 [
    (dbp.datastore) = {name: "multi_time"},
  ];
  string SomeInteger = 8 [ 
    (dbp.datastore).name = "my_integer",
    (dbp.index) = { single: True, composite: [ {id: "second", descending: True } ]
  ];
  int DeprecatedSizeParameter = 9 [ 
    (dbp.datastore).name = "size",
    (dbp.datastore).not_writeable = true,
  ];
}

In this example we see that the given proto Message is tied to a specific entity type. Each field of the message has an annotation that provides a canonical name for the field. Moreover, the field annotations control indexing: whether the given field forms an indiviually indexed column (single) or is part of a named composite object.

The typical Protobuf concepts apply: if the entity contains fields not present in the proto, they will be ignored (unless a message-level option strict_reading is used), and all non-default value (by proto3 semantics) fields will be written to the output.

Roadmap

• - working code generator for simple Datastore types and repeated fields
• - tests for compatibility between proto representation and Datastore canonical librayr
• - support for Enums and nested messages
• - grooming tools that use .proto files to assess compatibility of database tools
• - support for other Datastore types (GeoPoint, Key)
• - support for MongoDB and BSON serialization

Documentation

Overview

Package dbprotos is a generated protocol buffer package.

It is generated from these files:

dbprotos.proto

It has these top-level messages:

EntityMessageOpt
DatastoreEntityOpt
DatastoreFieldOpt
IndexFieldOpt
CompositeIndexOpt

Index

• Variables
• type CompositeIndexOpt
  • func (*CompositeIndexOpt) Descriptor() ([]byte, []int)
  • func (m *CompositeIndexOpt) GetDescending() bool
  • func (m *CompositeIndexOpt) GetId() string
  • func (*CompositeIndexOpt) ProtoMessage()
  • func (m *CompositeIndexOpt) Reset()
  • func (m *CompositeIndexOpt) String() string
• type DatastoreEntityOpt
  • func (*DatastoreEntityOpt) Descriptor() ([]byte, []int)
  • func (m *DatastoreEntityOpt) GetKind() string
  • func (m *DatastoreEntityOpt) GetStrictReading() bool
  • func (*DatastoreEntityOpt) ProtoMessage()
  • func (m *DatastoreEntityOpt) Reset()
  • func (m *DatastoreEntityOpt) String() string
• type DatastoreFieldOpt
  • func (*DatastoreFieldOpt) Descriptor() ([]byte, []int)
  • func (m *DatastoreFieldOpt) GetName() string
  • func (m *DatastoreFieldOpt) GetNotWriteable() bool
  • func (*DatastoreFieldOpt) ProtoMessage()
  • func (m *DatastoreFieldOpt) Reset()
  • func (m *DatastoreFieldOpt) String() string
• type EntityMessageOpt
  • func (*EntityMessageOpt) Descriptor() ([]byte, []int)
  • func (m *EntityMessageOpt) GetDatastore() *DatastoreEntityOpt
  • func (*EntityMessageOpt) ProtoMessage()
  • func (m *EntityMessageOpt) Reset()
  • func (m *EntityMessageOpt) String() string
• type IndexFieldOpt
  • func (*IndexFieldOpt) Descriptor() ([]byte, []int)
  • func (m *IndexFieldOpt) GetComposite() []*CompositeIndexOpt
  • func (m *IndexFieldOpt) GetSingle() bool
  • func (*IndexFieldOpt) ProtoMessage()
  • func (m *IndexFieldOpt) Reset()
  • func (m *IndexFieldOpt) String() string

Variables

var E_Datastore = &proto.ExtensionDesc{
	ExtendedType:  (*google_protobuf.FieldOptions)(nil),
	ExtensionType: (*DatastoreFieldOpt)(nil),
	Field:         1337110,
	Name:          "dbp.datastore",
	Tag:           "bytes,1337110,opt,name=datastore",
}

var E_Entity = &proto.ExtensionDesc{
	ExtendedType:  (*google_protobuf.MessageOptions)(nil),
	ExtensionType: (*EntityMessageOpt)(nil),
	Field:         1337001,
	Name:          "dbp.entity",
	Tag:           "bytes,1337001,opt,name=entity",
}

var E_Index = &proto.ExtensionDesc{
	ExtendedType:  (*google_protobuf.FieldOptions)(nil),
	ExtensionType: (*IndexFieldOpt)(nil),
	Field:         1337102,
	Name:          "dbp.index",
	Tag:           "bytes,1337102,opt,name=index",
}

var (
	UnknownFieldCallback = func(messageName string, propertyName string) {
		fmt.Printf(`YOLO callback on %v and prop %s`, messageName, propertyName)
	}
)

Types

type CompositeIndexOpt

type CompositeIndexOpt struct {
	// Id is the identifier of the composite index: other fields with the same id will make up the same composite index.
	Id *string `protobuf:"bytes,1,opt,name=id" json:"id,omitempty"`
	// Whether the order is descending (true) or ascending (false).
	Descending       *bool  `protobuf:"varint,2,opt,name=descending" json:"descending,omitempty"`
	XXX_unrecognized []byte `json:"-"`
}

func (*CompositeIndexOpt) Descriptor

func (*CompositeIndexOpt) Descriptor() ([]byte, []int)

func (*CompositeIndexOpt) GetDescending

func (m *CompositeIndexOpt) GetDescending() bool

func (*CompositeIndexOpt) GetId

func (m *CompositeIndexOpt) GetId() string

func (*CompositeIndexOpt) ProtoMessage

func (*CompositeIndexOpt) ProtoMessage()

func (*CompositeIndexOpt) Reset

func (m *CompositeIndexOpt) Reset()

func (*CompositeIndexOpt) String

func (m *CompositeIndexOpt) String() string

type DatastoreEntityOpt

type DatastoreEntityOpt struct {
	// Name of the Datastore Kind for this entity.
	Kind *string `protobuf:"bytes,1,opt,name=kind" json:"kind,omitempty"`
	// Strict reading defines whether the deserialization of Enities will fail if unknown fields are detected.
	StrictReading    *bool  `protobuf:"varint,2,opt,name=strict_reading,json=strictReading" json:"strict_reading,omitempty"`
	XXX_unrecognized []byte `json:"-"`
}

func (*DatastoreEntityOpt) Descriptor

func (*DatastoreEntityOpt) Descriptor() ([]byte, []int)

func (*DatastoreEntityOpt) GetKind

func (m *DatastoreEntityOpt) GetKind() string

func (*DatastoreEntityOpt) GetStrictReading

func (m *DatastoreEntityOpt) GetStrictReading() bool

func (*DatastoreEntityOpt) ProtoMessage

func (*DatastoreEntityOpt) ProtoMessage()

func (*DatastoreEntityOpt) Reset

func (m *DatastoreEntityOpt) Reset()

func (*DatastoreEntityOpt) String

func (m *DatastoreEntityOpt) String() string

type DatastoreFieldOpt

type DatastoreFieldOpt struct {
	// The name that we promise not to change for this field, even if the Protobuf field changes.
	Name *string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"`
	// If true, the field will not be written to Datastore, but it can be read from.
	// This can be used for backwards compatibility.
	NotWriteable     *bool  `protobuf:"varint,2,opt,name=not_writeable,json=notWriteable" json:"not_writeable,omitempty"`
	XXX_unrecognized []byte `json:"-"`
}

func (*DatastoreFieldOpt) Descriptor

func (*DatastoreFieldOpt) Descriptor() ([]byte, []int)

func (*DatastoreFieldOpt) GetName

func (m *DatastoreFieldOpt) GetName() string

func (*DatastoreFieldOpt) GetNotWriteable

func (m *DatastoreFieldOpt) GetNotWriteable() bool

func (*DatastoreFieldOpt) ProtoMessage

func (*DatastoreFieldOpt) ProtoMessage()

func (*DatastoreFieldOpt) Reset

func (m *DatastoreFieldOpt) Reset()

func (*DatastoreFieldOpt) String

func (m *DatastoreFieldOpt) String() string

type EntityMessageOpt

type EntityMessageOpt struct {
	Datastore        *DatastoreEntityOpt `protobuf:"bytes,1,opt,name=datastore" json:"datastore,omitempty"`
	XXX_unrecognized []byte              `json:"-"`
}

func (*EntityMessageOpt) Descriptor

func (*EntityMessageOpt) Descriptor() ([]byte, []int)

func (*EntityMessageOpt) GetDatastore

func (m *EntityMessageOpt) GetDatastore() *DatastoreEntityOpt

func (*EntityMessageOpt) ProtoMessage

func (*EntityMessageOpt) ProtoMessage()

func (*EntityMessageOpt) Reset

func (m *EntityMessageOpt) Reset()

func (*EntityMessageOpt) String

func (m *EntityMessageOpt) String() string

type IndexFieldOpt

type IndexFieldOpt struct {
	// If true, the field will have a single-key index.
	Single *bool `protobuf:"varint,1,opt,name=single" json:"single,omitempty"`
	// Composite indexes of the field, may be many.
	Composite        []*CompositeIndexOpt `protobuf:"bytes,2,rep,name=composite" json:"composite,omitempty"`
	XXX_unrecognized []byte               `json:"-"`
}

func (*IndexFieldOpt) Descriptor

func (*IndexFieldOpt) Descriptor() ([]byte, []int)

func (*IndexFieldOpt) GetComposite

func (m *IndexFieldOpt) GetComposite() []*CompositeIndexOpt

func (*IndexFieldOpt) GetSingle

func (m *IndexFieldOpt) GetSingle() bool

func (*IndexFieldOpt) ProtoMessage

func (*IndexFieldOpt) ProtoMessage()

func (*IndexFieldOpt) Reset

func (m *IndexFieldOpt) Reset()

func (*IndexFieldOpt) String

func (m *IndexFieldOpt) String() string