metadb

package
v0.1.0-beta0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 4, 2021 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BlobRef 

type BlobRef struct {
	// Key is the primary key for the blob entry
	Key uuid.UUID `datastore:"-"`
	// Size is the byte size of the blob
	Size int64
	// Status is the current status of the blob
	Status BlobRefStatus
	// StoreKey is the key of the store that the blob belongs to
	StoreKey string
	// RecordKey is the key of the record that the blob belongs to
	// It can be non-existent (e.g. deleted already) but then the Status
	// should not be Blob StatusReady.
	RecordKey string

	// Timestamps keeps track of creation and modification times and stores a randomly
	// generated UUID to maintain consistency.
	Timestamps Timestamps
}

BlobRef is a metadata document to keep track of blobs stored in an external blob store.

func NewBlobRef 

func NewBlobRef(size int64, storeKey, recordKey string) *BlobRef

NewBlobRef creates a new BlobRef as follows:

  • Set a new UUID to Key
  • Initialize Size and ObjectName as specified
  • Set Status to BlobRefStatusInitializing
  • Set current time to Timestamps (both created and updated at)

func (*BlobRef) Fail 

func (b *BlobRef) Fail() error

Fail marks the BlobRef as BlobRefStatusError and updates Timestamps. Any state can transition to BlobRefStatusError.

func (*BlobRef) Load 

func (b *BlobRef) Load(ps []datastore.Property) error

Load implements the Datastore PropertyLoadSaver interface and converts Datstore properties to the Properties field.

func (*BlobRef) LoadKey 

func (b *BlobRef) LoadKey(k *datastore.Key) error

LoadKey implements the KeyLoader interface and sets the value to the Key field.

func (*BlobRef) MarkForDeletion 

func (b *BlobRef) MarkForDeletion() error

MarkForDeletion marks the BlobRef as BlobRefStatusPendingDeletion and updates Timestamps. Returns an error if the current Status is not BlobRefStatusReady.

func (*BlobRef) ObjectPath 

func (b *BlobRef) ObjectPath() string

ObjectPath returns an object path for the backend blob storage.

func (*BlobRef) Ready 

func (b *BlobRef) Ready() error

Ready changes Status to BlobRefStatusReady and updates Timestamps. It returns an error if the current Status is not BlobRefStatusInitializing.

func (*BlobRef) Save 

func (b *BlobRef) Save() ([]datastore.Property, error)

Save implements the Datastore PropertyLoadSaver interface and converts the properties field in the struct to separate Datastore properties.

type BlobRefCursor 

type BlobRefCursor interface {
	// Next advances the iterator and returns the next value.
	// Returns nil and an iterator.Done at the end of the iterator.
	Next() (*BlobRef, error)
}

BlobRefCursor is a database cursor for BlobRef.

type BlobRefStatus 

type BlobRefStatus int16

BlobRefStatus represents a current blob status.

Life of a Blob

[New Record created] --> [new BlobRef entity with BlobRefStatusInitializing] 

                       /               \
                      / fail            \  success
                     v                   v
              [BlobRefStatusError]       [BlobRefStatusReady]
                     |       x            |
    Upload new blob  |        \ fail      | Record deleted or new blob uploaded
          or         |         \          v
   delete the record |          -----[BlobRefStatusPendingDeletion]
                     v                  /
[Delete the blob entity] <-------------/   Garbage collection

const (
	// BlobRefStatusUnknown represents internal error.
	BlobRefStatusUnknown BlobRefStatus = iota
	// BlobRefStatusInitializing means the blob is currently being prepared, i.e.
	// being uploaded to the blob store.
	BlobRefStatusInitializing
	// BlobRefStatusReady means the blob is committed and ready for use.
	BlobRefStatusReady
	// BlobRefStatusPendingDeletion means the blob is no longer referenced by
	// any Record entities and needs to be deleted.
	BlobRefStatusPendingDeletion
	// BlobRefStatusError means the blob was not uploaded due to client or server
	// errors and the corresponding record needs to be updated (either by
	// retrying blob upload or deleting the entry).
	BlobRefStatusError
)

type Driver 

type Driver interface {
	Connect(ctx context.Context) error
	Disconnect(ctx context.Context) error

	CreateStore(ctx context.Context, store *Store) (*Store, error)
	GetStore(ctx context.Context, key string) (*Store, error)
	FindStoreByName(ctx context.Context, name string) (*Store, error)
	DeleteStore(ctx context.Context, key string) error

	InsertRecord(ctx context.Context, storeKey string, record *Record) (*Record, error)
	UpdateRecord(ctx context.Context, storeKey string, key string, updater RecordUpdater) (*Record, error)
	GetRecord(ctx context.Context, storeKey, key string) (*Record, error)
	DeleteRecord(ctx context.Context, storeKey, key string) error

	InsertBlobRef(ctx context.Context, blob *BlobRef) (*BlobRef, error)
	UpdateBlobRef(ctx context.Context, blob *BlobRef) (*BlobRef, error)
	GetBlobRef(ctx context.Context, key uuid.UUID) (*BlobRef, error)
	GetCurrentBlobRef(ctx context.Context, storeKey, recordKey string) (*BlobRef, error)
	PromoteBlobRefToCurrent(ctx context.Context, blob *BlobRef) (*Record, *BlobRef, error)
	RemoveBlobFromRecord(ctx context.Context, storeKey string, recordKey string) (*Record, *BlobRef, error)
	DeleteBlobRef(ctx context.Context, key uuid.UUID) error
	ListBlobRefsByStatus(ctx context.Context, status BlobRefStatus, olderThan time.Time) (BlobRefCursor, error)

	TimestampPrecision() time.Duration
}

Driver interface defines common operations for the metadata server. MetaDB uses this interface to perform actual operations on the backend service.

type MetaDB 

type MetaDB struct {
	// contains filtered or unexported fields
}

MetaDB is a metadata database manager of Open Saves. It performs operations through the Driver interface. The methods return gRPC error codes. Here are some common error codes returned. For additional details, please look at the method help. Common errors:

  • NotFound: entity or object is not found
  • Aborted: transaction is aborted
  • InvalidArgument: key or value provided is not valid
  • Internal: internal unrecoverable error

func NewMetaDB 

func NewMetaDB(driver Driver) *MetaDB

NewMetaDB creates a new MetaDB instance with an initialized database Driver.

func (*MetaDB) Connect 

func (m *MetaDB) Connect(ctx context.Context) error

Connect initiates the database connection.

func (*MetaDB) CreateStore 

func (m *MetaDB) CreateStore(ctx context.Context, store *Store) (*Store, error)

CreateStore creates a new store.

func (*MetaDB) DeleteBlobRef 

func (m *MetaDB) DeleteBlobRef(ctx context.Context, key uuid.UUID) error

DeleteBlobRef deletes the BlobRef object from the database. Returned errors:

  • NotFound: the blobref object is not found
  • FailedPrecondition: the blobref status is Ready and can't be deleted

func (*MetaDB) DeleteRecord 

func (m *MetaDB) DeleteRecord(ctx context.Context, storeKey, key string) error

DeleteRecord deletes a record with key in store storeKey. It doesn't return error even if the key is not found in the database.

func (*MetaDB) DeleteStore 

func (m *MetaDB) DeleteStore(ctx context.Context, key string) error

DeleteStore deletes the store with specified key. Returns error if the store has any child records.

func (*MetaDB) Disconnect 

func (m *MetaDB) Disconnect(ctx context.Context) error

Disconnect terminates the database connection. Make sure to call this method to release resources (e.g. using defer). The MetaDB instance will not be available after Disconnect().

func (*MetaDB) FindStoreByName 

func (m *MetaDB) FindStoreByName(ctx context.Context, name string) (*Store, error)

FindStoreByName finds and fetch a store based on the name (complete match).

func (*MetaDB) GetBlobRef 

func (m *MetaDB) GetBlobRef(ctx context.Context, key uuid.UUID) (*BlobRef, error)

GetBlobRef returns a BlobRef object specified by the key. Returns errors:

  • NotFound: the object is not found.

func (*MetaDB) GetCurrentBlobRef 

func (m *MetaDB) GetCurrentBlobRef(ctx context.Context, storeKey, recordKey string) (*BlobRef, error)

GetCurrentBlobRef gets a BlobRef object associated with a record. Returned errors:

  • NotFound: the record is not found.
  • FailedPrecondition: the record doesn't have a blob.

func (*MetaDB) GetRecord 

func (m *MetaDB) GetRecord(ctx context.Context, storeKey, key string) (*Record, error)

GetRecord fetches and returns a record with key in store storeKey. Returns error if not found.

func (*MetaDB) GetStore 

func (m *MetaDB) GetStore(ctx context.Context, key string) (*Store, error)

GetStore fetches a store based on the key provided. Returns error if the key is not found.

func (*MetaDB) InsertBlobRef 

func (m *MetaDB) InsertBlobRef(ctx context.Context, blob *BlobRef) (*BlobRef, error)

InsertBlobRef inserts a new BlobRef object to the datastore.

func (*MetaDB) InsertRecord 

func (m *MetaDB) InsertRecord(ctx context.Context, storeKey string, record *Record) (*Record, error)

InsertRecord creates a new Record in the store specified with storeKey. Returns error if there is already a record with the same key.

func (*MetaDB) ListBlobRefsByStatus 

func (m *MetaDB) ListBlobRefsByStatus(ctx context.Context, status BlobRefStatus, olderThan time.Time) (BlobRefCursor, error)

ListBlobRefsByStatus returns a cursor that iterates over BlobRefs where Status = status and UpdatedAt < olderThan.

func (*MetaDB) PromoteBlobRefToCurrent 

func (m *MetaDB) PromoteBlobRefToCurrent(ctx context.Context, blob *BlobRef) (*Record, *BlobRef, error)

PromoteBlobRefToCurrent promotes the provided BlobRef object as a current external blob reference. Returned errors:

  • NotFound: the specified record or the blobref was not found
  • Internal: BlobRef status transition error

func (*MetaDB) RemoveBlobFromRecord 

func (m *MetaDB) RemoveBlobFromRecord(ctx context.Context, storeKey string, recordKey string) (*Record, *BlobRef, error)

RemoveBlobFromRecord removes the ExternalBlob from the record specified by storeKey and recordKey. It also changes the status of the blob object to BlobRefStatusPendingDeletion. Returned errors:

  • NotFound: the specified record or the blobref was not found
  • FailedPrecondition: the record doesn't have an external blob
  • Internal: BlobRef status transition error

func (*MetaDB) UpdateBlobRef 

func (m *MetaDB) UpdateBlobRef(ctx context.Context, blob *BlobRef) (*BlobRef, error)

UpdateBlobRef updates a BlobRef object with the new property values. Returns a NotFound error if the key is not found.

func (*MetaDB) UpdateRecord 

func (m *MetaDB) UpdateRecord(ctx context.Context, storeKey string, key string, updater RecordUpdater) (*Record, error)

UpdateRecord updates the record in the store specified with storeKey. Pass a callback function to updater and change values there. The callback will be protected by a transaction. Returns error if the store doesn't have a record with the key provided.

type PropertyMap 

type PropertyMap map[string]*PropertyValue

PropertyMap represents user-defined custom properties.

func NewPropertyMapFromProto 

func NewPropertyMapFromProto(proto map[string]*pb.Property) PropertyMap

NewPropertyMapFromProto creates a new Property instance from a proto. Passing nil returns an empty map.

func (*PropertyMap) Load 

func (m *PropertyMap) Load(ps []datastore.Property) error

Load implements the Datastore PropertyLoadSaver interface and converts individual properties to PropertyMap.

func (*PropertyMap) Save 

func (m *PropertyMap) Save() ([]datastore.Property, error)

Save implements the Datastore PropertyLoadSaver interface and converts PropertyMap to a slice of datastore Properties.

func (*PropertyMap) ToProto 

func (m *PropertyMap) ToProto() map[string]*pb.Property

ToProto converts the struct to a proto.

type PropertyValue 

type PropertyValue struct {
	Type         pb.Property_Type
	IntegerValue int64
	StringValue  string
	BooleanValue bool
}

PropertyValue is an internal representation of the user-defined property. See the Open Saves API definition for details.

func NewPropertyValueFromProto 

func NewPropertyValueFromProto(proto *pb.Property) *PropertyValue

NewPropertyValueFromProto creates a new Property instance from a proto. Passing nil returns a zero-initialized Property.

func (*PropertyValue) ToProto 

func (p *PropertyValue) ToProto() *pb.Property

ToProto converts the struct to a proto.

type Record 

type Record struct {
	Key          string `datastore:"-"`
	Blob         []byte `datastore:",noindex"`
	BlobSize     int64
	ExternalBlob uuid.UUID `datastore:"-"`
	Properties   PropertyMap
	OwnerID      string
	Tags         []string

	// Timestamps keeps track of creation and modification times and stores a randomly
	// generated UUID to maintain consistency.
	Timestamps Timestamps
}

Record represents a Open Saves record in the metadata database. See the Open Saves API definition for details.

func NewRecordFromProto 

func NewRecordFromProto(p *pb.Record) *Record

NewRecordFromProto creates a new Record instance from a proto. Passing nil returns a zero-initialized proto.

func (*Record) Load 

func (r *Record) Load(ps []datastore.Property) error

Load implements the Datastore PropertyLoadSaver interface and converts Datastore properties to corresponding struct fields.

func (*Record) LoadKey 

func (r *Record) LoadKey(k *datastore.Key) error

LoadKey implements the KeyLoader interface and sets the value to the Key field.

func (*Record) Save 

func (r *Record) Save() ([]datastore.Property, error)

Save implements the Datastore PropertyLoadSaver interface and converts struct fields to Datastore properties.

func (*Record) ToProto 

func (r *Record) ToProto() *pb.Record

ToProto converts the struct to a proto.

type RecordUpdater 

type RecordUpdater func(record *Record) (*Record, error)

RecordUpdater is a callback function for record updates. Returning a non-nil error aborts the transaction.

type Store 

type Store struct {
	Key     string `datastore:"-"`
	Name    string
	Tags    []string
	OwnerID string

	// Timestamps keeps track of creation and modification times and stores a randomly
	// generated UUID to maintain consistency.
	Timestamps Timestamps
}

Store represents a Open Saves store in the metadata database. See the Open Saves API definition for details.

func NewStoreFromProto 

func NewStoreFromProto(p *pb.Store) *Store

NewStoreFromProto creates a new Store instance from a proto. Passing nil returns a zero-initialized Store.

func (*Store) Load 

func (s *Store) Load(ps []datastore.Property) error

Load implements the Datastore PropertyLoadSaver interface and converts Datstore properties to the Properties field.

func (*Store) LoadKey 

func (s *Store) LoadKey(k *datastore.Key) error

LoadKey implements the KeyLoader interface and sets the value to the Key field.

func (*Store) Save 

func (s *Store) Save() ([]datastore.Property, error)

Save implements the Datastore PropertyLoadSaver interface and converts the properties field in the struct to separate Datastore properties.

func (*Store) ToProto 

func (s *Store) ToProto() *pb.Store

ToProto converts the structure a proto.

type Timestamps 

type Timestamps struct {
	// CreatedAt is the timestamp of the record creation time
	// Automatically set by MetaDB
	CreatedAt time.Time
	// UpdatedAt is the timestamp of the last modification time
	// Automatically set and managed by MetaDB
	UpdatedAt time.Time
	// Signature is a UUID that is randomly created each time the record is updated
	// Automatically set and managed by MetaDB
	Signature uuid.UUID `datastore:"-"`
}

Timestamps keeps keeps when each record is created or updated as well as a randomly generated UUID to keep consistency under concurrent writes. It should be embedded to metadata entities such as Record and Store.

func (*Timestamps) Load 

func (t *Timestamps) Load(ps []datastore.Property) error

Load implements the Datastore PropertyLoadSaver interface and converts Datastore properties to corresponding struct fields.

func (*Timestamps) NewTimestamps 

func (t *Timestamps) NewTimestamps(d time.Duration)

NewTimestamps sets CreatedAt and UpdatedAt to time.Now() and Signature to uuid.New().

func (*Timestamps) Save 

func (t *Timestamps) Save() ([]datastore.Property, error)

Save implements the Datastore PropertyLoadSaver interface and converts the properties field in the struct to separate Datastore properties.

func (*Timestamps) UpdateTimestamps 

func (t *Timestamps) UpdateTimestamps(d time.Duration)

UpdateTimestamps updates the UpdatedAt and Signature fields with time.Now() and uuid.New().

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.