model

package
v0.0.0-...-110cdec Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 22, 2024 License: Apache-2.0 Imports: 37 Imported by: 12

Documentation

Overview

Package model contains core CIPD datastore entities.

Index

Constants

View Source 
const (
	// Hidden can be used in place of 'true' when working with Package.Hidden flag.
	Hidden = true
	// Visible can be used in place of 'false' when working with Package.Hidden flag.
	Visible = false
)
View Source 
const NoInstance = "NONE"

NoInstance is used in place of empty Instance field in Event entity, so that we can query for all events that are not associated with instances and only them (these are package-level events).

Variables

View Source 
var EventsEpoch = time.Date(2018, time.January, 1, 0, 0, 0, 0, time.UTC)

EventsEpoch is used to calculate timestamps used to order entities in the datastore.

time.Time is stored with millisecond precision in the datastore. This is not enough to preserve the order of cipd.Event protos (that use nanoseconds for ordering events within a transactional group).

So instead we store (event.When-EventsEpoch).Nanos() as int64. The original timestamp is still available in cipd.Event proto. This is only for datastore index.

Functions

func AttachMetadata 

func AttachMetadata(ctx context.Context, inst *Instance, md []*api.InstanceMetadata) error

AttachMetadata transactionally attaches metadata to an instance.

Mutates `md` in place by calculating fingerprints and "guessing" content type if necessary.

Assumes inputs are already validated. Launches a transaction inside (and thus can't be a part of a transaction itself). Updates 'inst' in-place with the most recent instance state.

Returns gRPC-tagged errors: 

NotFound if there's no such instance or package.
FailedPrecondition if some processors are still running.
Aborted if some processors have failed.
Internal on fingerprint collision.

func AttachTags 

func AttachTags(ctx context.Context, inst *Instance, tags []*api.Tag) error

AttachTags transactionally attaches a bunch of tags to an instance.

Assumes inputs are already validated. Launches a transaction inside (and thus can't be a part of a transaction itself). Updates 'inst' in-place with the most recent instance state.

Returns gRPC-tagged errors: 

NotFound if there's no such instance or package.
FailedPrecondition if some processors are still running.
Aborted if some processors have failed.
Internal on tag ID collision.

func CheckInstanceExists 

func CheckInstanceExists(ctx context.Context, inst *Instance) error

CheckInstanceExists fetches the instance and verifies it exists.

Can be called as part of a transaction. Updates 'inst' in place.

Returns gRPC-tagged NotFound error if there's no such instance or package.

func CheckInstanceReady 

func CheckInstanceReady(ctx context.Context, inst *Instance) error

CheckInstanceReady fetches the instance and verifies it exists and has zero pending or failed processors.

Can be called as part of a transaction. Updates 'inst' in place.

Returns gRPC-tagged errors: 

NotFound if there's no such instance or package.
FailedPrecondition if some processors are still running.
Aborted if some processors have failed.

func CheckPackageExists 

func CheckPackageExists(ctx context.Context, pkg string) error

CheckPackageExists verifies the package exists.

Returns gRPC-tagged NotFound error if there's no such package.

func CheckPackages 

func CheckPackages(ctx context.Context, names []string, includeHidden bool) ([]string, error)

CheckPackages given a list of package names returns packages that exist, in the order they are listed in the list.

If includeHidden is false, omits hidden packages from the result.

Returns only transient errors.

func DeletePackage 

func DeletePackage(ctx context.Context, pkg string) error

DeletePackage deletes all entities associated with a package.

Deleting everything under a single transaction is generally not feasible, since deleting packages with lots of metadata exceeds transaction limits.

Instead we delete most of the stuff non-transactionally first, and then delete the rest inside the transaction (to make sure nothing stays, even if it was created while we were deleting stuff).

Returns grpc-tagged errors (in particular NotFound if there's no such package).

func DeleteRef 

func DeleteRef(ctx context.Context, pkg, ref string) error

DeleteRef removes the ref if it exists.

Does nothing if there's no such ref or package.

func DetachMetadata 

func DetachMetadata(ctx context.Context, inst *Instance, md []*api.InstanceMetadata) error

DetachMetadata detaches a bunch of metadata entries from an instance.

Assumes inputs are already validated. If Fingerprint is populated, uses it to identifies entries to detach. Otherwise calculates it from Key and Value (which must be populated in this case).

Launches a transaction inside (and thus can't be a part of a transaction itself).

func DetachTags 

func DetachTags(ctx context.Context, inst *Instance, tags []*api.Tag) error

DetachTags detaches a bunch of tags from an instance.

Assumes inputs are already validated. Launches a transaction inside (and thus can't be a part of a transaction itself).

'inst' is used only for its key.

func EmitEvent 

func EmitEvent(ctx context.Context, e *api.Event) error

EmitEvent adds a single event to the event log.

Prefer using Events to add multiple events at once.

func EmitMetadataEvents 

func EmitMetadataEvents(ctx context.Context, before, after *api.PrefixMetadata) error

EmitMetadataEvents compares two metadatum of a prefix and emits events for fields that have changed.

func FetchProcessors 

func FetchProcessors(ctx context.Context, inst *Instance) ([]*api.Processor, error)

FetchProcessors fetches results of all processors assigned to the instance and returns them as cipd.Processor proto messages (sorted by processor ID).

func IsTextContentType 

func IsTextContentType(ct string) bool

IsTextContentType checks the metadata content type against a list of known-good values.

func ListPackages 

func ListPackages(ctx context.Context, prefix string, includeHidden bool) (out []string, err error)

ListPackages returns a list of names of packages under the given prefix.

Lists all packages recursively. If there's package named as 'prefix' it is NOT included in the result. Only packaged under the prefix are included.

The result is sorted by the package name. Returns only transient errors.

func NewEventsQuery 

func NewEventsQuery() *datastore.Query

NewEventsQuery returns an unfiltered query for Event entities.

func PackageKey 

func PackageKey(ctx context.Context, pkg string) *datastore.Key

PackageKey returns a datastore key of some package, given its name.

func QueryEvents 

func QueryEvents(ctx context.Context, q *datastore.Query) ([]*api.Event, error)

QueryEvents fetches and deserializes all events matching the query, sorting them by timestamp (most recent first).

Start the query with NewEventsQuery.

func ResolveTag 

func ResolveTag(ctx context.Context, pkg string, tag *api.Tag) (string, error)

ResolveTag searches for a given tag among all instances of the package and returns an ID of the instance the tag is attached to.

Assumes inputs are already validated. Doesn't double check the instance exists.

Returns gRPC-tagged errors: 

NotFound if there's no such tag at all.
FailedPrecondition if the tag resolves to multiple instances.

func SetPackageHidden 

func SetPackageHidden(ctx context.Context, pkg string, hidden bool) error

SetPackageHidden updates Hidden field of the package.

If the package is missing returns datastore.ErrNoSuchEntity. All other errors are transient.

func SetRef 

func SetRef(ctx context.Context, ref string, inst *Instance) error

SetRef moves or creates a ref.

Assumes inputs are already validated. Launches a transaction inside (and thus can't be a part of a transaction itself). Updates 'inst' in-place with the most recent instance state.

Returns gRPC-tagged errors: 

NotFound if there's no such instance or package.
FailedPrecondition if some processors are still running.
Aborted if some processors have failed.

func TagID 

func TagID(t *api.Tag) string

TagID calculates Tag entity ID (SHA1 digest) from the given tag.

Panics if the tag is invalid.

func Txn 

func Txn(ctx context.Context, name string, cb func(context.Context) error) error

Txn runs the callback in a datastore transaction, marking commit errors with transient tag.

The given name will be used for logging and error messages.

Types

type BigQueryEventLogger 

type BigQueryEventLogger struct {
	// contains filtered or unexported fields
}

BigQueryEventLogger moves events from PubSub to BQ.

func NewBigQueryEventLogger 

func NewBigQueryEventLogger(ctx context.Context, projectID string) (*BigQueryEventLogger, error)

NewBigQueryEventLogger constructs a logger that writes to the given project.

func (*BigQueryEventLogger) HandlePubSubPush 

func (l *BigQueryEventLogger) HandlePubSubPush(ctx context.Context, body io.Reader) error

HandlePubSubPush handles incoming PubSub push messages produced by the sink.

func (*BigQueryEventLogger) RegisterSink 

func (l *BigQueryEventLogger) RegisterSink(ctx context.Context, disp *tq.Dispatcher, prod bool) context.Context

RegisterSink registers this event logger as events sink in the context.

It will use the given dispatcher to route messages between processes.

type Event 

type Event struct {
	ID      int64          `gae:"$id"`      // auto-generated
	Package *datastore.Key `gae:"$parent"`  // see PackageKey()
	Event   []byte         `gae:",noindex"` // serialized cipd.Event proto

	// Fields extracted from cipd.Event for indexing.
	Kind          api.EventKind
	Who           string
	When          int64  // nanoseconds since EventsEpoch
	Instance      string // literal 'NONE' for package events, to be able to query them
	Ref           string
	Tag           string
	MdKey         string
	MdFingerprint string
	// contains filtered or unexported fields
}

Event in a global structured event log.

It exists in both BigQuery (for adhoc queries) and in Datastore (for showing in web UI, e.g. for "recent tags" feature).

ID as auto-generated. The parent entity is the corresponding Package entity (so that events can be committed transactionally with actual changes).

func (*Event) FromProto 

func (e *Event) FromProto(ctx context.Context, p *api.Event) *Event

FromProto fills in the entity based on the proto message.

Panics if the proto can't be serialized. This should never happen.

Returns the entity itself for easier chaining.

type Events 

type Events struct {
	// contains filtered or unexported fields
}

Events collects events emitted in some transaction and then flushes them.

func (*Events) Emit 

func (t *Events) Emit(e *api.Event)

Emit adds an event to be flushed later in Flush.

'Who' and 'When' fields are populated in Flush using values from the context.

func (*Events) Flush 

func (t *Events) Flush(ctx context.Context) error

Flush sends all pending events to the datastore/bigquery.

Returned errors are tagged as transient. On success, clears the pending events queue.

type Instance 

type Instance struct {
	InstanceID string         `gae:"$id"`     // see common.ObjectRefToInstanceID()
	Package    *datastore.Key `gae:"$parent"` // see PackageKey()

	RegisteredBy string    `gae:"registered_by"` // who registered it
	RegisteredTs time.Time `gae:"registered_ts"` // when it was registered

	// Names of currently running or scheduled processors.
	ProcessorsPending []string `gae:"processors_pending"`
	// Names of processors that successfully finished the processing.
	ProcessorsSuccess []string `gae:"processors_success"`
	// Names of processors that returned fatal errors.
	ProcessorsFailure []string `gae:"processors_failure"`
	// contains filtered or unexported fields
}

Instance represents a package instance as it is stored in the datastore.

Contains some instance metadata, in particular a list of processors that scanned the instance (see below).

The parent entity is the corresponding package entity. ID is derived from package instance file digest, see common.ObjectRefToInstanceID().

Compatible with the python version of the backend.

func ListInstances 

func ListInstances(ctx context.Context, pkg string, pageSize int32, cursor datastore.Cursor) (out []*Instance, nextCur datastore.Cursor, err error)

ListInstances lists instances of a package, more recent first.

Only does a query over Instances entities. Doesn't check whether the Package entity exists. Returns up to pageSize entities, plus non-nil cursor (if there are more results). pageSize must be positive.

func RegisterInstance 

func RegisterInstance(ctx context.Context, inst *Instance, cb func(context.Context, *Instance) error) (reg bool, out *Instance, err error)

RegisterInstance transactionally registers an instance (and the corresponding package), if it isn't registered already.

Calls the given callback (inside the transaction) if it is indeed registering a new instance. The callback may mutate the instance entity before it is stored. The callback may be called multiple times in case of retries (each time it will be given a fresh instance to be mutated).

Returns (true, entity, nil) if the instance was registered just now or (false, entity, nil) if it existed before.

In either case, it returns the entity that is stored now in the datastore. It is either the new instance, or something that existed there before.

func ResolveVersion 

func ResolveVersion(ctx context.Context, pkg, version string) (*Instance, error)

ResolveVersion takes a version identifier (an instance ID, a ref or a tag) and resolves it into a concrete package instance, ensuring it exists.

Returns gRPC-tagged errors: 

InvalidArgument if the version string format is invalid.
NotFound if there's no such package or such version.
FailedPrecondition if the tag resolves to multiple instances.

func SearchInstances 

func SearchInstances(ctx context.Context, pkg string, tags []*api.Tag, pageSize int32, cursor datastore.Cursor) (out []*Instance, nextCur datastore.Cursor, err error)

SearchInstances lists instances of a package with all given tags attached.

Only does a query over Instances entities. Doesn't check whether the Package entity exists. Returns up to pageSize entities, plus non-nil cursor (if there are more results). 'pageSize' must be positive.

func (*Instance) CheckReady 

func (e *Instance) CheckReady() error

CheckReady returns an error if the instance has pending or failed processors.

func (*Instance) FromProto 

func (e *Instance) FromProto(ctx context.Context, p *api.Instance) *Instance

FromProto fills in the entity based on the proto message.

Returns the entity itself for easier chaining.

Doesn't touch output-only fields at all.

func (*Instance) Proto 

func (e *Instance) Proto() *api.Instance

Proto returns cipd.Instance proto with information from this entity.

type InstanceMetadata 

type InstanceMetadata struct {
	Fingerprint string         `gae:"$id"`     // see common.InstanceMetadataFingerprint
	Instance    *datastore.Key `gae:"$parent"` // a key of the corresponding Instance entity

	Key         string `gae:"key"`                  // the metadata key
	Value       []byte `gae:"value,noindex"`        // the metadata payload, can be big
	ContentType string `gae:"content_type,noindex"` // a content type (perhaps guessed)

	AttachedBy string    `gae:"attached_by"` // who added this metadata
	AttachedTs time.Time `gae:"attached_ts"` // when it was added
	// contains filtered or unexported fields
}

InstanceMetadata represents one instance metadata entry.

It is a key-value pair (along with some additional attributes).

The parent entity is the instance entity. ID is derived from the key-value pair, see common.InstanceMetadataFingerprint.

func ListMetadata 

func ListMetadata(ctx context.Context, inst *Instance) ([]*InstanceMetadata, error)

ListMetadata lists all instance metadata.

The result is ordered by AttachedTs (the most recent first).

func ListMetadataWithKeys 

func ListMetadataWithKeys(ctx context.Context, inst *Instance, keys []string) ([]*InstanceMetadata, error)

ListMetadataWithKeys lists instance metadata with any of the given keys.

The result is ordered by AttachedTs (the most recent first).

func (*InstanceMetadata) Proto 

func (md *InstanceMetadata) Proto() *api.InstanceMetadata

Proto returns cipd.InstanceMetadata proto with information from this entity.

Assumes the entity is valid.

type Package 

type Package struct {
	Name string `gae:"$id"` // e.g. "a/b/c"

	RegisteredBy string    `gae:"registered_by"` // who registered it
	RegisteredTs time.Time `gae:"registered_ts"` // when it was registered

	Hidden bool `gae:"hidden"` // if true, hide from the listings
	// contains filtered or unexported fields
}

Package represents a package as it is stored in the datastore.

It is mostly a marker that the package exists plus some minimal metadata about this specific package. Metadata for the package prefix is stored separately elsewhere (see 'metadata' package). Package instances, tags and refs are stored as child entities (see below).

Root entity. ID is the package name.

Compatible with the python version of the backend.

type ProcessingResult 

type ProcessingResult struct {
	ProcID   string         `gae:"$id"`     // processor that generated the result
	Instance *datastore.Key `gae:"$parent"` // instance it was generated from

	CreatedTs time.Time `gae:"created_ts"`     // when it was generated
	Success   bool      `gae:"success"`        // mostly for for indexing
	Error     string    `gae:"error,noindex"`  // for Success == false
	ResultRaw []byte    `gae:"result,noindex"` // for Success == true
	// contains filtered or unexported fields
}

ProcessingResult holds information extracted from the package instance file.

It is obtained during an asynchronous post processing step triggered after the instance is uploaded. Immutable.

Entity ID is a processor name used to extract it. Parent entity is PackageInstance the information was extracted from.

func (*ProcessingResult) Proto 

func (p *ProcessingResult) Proto() (*api.Processor, error)

Proto returns cipd.Processor proto with information from this entity.

func (*ProcessingResult) ReadResult 

func (p *ProcessingResult) ReadResult(r any) error

ReadResult deserializes the result into the given variable.

Does nothing if there's no results stored.

func (*ProcessingResult) ReadResultIntoStruct 

func (p *ProcessingResult) ReadResultIntoStruct(s *structpb.Struct) error

ReadResultIntoStruct deserializes the result into the protobuf.Struct.

Does nothing if there's no results stored.

func (*ProcessingResult) WriteResult 

func (p *ProcessingResult) WriteResult(r any) error

WriteResult overwrites ResultRaw field with compressed JSON-serialized 'r'.

'r' should serialize into a JSON object, e.g. '{...}'.

type Ref 

type Ref struct {
	Name    string         `gae:"$id"`     // e.g. "latest"
	Package *datastore.Key `gae:"$parent"` // see PackageKey()

	InstanceID string    `gae:"instance_id"` // see common.ObjectRefToInstanceID()
	ModifiedBy string    `gae:"modified_by"` // who moved it the last time
	ModifiedTs time.Time `gae:"modified_ts"` // when it was moved the last time
	// contains filtered or unexported fields
}

Ref represents a named pointer to some package instance.

ID is a ref name, the parent entity is the corresponding Package.

Compatible with the python version of the backend.

func GetRef 

func GetRef(ctx context.Context, pkg, ref string) (*Ref, error)

GetRef fetches the given ref.

Returns gRPC-tagged NotFound error if there's no such ref.

func ListInstanceRefs 

func ListInstanceRefs(ctx context.Context, inst *Instance) (out []*Ref, err error)

ListInstanceRefs returns all refs that point to a particular instance, most recently modified first.

This is a subset of the output of ListPackageRefs for the corresponding package.

Assumes 'inst' is a valid Instance, panics otherwise.

Returns an empty list if there's no such instance at all.

func ListPackageRefs 

func ListPackageRefs(ctx context.Context, pkg string) (out []*Ref, err error)

ListPackageRefs returns all refs in a package, most recently modified first.

Returns an empty list if there's no such package at all.

func (*Ref) Proto 

func (e *Ref) Proto() *api.Ref

Proto returns cipd.Ref proto with information from this entity.

type Tag 

type Tag struct {
	ID       string         `gae:"$id"`     // see TagID()
	Instance *datastore.Key `gae:"$parent"` // key of corresponding Instance entity

	Tag          string    `gae:"tag"`           // the tag itself, as "k:v" pair
	RegisteredBy string    `gae:"registered_by"` // who added this tag
	RegisteredTs time.Time `gae:"registered_ts"` // when it was added
	// contains filtered or unexported fields
}

Tag represents a "key:value" pair attached to some package instance.

Tags exist as separate entities (as opposed to being a repeated property of Instance entity) to allow essentially unlimited number of them. Also we often do not want to fetch all tags when fetching Instance entity.

ID is hex-encoded SHA1 of the tag. The parent entity is the corresponding Instance. The tag string can't be made a part of the key because the total key length (including entity kind names and all parent keys) is limited to 500 bytes and tags are allowed to be pretty long (up to 400 bytes).

There's a possibility someone tries to abuse a collision in SHA1 hash used by TagID() to attach or detach some wrong tag. The chance is minuscule, and it's not a big deal if it happens now, but it may become important in the future once tags have their own ACLs. For the sake of paranoia, we double check 'Tag' field in all entities we touch.

Compatible with the python version of the backend.

func ListInstanceTags 

func ListInstanceTags(ctx context.Context, inst *Instance) (out []*Tag, err error)

ListInstanceTags returns all tags attached to an instance, sorting them by the tag key first, and then by the timestamp (most recent first).

Returns an empty list if there's no such instance at all.

func (*Tag) Proto 

func (t *Tag) Proto() *api.Tag

Proto returns cipd.Tag proto with information from this entity.

Assumes the tag is valid.

Source Files

View all Source files

Directories

Path Synopsis
Package tasks contains task queue tasks definitions.
 Package tasks contains task queue tasks definitions.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.