coordinator

package
v0.0.0-...-4a11b79 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 2, 2020 License: Apache-2.0 Imports: 33 Imported by: 0

Documentation

Index

Constants

View Source 
const CurrentSchemaVersion = "3"

CurrentSchemaVersion is the current schema version of the LogStream. Changes that are not backward-compatible should update this field so migration logic and scripts can translate appropriately.

History: 

1 - Contained _Tags and _C queryable fields
2 - Removed _Tags and _C queryable fields and applied noindex to
    most fields, since query filtering is now implemented in-memory instead
    of via datastore filters.
3 - Removed all non-indexed fields which are redundant with content in
    Descriptor.
View Source 
const (
	// ProjectNamespacePrefix is the datastore namespace prefix for project
	// namespaces.
	ProjectNamespacePrefix = "luci."
)
View Source 
const (
	// RegistrationNonceTimeout is how long LogPrefix.IsRetry will consider
	// a matching nonce to be valid.
	RegistrationNonceTimeout = 15 * time.Minute
)

Variables

View Source 
var ArchivalStateKey = "_ArchivalState"

ArchivalStateKey is the name of the index key for the archival state.

View Source 
var ErrPathNotFound = grpcutil.Errf(codes.NotFound, "path not found")

ErrPathNotFound is the canonical error returned when a Log Stream Path is not found.

Functions

func CurrentProject 

func CurrentProject(ctx context.Context) string

CurrentProject returns the current project based on the currently-loaded namespace.

If there is no current namespace, or if the current namespace is not a valid project namespace, an empty string will be returned.

func CurrentProjectConfig 

func CurrentProjectConfig(ctx context.Context) (*svcconfig.ProjectConfig, error)

CurrentProjectConfig returns the project-specific configuration for the current project.

If there is no current project namespace, or if the current project has no configuration, config.ErrInvalidConfig will be returned.

func IsAdminUser 

func IsAdminUser(c context.Context) error

IsAdminUser tests whether the current user belongs to the administrative users group.

If the user is not, a MembershipError will be returned.

func IsMembershipError 

func IsMembershipError(e error) bool

IsMembershipError returns whether a given error is a membership error.

func IsProjectReader 

func IsProjectReader(c context.Context, pcfg *svcconfig.ProjectConfig) error

IsProjectReader tests whether the current user belongs to one of the project's declared reader groups.

If the user is not, a MembershipError will be returned.

func IsProjectWriter 

func IsProjectWriter(c context.Context, pcfg *svcconfig.ProjectConfig) error

IsProjectWriter tests whether the current user belongs to one of the project's declared writer groups.

If the user is not a member of any of the groups, a MembershipError will be returned.

func IsServiceUser 

func IsServiceUser(c context.Context) error

IsServiceUser tests whether the current user belongs to the backend services users group.

If the user is not, a MembershipError will be returned.

func Project 

func Project(c context.Context) string

Project returns the current project installed in the supplied Context's namespace.

This function is called with the expectation that the Context is in a namespace conforming to ProjectNamespace. If this is not the case, this method will panic.

func ProjectFromNamespace 

func ProjectFromNamespace(ns string) string

ProjectFromNamespace returns the current project installed in the supplied Context's namespace.

If the namespace does not have a project namespace prefix, this function will return an empty string.

func ProjectNamespace 

func ProjectNamespace(project string) string

ProjectNamespace returns the AppEngine namespace for a given luci-config project name.

func WithProjectNamespace 

func WithProjectNamespace(c *context.Context, project string, at NamespaceAccessType) error

WithProjectNamespace sets the current namespace to the project name.

It will return a user-facing wrapped gRPC error on failure:

  • InvalidArgument if the project name is invalid.
  • If the project exists, then
  • nil, if the user has the requested access.
  • Unauthenticated if the user does not have the requested access, but is also not authenticated. This lets them know they should try again after authenticating.
  • PermissionDenied if the user does not have the requested access.
  • PermissionDenied if the project doesn't exist.
  • Internal if an internal error occurred.

Types

type ArchivalState 

type ArchivalState int

ArchivalState describes the archival state of a LogStream. 

const (
	// NotArchived means that the stream is not archived, and that no archival has
	// been tasked.
	NotArchived ArchivalState = iota
	// ArchiveTasked is true if the log stream has an archival tasked, but has
	// not yet been archived.
	ArchiveTasked
	// ArchivedPartial means that the stream is archived, but that some log
	// entries are missing.
	ArchivedPartial
	// ArchivedComplete means that the stream is archived and all log entries are
	// present.
	ArchivedComplete
)

func (ArchivalState) Archived 

func (as ArchivalState) Archived() bool

Archived returns true if this ArchivalState implies that the log stream is archived.

type HashID 

type HashID string

HashID is a hex-encoded SHA256 hash.

func LogPrefixID 

func LogPrefixID(prefix types.StreamName) HashID

LogPrefixID returns the HashID for a specific prefix.

func LogStreamID 

func LogStreamID(path types.StreamPath) HashID

LogStreamID returns the HashID for a given log stream path.

func (*HashID) Normalize 

func (id *HashID) Normalize() error

Normalize normalizes the hash ID and verifies its integrity.

type LogPrefix 

type LogPrefix struct {
	// ID is the LogPrefix's ID. It is an encoded hash value generated from the
	// stream's Prefix field.
	ID HashID `gae:"$id"`

	// Schema is the datastore schema version for this object. This can be used
	// to facilitate schema migrations.
	//
	// The current schema is currentSchemaVersion.
	Schema string

	// Created is the time when this stream was created.
	Created time.Time `gae:",noindex"`

	// Prefix is this log stream's prefix value. Log streams with the same prefix
	// are logically grouped.
	//
	// This value should not be changed once populated, as it will invalidate the
	// HashID.
	Prefix string `gae:",noindex"`

	// Source is the (indexed) set of source strings sent by the prefix registrar.
	Source []string

	// Expiration is the time when this log prefix expires. Stream registrations
	// for this prefix will fail after this point.
	Expiration time.Time

	// Secret is the Butler secret value for this prefix. All streams within
	// the prefix share this secret value.
	//
	// This value may only be returned to LogDog services; it is not user-visible.
	Secret []byte `gae:",noindex"`

	// OpNonce is provided by the client when calling RegisterPrefix. If the
	// client provides the same nonce on a subsequent invocation of
	// RegisterPrefix, the server will respond with success instead of
	// AlreadyExists.
	//
	// This must have a length of either 0 or types.OpNonceLength.
	//
	// The nonce has a valid lifetime of RegistrationNonceTimeout after Created.
	OpNonce []byte `gae:",noindex"`
	// contains filtered or unexported fields
}

LogPrefix is a datastore model for a prefix space. All log streams sharing a prefix will have a LogPrefix entry to group under.

A LogPrefix is keyed on the hash of its Prefix property.

Prefix-scoped properties are used to control creation and modification attributes of log streams sharing the prefix.

func (*LogPrefix) IsRetry 

func (p *LogPrefix) IsRetry(c context.Context, nonce []byte) bool

IsRetry checks to see if this LogPrefix is still in the OpNonce window, and if nonce matches the one in this LogPrefix.

func (*LogPrefix) Load 

func (p *LogPrefix) Load(pmap ds.PropertyMap) error

Load implements ds.PropertyLoadSaver.

func (*LogPrefix) Save 

func (p *LogPrefix) Save(withMeta bool) (ds.PropertyMap, error)

Save implements ds.PropertyLoadSaver.

func (*LogPrefix) Validate 

func (p *LogPrefix) Validate() error

Validate evaluates the state and data contents of the LogPrefix and returns an error if it is invalid.

type LogStream 

type LogStream struct {
	// ID is the LogStream ID. It is generated from the stream's Prefix/Name
	// fields.
	ID HashID `gae:"$id"`

	// Schema is the datastore schema version for this object. This can be used
	// to facilitate schema migrations.
	//
	// The current schema is currentSchemaVersion.
	Schema string // index needed for batch conversions

	// Prefix is this log stream's prefix value. Log streams with the same prefix
	// are logically grouped.
	//
	// This value should not be changed once populated, as it will invalidate the
	// ID.
	Prefix string // index needed for Query RPC
	// Name is the unique name of this log stream within the Prefix scope.
	//
	// This value should not be changed once populated, as it will invalidate the
	// ID.
	Name string `gae:",noindex"`

	// Created is the time when this stream was created.
	Created time.Time // index needed for Query RPC

	// Purged, if true, indicates that this log stream has been marked as purged.
	// Non-administrative queries and requests for this stream will operate as
	// if this entry doesn't exist.
	Purged bool `gae:",noindex"`
	// PurgedTime is the time when this stream was purged.
	PurgedTime time.Time `gae:",noindex"`

	// ProtoVersion is the version string of the protobuf, as reported by the
	// Collector (and ultimately self-identified by the Butler).
	ProtoVersion string `gae:",noindex"`
	// Descriptor is the binary protobuf data LogStreamDescriptor.
	Descriptor []byte `gae:",noindex"`
	// contains filtered or unexported fields
}

LogStream is the primary datastore model containing information and state of an individual log stream.

func WithStream 

func WithStream(c context.Context, project string, path types.StreamPath, at NamespaceAccessType) (context.Context, *LogStream, error)

WithStream performs all ACL checks on a stream, including:

  • Project Namespace check
  • Logs purged check

This returns the stream metadata, and a new context tied to the project. The new context can be used to fetch the LogStreamState from the datastore. The returned error is a gRPC error, parsable with grpcutil.

func (*LogStream) DescriptorProto 

func (s *LogStream) DescriptorProto() (*logpb.LogStreamDescriptor, error)

DescriptorProto unmarshals a LogStreamDescriptor from the stream's Descriptor field. It will return an error if the unmarshalling fails.

func (*LogStream) Load 

func (s *LogStream) Load(pmap ds.PropertyMap) error

Load implements ds.PropertyLoadSaver.

func (*LogStream) LoadDescriptor 

func (s *LogStream) LoadDescriptor(desc *logpb.LogStreamDescriptor) error

LoadDescriptor loads the fields in the log stream descriptor into this LogStream entry. These fields are:

  • Prefix
  • Name
  • Descriptor

func (*LogStream) LogPrefix 

func (s *LogStream) LogPrefix() *LogPrefix

LogPrefix returns a keyed (but not loaded) LogPrefix struct for this LogStream's Prefix.

func (*LogStream) Path 

func (s *LogStream) Path() types.StreamPath

Path returns the LogDog path for this log stream.

func (*LogStream) PopulateState 

func (s *LogStream) PopulateState(c context.Context, lst *LogStreamState)

PopulateState populates the datastore key fields for the supplied LogStreamState, binding them to the current LogStream.

func (*LogStream) Save 

func (s *LogStream) Save(withMeta bool) (ds.PropertyMap, error)

Save implements ds.PropertyLoadSaver.

func (*LogStream) SetDSValidate 

func (s *LogStream) SetDSValidate(v bool)

SetDSValidate controls whether this LogStream is validated prior to being read from or written to datastore.

This is a testing parameter, and should NOT be used in production code.

func (*LogStream) State 

func (s *LogStream) State(c context.Context) *LogStreamState

State returns the LogStreamState keyed for this LogStream.

func (*LogStream) Validate 

func (s *LogStream) Validate() error

Validate evaluates the state and data contents of the LogStream and returns an error if it is invalid.

type LogStreamQuery 

type LogStreamQuery struct {
	// contains filtered or unexported fields
}

LogStreamQuery is a function returning `true` if the provided LogStream matches.

func NewLogStreamQuery 

func NewLogStreamQuery(pathGlob string) (*LogStreamQuery, error)

NewLogStreamQuery returns a new LogStreamQuery constrained to the prefix of `pathGlob`, and with a filter function for the stream name in `pathGlob`.

By default, it will exclude purged logs.

pathGlob must have a prefix without wildcards, and a stream name portion which can include `*` or `**` in any combination.

Returns an error if the supplied pathGlob string describes an invalid query.

func (*LogStreamQuery) IncludePurged 

func (lsp *LogStreamQuery) IncludePurged()

IncludePurged will have the LogStreamQuery return purged logs as well.

func (*LogStreamQuery) MustHaveTags 

func (lsp *LogStreamQuery) MustHaveTags(tags map[string]string)

MustHaveTags constrains LogStreams returned to have all of the given tags.

func (*LogStreamQuery) OnlyContentType 

func (lsp *LogStreamQuery) OnlyContentType(ctype string)

OnlyContentType constrains the LogStreamQuery to only return LogStreams of the given content type.

func (*LogStreamQuery) OnlyPurged 

func (lsp *LogStreamQuery) OnlyPurged()

OnlyPurged will have the LogStreamQuery return ONLY purged logs.

Will result in NO logs if IncludePurged hasn't been set.

func (*LogStreamQuery) OnlyStreamType 

func (lsp *LogStreamQuery) OnlyStreamType(stype logpb.StreamType) error

OnlyStreamType constrains the LogStreamQuery to only return LogStreams of the given stream type.

func (*LogStreamQuery) Run 

func (lsp *LogStreamQuery) Run(ctx context.Context, cb func(*LogStream, ds.CursorCB) error) error

Run executes the LogStreamQuery and calls `cb` with each LogStream which matches the LogStreamQuery.

If `cb` returns ds.Stop, the query will stop with a nil error. If `cb` returns a different error, the query will stop with the returned error. If `cb` returns nil, the query continues until it exhausts.

func (*LogStreamQuery) SetCursor 

func (lsp *LogStreamQuery) SetCursor(ctx context.Context, cursor string) error

SetCursor causes the LogStreamQuery to start from the given encoded cursor.

func (*LogStreamQuery) TimeBound 

func (lsp *LogStreamQuery) TimeBound(lower, upper *timestamp.Timestamp)

TimeBound constrains LogStreams returned to be bound by the given lower and upper creation timestamps.

type LogStreamState 

type LogStreamState struct {

	// Parent is the key of the corresponding LogStream.
	Parent *ds.Key `gae:"$parent"`

	// Schema is the datastore schema version for this object. This can be used
	// to facilitate schema migrations.
	//
	// The current schema is CurrentSchemaVersion.
	Schema string `gae:",noindex"`

	// Created is the last time that this state has been created.
	Created time.Time
	// Updated is the last time that this state has been updated.
	Updated time.Time `gae:",noindex"`

	// Secret is the Butler secret value for this stream.
	//
	// This value may only be returned to LogDog services; it is not user-visible.
	Secret []byte `gae:",noindex"`

	// TerminatedTime is the Coordinator's record of when this log stream was
	// terminated.
	TerminatedTime time.Time `gae:",noindex"`
	// TerminalIndex is the index of the last log entry in the stream.
	//
	// If this is <0, the log stream is either still streaming or has been
	// archived with no log entries.
	TerminalIndex int64 `gae:",noindex"`

	// ArchiveRetryCount is the number of times this stream has attempted
	// archival.
	ArchiveRetryCount int64

	// ArchivedTime is the Coordinator's record of when this log stream was
	// archived. If this is non-zero, it means that the log entry has been
	// archived.
	ArchivedTime time.Time
	// ArchiveLogEntryCount is the number of LogEntry records that were archived
	// for this log stream.
	//
	// This is valid only if the log stream is Archived.
	ArchiveLogEntryCount int64 `gae:",noindex"`
	// ArchivalKey is the archival key for this log stream. This is used to
	// differentiate the real archival request from those that were dispatched,
	// but that ultimately failed to update state.
	//
	// See createArchivalKey for details on its generation and usage.
	ArchivalKey []byte `gae:",noindex"`

	// ArchiveIndexURL is the Google Storage URL where the log stream's index is
	// archived.
	ArchiveIndexURL string `gae:",noindex"`
	// ArchiveIndexSize is the size, in bytes, of the archived Index. It will be
	// zero if the file is not archived.
	ArchiveIndexSize int64 `gae:",noindex"`
	// ArchiveStreamURL is the Google Storage URL where the log stream's raw
	// stream data is archived. If this is not empty, the log stream is considered
	// archived.
	ArchiveStreamURL string `gae:",noindex"`
	// ArchiveStreamSize is the size, in bytes, of the archived stream. It will be
	// zero if the file is not archived.
	ArchiveStreamSize int64 `gae:",noindex"`
	// contains filtered or unexported fields
}

LogStreamState contains the current state of a LogStream.

This structure has additional datastore fields imposed by the PropertyLoadSaver.

  • _Terminated is true if the LogStream has been terminated.
  • _ArchivePending is true if the LogStream currently has an archive task dispatched.
  • _ArchivalState is true if the LogStream has been archived.

See services API's LogStreamState message type.

func NewLogStreamState 

func NewLogStreamState(c context.Context, id HashID) *LogStreamState

NewLogStreamState returns a LogStreamState with its parent key populated to the LogStream with the supplied ID.

func (*LogStreamState) ArchivalState 

func (lst *LogStreamState) ArchivalState() ArchivalState

ArchivalState returns the archival state of the log stream.

func (*LogStreamState) ID 

func (lst *LogStreamState) ID() HashID

ID returns the LogStream ID for the LogStream that owns this LogStreamState.

func (*LogStreamState) Load 

func (lst *LogStreamState) Load(pmap ds.PropertyMap) error

Load implements ds.PropertyLoadSaver.

func (*LogStreamState) Save 

func (lst *LogStreamState) Save(withMeta bool) (ds.PropertyMap, error)

Save implements ds.PropertyLoadSaver.

func (*LogStreamState) Terminated 

func (lst *LogStreamState) Terminated() bool

Terminated returns true if this stream has been terminated.

func (*LogStreamState) Validate 

func (lst *LogStreamState) Validate() error

Validate evaluates the state and data contents of the LogStreamState and returns an error if it is invalid.

type MembershipError 

type MembershipError struct {
	Identity identity.Identity
	Groups   []string
}

MembershipError is an error returned by group membership checking functions if the current identity is not a member of the requested group.

func (*MembershipError) Error 

func (e *MembershipError) Error() string

type NamespaceAccessType 

type NamespaceAccessType int

NamespaceAccessType specifies the type of namespace access that is being requested for WithProjectNamespace. 

const (
	// NamespaceAccessNoAuth grants unconditional access to a project's namespace.
	// This bypasses all ACL checks, and must only be used by service endpoints
	// that explicitly apply ACLs elsewhere.
	NamespaceAccessNoAuth NamespaceAccessType = iota

	// NamespaceAccessAllTesting is an extension of NamespaceAccessNoAuth that,
	// in addition to doing no ACL checks, also does no project existence checks.
	//
	// This must ONLY be used for testing.
	NamespaceAccessAllTesting

	// NamespaceAccessREAD enforces READ permission access to a project's
	// namespace.
	NamespaceAccessREAD

	// NamespaceAccessWRITE enforces WRITE permission access to a project's
	// namespace.
	NamespaceAccessWRITE
)

type Settings 

type Settings struct {
	// OptimisticArchivalDelay controls the delay to set for optimistic archival.
	OptimisticArchivalDelay time.Duration

	// OptimisticArchivalPercent (0-100) Controls what percent of terminated streams
	// are tasked to the new pipeline
	OptimisticArchivalPercent uint32

	// PessimisticArchivalPercent (0-100) Controls what percent of registered
	// streams are tasked to the new pipeline after 47 hours.
	PessimisticArchivalPercent uint32

	// ArchivistBatchSize controls the batchsize per lease cycle.
	ArchivistBatchSize int64

	// ArchivistLeaseTime controls the lease time per cycle.
	ArchivistLeaseTime time.Duration
}

func GetSettings 

func GetSettings(c context.Context) *Settings

GetSettings returns the current settings.

It first tries to load it from settings. If no settings is installed, or if there is no configuration in settings, defaultSettings is returned.

type SigningStorage 

type SigningStorage interface {
	// Storage is the base Storage instance.
	storage.Storage

	// GetSignedURLs attempts to sign the storage's stream's RecordIO archive
	// stream storage URL.
	//
	// If signing is not supported by this Storage instance, this will return
	// a nil signing response and no error.
	GetSignedURLs(context.Context, *URLSigningRequest) (*URLSigningResponse, error)
}

SigningStorage is an interface to storage used by the Coordinator.

type URLSigningRequest 

type URLSigningRequest struct {
	// Lifetime is the signed URL expiration time.
	Lifetime time.Duration

	// Stream, if true, requests a signed log stream URL.
	Stream bool
	// Index, if true, requests a signed log stream index URL.
	Index bool
}

URLSigningRequest is the set of URL signing parameters passed to a SigningStorage.GetSignedURLs call.

func (*URLSigningRequest) HasWork 

func (r *URLSigningRequest) HasWork() bool

HasWork returns true if this signing request actually has work that is requested.

type URLSigningResponse 

type URLSigningResponse struct {
	// Expiration is the signed URL expiration time.
	Expiration time.Time

	// Stream is the signed URL for the log stream, if requested.
	Stream string
	// Index is the signed URL for the log stream index, if requested.
	Index string
}

URLSigningResponse is the resulting signed URLs from a SigningStorage.GetSignedURLs call.

Source Files

View all Source files

Directories

Path Synopsis
registration
services
logs

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.