coordinator

package
v0.0.0-...-add0f8d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 22, 2024 License: Apache-2.0 Imports: 27 Imported by: 28

Documentation

Index

Constants

View Source 
const CurrentSchemaVersion = "3"

CurrentSchemaVersion is the current schema version of the LogStream. Changes that are not backward-compatible should update this field so migration logic and scripts can translate appropriately.

History: 

1 - Contained _Tags and _C queryable fields
2 - Removed _Tags and _C queryable fields and applied noindex to
    most fields, since query filtering is now implemented in-memory instead
    of via datastore filters.
3 - Removed all non-indexed fields which are redundant with content in
    Descriptor.
View Source 
const LogStreamExpiry = 540 * 24 * time.Hour

LogStreamExpiry is the duration after creation that a LogStream record should persist for. After this duration it may be deleted.

View Source 
const LogStreamStateExpiry = 540 * 24 * time.Hour

LogStreamStateExpiry is the duration after creation that a LogStreamState record should persist for. After this duration it may be deleted.

View Source 
const (
	// ProjectNamespacePrefix is the datastore namespace prefix for project
	// namespaces.
	ProjectNamespacePrefix = "luci."
)
View Source 
const (
	// RegistrationNonceTimeout is how long LogPrefix.IsRetry will consider
	// a matching nonce to be valid.
	RegistrationNonceTimeout = 15 * time.Minute
)

Variables

View Source 
var (
	// PermLogsCreate is a permission required for RegisterPrefix RPC.
	PermLogsCreate = realms.RegisterPermission("logdog.logs.create")
	// PermLogsGet is a permission required for reading individual streams.
	PermLogsGet = realms.RegisterPermission("logdog.logs.get")
	// PermLogsList is a permission required for listing streams in a prefix.
	PermLogsList = realms.RegisterPermission("logdog.logs.list")
)
View Source 
var ArchivalStateKey = "_ArchivalState"

ArchivalStateKey is the name of the index key for the archival state.

View Source 
var ErrPathNotFound = status.Error(codes.NotFound, "path not found")

ErrPathNotFound is the canonical error returned when a Log Stream Path is not found.

Functions

func CheckAdminUser 

func CheckAdminUser(ctx context.Context) (bool, error)

CheckAdminUser tests whether the current user belongs to the administrative users group.

Logs the outcome inside. The error is non-nil only if the check itself fails.

func CheckPermission 

func CheckPermission(ctx context.Context, perm realms.Permission, prefix types.StreamName, realm string) error

CheckPermission checks the caller has the requested permission.

Logs the outcome inside (`prefix` is used only in this logging). Returns gRPC errors that can be returned directly to the caller.

func CheckServiceUser 

func CheckServiceUser(ctx context.Context) (bool, error)

CheckServiceUser tests whether the current user belongs to the backend services users group.

Logs the outcome inside. The error is non-nil only if the check itself fails.

func PermissionDeniedErr 

func PermissionDeniedErr(ctx context.Context) error

PermissionDeniedErr is a generic "doesn't exist or don't have access" error.

If the request is anonymous, it is an Unauthenticated error instead.

func Project 

func Project(ctx context.Context) string

Project returns the current project installed in the supplied Context's namespace.

This function is called with the expectation that the Context is in a namespace conforming to ProjectNamespace. If this is not the case, this method will panic.

func ProjectConfig 

func ProjectConfig(ctx context.Context) (*svcconfig.ProjectConfig, error)

ProjectConfig returns the project-specific configuration for the current project as set in WithProjectNamespace.

If there is no current project namespace, or if the current project has no configuration, config.ErrInvalidConfig will be returned.

func ProjectFromNamespace 

func ProjectFromNamespace(ns string) string

ProjectFromNamespace returns the current project installed in the supplied Context's namespace.

If the namespace does not have a project namespace prefix, this function will return an empty string.

func ProjectNamespace 

func ProjectNamespace(project string) string

ProjectNamespace returns the AppEngine namespace for a given luci-config project name.

func WithProjectNamespace 

func WithProjectNamespace(c *context.Context, project string) error

WithProjectNamespace sets the current namespace to the project name.

Checks the project exists, but doesn't do any ACL checks.

It will return a user-facing wrapped gRPC error on failure:

  • InvalidArgument if the project name is invalid.
  • PermissionDenied/Unauthenticated if the project doesn't exist.
  • Internal if an internal error occurred.

Types

type ArchivalState 

type ArchivalState int

ArchivalState describes the archival state of a LogStream. 

const (
	// NotArchived means that the stream is not archived, and that no archival has
	// been tasked.
	NotArchived ArchivalState = iota
	// ArchiveTasked is true if the log stream has an archival tasked, but has
	// not yet been archived.
	ArchiveTasked
	// ArchivedPartial means that the stream is archived, but that some log
	// entries are missing.
	ArchivedPartial
	// ArchivedComplete means that the stream is archived and all log entries are
	// present.
	ArchivedComplete
)

func (ArchivalState) Archived 

func (as ArchivalState) Archived() bool

Archived returns true if this ArchivalState implies that the log stream is archived.

type HashID 

type HashID string

HashID is a hex-encoded SHA256 hash.

func LogPrefixID 

func LogPrefixID(prefix types.StreamName) HashID

LogPrefixID returns the HashID for a specific prefix.

func LogStreamID 

func LogStreamID(path types.StreamPath) HashID

LogStreamID returns the HashID for a given log stream path.

func (*HashID) Normalize 

func (id *HashID) Normalize() error

Normalize normalizes the hash ID and verifies its integrity.

type LogPrefix 

type LogPrefix struct {
	// ID is the LogPrefix's ID. It is an encoded hash value generated from the
	// stream's Prefix field.
	ID HashID `gae:"$id"`

	// Schema is the datastore schema version for this object. This can be used
	// to facilitate schema migrations.
	//
	// The current schema is currentSchemaVersion.
	Schema string

	// Created is the time when this stream was created.
	Created time.Time `gae:",noindex"`

	// Prefix is this log stream's prefix value. Log streams with the same prefix
	// are logically grouped.
	//
	// This value should not be changed once populated, as it will invalidate the
	// HashID.
	Prefix string `gae:",noindex"`

	// Realm is a full realm name ("<project>:<realm>") with ACLs for this prefix.
	//
	// It is set in RegisterStream and can't be changed afterwards.
	Realm string

	// Source is the (indexed) set of source strings sent by the prefix registrar.
	Source []string

	// Expiration is the time when this log prefix expires. Stream registrations
	// for this prefix will fail after this point.
	Expiration time.Time

	// Secret is the Butler secret value for this prefix. All streams within
	// the prefix share this secret value.
	//
	// This value may only be returned to LogDog services; it is not user-visible.
	Secret []byte `gae:",noindex"`

	// OpNonce is provided by the client when calling RegisterPrefix. If the
	// client provides the same nonce on a subsequent invocation of
	// RegisterPrefix, the server will respond with success instead of
	// AlreadyExists.
	//
	// This must have a length of either 0 or types.OpNonceLength.
	//
	// The nonce has a valid lifetime of RegistrationNonceTimeout after Created.
	OpNonce []byte `gae:",noindex"`
	// contains filtered or unexported fields
}

LogPrefix is a datastore model for a prefix space. All log streams sharing a prefix will have a LogPrefix entry to group under.

A LogPrefix is keyed on the hash of its Prefix property.

Prefix-scoped properties are used to control creation and modification attributes of log streams sharing the prefix.

func (*LogPrefix) IsRetry 

func (p *LogPrefix) IsRetry(c context.Context, nonce []byte) bool

IsRetry checks to see if this LogPrefix is still in the OpNonce window, and if nonce matches the one in this LogPrefix.

func (*LogPrefix) Load 

func (p *LogPrefix) Load(pmap ds.PropertyMap) error

Load implements ds.PropertyLoadSaver.

func (*LogPrefix) Save 

func (p *LogPrefix) Save(withMeta bool) (ds.PropertyMap, error)

Save implements ds.PropertyLoadSaver.

func (*LogPrefix) Validate 

func (p *LogPrefix) Validate() error

Validate evaluates the state and data contents of the LogPrefix and returns an error if it is invalid.

type LogStream 

type LogStream struct {
	// ID is the LogStream ID. It is generated from the stream's Prefix/Name
	// fields.
	ID HashID `gae:"$id"`

	// Schema is the datastore schema version for this object. This can be used
	// to facilitate schema migrations.
	//
	// The current schema is currentSchemaVersion.
	Schema string // index needed for batch conversions

	// Prefix is this log stream's prefix value. Log streams with the same prefix
	// are logically grouped.
	//
	// This value should not be changed once populated, as it will invalidate the
	// ID.
	Prefix string // index needed for Query RPC
	// Name is the unique name of this log stream within the Prefix scope.
	//
	// This value should not be changed once populated, as it will invalidate the
	// ID.
	Name string `gae:",noindex"`

	// Created is the time when this stream was created.
	Created time.Time `gae:",noindex"`
	// ExpireAt is time after which the datastore entry for the stream will be deleted.
	ExpireAt time.Time `gae:",noindex"`

	// Purged, if true, indicates that this log stream has been marked as purged.
	// Non-administrative queries and requests for this stream will operate as
	// if this entry doesn't exist.
	Purged bool `gae:",noindex"`
	// PurgedTime is the time when this stream was purged.
	PurgedTime time.Time `gae:",noindex"`

	// ProtoVersion is the version string of the protobuf, as reported by the
	// Collector (and ultimately self-identified by the Butler).
	ProtoVersion string `gae:",noindex"`
	// Descriptor is the binary protobuf data LogStreamDescriptor.
	Descriptor []byte `gae:",noindex"`
	// contains filtered or unexported fields
}

LogStream is the primary datastore model containing information and state of an individual log stream.

func (*LogStream) DescriptorProto 

func (s *LogStream) DescriptorProto() (*logpb.LogStreamDescriptor, error)

DescriptorProto unmarshals a LogStreamDescriptor from the stream's Descriptor field. It will return an error if the unmarshalling fails.

func (*LogStream) Load 

func (s *LogStream) Load(pmap ds.PropertyMap) error

Load implements ds.PropertyLoadSaver.

func (*LogStream) LoadDescriptor 

func (s *LogStream) LoadDescriptor(desc *logpb.LogStreamDescriptor) error

LoadDescriptor loads the fields in the log stream descriptor into this LogStream entry. These fields are:

  • Prefix
  • Name
  • Descriptor

func (*LogStream) Path 

func (s *LogStream) Path() types.StreamPath

Path returns the LogDog path for this log stream.

func (*LogStream) PopulateState 

func (s *LogStream) PopulateState(c context.Context, lst *LogStreamState)

PopulateState populates the datastore key fields for the supplied LogStreamState, binding them to the current LogStream.

func (*LogStream) Save 

func (s *LogStream) Save(withMeta bool) (ds.PropertyMap, error)

Save implements ds.PropertyLoadSaver.

func (*LogStream) SetDSValidate 

func (s *LogStream) SetDSValidate(v bool)

SetDSValidate controls whether this LogStream is validated prior to being read from or written to datastore.

This is a testing parameter, and should NOT be used in production code.

func (*LogStream) State 

func (s *LogStream) State(c context.Context) *LogStreamState

State returns the LogStreamState keyed for this LogStream.

func (*LogStream) Validate 

func (s *LogStream) Validate() error

Validate evaluates the state and data contents of the LogStream and returns an error if it is invalid.

type LogStreamQuery 

type LogStreamQuery struct {
	Prefix types.StreamName // the prefix being queried
	// contains filtered or unexported fields
}

LogStreamQuery is a function returning `true` if the provided LogStream matches.

func NewLogStreamQuery 

func NewLogStreamQuery(pathGlob string) (*LogStreamQuery, error)

NewLogStreamQuery returns a new LogStreamQuery constrained to the prefix of `pathGlob`, and with a filter function for the stream name in `pathGlob`.

By default, it will exclude purged logs.

pathGlob must have a prefix without wildcards, and a stream name portion which can include `*` or `**` in any combination.

Returns an error if the supplied pathGlob string describes an invalid query.

func (*LogStreamQuery) IncludePurged 

func (lsp *LogStreamQuery) IncludePurged()

IncludePurged will have the LogStreamQuery return purged logs as well.

func (*LogStreamQuery) MustHaveTags 

func (lsp *LogStreamQuery) MustHaveTags(tags map[string]string)

MustHaveTags constrains LogStreams returned to have all of the given tags.

func (*LogStreamQuery) OnlyContentType 

func (lsp *LogStreamQuery) OnlyContentType(ctype string)

OnlyContentType constrains the LogStreamQuery to only return LogStreams of the given content type.

func (*LogStreamQuery) OnlyPurged 

func (lsp *LogStreamQuery) OnlyPurged()

OnlyPurged will have the LogStreamQuery return ONLY purged logs.

Will result in NO logs if IncludePurged hasn't been set.

func (*LogStreamQuery) OnlyStreamType 

func (lsp *LogStreamQuery) OnlyStreamType(stype logpb.StreamType) error

OnlyStreamType constrains the LogStreamQuery to only return LogStreams of the given stream type.

func (*LogStreamQuery) Run 

func (lsp *LogStreamQuery) Run(ctx context.Context, cb func(*LogStream, ds.CursorCB) error) error

Run executes the LogStreamQuery and calls `cb` with each LogStream which matches the LogStreamQuery.

If `cb` returns ds.Stop, the query will stop with a nil error. If `cb` returns a different error, the query will stop with the returned error. If `cb` returns nil, the query continues until it exhausts.

func (*LogStreamQuery) SetCursor 

func (lsp *LogStreamQuery) SetCursor(ctx context.Context, cursor string) error

SetCursor causes the LogStreamQuery to start from the given encoded cursor.

type LogStreamState 

type LogStreamState struct {

	// Parent is the key of the corresponding LogStream.
	Parent *ds.Key `gae:"$parent"`

	// Schema is the datastore schema version for this object. This can be used
	// to facilitate schema migrations.
	//
	// The current schema is CurrentSchemaVersion.
	Schema string `gae:",noindex"`

	// Created is the last time that this state has been created.
	Created time.Time `gae:",noindex"`
	// Updated is the last time that this state has been updated.
	Updated time.Time `gae:",noindex"`
	// ExpireAt is time after which the state will be deleted.
	ExpireAt time.Time `gae:",noindex"`

	// Secret is the Butler secret value for this stream.
	//
	// This value may only be returned to LogDog services; it is not user-visible.
	Secret []byte `gae:",noindex"`

	// TerminatedTime is the Coordinator's record of when this log stream was
	// terminated.
	TerminatedTime time.Time `gae:",noindex"`
	// TerminalIndex is the index of the last log entry in the stream.
	//
	// If this is <0, the log stream is either still streaming or has been
	// archived with no log entries.
	TerminalIndex int64 `gae:",noindex"`

	// ArchiveRetryCount is the number of times this stream has attempted
	// archival.
	ArchiveRetryCount int64

	// ArchivedTime is the Coordinator's record of when this log stream was
	// archived. If this is non-zero, it means that the log entry has been
	// archived.
	ArchivedTime time.Time `gae:",noindex"`
	// ArchiveLogEntryCount is the number of LogEntry records that were archived
	// for this log stream.
	//
	// This is valid only if the log stream is Archived.
	ArchiveLogEntryCount int64 `gae:",noindex"`
	// ArchivalKey is the archival key for this log stream. This is used to
	// differentiate the real archival request from those that were dispatched,
	// but that ultimately failed to update state.
	//
	// See createArchivalKey for details on its generation and usage.
	ArchivalKey []byte `gae:",noindex"`

	// ArchiveIndexURL is the Google Storage URL where the log stream's index is
	// archived.
	ArchiveIndexURL string `gae:",noindex"`
	// ArchiveIndexSize is the size, in bytes, of the archived Index. It will be
	// zero if the file is not archived.
	ArchiveIndexSize int64 `gae:",noindex"`
	// ArchiveStreamURL is the Google Storage URL where the log stream's raw
	// stream data is archived. If this is not empty, the log stream is considered
	// archived.
	ArchiveStreamURL string `gae:",noindex"`
	// ArchiveStreamSize is the size, in bytes, of the archived stream. It will be
	// zero if the file is not archived.
	ArchiveStreamSize int64 `gae:",noindex"`
	// contains filtered or unexported fields
}

LogStreamState contains the current state of a LogStream.

This structure has additional datastore fields imposed by the PropertyLoadSaver.

  • _Terminated is true if the LogStream has been terminated.
  • _ArchivePending is true if the LogStream currently has an archive task dispatched.
  • _ArchivalState is true if the LogStream has been archived.

See services API's LogStreamState message type.

func NewLogStreamState 

func NewLogStreamState(c context.Context, id HashID) *LogStreamState

NewLogStreamState returns a LogStreamState with its parent key populated to the LogStream with the supplied ID.

func (*LogStreamState) ArchivalState 

func (lst *LogStreamState) ArchivalState() ArchivalState

ArchivalState returns the archival state of the log stream.

func (*LogStreamState) ID 

func (lst *LogStreamState) ID() HashID

ID returns the LogStream ID for the LogStream that owns this LogStreamState.

func (*LogStreamState) Load 

func (lst *LogStreamState) Load(pmap ds.PropertyMap) error

Load implements ds.PropertyLoadSaver.

func (*LogStreamState) Save 

func (lst *LogStreamState) Save(withMeta bool) (ds.PropertyMap, error)

Save implements ds.PropertyLoadSaver.

func (*LogStreamState) Terminated 

func (lst *LogStreamState) Terminated() bool

Terminated returns true if this stream has been terminated.

func (*LogStreamState) Validate 

func (lst *LogStreamState) Validate() error

Validate evaluates the state and data contents of the LogStreamState and returns an error if it is invalid.

type MetadataFetcher 

type MetadataFetcher struct {
	Path      types.StreamPath // the log stream to fetch
	WithState bool             // if true, fetch LogStreamState as well

	Prefix *LogPrefix      // the fetched prefix
	Stream *LogStream      // the fetched stream
	State  *LogStreamState // the fetched state if WithState was true
}

MetadataFetcher fetches LogStream and LogPrefix metadata and checks ACLs.

func (*MetadataFetcher) FetchWithACLCheck 

func (f *MetadataFetcher) FetchWithACLCheck(ctx context.Context) error

FetchWithACLCheck fetches the log stream entities and checks ACLs.

Must be called within some project namespace. Returns gRPC errors, logs them inside.

type SigningStorage 

type SigningStorage interface {
	// Storage is the base Storage instance.
	storage.Storage

	// GetSignedURLs attempts to sign the storage's stream's RecordIO archive
	// stream storage URL.
	//
	// If signing is not supported by this Storage instance, this will return
	// a nil signing response and no error.
	GetSignedURLs(context.Context, *URLSigningRequest) (*URLSigningResponse, error)
}

SigningStorage is an interface to storage used by the Coordinator.

type URLSigningRequest 

type URLSigningRequest struct {
	// Lifetime is the signed URL expiration time.
	Lifetime time.Duration

	// Stream, if true, requests a signed log stream URL.
	Stream bool
	// Index, if true, requests a signed log stream index URL.
	Index bool
}

URLSigningRequest is the set of URL signing parameters passed to a SigningStorage.GetSignedURLs call.

func (*URLSigningRequest) HasWork 

func (r *URLSigningRequest) HasWork() bool

HasWork returns true if this signing request actually has work that is requested.

type URLSigningResponse 

type URLSigningResponse struct {
	// Expiration is the signed URL expiration time.
	Expiration time.Time

	// Stream is the signed URL for the log stream, if requested.
	Stream string
	// Index is the signed URL for the log stream index, if requested.
	Index string
}

URLSigningResponse is the resulting signed URLs from a SigningStorage.GetSignedURLs call.

Source Files

View all Source files

Directories

Path Synopsis
registration
services
logs

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.