firestoreevents

README

Firestore Events backend implementation for Teleport.

Introduction

This package enables Teleport auth server to store secrets in Firestore on GCP.

WARNING: Using Firestore events involves recurring charge from GCP.

Building

Firestore events is not enabled by default. To enable it you have to compile Teleport with firestore build flag.

To build Teleport with Firestore enabled, run:

ADDFLAGS='-tags firestore' make teleport

Quick Start

There are currently two Firestore mode options for any given GCP Project; Native mode and Datastore Mode. This storage backend uses Real-time updates to keep individual auth instances in sync and requires Firestore configured in Native mode.

Add this storage configuration in teleport section of the config file (by default it's /etc/teleport.yaml):

teleport:
  storage:
    audit_events_uri: 'firestore://events?projectID=gcp-proj-with-firestore-enabled&credentialsPath=/var/lib/teleport/gcs_creds'

Collections are automatically created by the Firestore APIs and the required indexes are created by the event backend on first start, if they do not exist.

Full Properties

The full list of configurable properties for this backend are:

• host portion of URI is the Firestore collection used to persist stored events
• credentialsPath (string, path to GCP creds for Firestore, not-required)
• projectID (string, project ID, required)
• purgeInterval (time duration, poll interval to sweep expired documents, not-required, defaults to once per minute)
• retryPeriod (time duration, retry period for all background tasks, not-required, defaults to 10 seconds)
• disableExpiredDocumentPurge (bool, disables expired document purging, not-required, defaults to false)
• eventRetentionPeriod (int, buffer size for watched events, not-required, defaults to 1024)
• endpoint (string, firestore client endpoint, not-required, ex: localhost:8618)

Firestore Client Authentication Options

There are three authentication/authorization modes available;

1. With no credentialsPath and no endpoint defined, the Firestore clients will use Google Application Default Credentials for authentication. This only works in cases where Teleport is installed on GCE instances and have service accounts with IAM role/profile associations authorizing that GCE instance to use Firestore.
2. With endpoint defined, Firestore will create clients no auth, GRPC in-secure, clients pointed at the specified endpoint. This is only used for tests, see Tests section below.
3. With credentialsPath defined, Firestore will create clients authenticating against live systems with the Service Account bound to the JSON key file referenced in the option.

Implementation Details

Firestore Document IDs must be unique, cannot start with periods, and cannot contain forward slashes. In order to support more straight forward fetching but work within the requirements of Firestore, Document IDs are the concatenation of the session ID (a UUID) and event type joined with a dash -, ex: 13498a42-69a8-4fa2-b39d-b0c49e346713-user.login.

Expired event purging should be enabled on as few instances as possible to reduce query costs, though there's no harm in having every instance query and purge. Purging is enabled based on the purgeExpiredDocuments property, which defaults to true. Purging is done based on the configurable eventRetentionPeriod property, which defaults to a year. Add this property to the URI to change the retention period.

Two composite indexes are required for this implementation:

1. EventNamespace ascending, then on CreatedAt ascending
2. SessionID ascending, then on EventIndex ascending

Composite indexes should be limited to the specific collection set in the configuration (in the aforementioned example is events).

Tests

Tests must execute one of two ways:

1. With gcloud installed in test infrastructure and the firestore emulator enabled and running to a dynamic port a pre-defined port used in the config. Ex: gcloud beta emulators firestore start --host-port=localhost:8618. This is where the Firestore config parameter endpoint is used.
2. With a service account pointed a test GCP project and or test collections.

Get Help

This backend has been contributed by https://github.com/joshdurbin

Documentation

Overview

Package firestoreeventsLog implements Firestore storage backend for Teleport event storage.

firestoreevents package implements the Log storage back-end for the auth server. Originally contributed by https://github.com/joshdurbin

Index

• type EventsConfig
  • func (cfg *EventsConfig) SetFromParams(params backend.Params) error
  • func (cfg *EventsConfig) SetFromURL(url *url.URL) error
• type Log
  • func New(cfg EventsConfig) (*Log, error)
  • func (l *Log) Close() error
  • func (l *Log) EmitAuditEvent(ev events.Event, fields events.EventFields) error
  • func (l *Log) GetSessionChunk(namespace string, sid session.ID, offsetBytes, maxBytes int) ([]byte, error)
  • func (l *Log) GetSessionEvents(namespace string, sid session.ID, after int, inlcudePrintEvents bool) ([]events.EventFields, error)
  • func (l *Log) PostSessionSlice(slice events.SessionSlice) error
  • func (l *Log) SearchEvents(fromUTC, toUTC time.Time, filter string, limit int) ([]events.EventFields, error)
  • func (l *Log) SearchSessionEvents(fromUTC time.Time, toUTC time.Time, limit int) ([]events.EventFields, error)
  • func (l *Log) UploadSessionRecording(events.SessionRecording) error
  • func (l *Log) WaitForDelivery(ctx context.Context) error

Types

type EventsConfig

type EventsConfig struct {
	firestorebk.Config
	// RetentionPeriod is a default retention period for events
	RetentionPeriod time.Duration
	// Clock is a clock interface, used in tests
	Clock clockwork.Clock
	// UIDGenerator is unique ID generator
	UIDGenerator utils.UID
}

Config structure represents Firestore configuration as appears in `storage` section of Teleport YAML

func (*EventsConfig) SetFromParams

func (cfg *EventsConfig) SetFromParams(params backend.Params) error

SetFromParams establishes values on an EventsConfig from the supplied params

func (*EventsConfig) SetFromURL

func (cfg *EventsConfig) SetFromURL(url *url.URL) error

SetFromURL establishes values on an EventsConfig from the supplied URI

type Log

type Log struct {
	// Entry is a log entry
	*log.Entry
	// Config is a backend configuration
	EventsConfig
	// contains filtered or unexported fields
}

Log is a firestore-db backed storage of events

func New

func New(cfg EventsConfig) (*Log, error)

New returns new instance of Firestore backend. It's an implementation of backend API's NewFunc

func (*Log) Close

func (l *Log) Close() error

Close the Firestore driver

func (*Log) EmitAuditEvent

func (l *Log) EmitAuditEvent(ev events.Event, fields events.EventFields) error

EmitAuditEvent emits audit event

func (*Log) GetSessionChunk

func (l *Log) GetSessionChunk(namespace string, sid session.ID, offsetBytes, maxBytes int) ([]byte, error)

GetSessionChunk returns a reader which can be used to read a byte stream of a recorded session starting from 'offsetBytes' (pass 0 to start from the beginning) up to maxBytes bytes.

If maxBytes > MaxChunkBytes, it gets rounded down to MaxChunkBytes

func (*Log) GetSessionEvents

func (l *Log) GetSessionEvents(namespace string, sid session.ID, after int, inlcudePrintEvents bool) ([]events.EventFields, error)

Returns all events that happen during a session sorted by time (oldest first).

after tells to use only return events after a specified cursor Id

This function is usually used in conjunction with GetSessionReader to replay recorded session streams.

func (*Log) PostSessionSlice

func (l *Log) PostSessionSlice(slice events.SessionSlice) error

PostSessionSlice sends chunks of recorded session to the event log

func (*Log) SearchEvents

func (l *Log) SearchEvents(fromUTC, toUTC time.Time, filter string, limit int) ([]events.EventFields, error)

SearchEvents is a flexible way to find The format of a query string depends on the implementing backend. A recommended format is urlencoded (good enough for Lucene/Solr)

Pagination is also defined via backend-specific query format.

The only mandatory requirement is a date range (UTC). Results must always show up sorted by date (newest first)

func (*Log) SearchSessionEvents

func (l *Log) SearchSessionEvents(fromUTC time.Time, toUTC time.Time, limit int) ([]events.EventFields, error)

SearchSessionEvents returns session related events only. This is used to find completed session.

func (*Log) UploadSessionRecording

func (l *Log) UploadSessionRecording(events.SessionRecording) error

func (*Log) WaitForDelivery

func (l *Log) WaitForDelivery(ctx context.Context) error

WaitForDelivery waits for resources to be released and outstanding requests to complete after calling Close method