firestore

README

Firestore backend implementation for Teleport.

Introduction

This package enables Teleport auth server to store secrets in Firestore on GCP.

WARNING: Using Firestore involves recurring charge from GCP.

Building

Firestore backend is not enabled by default. To enable it you have to compile Teleport with firestore build flag.

To build Teleport with Firestore enabled, run:

ADDFLAGS='-tags firestore' make teleport

Quick Start

There are currently two Firestore mode options for any given GCP Project; Native mode and Datastore Mode. This storage backend uses Real-time updates to keep individual auth instances in sync and requires Firestore configured in Native mode.

Add this storage configuration in teleport section of the config file (by default it's /etc/teleport.yaml):

teleport:
  storage:
    type: firestore
    collection_name: cluster-data
    credentials_path: /var/lib/teleport/firestore_creds
    project_id: gcp-proj-with-firestore-enabled

Collections are automatically created by the Firestore APIs and the required indexes are created by the backend on first start, if they do not exist.

Encryption

Lifted from Google docs; Cloud Firestore automatically encrypts all data before it is written to disk. There is no setup or configuration required and no need to modify the way you access the service. The data is automatically and transparently decrypted when read by an authorized user.

With server-side encryption, Google manages the cryptographic keys on your behalf using the same hardened key management systems that we use for our own encrypted data, including strict key access controls and auditing. Each Cloud Firestore object's data and metadata is encrypted under the 256-bit Advanced Encryption Standard, and each encryption key is itself encrypted with a regularly rotated set of master keys.

Full Properties

The full list of configurable properties for this backend are:

• credentials_path (string, path to GCP creds for Firestore, not-required)
• project_id (string, project ID, required)
• collection_name (string, collection for cluster information, required)
• purge_expired_documents_poll_interval (time duration, poll interval to sweep expired documents, not-required, defaults to once per minute)
• retry_period (time duration, retry period for all background tasks, not-required, defaults to 10 seconds)
• disable_expired_document_purge (bool, disables expired document purging, not-required, defaults to false)
• buffer_size (int, buffer size for watched events, not-required, defaults to 1024)
• endpoint (string, firestore client endpoint, not-required, ex: localhost:8618)
• limit_watch_query (bool, forces the watcher to start querying only for records from current time forward, not-required, defaults to false)

Firestore Client Authentication Options

There are three authentication/authorization modes available;

1. With no credentialsPath and no endpoint defined, the Firestore clients will use Google Application Default Credentials for authentication. This only works in cases where Teleport is installed on GCE instances and have service accounts with IAM role/profile associations authorizing that GCE instance to use Firestore.
2. With endpoint defined, Firestore will create clients no auth, GRPC in-secure, clients pointed at the specified endpoint. This is only used for tests, see Tests section below.
3. With credentialsPath defined, Firestore will create clients authenticating against live systems with the Service Account bound to the JSON key file referenced in the option.

Implementation Details

Firestore Document IDs must be unique, cannot start with periods, and cannot contain forward slashes. In order to support more straight forward fetching but work within the requirements of Firestore, Document IDs are SHA1 hashed from the records key.

Realtime updates are consumed across the cluster via Firestore's ability to watch for document updates.

One composite indexes is required for this implementation:

1. key ascending, then on expires ascending

Composite indexes should be limited to the specific collection set in the configuration (in the aforementioned example is cluster-data).

Tests

Tests must execute one of two ways:

1. With gcloud installed in test infrastructure and the firestore emulator enabled and running to a dynamic port a pre-defined port used in the config. Ex: gcloud beta emulators firestore start --host-port=localhost:8618. This is where the Firestore config parameter endpoint is used.
2. With a service account pointed a test GCP project and or test collections.

Get Help

This backend has been contributed by https://github.com/joshdurbin

Documentation

Overview

Package firestoreFirestoreBackend implements Firestore storage backend for Teleport auth service, similar to DynamoDB backend.

firestore package implements the FirestoreBackend storage back-end for the auth server. Originally contributed by https://github.com/joshdurbin

Index

• Constants
• func ConvertGRPCError(err error, args ...interface{}) error
• func CreateFirestoreClients(ctx context.Context, projectID string, endPoint string, credentialsFile string) (*apiv1.FirestoreAdminClient, *firestore.Client, error)
• func DeleteAllDocuments(ctx context.Context, svc *firestore.Client, collectionName string)
• func EnsureIndexes(ctx context.Context, adminSvc *apiv1.FirestoreAdminClient, ...) error
• func GetName() string
• func RetryingAsyncFunctionRunner(ctx context.Context, retryConfig utils.LinearConfig, logger *log.Logger, ...)
• type Config
• type FirestoreBackend
  • func New(ctx context.Context, params backend.Params) (*FirestoreBackend, error)
  • func (cfg *FirestoreBackend) CheckAndSetDefaults() error
  • func (b *FirestoreBackend) Clock() clockwork.Clock
  • func (b *FirestoreBackend) Close() error
  • func (b *FirestoreBackend) CloseWatchers()
  • func (b *FirestoreBackend) CompareAndSwap(ctx context.Context, expected backend.Item, replaceWith backend.Item) (*backend.Lease, error)
  • func (b *FirestoreBackend) Create(ctx context.Context, item backend.Item) (*backend.Lease, error)
  • func (b *FirestoreBackend) Delete(ctx context.Context, key []byte) error
  • func (b *FirestoreBackend) DeleteRange(ctx context.Context, startKey, endKey []byte) error
  • func (b *FirestoreBackend) Get(ctx context.Context, key []byte) (*backend.Item, error)
  • func (b *FirestoreBackend) GetRange(ctx context.Context, startKey []byte, endKey []byte, limit int) (*backend.GetResult, error)
  • func (b *FirestoreBackend) KeepAlive(ctx context.Context, lease backend.Lease, expires time.Time) error
  • func (b *FirestoreBackend) NewWatcher(ctx context.Context, watch backend.Watch) (backend.Watcher, error)
  • func (b *FirestoreBackend) Put(ctx context.Context, item backend.Item) (*backend.Lease, error)
  • func (b *FirestoreBackend) Update(ctx context.Context, item backend.Item) (*backend.Lease, error)
• type IndexTuple

Constants

const (
	// BackendName is the name of this backend
	BackendName = "firestore"
)

Functions

func ConvertGRPCError

func ConvertGRPCError(err error, args ...interface{}) error

ConvertGRPCError converts GRPC errors

func CreateFirestoreClients

func CreateFirestoreClients(ctx context.Context, projectID string, endPoint string, credentialsFile string) (*apiv1.FirestoreAdminClient, *firestore.Client, error)

CreateFirestoreClients creates a firestore admin and normal client given the supplied parameters

func DeleteAllDocuments

func DeleteAllDocuments(ctx context.Context, svc *firestore.Client, collectionName string)

DeleteAllDocuments will delete all documents in a collection.

func EnsureIndexes

func EnsureIndexes(ctx context.Context, adminSvc *apiv1.FirestoreAdminClient, tuples []*IndexTuple, indexParent string) error

EnsureIndexes is a function used by Firestore events and backend to generate indexes and will block until indexes are reported as created

func GetName

func GetName() string

GetName is a part of backend API and it returns Firestore backend type as it appears in `storage/type` section of Teleport YAML

func RetryingAsyncFunctionRunner

func RetryingAsyncFunctionRunner(ctx context.Context, retryConfig utils.LinearConfig, logger *log.Logger, task func() error, taskName string)

RetryingAsyncFunctionRunner wraps a task target in retry logic

Types

type Config

type Config struct {
	// Credentials path for the Firestore client
	CredentialsPath string `json:"credentials_path,omitempty"`
	// Google Project ID of Collection containing events
	ProjectID string `json:"project_id,omitempty"`
	// CollectName is the name of the collection containing events
	CollectionName string `json:"collection_name,omitempty"`
	// PurgeExpiredDocumentsPollInterval is the poll interval used to purge expired documents
	PurgeExpiredDocumentsPollInterval time.Duration `json:"purge_expired_documents_poll_interval,omitempty"`
	// RetryPeriod is a period between retry executions of long-lived document snapshot queries and purging expired records
	RetryPeriod time.Duration `json:"retry_period,omitempty"`
	// DisableExpiredDocumentPurge
	DisableExpiredDocumentPurge bool `json:"disable_expired_document_purge,omitempty"`
	// EndPoint is used to point the Firestore clients at emulated Firestore storage.
	EndPoint string `json:"endpoint,omitempty"`
}

FirestoreConfig structure represents Firestore configuration as appears in `storage` section of Teleport YAML

type FirestoreBackend

type FirestoreBackend struct {
	*log.Entry
	// contains filtered or unexported fields
}

FirestoreBackend is a Firestore-backed key value backend implementation.

func New

func New(ctx context.Context, params backend.Params) (*FirestoreBackend, error)

New returns new instance of Firestore backend. It's an implementation of backend API's NewFunc

func (*FirestoreBackend) CheckAndSetDefaults

func (cfg *FirestoreBackend) CheckAndSetDefaults() error

CheckAndSetDefaults is a helper returns an error if the supplied configuration is not enough to connect to Firestore

func (*FirestoreBackend) Clock

func (b *FirestoreBackend) Clock() clockwork.Clock

Clock returns wall clock

func (*FirestoreBackend) Close

func (b *FirestoreBackend) Close() error

Close closes the Firestore client contexts and releases associated resources

func (*FirestoreBackend) CloseWatchers

func (b *FirestoreBackend) CloseWatchers()

CloseWatchers closes all the watchers without closing the backend

func (*FirestoreBackend) CompareAndSwap

func (b *FirestoreBackend) CompareAndSwap(ctx context.Context, expected backend.Item, replaceWith backend.Item) (*backend.Lease, error)

CompareAndSwap compares and swap values in atomic operation CompareAndSwap compares item with existing item and replaces is with replaceWith item

func (*FirestoreBackend) Create

func (b *FirestoreBackend) Create(ctx context.Context, item backend.Item) (*backend.Lease, error)

Create creates item if it does not exist

func (*FirestoreBackend) Delete

func (b *FirestoreBackend) Delete(ctx context.Context, key []byte) error

Delete deletes item by key

func (*FirestoreBackend) DeleteRange

func (b *FirestoreBackend) DeleteRange(ctx context.Context, startKey, endKey []byte) error

DeleteRange deletes range of items with keys between startKey and endKey

func (*FirestoreBackend) Get

func (b *FirestoreBackend) Get(ctx context.Context, key []byte) (*backend.Item, error)

Get returns a single item or not found error

func (*FirestoreBackend) GetRange

func (b *FirestoreBackend) GetRange(ctx context.Context, startKey []byte, endKey []byte, limit int) (*backend.GetResult, error)

GetRange returns range of elements

func (*FirestoreBackend) KeepAlive

func (b *FirestoreBackend) KeepAlive(ctx context.Context, lease backend.Lease, expires time.Time) error

KeepAlive keeps object from expiring, updates lease on the existing object, expires contains the new expiry to set on the lease, some backends may ignore expires based on the implementation in case if the lease managed server side

func (*FirestoreBackend) NewWatcher

func (b *FirestoreBackend) NewWatcher(ctx context.Context, watch backend.Watch) (backend.Watcher, error)

NewWatcher returns a new event watcher

func (*FirestoreBackend) Put

func (b *FirestoreBackend) Put(ctx context.Context, item backend.Item) (*backend.Lease, error)

Put puts value into backend (creates if it does not exists, updates it otherwise)

func (*FirestoreBackend) Update

func (b *FirestoreBackend) Update(ctx context.Context, item backend.Item) (*backend.Lease, error)

Update updates value in the backend

type IndexTuple

type IndexTuple struct {
	FirstField  string
	SecondField string
}