quota

README

User Quota

Quota module allows to protect the system from overloading and enforce per user resource limits. The module is composable and allow to control the following

• Per user overall storage usage
• Allows to protect the node from overloading, by enforcing absolute maximum of request rate single node of Tigris service allowed to handle.
• Coordinated per-user request rate limiting across all Tigris service nodes, by leveraging Datadog metrics backend.

Every limiter has Enabled config variable and can be enabled and disabled independently. Quota is checked and enforced in the server middleware.

Storage size limiting

Storage limiter keeps per-user in memory cache of current storage used, the cache is updated in the background thread every config.DefaultConfig.Quota.Storage.RefreshInterval. Background thread also updates collection sizes, database sizes and namespaces sizes and sends them to observability service as metrics.

Current storage size is checked against per namespace or default storage size limit and request is rejected with datasize limit exceeded (HTTP: 429) error if storage size exceeds the limit.

Request limiter

Request limiters are used to enforce node and namespace quotas. There are two interface methods limiter provides: 1. Allow: this method return and error immediately if the request rate exceeded configured values. 2. Wait: Even if the limits are exceeded at this particular moment, wait allows to reserve a token and schedule execution at some point-in-time in the future. The execution can be delayed upto the request context timeout.

Request rate limiter is configured by the number of:

• ReadUnits
• WriteUnits

Every request consumes at least one request unit. Number of units consumed calculated by the following formula:

readUnits = {request size in bytes} / ReadUnitSize writeUnits = {request size in bytes} / WriteUnitSize

plus one unit if remainder of above division is not zero.

where:

• ReadUnitSize = 4096 bytes
• WriteUnitSize = 1024 bytes

Namespace request rate limiting

Namespace rate limiter enforces cluster wide quota using current performance rate from the metrics backend, currently only Datadog backend is supported. By default, per cluster per namespace limits are configured by config.DefaultConfig.Quota.Namespace.Default.(Read|Write)Units. Specific per namespace limits can be set by config.DefaultConfig.Quota.Namespace.Namespaces[{ns}].(Read|Write)Units.

Namespace rate limiter periodically updates current per namespace rates in the background thread. Depending on the current rates received from the metrics backend background thread reduce or increase allowed limits. Limits are increased or reduced by 10% of maximum limits allowed per namespace per node. Maximum limit is configured by config.DefaultConfig.Quota.Namespace.Node.(Read|Write)Units

Maximum per node rate limiting

This rate limiter is used to enforce maximum request rate across all the user of single node. Configured by config.DefaultConfig.Quota.Node.(Read|Write)Units

Documentation

Index

• Variables
• func Allow(ctx context.Context, namespace string, size int, isWrite bool) error
• func Cleanup()
• func Init(tm *metadata.TenantManager, cfg *config.Config) error
• func Wait(ctx context.Context, namespace string, size int, isWrite bool) error
• type Backend
• type Datadog
  • func (d *Datadog) CurRates(ctx context.Context, namespace string) (int64, int64, error)
• type Limiter
  • func (l *Limiter) Allow(size int) (err error)
  • func (l *Limiter) SetBurst(ratel int)
  • func (l *Limiter) SetLimit(ratel int)
  • func (l *Limiter) Wait(ctx context.Context, size int) (err error)
• type LimiterConfig
• type Manager
• type NoopMetrics
  • func (d *NoopMetrics) CurRates(_ context.Context, _ string) (int64, int64, error)
• type Quota
• type State
  • func (s *State) Allow(size int, isWrite bool) error
  • func (s *State) Wait(ctx context.Context, size int, isWrite bool) error

Variables

var (
	ErrReadUnitsExceeded      = errors.ResourceExhausted("request read rate exceeded")
	ErrWriteUnitsExceeded     = errors.ResourceExhausted("request write rate exceeded")
	ErrStorageSizeExceeded    = errors.ResourceExhausted("data size limit exceeded")
	ErrMaxRequestSizeExceeded = errors.ResourceExhausted("maximum request size limit exceeded")
)

var RunningAverageLength = 60 * time.Second

Functions

func Allow

func Allow(ctx context.Context, namespace string, size int, isWrite bool) error

Allow checks read, write rates and storage size limits for the namespace and returns error if at least one of them is exceeded.

func Cleanup

func Cleanup()

func Init

func Init(tm *metadata.TenantManager, cfg *config.Config) error

func Wait

func Wait(ctx context.Context, namespace string, size int, isWrite bool) error

Types

type Backend

type Backend interface {
	CurRates(ctx context.Context, namespace string) (int64, int64, error)
}

type Datadog

type Datadog struct {
	Datadog *metrics.Datadog
}

func (*Datadog) CurRates

func (d *Datadog) CurRates(ctx context.Context, namespace string) (int64, int64, error)

type Limiter

type Limiter struct {
	Rate *rate.Limiter
	// contains filtered or unexported fields
}

func (*Limiter) Allow

func (l *Limiter) Allow(size int) (err error)

Allow checks if the request with given size can be executed at this moment. It returns immediately. If no error returned the request is allowed to proceed. In the case of error, it returns specific error indicating, which limiter parameter is violated, whether read or write.

func (*Limiter) SetBurst

func (l *Limiter) SetBurst(ratel int)

SetBurst sets new limiter burst values.

func (*Limiter) SetLimit

func (l *Limiter) SetLimit(ratel int)

SetLimit sets new limiter limits values.

func (*Limiter) Wait

func (l *Limiter) Wait(ctx context.Context, size int) (err error)

Wait checks if the request with given size can be executed at this moment. It returns immediately if the request allowed to proceed. But, in the case of limits violation, unlike Allow, it doesn't return immediately, but reserves a token and delays the execution till the moment when request can proceed without violating the limits. The execution can be delayed upto the timeout set in the context.

type LimiterConfig

type LimiterConfig struct {
	Rate      int
	BurstRate int
}

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

type NoopMetrics

type NoopMetrics struct{}

func (*NoopMetrics) CurRates

func (d *NoopMetrics) CurRates(_ context.Context, _ string) (int64, int64, error)

type Quota

type Quota interface {
	Allow(ctx context.Context, namespace string, size int, isWrite bool) error
	Wait(ctx context.Context, namespace string, size int, isWrite bool) error
	Cleanup()
}

type State

type State struct {
	Read  Limiter
	Write Limiter
}

func (*State) Allow

func (s *State) Allow(size int, isWrite bool) error

func (*State) Wait

func (s *State) Wait(ctx context.Context, size int, isWrite bool) error