dlock

README

dlock

dlock is a library which provides safe exclusive locking in distributed systems, based on well synchronized clocks and AWS DynamoDB - making it the perfect choice for your AWS Lambda functions.

Features

Safety with fencing tokens

Locks returned by dlock ensure that no other parallel process can acquire the same lock. Since it is built for distributed systems, there are some situations where dlock cannot be sure if it still has the lock (e.g. network issues to DynamoDB) - in these cases dlock always errs on the side of caution, rather marking a lock as expired than proceeding if unsure. Note that this guarantee only holds as long as either fencing tokens are used or the process holding the lock does not get paused for a time greater than the lease time. See Martin Kleppmanns post for more details on distributed locking and fencing tokens. dlock supports generating fencing tokens.

The safety guarantees are ultimately based on the guarantees that DynamoDB gives for conditional writes, see AWS docs. dlock only executes conditional writes.

Absolute timestamps

dlock stores absolute timestamps in DynamoDB for each active lock ("lease until"), in contrast to other libraries which store the duration during which the lock is active. The participating clients in these other libraries need to (actively) wait the lease duration of the lock, until they are safe to steal the lock - imagine a process that crashed before it was able to release the lock. Choosing to store absolute timestamps not only improves throughput, but also allows to use dlock in environments where runtimes of programs can be very short-lived, e.g. AWS Lambda. Having Lambdas run for some time just to wait for a duration to pass in order to proceed on a stale lock incurs costs that can easily be prevented.

This requires that the clocks of the systems that use dlock have to be synchronized well and must not "jump". Use NTP or some other mechanism to keep the clocks in sync and use WithMaxClockSkew to define the maximum amount of time that clocks might differ from each other.

Since AWS Lambda is a managed service that synchronizes the system clocks, you do not need to care about this when using dlock.

Automatic heartbeats

A lock once acquired by dlock stays valid until it is unlocked. In order to refresh the absolute "lease until" timestamp in DynamoDB, automatic heartbeats are in place. If these keep failing, e.g. due to unavailability of DynamoDB, the locks will expire.

Warning before locks are about to expire

Each lock has a "WarnChan" which posts a message at a configurable time before the lock expires. This gives the process time to gracefully shut down all operations it might be doing where it relies on having the exclusive lock.

Usage

1. Get it go get -u https://github.com/scailio-oss/dlock
2. Create DynamoDB table with a partition key of type string and name key. Default name of the table is dlock.
3. Use:

package main

import (
	// snip

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/dynamodb"

	"github.com/scailio-oss/dlock"
)

func main() {
	awsConfig := aws.Config{} // Whatever you need to create the config
	dynamoDbClient := dynamodb.NewFromConfig(awsConfig)

	// Ensure this is unique for this program instance. E.g. use AWS RequestId in Lambda.
	ownerName := strconv.FormatUint(rand.Uint64(), 16)

	locker := dlock.NewLocker(dynamoDbClient, ownerName,
		// This locker locks objects of type 'streets in NYC'
		dlock.WithLockIdPrefix("nyc-street-"),
		dlock.WithLease(10*time.Second),
		dlock.WithHeartbeat(2*time.Second),
		dlock.WithMaxClockSkew(10*time.Second),
		dlock.WithWarnAfter(9*time.Second),
		dlock.WithDynamoDbTimeout(1*time.Second),
	)
	defer locker.Close()

	// Try to acquire a lock on 'wallstreet'
	lock, err := locker.TryLock(context.Background(), "wallstreet")
	if err != nil {
		fmt.Printf("Could not lock: %v\n", err)
		return
	}

	// TODO do things exclusively on object 'wallstreet'

	select {
	case <-lock.WarnChan():
		// Will fire after 9 seconds after the last successful heartbeat
		fmt.Printf("WARNING, the lock is about to expire\n")
	default:
	}

	lock.Unlock(context.Background())
}

Integration tests

dlock contains integration tests, which locally start a DynamoDB in a Docker container. See internal/itest/ for details.

Comparison to other libraries

• https://github.com/cirello-io/dynamolock v2
  • Instead of an absolute "lease until" timestamp, dynamolock stores a "duration" in the database with the downsides for short-lived processes as noted above
  • Does not generate fencing tokens
• https://github.com/Clever/dynamodb-lock-go
  • dynamo-lock-go uses AWS SDK v1 instead of v2
  • Does not generate fencing tokens

License

Apache 2.0

Documentation

Index

• func NewLocker(dynamodbClient *dynamodb.Client, ownerName string, options ...LockerOption) locker.Locker
• type LockerOption
  • func WithDynamoDbTimeout(dynamoDbTimeout time.Duration) LockerOption
  • func WithFencingEnabled(fencingEnabled bool) LockerOption
  • func WithHeartbeat(heartbeat time.Duration) LockerOption
  • func WithLease(lease time.Duration) LockerOption
  • func WithLockIdPrefix(lockIdPrefix string) LockerOption
  • func WithLogger(logger logger.Logger) LockerOption
  • func WithMaxClockSkew(maxClockSkew time.Duration) LockerOption
  • func WithTableName(tableName string) LockerOption
  • func WithWarnAfter(warnAfter time.Duration) LockerOption
  • func WithWarnDisabled() LockerOption
• type LockerParams

Functions

func NewLocker

func NewLocker(dynamodbClient *dynamodb.Client, ownerName string, options ...LockerOption) locker.Locker

NewLocker creates a new Locker based on DynamoDB. ownerName: unique name identifying this Locker instance - this information will be written into the DynamoDB options: Additional, optional options.

Types

type LockerOption

type LockerOption func(params *LockerParams)

func WithDynamoDbTimeout

func WithDynamoDbTimeout(dynamoDbTimeout time.Duration) LockerOption

Use this timeout for dynamoDb calls instead of the default. Locker will call dynamoDb both in methods directly triggered by the user, but also in goroutines (e.g. heartbeat). This timeout will be used for the remote calls.

func WithFencingEnabled

func WithFencingEnabled(fencingEnabled bool) LockerOption

Use this fencing state instead of the default.

When fencing is enabled, each lock provides a fencing token, otherwise not. Maintaining fencing tokens has downsides:

• More requests to DynamoDB needed
• Locks are not actually deleted in DynamoDB but need to be kept there forever, in order to ensure that the fencing tokens fulfill the promises as given in Lock.FencingToken() (monotonically increasing).

func WithHeartbeat

func WithHeartbeat(heartbeat time.Duration) LockerOption

Use the given heartbeat duration instead of the default defaultHeartbeat. See WithLease. Also, at heartbeat time, internal cleanup of Locker datastructures is executed, if it did not happen before.

func WithLease

func WithLease(lease time.Duration) LockerOption

Use the given Lease time instead of the default defaultLease. The lease time is the duration after which a lock will timeout automatically and other owners can steal a lock (after an additional wait for the MaxClockSkew). The Locker runs a heartbeat loop which auto-refreshes all locks regularly, extending their lifetime to the leasetime as long as this locker is running and as long as we have a connection to DynamoDB. The lease time and heartbeat time should be chosen in a way that multiple heartbeats are sent during a normal lease time, which allows single heartbeats to fail e.g. due to temporary connection issues, but the lock not being lost immediately. Note that Locker will store a fixed internal leaseUntilTime for each lock, i.e. a timestamp that identifies when the lock will timeout if it is not refreshed with heartbeats. The guarantees that the Locks give, require that system clocks are synchronized well, see WithMaxClockSkew.

func WithLockIdPrefix

func WithLockIdPrefix(lockIdPrefix string) LockerOption

Use this prefix to all lockIds that are used within TryLock. Since a single Locker should lock only items of the same type, this prefix can be used to re-use the same DynamoDB table for different Lockers locking different kinds of objects.

func WithLogger

func WithLogger(logger logger.Logger) LockerOption

Use the given Logger instead of a default one

func WithMaxClockSkew

func WithMaxClockSkew(maxClockSkew time.Duration) LockerOption

Use this maximum clock skew instead of the default defaultMaxClockSkew. Locker relies on well-synchronized system clocks, which in reality is very hard to achieve. To compensate, this parameter specifies an upper bound of the time difference of the system clocks of all participanting systems, i.e. all systems that execute a Locker with the same LockIdPrefix on the same dynamoDB table. The Locker will not steal an existing lock for leaseUntilTime+maxClockSkew.

func WithTableName

func WithTableName(tableName string) LockerOption

Use the given DynamoDB table name instead of the default defaultTableName

func WithWarnAfter

func WithWarnAfter(warnAfter time.Duration) LockerOption

Use the given warn time instead of the default defaultWarnAfter. A lock that has been acquired has a WarnChan that will receive a message after this amount of time has passed since acquiring the lock or the last successful heartbeat. This allows the program to stop working on resources that are locked by the distributed locks, since after the lock actually expired, there is no way to guarantee exclusive access to those resources anymore.

Example: leaseTime = 1 min, heartbeat = 15 sec, warnAfter = 50 sec After the lock is acquired, the heartbeats start extending the lifetime of the lock, each setting the internal leaseUntilTime of the lock to currentTime + lease duration. If all heartbeats succeed, there will be no warning issued. If heartbeats start failing, e.g. due to a network connection issue to DynamoDB, then 50 sec after the last update to the internal leaseUntilTime, the lock will send a message on the WarnChan.

See also WithWarnDisabled

func WithWarnDisabled

func WithWarnDisabled() LockerOption

Disables warnChans fully. The corresponding methods will return nil.

See also WithWarnAfter.

type LockerParams

type LockerParams struct {
	// contains filtered or unexported fields
}