rlock

README

Overview

rlock is a "remote lock" lib that uses an SQL DB for its backend.

Think "remote mutex" with some bells and whistles.

WARNING: Using a non-distributed backend for a remote lock is dangerous! If you are concerned about this, add support for a distributed backend and submit a PR .. or re-evaluate your approach and do not use this library.

Bells and Whistles

A basic remote mutex lock implementation is pretty simple: try to acquire a lock by continuously trying to insert (or update) a lock record until you hit an error, timeout or success.

On top of the above, rlock also enables the lock holder to pass state to any potential future lock owners via the Unlock(err Error) method.

When a future lock owner acquires a lock, it can check to see what (if any) error a previous lock owner ran into by using LastError(). By examining the error, the future lock owner can determine if the previous lock owner ran into a fatal error or an error that the current lock holder may potentially be able to avoid.

Neat!

Use Case / Example Scenario

Imagine you have ten instances of a service that are all load balanced. Each one of these instances is able to create some sort of a resource that takes 1+ minutes to create.

1. Request A comes in and is load balanced to instance #1
2. Instance #1 checks if requested resource exists -- it does not
3. Instance #1 starts creating resource
4. Request B comes in and is load balanced to instance #2
5. Instance #2 checks if requested resource exists -- it does not (because it is being actively created by instance #1)
6. Instance #2 starts creating resource
7. We have a race -- Both #1 and #2 are creating the same resource that is likely to result in a bad outcome

The above problem case can be mitigated by introducing a remote lock. Going off the previous scenario, the sequence of events would look something like this:

1. Request A comes in and is load balanced to instance #1
2. Instance #1 acquires lock via Lock("MyLock", 2 * time.Minute)
3. Instance #1 starts creating resource
4. Request B comes in and is load balanced to instance #2
5. Instance #2 attempts to acquire lock via Lock("MyLock", 2 * time.Minute)
6. Instance #2 blocks waiting on lock acquire until either:
   • IF instance #1 finishes work and unlocks "MyLock"
     1. Instance #2 acquires the lock
     2. Instance #2 checks if resource exists -- IT DOES
     3. Instance #2 avoids creating resource and moves on to next step
   • IF Instance #1 doesn't finish work and/or doesn't unlock "MyLock"
     1. Instance #2 receives an AcquireTimeoutErr and errors out
   • IF Instance #1 runs into a recoverable error and unlocks "MyLock" but with an error (that instance #2 can look at and determine if it should re-attempt to do the "work" once more)
     1. Instance #2 acquires the Lock and checks to see if the previous lock user ran into an error via LastError()
     2. Instance #2 sees that the last lock user indeed ran into an error but instance #2 knows how to mitigate the error (for example, this could be a temporary network error that is likely to go away)
     3. Instance #2 attempts to create the resource AND succeeds
     4. Instance #2 releases the lock

Contrived Example

1. Launch two goroutines
2. One goroutine is told to Unlock the lock WITH an error
3. Second goroutine is told to Unlock WITHOUT an error
4. Both goroutines check if previous lock owners ran into an error

import (
    "fmt"
    "os"
    "time"
    
    "github.com/dselans/rlock"
    _ "github.com/go-sql-driver/mysql"
    "github.com/jmoiron/sqlx"
)

var (
    AcquireTimeout   = 10 * time.Second
    RecoverableError = errors.New("recoverable error")
)
    
func main() {
    // Connect to a DB using sqlx
    db, _ := sqlx.Connect("mysql", "user:pass@tcp(127.0.0.1:3306)/dbname?parseTime=true"
    
    // Create an rlock instance
    rl, _ := rlock.New(db)
    
    go createResource(rl, RecoverableError)
    go createResource(rl, nil)
    
    time.Sleep(12 * time.Second)
    
    fmt.Printn("Done!")
}

func createResource(rl *rlock.Lock, stateError error) {
    l, _ := rl.Lock("MyLock", AcquireTimeout)
    lastError = l.LastError()
    
    if lastError != nil {
        if lastError == RecoverableError {
            // Do recovery work
        } else {
           // Fatal error, quit
           os.Exit(1)
        }
    }
    
    // Do actual work
    // ... 
    
    // Release lock
    l.Unlock(stateError)
}

Documentation

Index

• Constants
• Variables
• type IRLock
• type Lock
  • func (l *Lock) LastError() error
  • func (l *Lock) Unlock(lastError error) error
• type LockEntry
• type RLock
  • func New(db *sqlx.DB) (*RLock, error)
  • func (r *RLock) Lock(name string, acquireTimeout time.Duration) (*Lock, error)

Constants

const (
	TableName    = "rlock"
	PollInterval = 1 * time.Second
	MaxAge       = 1 * time.Hour
)

Variables

var (
	AcquireTimeoutErr = errors.New("reached timeout while waiting on lock")
	KeyNotFoundErr    = errors.New("no such lock")
)

Types

type IRLock

type IRLock interface {
	Lock(name string, acquireTimeout time.Duration) (*Lock, error)
}

type Lock

type Lock struct {
	// contains filtered or unexported fields
}

func (*Lock) LastError

func (l *Lock) LastError() error

LastError returns nil if `last_used` is empty or an error if `last_used` is not empty.

`last_error` (in the db) is used by *current* lock holders to convey state such as whether they ran into an error. A subsequent lock holder can then determine whether the previous lock user ran into issues AND potentially perform additional steps based on the answer.

func (*Lock) Unlock

func (l *Lock) Unlock(lastError error) error

If an error is passed to unlock, upon unlocking the row in the db, we will also update `last_used` to the passed error. This way, subsequent lock holders can call on LastError() and see what (if any) error previous lock holder(s) ran into.

type LockEntry

type LockEntry struct {
	ID        int           `db:"id"`
	Name      string        `db:"name"`
	Owner     string        `db:"owner"`
	InUse     types.BitBool `db:"in_use"`
	LastError string        `db:"last_error"`
	LastUsed  time.Time     `db:"last_used"`
	CreatedAt time.Time     `db:"created_at"`
}

type RLock

type RLock struct {
	// contains filtered or unexported fields
}

func New

func New(db *sqlx.DB) (*RLock, error)

func (*RLock) Lock

func (r *RLock) Lock(name string, acquireTimeout time.Duration) (*Lock, error)