kvsync

README

KVSync

KVSync is a Go package that provides a simple and efficient way to synchronize your GORM models with a key-value store.

There can be multiple key definitions for each model. For example, you can have a key for fetching by ID, a key for fetching by UUID, and a composite key for fetching by both ID and UUID. For each key, the model data is replicated accordingly to the key-value store.

Installation

To install KVSync, use the go get command:

go get github.com/ndthuan/kvsync

Sync Setup

Define Synced Models

Implement kvsync.Syncable and provide sync keys mapped by a name for fetching later. Each key is unique on the key-value store.

type SyncedUser struct {
	gorm.Model
	UUID     string
	Username string
}

func (u SyncedUser) SyncKeys() map[string]string {
	return map[string]string{
		"id":        fmt.Sprintf("user:id:%d", u.ID),
		"uuid":      fmt.Sprintf("user:uuid:%s", u.UUID),
		"composite": fmt.Sprintf("user:composite:%d_%s", u.ID, u.UUID),
	}
}

Configure Key-Value Store

With Redis for example, you can use the provided RedisStore. Steps:

• Init GORM DB instance
• Init Redis client
• Create a new RedisStore instance
• Create a new KVSync instance
• Register GORM callbacks

package main

import (
	"context"
	"fmt"
	"github.com/ndthuan/kvsync"
	"github.com/redis/go-redis/v9"
	"gorm.io/gorm"
	"time"
)

func main() {
	db, err := gorm.Open(...)
	if err != nil {
		panic(fmt.Sprintf("Failed to connect to database: %v", err))
	}

	clusterClient := redis.NewClusterClient(&redis.ClusterOptions{
		Addrs: []string{},
	})

	// Create a new RedisStore
	store := &kvsync.RedisStore{
		Client:     clusterClient,
		Expiration: time.Hour * 24 * 365,               // Set the expiration time for the keys
		Prefix:     "kvsync:",                          // Optional, defaults to "kvsync:"
		Marshaler:  &kvsync.BSONMarshalingAdapter{},    // Optional, BSONMarshalingAdapter (using Mongo's BSON) is the default and recommended
	}

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	kvSync := kvsync.NewKVSync(ctx, kvsync.Options{
		Store:   store,
		Workers: 4,
		ReportCallback: func(r kvsync.Report) {
			if r.Err == nil {
				actualDoneCount++
			}
		},
	})
	
	// Register the GORM callbacks for automated synchronization
	db.Callback().Create().After("gorm:create").Register("kvsync:create", kvSync.GormCallback())
	db.Callback().Update().After("gorm:update").Register("kvsync:update", kvSync.GormCallback())

}

And create/update your model as usual

// Create a new SyncedUser
db.Create(&SyncedUser{
    UUID:     "test-uuid",
    Username: "test-username",
})
// The SyncedUser is automatically synchronized with the key-value store

Fetching Synced Models

You can fetch the model by any of the keys you defined. You must provide a struct with non-zero values for the keys you want to fetch by.

By ID

user := SyncedUser{
    Model: gorm.Model{ID: 1},
}
kvSync.Fetch(&user, "id")

By UUID

user := SyncedUser{
    UUID: "test-uuid",
}
kvSync.Fetch(&user, "uuid")

By composite key

user := SyncedUser{
    Model: gorm.Model{ID: 1},
    UUID:  "test-uuid",
}
kvSync.Fetch(&user, "composite")

License

KVSync is licensed under the MIT License. See the LICENSE file for more information.

Documentation

Index

• type BSONMarshalingAdapter
  • func (b *BSONMarshalingAdapter) Marshal(v any) ([]byte, error)
  • func (b *BSONMarshalingAdapter) Unmarshal(data []byte, v any) error
• type InMemoryStore
  • func (m *InMemoryStore) Fetch(key string, dest any) error
  • func (m *InMemoryStore) Put(key string, value any) error
• type KVStore
• type KVSync
  • func NewKVSync(ctx context.Context, options Options) KVSync
• type MarshalingAdapter
• type Options
• type RedisStore
  • func (r *RedisStore) Fetch(key string, dest any) error
  • func (r *RedisStore) Put(key string, value any) error
• type Report
• type ReportCallback
• type Syncable

Types

type BSONMarshalingAdapter

type BSONMarshalingAdapter struct{}

BSONMarshalingAdapter is a BSON implementation of MarshalingAdapter

func (*BSONMarshalingAdapter) Marshal

func (b *BSONMarshalingAdapter) Marshal(v any) ([]byte, error)

func (*BSONMarshalingAdapter) Unmarshal

func (b *BSONMarshalingAdapter) Unmarshal(data []byte, v any) error

type InMemoryStore

type InMemoryStore struct {
	Store map[string]any
	// contains filtered or unexported fields
}

InMemoryStore is an in-memory implementation of KVStore

func (*InMemoryStore) Fetch

func (m *InMemoryStore) Fetch(key string, dest any) error

func (*InMemoryStore) Put

func (m *InMemoryStore) Put(key string, value any) error

type KVStore

type KVStore interface {
	Put(key string, value any) error
	Fetch(key string, dest any) error
}

KVStore is the interface for a key-value store

type KVSync

type KVSync interface {
	Fetch(dest Syncable, keyName string) error
	GormCallback() func(db *gorm.DB)
	Sync(entity any) error
}

KVSync is the interface for a service that syncs Gorm models with a KVStore

func NewKVSync

func NewKVSync(ctx context.Context, options Options) KVSync

NewKVSync creates a new KVSync instance

type MarshalingAdapter

type MarshalingAdapter interface {
	Marshal(v any) ([]byte, error)
	Unmarshal(data []byte, v any) error
}

MarshalingAdapter is an interface for marshaling and unmarshaling data

type Options

type Options struct {
	Store          KVStore
	Workers        int
	ReportCallback ReportCallback
}

Options is a struct that contains options for creating a KVSync instance

type RedisStore

type RedisStore struct {
	Client     *redis.ClusterClient
	Prefix     string
	Expiration time.Duration
	Marshaler  MarshalingAdapter
}

RedisStore is a Redis implementation of KVStore

func (*RedisStore) Fetch

func (r *RedisStore) Fetch(key string, dest any) error

func (*RedisStore) Put

func (r *RedisStore) Put(key string, value any) error

type Report

type Report struct {
	Model   any
	KeyName string
	Key     string
	Err     error
}

Report is a struct that represents a report of a sync operation

type ReportCallback

type ReportCallback func(Report)

type Syncable

type Syncable interface {
	SyncKeys() map[string]string
}

Syncable is the interface for a Gorm model that can be synced with a KVStore