mutli_cache

package module
v0.0.0-...-a0b935e Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 6, 2022 License: Apache-2.0 Imports: 15 Imported by: 0

README

go-multi-lev-cache

介绍

分布式多级缓存方案g2cache

软件架构

软件架构说明
主要内容：

模块 功能 特点 注意
local interface 本机内存高速缓存 纳秒，毫秒级别响应速度；有效期较短 实现LocalCache接口，注意控制内存
out interface 外部高速缓存，如Redis 较快的访问速度，容量很足 实现OutCache接口
LoadDataSourceFunc 外部数据源 外部加载函数，灵活，并发保护 需要自行处理panic
PubSub 外部缓存发布订阅 外部缓存可以自选实现 未实现就不支持分布式

local 本地内存高速缓存接口

// Local memory cache，Local memory cache with high access speed
type LocalCache interface {
	Get(key string, obj interface{}) (*Entry, bool, error) // obj represents the internal structure of the real object
	Set(key string, e *Entry) error                        // local storage should set Entry.Obsolete
	Del(key string) error
	ThreadSafe() // Need to ensure thread safety
	Close()
}

out 外部缓存接口，值得一提的是外部缓存需要实现发布订阅接口，这是因为g2cache支持分布式多级缓存

// External cache has faster access speed, such as Redis
type OutCache interface {
	Get(key string, obj interface{}) (*Entry, bool, error) // obj represents the internal structure of the real object
	Set(key string, e *Entry) error                        // out storage should set Entry.Expiration
	Subscribe(data chan *ChannelMeta) error
	Publish(gid ,key string, action int8, data *Entry) error
	Del(key string) error
	ThreadSafe() // Need to ensure thread safety
	Close()
}

外部缓存发布订阅

// only out storage pub sub
type PubSub interface {
	Subscribe(data chan *ChannelMeta) error
	Publish(gid, key string, action int8, data *Entry) error
}

LoadDataSourceFunc 原始数据加载函数，请自行处理panic并以error形式返回
加载函数支持返回string，map，slice，struct，ptr类型

// Shouldn't throw a panic, please return an error
type LoadDataSourceFunc func() (interface{}, error)

缓存效果-初始

[G2CACHE] 2021/04/21 11:46:21 [debug] key:[[g2cache-key:61               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:21 [debug] key:[[g2cache-key:222               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:21 [debug] key:[[g2cache-key:61                ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 11:46:21 [debug] key:[[g2cache-key:183                ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:83                ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:18                ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:103               ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:90                ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:170               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:101               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:165               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:23 [debug] statistics [local] hit percentage [[5.8824]]
[G2CACHE] 2021/04/21 11:46:23 [debug] statistics [out] hit percentage [[1.9608]]
[G2CACHE] 2021/04/21 11:46:23 [debug] statistics [data source] hit percentage [[90.1961]]

缓存效果-N分钟后

[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:106               ]] => [ hit out storage ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:136               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:100               ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:219               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:13                ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:200               ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:172               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 12:26:53 [debug] statistics [local] hit percentage [[45.3865]]
[G2CACHE] 2021/04/21 12:26:53 [debug] statistics [out] hit percentage [[6.7332]]
[G2CACHE] 2021/04/21 12:26:53 [debug] statistics [data source] hit percentage [[48.1297]]

缓存效果-N+M分钟后

[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:2                 ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:2                 ]] => [ hit out storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:113               ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:113               ]] => [ hit out storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:43                ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:43                ]] => [ hit out storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] statistics [local] hit percentage [[82.3877]]
[G2CACHE] 2021/04/21 13:18:53 [debug] statistics [out] hit percentage [[16.2689]]
[G2CACHE] 2021/04/21 13:18:53 [debug] statistics [data source] hit percentage [[1.3641]]
Usage

go get -u gitee.com/kelvins-io/g2cache@v4.0.5

模拟测试

修改example/main.go中Redis的配置
sh example-run.sh

分支说明
  1. copyobj分支支持返回特定object，为不稳定版本
  2. dev分支Get返回object序列化值，过渡版
  3. master分支只提供sync.Map的local实现，早期版
  4. release分支提供发布版，与copyobj有较大变化

Documentation

Index

Constants

View Source 
const (
	SetPublishType int8 = iota // 存入数据类型
	DelPublishType             // 删除数据类型
)

Entry expire is UnixNano

Variables

View Source 
var (
	CacheKeyEmpty           = errors.New("cache key is empty")
	CacheObjNil             = errors.New("cache object is nil")
	LoadDataSourceFuncNil   = errors.New("cache load func is nil")
	LocalStorageClose       = errors.New("local storage close !!! ")
	OutStorageClose         = errors.New("out storage close !!! ")
	CacheClose              = errors.New("g2cache close !!! ")
	DataSourceLoadNil       = errors.New("data source load nil")
	OutStorageLoadNil       = errors.New("out storage load nil")
	CacheNotImplementPubSub = errors.New("cache not implement pubsub interface")
)
View Source 
var (
	CacheDebug                  bool   // 是否开启Debug模式,debug日志输出
	CacheMonitor                bool   // 缓存监视器,负责监视缓存是否过期
	OutCachePubSub              bool   // 缓存数据源,发布订阅模式是否开启
	CacheMonitorSecond          = 5    // 监视器默认扫描时间5秒
	DefaultGPoolWorkerNum       = 200  // 默认工作协程数目
	DefaultGPoolJobQueueChanLen = 1000 // 默认工作协程任务数量
)
View Source 
var (
	DefaultFreeCacheSize = 50 * 1024 * 1024 // 50MB
)
View Source 
var DefaultPubSubRedisChannel = "g2cache-pubsub-channel"
View Source 
var (
	EntryLazyFactor = 32
)

Functions

func GenKey 

func GenKey(args ...interface{}) string

func GetRedisPool 

func GetRedisPool(conf *RedisConf) (*redis.Pool, error)

func LogDebug 

func LogDebug(s ...interface{})

func LogDebugF 

func LogDebugF(f string, s ...interface{})

func LogErr 

func LogErr(s ...interface{})

func LogErrF 

func LogErrF(f string, s ...interface{})

func LogInfo 

func LogInfo(s ...interface{})

func LogInfoF 

func LogInfoF(f string, s ...interface{})

func NewUUID 

func NewUUID() (string, error)

func RedisDelKey 

func RedisDelKey(key string, pool *redis.Pool) error

func RedisGetString 

func RedisGetString(key string, pool *redis.Pool) (string, error)

func RedisPublish 

func RedisPublish(channel, message string, pool *redis.Pool) error

func RedisSetString 

func RedisSetString(key, value string, ttl int, pool *redis.Pool) error

Types

type ChannelMeta 

type ChannelMeta struct {
	Key     string `json:"key"`    // 数据对应的key
	CacheID string `json:"gid"`    // 数据存入的缓存GID
	Action  int8   `json:"action"` // 数据执行类型
	Data    *Entry `json:"data"`   // 对应数据
}

type Entry 

type Entry struct {
	Value      interface{} `json:"value"`      // 数据
	TtlSecond  int         `json:"ttl"`        // 过期时间段,x秒
	Obsolete   int64       `json:"obsolete"`   // 失效时间,本地缓存使用
	Expiration int64       `json:"expiration"` // 过期时间,外部数据源使用
}

func NewEntry 

func NewEntry(v interface{}, second int) *Entry

NewEntry 创建数据对象,包含过期时间

func (*Entry) Expired 

func (e *Entry) Expired() bool

Expired 外部数据源数据是否过期

func (*Entry) GetExpireTTL 

func (e *Entry) GetExpireTTL() (second int64)

GetExpireTTL 获取过期时间

func (*Entry) GetObsoleteTTL 

func (e *Entry) GetObsoleteTTL() (second int64)

GetObsoleteTTL 获取失效时间

func (*Entry) Obsoleted 

func (e *Entry) Obsoleted() bool

Obsoleted 判断本地数据源数据是否需要更新

func (*Entry) String 

func (e *Entry) String() string

type FreeCache 

type FreeCache struct {
	// contains filtered or unexported fields
}

func NewFreeCache 

func NewFreeCache() *FreeCache

func (*FreeCache) Close 

func (c *FreeCache) Close()

func (*FreeCache) Del 

func (c *FreeCache) Del(key string) error

func (*FreeCache) Get 

func (c *FreeCache) Get(key string, obj interface{}) (*Entry, bool, error)

func (*FreeCache) Set 

func (c *FreeCache) Set(key string, e *Entry) error

type Harsher 

type Harsher interface {
	Sum64(string) uint64
}

type HitStatistics 

type HitStatistics struct {
	HitOutStorageTotalRate   float64 `json:"hit_out_storage_total_rate"`   // 命中外部数据源速率
	HitDataSourceTotalRate   float64 `json:"hit_data_source_total_rate"`   // 命中普通数据源速率
	HitLocalStorageTotalRate float64 `json:"hit_local_storage_total_rate"` // 命中本地缓存速率
	HitDataSourceTotal       int64   `json:"hit_data_source_total"`        // 命中普通数据源次数
	HitLocalStorageTotal     int64   `json:"hit_local_storage_total"`      // 命中本地缓存次数
	HitOutStorageTotal       int64   `json:"hit_out_storage_total"`        // 命中外部数据源次数
	AccessGetTotal           int64   `json:"access_get_total"`             // 获取数据总数
}
var HitStatisticsOut HitStatistics // 统计变量

func (*HitStatistics) Calculation 

func (h *HitStatistics) Calculation()

Calculation 更新命中率

func (*HitStatistics) StatisticsDataSource 

func (h *HitStatistics) StatisticsDataSource()

StatisticsDataSource 计算外部数据源命中率

func (*HitStatistics) StatisticsLocalStorage 

func (h *HitStatistics) StatisticsLocalStorage()

StatisticsLocalStorage 计算本地数据源命中率

func (*HitStatistics) StatisticsOutStorage 

func (h *HitStatistics) StatisticsOutStorage()

StatisticsOutStorage 计算缓存数据源命中率

func (*HitStatistics) String 

func (h *HitStatistics) String() string

type Job 

type Job func()

Job 任务结构体

type LoadDataSourceFunc 

type LoadDataSourceFunc func() (interface{}, error)

LoadDataSourceFunc 外部数据源(mysql),兜底策略

type LocalCache 

type LocalCache interface {
	Get(key string, obj interface{}) (*Entry, bool, error) // 返回真实数据
	Set(key string, e *Entry) error                        // 本机缓存需要设置过期时间
	Del(key string) error                                  // 删除指定Key对应数据
	Close()                                                // 关闭本机缓存
}

LocalCache 本地缓存数据源

type LoggerInterface 

type LoggerInterface interface {
	LogInfoF(f string, s ...interface{})
	LogInfo(s ...interface{})
	LogDebug(s ...interface{})
	LogDebugF(f string, s ...interface{})
	LogErr(s ...interface{})
	LogErrF(f string, s ...interface{})
}

LoggerInterface 外部调用者可实现此日志接口用于将日志导出 

var (
	Logger LoggerInterface = &sysLogger{}
)

type MultiCache 

type MultiCache struct {
	CacheID string // 缓存唯一GID
	// contains filtered or unexported fields
}

func New 

func New(out OutCache, local LocalCache) (g *MultiCache, err error)

func (*MultiCache) Close 

func (g *MultiCache) Close()

Close 使用once确保所有线程只能关闭一次

func (*MultiCache) Del 

func (g *MultiCache) Del(key string, wait bool) (err error)

Del 删除数据 key 数据删除数据的key wait 是否同步删除

func (*MultiCache) Get 

func (g *MultiCache) Get(key string, ttlSecond int, obj interface{}, fn LoadDataSourceFunc) error

Get 获取数据 key 数据对应key ttlSecond 超时时间 obj 赋值对象,外部传入obj,obj会被深拷贝为对于的数据 fn 加载数据的函数

func (*MultiCache) Set 

func (g *MultiCache) Set(key string, obj interface{}, ttlSecond int, wait bool) (err error)

Set 存入数据 key 存入数据对应key obj 存入数据对应值 wait 是否同步等待数据存入 true则等待数据存入完成同步返回,false异步存入数据

type OutCache 

type OutCache interface {
	Get(key string, obj interface{}) (*Entry, bool, error) // 返回真实数据
	Set(key string, e *Entry) error                        // 外部缓存需要设置过期时间
	Del(key string) error                                  // 删除指定Key对应数据
	Close()                                                // 关闭缓存数据源
}

OutCache 缓存数据源,如Redis

type PubSub 

type PubSub interface {
	Subscribe(data chan<- *ChannelMeta) error
	Publish(cacheID, key string, action int8, data *Entry) error
}

PubSub 缓存数据源发布订阅接口

type RedisCache 

type RedisCache struct {
	// contains filtered or unexported fields
}

func NewRedisCache 

func NewRedisCache() (*RedisCache, error)

func (*RedisCache) Close 

func (r *RedisCache) Close()

func (*RedisCache) Del 

func (r *RedisCache) Del(key string) error

func (*RedisCache) DistributedEnable 

func (r *RedisCache) DistributedEnable() bool

func (*RedisCache) Get 

func (r *RedisCache) Get(key string, obj interface{}) (*Entry, bool, error)

func (*RedisCache) Publish 

func (r *RedisCache) Publish(gid, key string, action int8, value *Entry) error

func (*RedisCache) Set 

func (r *RedisCache) Set(key string, obj *Entry) error

func (*RedisCache) Subscribe 

func (r *RedisCache) Subscribe(ch chan<- *ChannelMeta) error

Subscribe Redis实现的发布订阅获取数据函数

type RedisConf 

type RedisConf struct {
	DSN     string
	Pwd     string
	DB      int
	MaxConn int
}
var DefaultPubSubRedisConf RedisConf
var DefaultRedisConf RedisConf

type WorkerPool 

type WorkerPool struct {
	// contains filtered or unexported fields
}

func NewPool 

func NewPool(numWorkers int, jobQueueLen int) *WorkerPool

NewPool 创建缓存池 numWorkers 创建的工作线程数量 queueLen 阻塞时能接受多少任务

func (*WorkerPool) Release 

func (p *WorkerPool) Release()

Release 关闭线程池,上锁,多线程只允许一次

func (*WorkerPool) SendJob 

func (p *WorkerPool) SendJob(job func())

SendJob 运行任务

func (*WorkerPool) SendJobWithDeadline 

func (p *WorkerPool) SendJobWithDeadline(job func(), t time.Time) bool

SendJobWithDeadline 运行任务,在期限内完成,没有完成则返回false

func (*WorkerPool) SendJobWithTimeout 

func (p *WorkerPool) SendJobWithTimeout(job func(), t time.Duration) bool

SendJobWithTimeout 运行任务,在t时间内没有完成则返回false

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.