Documentation
¶
Index ¶
- Constants
- Variables
- func DateToTime(d uint32) time.Time
- func NormalizeTTL(t time.Time) time.Time
- func RecordsEqualish(a, b Record) bool
- func TimeToDateDown(t time.Time) uint32
- func TimeToDateUp(t time.Time) uint32
- type DB
- type DBStats
- type Expiration
- type HashTbl
- func (h *HashTbl) Close()
- func (h *HashTbl) Insert(ctx context.Context, rec Record) (_ bool, err error)
- func (h *HashTbl) Load() float64
- func (h *HashTbl) Lookup(ctx context.Context, key Key) (_ Record, _ bool, err error)
- func (h *HashTbl) Range(ctx context.Context, fn func(Record, error) bool)
- func (h *HashTbl) Stats() HashTblStats
- type HashTblStats
- type Key
- type Reader
- func (l *Reader) Close() error
- func (l *Reader) Key() Key
- func (l *Reader) Read(p []byte) (int, error)
- func (l *Reader) ReadAt(p []byte, off int64) (int, error)
- func (l *Reader) Release()
- func (l *Reader) Seek(offset int64, whence int) (int64, error)
- func (l *Reader) Size() int64
- func (l *Reader) Trash() bool
- type Record
- type Store
- func (s *Store) Close()
- func (s *Store) Compact(ctx context.Context, ...) (err error)
- func (s *Store) Create(ctx context.Context, key Key, expires time.Time) (_ *Writer, err error)
- func (s *Store) Load() float64
- func (s *Store) Read(ctx context.Context, key Key) (_ *Reader, err error)
- func (s *Store) Stats() StoreStats
- type StoreStats
- type Writer
Constants ¶
const (
// RecordSize is the size of a serialized record in bytes.
RecordSize = 64
)
Variables ¶
var Error = errs.Class("hashstore")
Error is the class that wraps all errors generated by the hashstore package.
Functions ¶
func DateToTime ¶ added in v1.119.2
DateToTime returns the earliest time in the day for the given date.
func NormalizeTTL ¶ added in v1.119.2
NormalizeTTL takes a time and returns a time that is an output of DateToTime that is larger than or equal to the input time, for all times before the year 24937. In other words, it rounds up to the closest time representable for a TTL, and rounds down if no such time is possible (i.e. times after year 24937).
func RecordsEqualish ¶
RecordsEqualish returns true if the records are equalish. Records are equalish if they are equal except for the expires time and checksums.
func TimeToDateDown ¶ added in v1.119.2
TimeToDateDown returns a number of days past the epoch that is less than or equal to t.
func TimeToDateUp ¶ added in v1.119.2
TimeToDateUp returns a number of days past the epoch that is greater than or equal to t.
Types ¶
type DB ¶
type DB struct {
// contains filtered or unexported fields
}
DB is a database that stores pieces.
func New ¶
func New( dir string, log *zap.Logger, shouldTrash func(context.Context, Key, time.Time) bool, lastRestore func(context.Context) time.Time, ) (_ *DB, err error)
New makes or opens an existing database in the directory allowing for nlogs concurrent writes.
func (*DB) Close ¶
func (d *DB) Close()
Close closes down the database and blocks until all background processes have stopped.
func (*DB) Compact ¶ added in v1.119.2
Compact waits for any background compaction to finish and then calls Compact on both stores. After a call to Compact, you can be sure that each Store was fully compacted at least once.
func (*DB) Create ¶
Create adds an entry to the database with the given key and expiration time. Close or Cancel must be called on the Writer when done. It is safe to call either of them multiple times.
type DBStats ¶
type DBStats struct {
NumSet uint64 // number of set records.
LenSet uint64 // sum of lengths in set records.
AvgSet float64 // average size of length of records.
NumTrash uint64 // number of set trash records.
LenTrash uint64 // sum of lengths in set trash records.
AvgTrash float64 // average size of length of trash records.
NumSlots uint64 // total number of records available.
TableSize uint64 // total number of bytes in the hash table.
Load float64 // percent of slots that are set.
NumLogs uint64 // total number of log files.
LenLogs uint64 // total number of bytes in the log files.
NumLogsTTL uint64 // total number of log files with ttl set.
LenLogsTTL uint64 // total number of bytes in log files with ttl set.
SetPercent float64 // percent of bytes that are set in the log files.
TrashPercent float64 // percent of bytes that are trash in the log files.
Compacting bool // if true, a background compaction is in progress.
Compactions uint64 // total number of compactions that finished on either store.
Active int // which store is currently active
S0 StoreStats // statistics about the s0 store.
S1 StoreStats // statistics about the s1 store.
}
DBStats is a collection of statistics about a database.
type Expiration ¶
type Expiration uint32
Expiration is a 23-bit timestamp with a 1-bit flag for trash.
func MaxExpiration ¶
func MaxExpiration(a, b Expiration) Expiration
MaxExpiration returns the larger of two expirations. All expirations with the trash bit set are larger than expirations without the trash bit set. If both have the same trash bit setting, then the larger of the timestamps is returned. An unset expiration is always largest.
func NewExpiration ¶
func NewExpiration(t uint32, trash bool) Expiration
NewExpiration constructs an Expiration from a timestamp and a trash flag.
func (Expiration) Set ¶
func (e Expiration) Set() bool
Set returns true if the expiration is set: not zero.
func (Expiration) Time ¶
func (e Expiration) Time() uint32
Time returns the timestamp part of the expiration.
func (Expiration) Trash ¶
func (e Expiration) Trash() bool
Trash returns true if the trash bit is set.
type HashTbl ¶
type HashTbl struct {
// contains filtered or unexported fields
}
HashTbl is an on disk hash table of records.
func CreateHashtbl ¶
CreateHashtbl allocates a new hash table with the given log base 2 number of records and created timestamp. The file is truncated and allocated to the correct size.
func OpenHashtbl ¶
OpenHashtbl opens an existing hash table stored in the given file handle.
func (*HashTbl) Close ¶
func (h *HashTbl) Close()
Close closes the hash table and returns when no more operations are running.
func (*HashTbl) Insert ¶
Insert adds a record to the hash table. It returns (true, nil) if the record was inserted, it returns (false, nil) if the hash table is full, and (false, err) if any errors happened trying to insert the record.
func (*HashTbl) Lookup ¶
Lookup returns the record for the given key if it exists in the hash table. It returns (rec, true, nil) if the record existed, (rec{}, false, nil) if it did not exist, and (rec{}, false, err) if any errors happened trying to look up the record.
func (*HashTbl) Stats ¶
func (h *HashTbl) Stats() HashTblStats
Stats returns a HashTblStats about the hash table.
type HashTblStats ¶
type HashTblStats struct {
NumSet uint64 // number of set records.
LenSet uint64 // sum of lengths in set records.
AvgSet float64 // average size of length of records.
NumTrash uint64 // number of set trash records.
LenTrash uint64 // sum of lengths in set trash records.
AvgTrash float64 // average size of length of trash records.
NumSlots uint64 // total number of records available.
TableSize uint64 // total number of bytes in the hash table.
Load float64 // percent of slots that are set.
Created uint32 // date that the hashtbl was created.
}
HashTblStats contains statistics about the hash table.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader is a type that reads a section from a log file.
func (*Reader) Close ¶
Close is like Release but implements io.Closer. The returned error is always nil.
func (*Reader) Release ¶
func (l *Reader) Release()
Release returns the resources associated with the reader. It must be called when done.
type Record ¶
type Record struct {
Key Key // 256 bits (32b) of key
Offset uint64 // 48 bits (6b) of offset (256TB max file size)
Log uint64 // 64 bits (8b) of log id (effectively unlimited number of logs)
Length uint32 // 32 bits (4b) of length (4GB max piece size)
Created uint32 // 23 bits (3b) of days since epoch (~22900 years), 1 bit reserved
Expires Expiration // 23 bits (3b) of days since epoch (~22900 years), 1 bit flag for trash
}
Record contains metadata about a piece stored in a hash table.
func (*Record) ReadFrom ¶
func (r *Record) ReadFrom(buf *[RecordSize]byte) bool
ReadFrom updates the record with the values from the buffer and returns true if the checksum is valid.
func (*Record) WriteTo ¶
func (r *Record) WriteTo(buf *[RecordSize]byte)
WriteTo stores the record and its checksum into the buffer.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is a hash table based key-value store with compaction.
func (*Store) Close ¶
func (s *Store) Close()
Close interrupts any compactions and closes the store.
func (*Store) Compact ¶
func (s *Store) Compact( ctx context.Context, shouldTrash func(ctx context.Context, key Key, created time.Time) bool, lastRestore time.Time, ) (err error)
Compact removes keys and files that are definitely expired, and marks keys that are determined trash by the callback to expire in the future. It also rewrites any log files that have too much dead data.
func (*Store) Create ¶
Create returns a Handle that writes data to the store. The error on Close must be checked. Expires is when the data expires, or zero if it never expires.
func (*Store) Load ¶
Load returns the estimated load factor of the hash table. If it's too large, a Compact call is indicated.
func (*Store) Read ¶
Read returns a Reader that reads data from the store. The Reader will be nil if the key does not exist.
func (*Store) Stats ¶
func (s *Store) Stats() StoreStats
Stats returns a StoreStats about the store.
type StoreStats ¶
type StoreStats struct {
NumLogs uint64 // total number of log files.
LenLogs uint64 // total number of bytes in the log files.
NumLogsTTL uint64 // total number of log files with ttl set.
LenLogsTTL uint64 // total number of bytes in log files with ttl set.
SetPercent float64 // percent of bytes that are set in the log files.
TrashPercent float64 // percent of bytes that are trash in the log files.
Compacting bool // if true, a compaction is in progress.
Compactions uint64 // number of compaction calls that finished
TableFull uint64 // number of times the hashtbl was full trying to insert
Today uint32 // the current date.
LastCompact uint32 // the date of the last compaction.
Table HashTblStats // stats about the hash table.
}
StoreStats is a collection of statistics about a store.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer is a type that allows one to write a piece to a log file.
func (*Writer) Cancel ¶
func (h *Writer) Cancel()
Cancel discards the writes that have happened. Close or Cancel must be called at least once.
func (*Writer) Close ¶
Close commits the writes that have happened. Close or Cancel must be called at least once.
Source Files