Documentation
¶
Index ¶
- Constants
- Variables
- func DateToTime(d uint32) time.Time
- func NormalizeTTL(t time.Time) time.Time
- func RecordsEqualish(a, b Record) bool
- func TimeToDateDown(t time.Time) uint32
- func TimeToDateUp(t time.Time) uint32
- type DB
- type DBStats
- type Expiration
- type HashTbl
- func (h *HashTbl) Close()
- func (h *HashTbl) ExpectOrdered() (flush func() error, done func(), err error)
- func (h *HashTbl) Insert(rec Record) (_ bool, err error)
- func (h *HashTbl) Load() float64
- func (h *HashTbl) Lookup(key Key) (_ Record, _ bool, err error)
- func (h *HashTbl) Range(fn func(Record, error) bool)
- func (h *HashTbl) Stats() HashTblStats
- type HashTblStats
- type Key
- type Reader
- func (l *Reader) Close() error
- func (l *Reader) Key() Key
- func (l *Reader) Read(p []byte) (int, error)
- func (l *Reader) ReadAt(p []byte, off int64) (int, error)
- func (l *Reader) Release()
- func (l *Reader) Seek(offset int64, whence int) (int64, error)
- func (l *Reader) Size() int64
- func (l *Reader) Trash() bool
- type Record
- type Store
- func (s *Store) Close()
- func (s *Store) Compact(ctx context.Context, ...) (err error)
- func (s *Store) Create(ctx context.Context, key Key, expires time.Time) (w *Writer, err error)
- func (s *Store) Load() float64
- func (s *Store) Read(ctx context.Context, key Key) (_ *Reader, err error)
- func (s *Store) Stats() StoreStats
- type StoreStats
- type Writer
Constants ¶
const (
// RecordSize is the size of a serialized record in bytes.
RecordSize = 64
)
Variables ¶
var ( // Error is the class that wraps all errors generated by the // hashstore package. Error = errs.Class("hashstore") // ErrCollision represents collision errors returned while // committing to the store. ErrCollision = errs.New("collision detected") )
Functions ¶
func DateToTime ¶ added in v1.119.2
DateToTime returns the earliest time in the day for the given date.
func NormalizeTTL ¶ added in v1.119.2
NormalizeTTL takes a time and returns a time that is an output of DateToTime that is larger than or equal to the input time, for all times before the year 24937. In other words, it rounds up to the closest time representable for a TTL, and rounds down if no such time is possible (i.e. times after year 24937).
func RecordsEqualish ¶
RecordsEqualish returns true if the records are equalish. Records are equalish if they are equal except for the expires time and checksums.
func TimeToDateDown ¶ added in v1.119.2
TimeToDateDown returns a number of days past the epoch that is less than or equal to t.
func TimeToDateUp ¶ added in v1.119.2
TimeToDateUp returns a number of days past the epoch that is greater than or equal to t.
Types ¶
type DB ¶
type DB struct {
// contains filtered or unexported fields
}
DB is a database that stores pieces.
func New ¶
func New( dir string, log *zap.Logger, shouldTrash func(context.Context, Key, time.Time) bool, lastRestore func(context.Context) time.Time, ) (_ *DB, err error)
New makes or opens an existing database in the directory allowing for nlogs concurrent writes.
func (*DB) Close ¶
func (d *DB) Close()
Close closes down the database and blocks until all background processes have stopped.
func (*DB) Compact ¶ added in v1.119.2
Compact waits for any background compaction to finish and then calls Compact on both stores. After a call to Compact, you can be sure that each Store was fully compacted at least once.
func (*DB) Create ¶
Create adds an entry to the database with the given key and expiration time. Close or Cancel must be called on the Writer when done. It is safe to call either of them multiple times.
func (*DB) Read ¶
Read returns a reader for the given key. If the key is not present the returned Reader will be nil and the error will be a wrapped fs.ErrNotExist. Close must be called on the non-nil Reader when done.
func (*DB) Stats ¶
func (d *DB) Stats() (DBStats, StoreStats, StoreStats)
Stats returns statistics about the database and underlying stores.
type DBStats ¶
type DBStats struct {
NumSet uint64 // number of set records.
LenSet uint64 // sum of lengths in set records.
AvgSet float64 // average size of length of records.
NumTrash uint64 // number of set trash records.
LenTrash uint64 // sum of lengths in set trash records.
AvgTrash float64 // average size of length of trash records.
NumSlots uint64 // total number of records available.
TableSize uint64 // total number of bytes in the hash table.
Load float64 // percent of slots that are set.
NumLogs uint64 // total number of log files.
LenLogs uint64 // total number of bytes in the log files.
NumLogsTTL uint64 // total number of log files with ttl set.
LenLogsTTL uint64 // total number of bytes in log files with ttl set.
SetPercent float64 // percent of bytes that are set in the log files.
TrashPercent float64 // percent of bytes that are trash in the log files.
Compacting bool // if true, a background compaction is in progress.
Compactions uint64 // total number of compactions that finished on either store.
Active int // which store is currently active
}
DBStats is a collection of statistics about a database.
type Expiration ¶
type Expiration uint32
Expiration is a 23-bit timestamp with a 1-bit flag for trash.
func MaxExpiration ¶
func MaxExpiration(a, b Expiration) Expiration
MaxExpiration returns the larger of two expirations. All expirations with the trash bit set are larger than expirations without the trash bit set. If both have the same trash bit setting, then the larger of the timestamps is returned. An unset expiration is always largest.
func NewExpiration ¶
func NewExpiration(t uint32, trash bool) Expiration
NewExpiration constructs an Expiration from a timestamp and a trash flag.
func (Expiration) Set ¶
func (e Expiration) Set() bool
Set returns true if the expiration is set: not zero.
func (Expiration) Time ¶
func (e Expiration) Time() uint32
Time returns the timestamp part of the expiration.
func (Expiration) Trash ¶
func (e Expiration) Trash() bool
Trash returns true if the trash bit is set.
type HashTbl ¶
type HashTbl struct {
// contains filtered or unexported fields
}
HashTbl is an on disk hash table of records.
func CreateHashtbl ¶
CreateHashtbl allocates a new hash table with the given log base 2 number of records and created timestamp. The file is truncated and allocated to the correct size.
func OpenHashtbl ¶
OpenHashtbl opens an existing hash table stored in the given file handle.
func (*HashTbl) Close ¶
func (h *HashTbl) Close()
Close closes the hash table and returns when no more operations are running.
func (*HashTbl) ExpectOrdered ¶ added in v1.120.1
ExpectOrdered signals that incoming writes to the hashtbl will be ordered so that a large shared buffer across Insert calls would be effective. This is useful when rewriting a hashtbl during a Compaction, for instance. It returns a flush callback that both flushes any potentially buffered records and disables the expectation. Additionally, Lookups may not find entries written until after the flush callback is called. If flush returns an error there is no guarantee about what records were written. It returns a done callback that discards any potentially buffered records and disables the expectation. At least one of flush or done must be called. It returns an error if called again before flush or done is called.
func (*HashTbl) Insert ¶
Insert adds a record to the hash table. It returns (true, nil) if the record was inserted, it returns (false, nil) if the hash table is full, and (false, err) if any errors happened trying to insert the record.
func (*HashTbl) Lookup ¶
Lookup returns the record for the given key if it exists in the hash table. It returns (rec, true, nil) if the record existed, (rec{}, false, nil) if it did not exist, and (rec{}, false, err) if any errors happened trying to look up the record.
func (*HashTbl) Stats ¶
func (h *HashTbl) Stats() HashTblStats
Stats returns a HashTblStats about the hash table.
type HashTblStats ¶
type HashTblStats struct {
NumSet uint64 // number of set records.
LenSet uint64 // sum of lengths in set records.
AvgSet float64 // average size of length of records.
NumTrash uint64 // number of set trash records.
LenTrash uint64 // sum of lengths in set trash records.
AvgTrash float64 // average size of length of trash records.
NumSlots uint64 // total number of records available.
TableSize uint64 // total number of bytes in the hash table.
Load float64 // percent of slots that are set.
Created uint32 // date that the hashtbl was created.
}
HashTblStats contains statistics about the hash table.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader is a type that reads a section from a log file.
func (*Reader) Close ¶
Close is like Release but implements io.Closer. The returned error is always nil.
func (*Reader) Release ¶
func (l *Reader) Release()
Release returns the resources associated with the reader. It must be called when done.
type Record ¶
type Record struct {
Key Key // 256 bits (32b) of key
Offset uint64 // 48 bits (6b) of offset (256TB max file size)
Log uint64 // 64 bits (8b) of log id (effectively unlimited number of logs)
Length uint32 // 32 bits (4b) of length (4GB max piece size)
Created uint32 // 23 bits (3b) of days since epoch (~22900 years), 1 bit reserved
Expires Expiration // 23 bits (3b) of days since epoch (~22900 years), 1 bit flag for trash
}
Record contains metadata about a piece stored in a hash table.
func (*Record) ReadFrom ¶
func (r *Record) ReadFrom(buf *[RecordSize]byte) bool
ReadFrom updates the record with the values from the buffer and returns true if the checksum is valid.
func (*Record) WriteTo ¶
func (r *Record) WriteTo(buf *[RecordSize]byte)
WriteTo stores the record and its checksum into the buffer.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is a hash table based key-value store with compaction.
func (*Store) Close ¶
func (s *Store) Close()
Close interrupts any compactions and closes the store.
func (*Store) Compact ¶
func (s *Store) Compact( ctx context.Context, shouldTrash func(ctx context.Context, key Key, created time.Time) bool, lastRestore time.Time, ) (err error)
Compact removes keys and files that are definitely expired, and marks keys that are determined trash by the callback to expire in the future. It also rewrites any log files that have too much dead data.
func (*Store) Create ¶
Create returns a Handle that writes data to the store. The error on Close must be checked. Expires is when the data expires, or zero if it never expires.
func (*Store) Load ¶
Load returns the estimated load factor of the hash table. If it's too large, a Compact call is indicated.
func (*Store) Read ¶
Read returns a Reader that reads data from the store. The Reader will be nil if the key does not exist.
func (*Store) Stats ¶
func (s *Store) Stats() StoreStats
Stats returns a StoreStats about the store.
type StoreStats ¶
type StoreStats struct {
NumLogs uint64 // total number of log files.
LenLogs uint64 // total number of bytes in the log files.
NumLogsTTL uint64 // total number of log files with ttl set.
LenLogsTTL uint64 // total number of bytes in log files with ttl set.
SetPercent float64 // percent of bytes that are set in the log files.
TrashPercent float64 // percent of bytes that are trash in the log files.
Compacting bool // if true, a compaction is in progress.
Compactions uint64 // number of compaction calls that finished
TableFull uint64 // number of times the hashtbl was full trying to insert
Today uint32 // the current date.
LastCompact uint32 // the date of the last compaction.
Table HashTblStats // stats about the hash table.
Compaction struct {
Elapsed float64 // number of seconds elapsed in the compaction
Remaining float64 // estimated number of seconds remaining in the compaction
TotalRecords uint64 // total number of records expected to be processed in the compaction
ProcessedRecords uint64 // total number of records processed in the compaction
}
}
StoreStats is a collection of statistics about a store.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer is a type that allows one to write a piece to a log file.
func (*Writer) Cancel ¶
func (h *Writer) Cancel()
Cancel discards the writes that have happened. Close or Cancel must be called at least once.
func (*Writer) Close ¶
Close commits the writes that have happened. Close or Cancel must be called at least once.
Source Files