Documentation
¶
Overview ¶
Package datafile contains structures for building and reading static mappings from keys to data that may be too big to fit in memory or too expensive to compute on each process start.
A datafile generally looks like:
┌───────────────────┐ │ file header │ ├───────────────────┤ │ repeated KV pairs │ │ │ │ │ │ │ │ │ │ │ │ │ ├───────────────────┤ │ padding │ ├───────────────────┤ │ minimal perfect │ │ hashmap │ │ │ └───────────────────┘
Individual KV pairs stat with a fixed 7-byte header and are variable length. An entry for a 4-byte key and 15-byte value would look like:
0 1 2 3 4 5 6 7 8 9 A B C D E F +----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+ | value checksum |klen| vlen |key | value... | +----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+ | value... | +----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
This gives us a 255-byte max length for keys, and a 65-KB max length for values. The checksum is calculated from the bytes of the value, and is used to ensure we don't have un-detected on-disk corruption (with high probability).
Index ¶
Constants ¶
const (
MaxKeyLen = (1 << 8) - 1
)
Variables ¶
var (
InvalidOffset = errors.New("invalid offset")
)
Functions ¶
This section is empty.
Types ¶
type FileWriter ¶
FileWriter is usually an *os.File, but specified as an interface for easier testing.
type Iter ¶
type Iter interface {
Close()
Len() int64
ReadAt(off PackedOffset) (key []byte, value []byte, err error)
Next() (IterItem, bool)
}
Iter iterates over the contents in a logfile. Make sure to `defer it.Close()`.
type IterItem ¶
func (IterItem) PackedOffset ¶
func (ii IterItem) PackedOffset() PackedOffset
type MmapReader ¶
type MmapReader struct {
// contains filtered or unexported fields
}
func NewMMapReaderWithPath ¶
func NewMMapReaderWithPath(path string) (*MmapReader, error)
func (*MmapReader) Index ¶
func (r *MmapReader) Index() (level0Count, levellCount uint64, indexBytes []byte)
func (*MmapReader) Iter ¶
func (r *MmapReader) Iter() Iter
func (*MmapReader) Len ¶
func (r *MmapReader) Len() int64
func (*MmapReader) ReadAt ¶
func (r *MmapReader) ReadAt(poff PackedOffset) (key, value []byte, err error)
type OsFileReader ¶
type OsFileReader struct {
// contains filtered or unexported fields
}
func NewOsFileReader ¶
func NewOsFileReader(path string) (*OsFileReader, error)
func (*OsFileReader) Close ¶
func (r *OsFileReader) Close() error
func (*OsFileReader) Index ¶
func (r *OsFileReader) Index() (level0Count, levellCount uint64, indexBytes []byte, err error)
Index will return index metadata and the index bytes on-heap (not mmap'ed).
func (*OsFileReader) ReadAt ¶
func (r *OsFileReader) ReadAt(poff PackedOffset) (key, value []byte, err error)
type PackedOffset ¶
type PackedOffset uint64
PackedOffset packs datafile offset + record length into a 64-bit value
func NewPackedOffset ¶
func NewPackedOffset(off uint64, keyLen uint8, valueLen uint16) PackedOffset
func (PackedOffset) Unpack ¶
func (po PackedOffset) Unpack() (off int64, recordLen uint64)
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
func NewWriter ¶
func NewWriter(f FileWriter) (*Writer, error)
Source Files