README
¶
BenchmarkRead
$ go test -bench Read$ -count 3
Size of table: 105843444
BenchmarkRead-8 3 343846914 ns/op
BenchmarkRead-8 3 351790907 ns/op
BenchmarkRead-8 3 351762823 ns/op
Size of table is 105,843,444 bytes, which is ~101M.
The rate is ~287M/s which matches our read speed. This is using mmap.
To read a 64M table, this would take ~0.22s, which is negligible.
$ go test -bench BenchmarkReadAndBuild -count 3
BenchmarkReadAndBuild-8 1 2341034225 ns/op
BenchmarkReadAndBuild-8 1 2346349671 ns/op
BenchmarkReadAndBuild-8 1 2364064576 ns/op
The rate is ~43M/s. To build a ~64M table, this would take ~1.5s. Note that this does NOT include the flushing of the table to disk. All we are doing above is to read one table (mmaped) and write one table in memory.
The table building takes 1.5-0.22 ~ 1.3s.
If we are writing out up to 10 tables, this would take 1.5*10 ~ 15s, and ~13s is spent building the tables.
When running populate, building one table in memory tends to take ~1.5s to ~2.5s on my system. Where does this overhead come from? Let's investigate the merging.
Below, we merge 5 tables. The total size remains unchanged at ~101M.
$ go test -bench ReadMerged -count 3
BenchmarkReadMerged-8 1 1321190264 ns/op
BenchmarkReadMerged-8 1 1296958737 ns/op
BenchmarkReadMerged-8 1 1314381178 ns/op
The rate is ~76M/s. To build a 64M table, this would take ~0.84s. The writing takes ~1.3s as we saw above. So in total, we expect around 0.84+1.3 ~ 2.1s. This roughly matches what we observe when running populate. There might be some additional overhead due to the concurrent writes going on, in flushing the table to disk. Also, the tables tend to be slightly bigger than 64M/s.
Documentation
¶
Index ¶
- func IDToFilename(id uint64) string
- func IndexFilename(tableFilename string) string
- func NewFilename(id uint64, dir string) string
- func OnEvict(key uint64, value interface{})
- func ParseFileID(name string) (uint64, bool)
- type BuildResult
- type Builder
- func (b *Builder) Add(key y.Key, value y.ValueStruct) error
- func (b *Builder) Close()
- func (b *Builder) Empty() bool
- func (b *Builder) EstimateSize() int
- func (b *Builder) Finish() (*BuildResult, error)
- func (b *Builder) ReachedCapacity(capacity int64) bool
- func (b *Builder) Reset(f *os.File)
- func (b *Builder) SetIsManaged()
- type Iterator
- func (itr *Iterator) Close() error
- func (itr *Iterator) Error() error
- func (itr *Iterator) FillValue(vs *y.ValueStruct)
- func (itr *Iterator) Key() y.Key
- func (itr *Iterator) Next()
- func (itr *Iterator) NextVersion() bool
- func (itr *Iterator) Rewind()
- func (itr *Iterator) Seek(key []byte)
- func (itr *Iterator) Valid() bool
- func (itr *Iterator) Value() (ret y.ValueStruct)
- type Table
- func (t *Table) Biggest() y.Key
- func (t *Table) Close() error
- func (t *Table) CompressionType() options.CompressionType
- func (t *Table) Delete() error
- func (t *Table) Filename() string
- func (t *Table) Get(key y.Key, keyHash uint64) (y.ValueStruct, error)
- func (t *Table) HasGlobalTs() bool
- func (t *Table) HasOverlap(start, end y.Key, includeEnd bool) bool
- func (t *Table) ID() uint64
- func (t *Table) IsCompacting() bool
- func (t *Table) MarkCompacting(flag bool)
- func (t *Table) NewIterator(reversed bool) y.Iterator
- func (t *Table) SetGlobalTs(ts uint64) error
- func (t *Table) Size() int64
- func (t *Table) Smallest() y.Key
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func IDToFilename ¶
IDToFilename does the inverse of ParseFileID
func IndexFilename ¶
func NewFilename ¶
NewFilename should be named TableFilepath -- it combines the dir with the ID to make a table filepath.
func ParseFileID ¶
ParseFileID reads the file id out of a filename.
Types ¶
type BuildResult ¶
BuildResult contains the build result info, if it's file based compaction, fileName should be used to open Table. If it's in memory compaction, FileData and IndexData contains the data.
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
Builder is used in building a table.
func NewExternalTableBuilder ¶
func NewExternalTableBuilder(f *os.File, limiter *rate.Limiter, opt options.TableBuilderOptions, compression options.CompressionType) *Builder
func NewTableBuilder ¶
func NewTableBuilder(f *os.File, limiter *rate.Limiter, level int, opt options.TableBuilderOptions) *Builder
NewTableBuilder makes a new TableBuilder. If the f is nil, the builder builds in-memory result. If the limiter is nil, the write speed during table build will not be limited.
func (*Builder) Add ¶
Add adds a key-value pair to the block. If doNotRestart is true, we will not restart even if b.counter >= restartInterval.
func (*Builder) EstimateSize ¶
EstimateSize returns the size of the SST to build.
func (*Builder) Finish ¶
func (b *Builder) Finish() (*BuildResult, error)
Finish finishes the table by appending the index.
func (*Builder) ReachedCapacity ¶
ReachedCapacity returns true if we... roughly (?) reached capacity?
func (*Builder) SetIsManaged ¶
func (b *Builder) SetIsManaged()
SetIsManaged should be called when ingesting a table into a managed DB.
type Iterator ¶
type Iterator struct {
// contains filtered or unexported fields
}
Iterator is an iterator for a Table.
func (*Iterator) FillValue ¶
func (itr *Iterator) FillValue(vs *y.ValueStruct)
FillValue fill the value struct.
func (*Iterator) NextVersion ¶
func (*Iterator) Value ¶
func (itr *Iterator) Value() (ret y.ValueStruct)
Value follows the y.Iterator interface
type Table ¶
Table represents a loaded table file with the info we have about it
func OpenInMemoryTable ¶
OpenInMemoryTable opens a table that has data in memory.
func OpenTable ¶
OpenTable assumes file has only one table and opens it. Takes ownership of fd upon function entry. Returns a table with one reference count on it (decrementing which may delete the file! -- consider t.Close() instead). The fd has to writeable because we call Truncate on it before deleting.
func (*Table) CompressionType ¶
func (t *Table) CompressionType() options.CompressionType
CompressionType returns the compression algorithm used for block compression.
func (*Table) HasGlobalTs ¶
HasGlobalTs returns table does set global ts.
func (*Table) IsCompacting ¶
func (*Table) MarkCompacting ¶
func (*Table) SetGlobalTs ¶
SetGlobalTs update the global ts of external ingested tables.
Source Files