packed

package
v0.0.0-...-53ff736 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 27, 2024 License: Apache-2.0 Imports: 14 Imported by: 2

README

packed

Packed integer arrays and streams.

The packed package provides

  • sequential and random access capable arrays of positive longs,
  • routines for efficient serialization and deserialization of streams of packed integers.

The implementations provide different trade-offs between memory usage and access speed. The standard usage scenario is replacing large int or long arrays in order to reduce the memory footprint.

In-memory structures

Mutable
  • Only supports positive longs.
  • Requires the number of bits per value to be known in advance.
  • Random-access for both writing and reading.
GrowableWriter
  • Same as PackedInts.Mutable but grows the number of bits per values when needed.
  • Useful to build a PackedInts.Mutable from a read-once stream of longs.
PagedGrowableWriter
  • Slices data into fixed-size blocks stored in GrowableWriters.
  • Supports more than 2B values.
  • You should use PackedLongValues instead if you don't need random write access.
PackedLongValues.deltaPackedBuilder
  • Can store any sequence of longs.
  • Compression is good when values are close to each other.
  • Supports random reads, but only sequential writes.
  • Can address up to 2^42 values.
PackedLongValues.packedBuilder
  • Same as deltaPackedBuilder but assumes values are 0-based.
PackedLongValues.monotonicBuilder
  • Same as deltaPackedBuilder except that compression is good when the stream is a succession of affine functions.

Disk-based structures

Writer/Reader/ReaderIterator
  • Only supports positive longs.
  • Requires the number of bits per value to be known in advance.
  • Splits the stream into fixed-size blocks.
  • Supports both fast sequential access with low memory footprint with ReaderIterator and random-access by either loading values in memory or leaving them on disk with Reader.
BlockPackedWriter/BlockPackedReader/BlockPackedReaderIterator
  • Splits the stream into fixed-size blocks.
  • Compression is good when values are close to each other.
  • Can address up to 2B - blockSize values.
MonotonicBlockPackedWriter/MonotonicBlockPackedReader
  • Same as the non-monotonic variants except that compression is good when the stream is a succession of affine functions.
  • The reason why there is no sequential access is that if you need sequential access, you should rather delta-encode and use BlockPackedWriter.
PackedDataOutput/PackedDataInput
  • Writes sequences of longs where each long can use any number of bits.

Documentation

Index

Constants

View Source
const (
	ABP_MIN_BLOCK_SIZE     = 64
	ABP_MAX_BLOCK_SIZE     = 1 << (30 - 3)
	ABP_MIN_VALUE_EQUALS_0 = 1 << 0
	ABP_BPV_SHIFT          = 1
)
View Source
const (
	Packed64BlockSize = 64                            // 32 = int, 64 = long
	Packed64BlockBits = 6                             // The #bits representing BLOCK_SIZE
	MOD_MASK          = uint64(Packed64BlockSize - 1) // x % BLOCK_SIZE
)
View Source
const (
	// FASTEST At most 700% memory overhead, always select a direct implementation.
	FASTEST = 7.0

	// FAST At most 50% memory overhead, always select a reasonably fast implementation.
	FAST = 0.5

	// DEFAULT At most 25% memory overhead.
	DEFAULT = 0.25

	// COMPACT No memory overhead at all, but the returned implementation may be slow.
	COMPACT = 0.0

	// DEFAULT_BUFFER_SIZE Default amount of memory to use for bulk operations.
	DEFAULT_BUFFER_SIZE = 1024

	CODEC_NAME = "PackedInts"

	VERSION_MONOTONIC_WITHOUT_ZIGZAG = 2
	VERSION_START                    = VERSION_MONOTONIC_WITHOUT_ZIGZAG
	VERSION_CURRENT                  = VERSION_MONOTONIC_WITHOUT_ZIGZAG
)
View Source
const (
	MIN_PAGE_SIZE      = 64
	MAX_PAGE_SIZE      = 1 << 20
	INITIAL_PAGE_COUNT = 16
)
View Source
const (
	MIN_BLOCK_SIZE = 1 << 6
	MAX_BLOCK_SIZE = 1 << 30
)
View Source
const (
	Packed16ThreeBlocks_MAX_SIZE = math.MaxInt32 / 3
)
View Source
const (
	Packed8ThreeBlocks_MAX_SIZE = math.MaxInt32 / 3
)

Variables

View Source
var (
	FormatPacked            = newFormatPacked(0)
	FormatPackedSingleBlock = newFormatPackedSingleBlock(1)
)
View Source
var (
	//SUPPORTED_BITS_PER_VALUE = []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 16, 21, 32}
	SUPPORTED_BITS_PER_VALUE = map[int]bool{
		1: true, 2: true, 3: true, 4: true, 5: true, 6: true, 7: true, 8: true, 9: true, 10: true, 12: true, 16: true, 21: true, 32: true,
	}
)

Functions

func BitsRequired

func BitsRequired(maxValue int64) (int, error)

BitsRequired Returns how many bits are required to hold values up to and including maxValue NOTE: This method returns at least 1. Params: maxValue – the maximum value that should be representable. Returns: the amount of bits needed to represent values from 0 to maxValue. lucene.internal

func CloneI64ToU64

func CloneI64ToU64(src []int64, dest []uint64) int

func CopyValues

func CopyValues(src Reader, srcPos int, dest Mutable, destPos, size, mem int)

CopyValues Copy src[srcPos:srcPos+len] into dest[destPos:destPos+len] using at most mem bytes.

func CopyValuesWithBuffer

func CopyValuesWithBuffer(src Reader, srcPos int, dest Mutable, destPos, size int, buf []uint64)

CopyValuesWithBuffer Same as copy(PackedInts.Reader, int, PackedInts.Mutable, int, int, int) but using a pre-allocated buffer.

func Fill

func Fill(spi mutableSPI, fromIndex, toIndex int, value uint64)

func GetBulk

func GetBulk(reader Reader, index int, arr []uint64) int

func MaxValue

func MaxValue(bitsPerValue int) uint64

func Of

func Of(format Format, bitsPerValue int) (common.BulkOperation, error)

func SetBulk

func SetBulk(spi mutableSPI, index int, arr []uint64) int

func UnsignedBitsRequired

func UnsignedBitsRequired(v uint64) int

Types

type BaseWriter

type BaseWriter struct {
	// contains filtered or unexported fields
}

func (*BaseWriter) BitsPerValue

func (w *BaseWriter) BitsPerValue() int

func (*BaseWriter) WriteHeader

func (w *BaseWriter) WriteHeader(ctx context.Context) error

type BlockPackedFlusher

type BlockPackedFlusher interface {
	Flush(ctx context.Context) error
}

type BlockPackedReader

type BlockPackedReader struct {
	// contains filtered or unexported fields
}

BlockPackedReader Provides random access to a stream written with BlockPackedWriter. lucene.internal

func NewBlockPackedReader

func NewBlockPackedReader(ctx context.Context, in store.IndexInput,
	packedIntsVersion, blockSize, valueCount int, direct bool) (*BlockPackedReader, error)

func (*BlockPackedReader) Get

func (b *BlockPackedReader) Get(index int) (uint64, error)

type BlockPackedReaderIterator

type BlockPackedReaderIterator struct {
	// contains filtered or unexported fields
}

BlockPackedReaderIterator Reader for sequences of longs written with BlockPackedWriter. BlockPackedWriter lucene.internal

func NewBlockPackedReaderIterator

func NewBlockPackedReaderIterator(in store.DataInput,
	packedIntsVersion, blockSize, valueCount int) *BlockPackedReaderIterator

NewBlockPackedReaderIterator Sole constructor. blockSize: the number of values of a block, must be equal to the block size of the BlockPackedWriter which has been used to write the stream

func (*BlockPackedReaderIterator) Next

Next Read the next value.

func (*BlockPackedReaderIterator) NextSlices

func (b *BlockPackedReaderIterator) NextSlices(ctx context.Context, count int) ([]uint64, error)

func (*BlockPackedReaderIterator) Ord

func (b *BlockPackedReaderIterator) Ord() int

Ord Return the offset of the next value to read.

func (*BlockPackedReaderIterator) Reset

func (b *BlockPackedReaderIterator) Reset(in store.DataInput, valueCount int)

Reset the current reader to wrap a stream of valueCount values contained in in. The block size remains unchanged.

func (*BlockPackedReaderIterator) Skip

func (b *BlockPackedReaderIterator) Skip(ctx context.Context, count int) error

Skip exactly count values.

type BlockPackedWriter

type BlockPackedWriter struct {
	// contains filtered or unexported fields
}

BlockPackedWriter A writer for large sequences of longs. The sequence is divided into fixed-size blocks and for each block, the difference between each value and the minimum value of the block is encoded using as few bits as possible. Memory usage of this class is proportional to the block size. Each block has an overhead between 1 and 10 bytes to store the minimum value and the number of bits per value of the block. Format: <BLock>BlockCount BlockCount: ⌈ ValueCount / BlockSize ⌉ Block: <Header, (Ints)> Header: <Token, (MinValue)> Token: a byte, first 7 bits are the number of bits per value (bitsPerValue). If the 8th bit is 1, then MinValue (see next) is 0, otherwise MinValue and needs to be decoded MinValue: a zigzag-encoded variable-length long whose value should be added to every int from the block to restore the original values Ints: If the number of bits per value is 0, then there is nothing to decode and all ints are equal to MinValue. Otherwise: BlockSize packed ints encoded on exactly bitsPerValue bits per value. They are the subtraction of the original values and MinValue 请参阅: BlockPackedReaderIterator, BlockPackedReader lucene.internal

func NewBlockPackedWriter

func NewBlockPackedWriter(out store.DataOutput, blockSize int) *BlockPackedWriter

func (*BlockPackedWriter) Add

func (b *BlockPackedWriter) Add(ctx context.Context, u uint64) error

func (BlockPackedWriter) Finish

func (a BlockPackedWriter) Finish(ctx context.Context) error

func (*BlockPackedWriter) Flush

func (b *BlockPackedWriter) Flush(ctx context.Context) error

type Decoder

type Decoder interface {

	// LongBlockCount
	// The minimum number of long blocks to encode in a single iteration, when using long encoding.
	LongBlockCount() int

	// LongValueCount
	// The number of values that can be stored in longBlockCount() long blocks.
	LongValueCount() int

	// ByteBlockCount
	// The minimum number of byte blocks to encode in a single iteration, when using byte encoding.
	ByteBlockCount() int

	// ByteValueCount
	// The number of values that can be stored in byteBlockCount() byte blocks.
	ByteValueCount() int

	// DecodeUint64
	// Read iterations * blockCount() blocks from blocks,
	// decode them and write iterations * valueCount() values into values.
	// blocks: the long blocks that hold packed integer values
	// values: the values buffer
	// iterations: controls how much data to decode
	DecodeUint64(blocks []uint64, values []uint64, iterations int)

	// DecodeBytes
	// Read 8 * iterations * blockCount() blocks from blocks,
	// decode them and write iterations * valueCount() values into values.
	// blocks: the long blocks that hold packed integer values
	// values: the values buffer
	// iterations: controls how much data to decode
	DecodeBytes(blocks []byte, values []uint64, iterations int)
}

Decoder A decoder for packed integers.

func GetDecoder

func GetDecoder(format Format, version, bitsPerValue int) (Decoder, error)

type DeltaPackedLongValues

type DeltaPackedLongValues struct {
	*PackedLongValues
	// contains filtered or unexported fields
}

func NewDeltaPackedLongValues

func NewDeltaPackedLongValues(pageShift int, pageMask uint64, values []Reader, mins []int64, size int) *DeltaPackedLongValues

func (*DeltaPackedLongValues) Get

func (d *DeltaPackedLongValues) Get(index int) (uint64, error)

func (*DeltaPackedLongValues) Iterator

type DeltaPackedLongValuesBuilder

type DeltaPackedLongValuesBuilder struct {
	*PackedLongValuesBuilder
	// contains filtered or unexported fields
}

func NewDeltaPackedLongValuesBuilder

func NewDeltaPackedLongValuesBuilder(pageSize int, acceptableOverheadRatio float64) *DeltaPackedLongValuesBuilder

func (*DeltaPackedLongValuesBuilder) Add

func (*DeltaPackedLongValuesBuilder) Build

type Direct16

type Direct16 struct {
	// contains filtered or unexported fields
}

func NewDirect16

func NewDirect16(valueCount int) *Direct16

func NewDirect16V1

func NewDirect16V1(ctx context.Context, packedIntsVersion int, in store.DataInput, valueCount int) (*Direct16, error)

func (*Direct16) Clear

func (d *Direct16) Clear()

func (*Direct16) Fill

func (d *Direct16) Fill(fromIndex, toIndex int, value uint64)

func (*Direct16) Get

func (d *Direct16) Get(index int) (uint64, error)

func (Direct16) GetBitsPerValue

func (m Direct16) GetBitsPerValue() int

func (*Direct16) GetBulk

func (d *Direct16) GetBulk(index int, arr []uint64) int

func (*Direct16) GetTest

func (d *Direct16) GetTest(index int) uint64

func (*Direct16) Set

func (d *Direct16) Set(index int, value uint64)

func (*Direct16) SetBulk

func (d *Direct16) SetBulk(index int, arr []uint64) int

func (Direct16) Size

func (m Direct16) Size() int

type Direct32

type Direct32 struct {
	// contains filtered or unexported fields
}

func NewDirect32

func NewDirect32(valueCount int) *Direct32

func NewDirect32V1

func NewDirect32V1(ctx context.Context, packedIntsVersion int, in store.DataInput, valueCount int) (*Direct32, error)

func (*Direct32) Clear

func (d *Direct32) Clear()

func (*Direct32) Fill

func (d *Direct32) Fill(fromIndex, toIndex int, value uint64)

func (*Direct32) Get

func (d *Direct32) Get(index int) (uint64, error)

func (Direct32) GetBitsPerValue

func (m Direct32) GetBitsPerValue() int

func (*Direct32) GetBulk

func (d *Direct32) GetBulk(index int, arr []uint64) int

func (*Direct32) GetTest

func (d *Direct32) GetTest(index int) uint64

func (*Direct32) Set

func (d *Direct32) Set(index int, value uint64)

func (*Direct32) SetBulk

func (d *Direct32) SetBulk(index int, arr []uint64) int

func (Direct32) Size

func (m Direct32) Size() int

type Direct64

type Direct64 struct {
	// contains filtered or unexported fields
}

func NewDirect64

func NewDirect64(valueCount int) *Direct64

func NewDirect64V1

func NewDirect64V1(ctx context.Context, packedIntsVersion int, in store.DataInput, valueCount int) (*Direct64, error)

func (*Direct64) Clear

func (d *Direct64) Clear()

func (*Direct64) Fill

func (d *Direct64) Fill(fromIndex, toIndex int, value uint64)

func (*Direct64) Get

func (d *Direct64) Get(index int) (uint64, error)

func (Direct64) GetBitsPerValue

func (m Direct64) GetBitsPerValue() int

func (*Direct64) GetBulk

func (d *Direct64) GetBulk(index int, arr []uint64) int

func (*Direct64) GetTest

func (d *Direct64) GetTest(index int) uint64

func (*Direct64) Set

func (d *Direct64) Set(index int, value uint64)

func (*Direct64) SetBulk

func (d *Direct64) SetBulk(index int, arr []uint64) int

func (Direct64) Size

func (m Direct64) Size() int

type Direct8

type Direct8 struct {
	// contains filtered or unexported fields
}

Direct8 Direct wrapping of 8-bits values to a backing array. lucene.internal

func NewDirect8

func NewDirect8(valueCount int) *Direct8

func NewDirect8V1

func NewDirect8V1(_ context.Context, packedIntsVersion int, in store.DataInput, valueCount int) (*Direct8, error)

func (*Direct8) Clear

func (d *Direct8) Clear()

func (*Direct8) Fill

func (d *Direct8) Fill(fromIndex, toIndex int, value uint64)

func (*Direct8) Get

func (d *Direct8) Get(index int) (uint64, error)

func (Direct8) GetBitsPerValue

func (m Direct8) GetBitsPerValue() int

func (*Direct8) GetBulk

func (d *Direct8) GetBulk(index int, arr []uint64) int

func (*Direct8) GetTest

func (d *Direct8) GetTest(index int) uint64

func (*Direct8) Set

func (d *Direct8) Set(index int, value uint64)

func (*Direct8) SetBulk

func (d *Direct8) SetBulk(index int, arr []uint64) int

func (Direct8) Size

func (m Direct8) Size() int

type DirectPacked64SingleBlockReader

type DirectPacked64SingleBlockReader struct {
	// contains filtered or unexported fields
}

func NewDirectPacked64SingleBlockReader

func NewDirectPacked64SingleBlockReader(bitsPerValue, valueCount int,
	in store.IndexInput) *DirectPacked64SingleBlockReader

func (*DirectPacked64SingleBlockReader) Get

func (*DirectPacked64SingleBlockReader) GetBulk

func (d *DirectPacked64SingleBlockReader) GetBulk(index int, arr []uint64) int

func (*DirectPacked64SingleBlockReader) Size

type DirectPackedReader

type DirectPackedReader struct {
	// contains filtered or unexported fields
}

DirectPackedReader Reads directly from disk on each get just for back compat, use DirectReader/DirectWriter for more efficient impl

func NewDirectPackedReader

func NewDirectPackedReader(bitsPerValue, valueCount int, in store.IndexInput) *DirectPackedReader

func (*DirectPackedReader) Get

func (d *DirectPackedReader) Get(index int) (uint64, error)

func (*DirectPackedReader) GetBulk

func (d *DirectPackedReader) GetBulk(index int, arr []uint64) int

func (*DirectPackedReader) Size

func (d *DirectPackedReader) Size() int

type DirectPackedReader1

type DirectPackedReader1 struct {
	// contains filtered or unexported fields
}

func NewDirectPackedReader1

func NewDirectPackedReader1(in store.RandomAccessInput, offset int64) *DirectPackedReader1

func (*DirectPackedReader1) Get

func (d *DirectPackedReader1) Get(index int64) (uint64, error)

type DirectPackedReader12

type DirectPackedReader12 struct {
	// contains filtered or unexported fields
}

func NewDirectPackedReader12

func NewDirectPackedReader12(in store.RandomAccessInput, offset int64) *DirectPackedReader12

func (*DirectPackedReader12) Get

func (d *DirectPackedReader12) Get(index int64) (uint64, error)

type DirectPackedReader16

type DirectPackedReader16 struct {
	// contains filtered or unexported fields
}

func NewDirectPackedReader16

func NewDirectPackedReader16(in store.RandomAccessInput, offset int64) *DirectPackedReader16

func (*DirectPackedReader16) Get

func (d *DirectPackedReader16) Get(index int64) (uint64, error)

type DirectPackedReader2

type DirectPackedReader2 struct {
	// contains filtered or unexported fields
}

func NewDirectPackedReader2

func NewDirectPackedReader2(in store.RandomAccessInput, offset int64) *DirectPackedReader2

func (*DirectPackedReader2) Get

func (d *DirectPackedReader2) Get(index int64) (uint64, error)

type DirectPackedReader20

type DirectPackedReader20 struct {
	// contains filtered or unexported fields
}

func NewDirectPackedReader20

func NewDirectPackedReader20(in store.RandomAccessInput, offset int64) *DirectPackedReader20

func (*DirectPackedReader20) Get

func (d *DirectPackedReader20) Get(index int64) (uint64, error)

type DirectPackedReader24

type DirectPackedReader24 struct {
	// contains filtered or unexported fields
}

func NewDirectPackedReader24

func NewDirectPackedReader24(in store.RandomAccessInput, offset int64) *DirectPackedReader24

func (*DirectPackedReader24) Get

func (d *DirectPackedReader24) Get(index int64) (uint64, error)

type DirectPackedReader28

type DirectPackedReader28 struct {
	// contains filtered or unexported fields
}

func NewDirectPackedReader28

func NewDirectPackedReader28(in store.RandomAccessInput, offset int64) *DirectPackedReader28

func (*DirectPackedReader28) Get

func (d *DirectPackedReader28) Get(index int64) (uint64, error)

type DirectPackedReader32

type DirectPackedReader32 struct {
	// contains filtered or unexported fields
}

func NewDirectPackedReader32

func NewDirectPackedReader32(in store.RandomAccessInput, offset int64) *DirectPackedReader32

func (*DirectPackedReader32) Get

func (d *DirectPackedReader32) Get(index int64) (uint64, error)

type DirectPackedReader4

type DirectPackedReader4 struct {
	// contains filtered or unexported fields
}

func NewDirectPackedReader4

func NewDirectPackedReader4(in store.RandomAccessInput, offset int64) *DirectPackedReader4

func (*DirectPackedReader4) Get

func (d *DirectPackedReader4) Get(index int64) (uint64, error)

type DirectPackedReader40

type DirectPackedReader40 struct {
	// contains filtered or unexported fields
}

func NewDirectPackedReader40

func NewDirectPackedReader40(in store.RandomAccessInput, offset int64) *DirectPackedReader40

func (*DirectPackedReader40) Get

func (d *DirectPackedReader40) Get(index int64) (uint64, error)

type DirectPackedReader48

type DirectPackedReader48 struct {
	// contains filtered or unexported fields
}

func NewDirectPackedReader48

func NewDirectPackedReader48(in store.RandomAccessInput, offset int64) *DirectPackedReader48

func (*DirectPackedReader48) Get

func (d *DirectPackedReader48) Get(index int64) (uint64, error)

type DirectPackedReader56

type DirectPackedReader56 struct {
	// contains filtered or unexported fields
}

func NewDirectPackedReader56

func NewDirectPackedReader56(in store.RandomAccessInput, offset int64) *DirectPackedReader56

func (*DirectPackedReader56) Get

func (d *DirectPackedReader56) Get(index int64) (uint64, error)

type DirectPackedReader64

type DirectPackedReader64 struct {
	// contains filtered or unexported fields
}

func NewDirectPackedReader64

func NewDirectPackedReader64(in store.RandomAccessInput, offset int64) *DirectPackedReader64

func (*DirectPackedReader64) Get

func (d *DirectPackedReader64) Get(index int64) (uint64, error)

type DirectPackedReader8

type DirectPackedReader8 struct {
	// contains filtered or unexported fields
}

func NewDirectPackedReader8

func NewDirectPackedReader8(in store.RandomAccessInput, offset int64) *DirectPackedReader8

func (*DirectPackedReader8) Get

func (d *DirectPackedReader8) Get(index int64) (uint64, error)

type DirectReader

type DirectReader struct {
}

DirectReader Retrieves an instance previously written by DirectWriter

func NewDirectReader

func NewDirectReader() *DirectReader

func (*DirectReader) GetInstance

func (d *DirectReader) GetInstance(slice store.RandomAccessInput, bitsPerValue int, offset int64) (LongValuesReader, error)

type DirectWriter

type DirectWriter struct {
	// contains filtered or unexported fields
}

DirectWriter Class for writing packed integers to be directly read from Directory. Integers can be read on-the-fly via DirectReader. Unlike PackedInts, it optimizes for read i/o operations and supports > 2B values.

func NewDirectWriter

func NewDirectWriter(output store.DataOutput, numValues, bitsPerValue int) (*DirectWriter, error)

func (*DirectWriter) Add

func (d *DirectWriter) Add(v uint64) error

func (*DirectWriter) Finish

func (d *DirectWriter) Finish() error

Finish finishes writing

type Encoder

type Encoder interface {
	// LongBlockCount
	// The minimum number of long blocks to encode in a single iteration, when using long encoding.
	LongBlockCount() int

	// LongValueCount
	// The number of values that can be stored in longBlockCount() long blocks.
	LongValueCount() int

	// ByteBlockCount
	// The minimum number of byte blocks to encode in a single iteration, when using byte encoding.
	ByteBlockCount() int

	// ByteValueCount
	// The number of values that can be stored in byteBlockCount() byte blocks.
	ByteValueCount() int

	// EncodeUint64
	// Read iterations * valueCount() values from values, encode them and write
	// iterations * blockCount() blocks into blocks.
	// values: the values buffer
	// blocks: the long blocks that hold packed integer values
	// iterations: controls how much data to encode
	EncodeUint64(values []uint64, blocks []uint64, iterations int)

	// EncodeBytes
	// Read iterations * valueCount() values from values,
	// encode them and write 8 * iterations * blockCount() blocks into blocks.
	// values: the values buffer
	// blocks: the long blocks that hold packed integer values
	// iterations: controls how much data to encode
	EncodeBytes(values []uint64, blocks []byte, iterations int)
}

Encoder An encoder for packed integers.

func GetEncoder

func GetEncoder(format Format, version, bitsPerValue int) (Encoder, error)

GetEncoder Get an PackedInts.Encoder. format: the format used to store packed ints version – the compatibility version bitsPerValue – the number of bits per value

type FixSizePagedMutable

type FixSizePagedMutable struct {
	// contains filtered or unexported fields
}

FixSizePagedMutable A FixSizePagedMutable. This class slices data into fixed-size blocks which have the same number of bits per value. It can be a useful replacement for PackedInts.Mutable to store more than 2B values.

func (FixSizePagedMutable) Get

func (a FixSizePagedMutable) Get(index int) (uint64, error)

func (FixSizePagedMutable) GetSubMutableByIndex

func (a FixSizePagedMutable) GetSubMutableByIndex(index int) Mutable

func (FixSizePagedMutable) GetTest

func (a FixSizePagedMutable) GetTest(index int) uint64

func (FixSizePagedMutable) Grow

func (a FixSizePagedMutable) Grow(minSize int) PagedMutable

func (FixSizePagedMutable) GrowOne

func (a FixSizePagedMutable) GrowOne() PagedMutable

GrowOne Similar to ArrayUtil.grow(long[]).

func (*FixSizePagedMutable) NewMutable

func (p *FixSizePagedMutable) NewMutable(valueCount, bitsPerValue int) Mutable

func (*FixSizePagedMutable) NewUnfilledCopy

func (p *FixSizePagedMutable) NewUnfilledCopy(newSize int) (PagedMutable, error)

func (FixSizePagedMutable) Resize

func (a FixSizePagedMutable) Resize(newSize int) PagedMutable

Resize Create a new copy of size newSize based on the content of this buffer. This method is much more efficient than creating a new instance and copying values one by one.

func (FixSizePagedMutable) Set

func (a FixSizePagedMutable) Set(index int, value uint64)

func (FixSizePagedMutable) SetSubMutableByIndex

func (a FixSizePagedMutable) SetSubMutableByIndex(index int, value Mutable)

func (FixSizePagedMutable) Size

func (a FixSizePagedMutable) Size() int

func (FixSizePagedMutable) SubMutables

func (a FixSizePagedMutable) SubMutables() []Mutable

type FixSizePagedMutableBuilder

type FixSizePagedMutableBuilder struct {
}

func NewFixSizePagedMutableBuilder

func NewFixSizePagedMutableBuilder() FixSizePagedMutableBuilder

func (*FixSizePagedMutableBuilder) New

func (b *FixSizePagedMutableBuilder) New(size, pageSize, bitsPerValue int, acceptableOverheadRatio float64) (*FixSizePagedMutable, error)

func (*FixSizePagedMutableBuilder) NewWithFormat

func (b *FixSizePagedMutableBuilder) NewWithFormat(size, pageSize, bitsPerValue int, format Format) (*FixSizePagedMutable, error)

func (*FixSizePagedMutableBuilder) NewWithFormatAndBits

func (b *FixSizePagedMutableBuilder) NewWithFormatAndBits(size, pageSize int, formatAndBits *FormatAndBits) (*FixSizePagedMutable, error)

type FnPackValues

type FnPackValues func(values []int64, numValues int, acceptableOverheadRatio float64) error

type Format

type Format interface {
	GetId() int

	// ByteCount Computes how many byte blocks are needed to store values values of size bitsPerValue.
	ByteCount(packedIntsVersion, valueCount, bitsPerValue int) int

	// LongCount Computes how many long blocks are needed to store values values of size bitsPerValue.
	LongCount(packedIntsVersion, valueCount, bitsPerValue int) int

	// IsSupported Tests whether the provided number of bits per value is supported by the format.
	IsSupported(bitsPerValue int) bool

	// OverheadPerValue Returns the overhead per value, in bits.
	OverheadPerValue(bitsPerValue int) float64

	// OverheadRatio Returns the overhead ratio (overhead per value / bits per value).
	OverheadRatio(bitsPerValue int) float64
}

func GetFormatById

func GetFormatById(id int) (Format, error)

type FormatAndBits

type FormatAndBits struct {
	// contains filtered or unexported fields
}

FormatAndBits Simple class that holds a format and a number of bits per value.

func NewFormatAndBits

func NewFormatAndBits(format Format, bitsPerValue int) *FormatAndBits

type FuncGetMutable

type FuncGetMutable func(startBitsPerValue, valueCount int, acceptableOverheadRatio float64) Mutable

type GrowableWriter

type GrowableWriter struct {
	// contains filtered or unexported fields
}

GrowableWriter Implements Mutable, but grows the bit count of the underlying packed ints on-demand. Beware that this class will accept to set negative values but in order to do this, it will grow the number of bits per value to 64. @lucene.internal

func NewGrowableWriter

func NewGrowableWriter(startBitsPerValue, valueCount int, acceptableOverheadRatio float64) *GrowableWriter

NewGrowableWriter startBitsPerValue: the initial number of bits per value, may grow depending on the data valueCount: the number of values acceptableOverheadRatio: an acceptable overhead ratio

func (*GrowableWriter) Clear

func (g *GrowableWriter) Clear()

func (*GrowableWriter) Fill

func (g *GrowableWriter) Fill(fromIndex, toIndex int, value uint64)

func (*GrowableWriter) Get

func (g *GrowableWriter) Get(index int) (uint64, error)

func (*GrowableWriter) GetBitsPerValue

func (g *GrowableWriter) GetBitsPerValue() int

func (*GrowableWriter) GetBulk

func (g *GrowableWriter) GetBulk(index int, arr []uint64) int

func (*GrowableWriter) GetFormat

func (g *GrowableWriter) GetFormat() Format

func (*GrowableWriter) GetMutable

func (g *GrowableWriter) GetMutable() Mutable

func (*GrowableWriter) GetTest

func (g *GrowableWriter) GetTest(index int) uint64

func (*GrowableWriter) Resize

func (g *GrowableWriter) Resize(newSize int) *GrowableWriter

func (*GrowableWriter) Save

func (g *GrowableWriter) Save(ctx context.Context, out store.DataOutput) error

func (*GrowableWriter) Set

func (g *GrowableWriter) Set(index int, value uint64)

func (*GrowableWriter) SetBulk

func (g *GrowableWriter) SetBulk(index int, values []uint64) int

func (*GrowableWriter) Size

func (g *GrowableWriter) Size() int

type LongValuesIteratorSPI

type LongValuesIteratorSPI interface {
	Get(index int) (uint64, error)
	Size() int
}

type LongValuesReader

type LongValuesReader interface {
	Get(index int64) (uint64, error)
}

type MonotonicBlockPackedReader

type MonotonicBlockPackedReader struct {
	// contains filtered or unexported fields
}

MonotonicBlockPackedReader Provides random access to a stream written with MonotonicBlockPackedWriter. lucene.internal

func NewMonotonicBlockPackedReader

func NewMonotonicBlockPackedReader(ctx context.Context, in store.IndexInput,
	packedIntsVersion, blockSize, valueCount int, direct bool) (*MonotonicBlockPackedReader, error)

NewMonotonicBlockPackedReader IndexInput in, int packedIntsVersion, int blockSize, long valueCount, boolean direct

func (*MonotonicBlockPackedReader) Get

func (m *MonotonicBlockPackedReader) Get(index int) (uint64, error)

func (*MonotonicBlockPackedReader) Size

func (m *MonotonicBlockPackedReader) Size() int

Size Returns the number of values

type MonotonicBlockPackedWriter

type MonotonicBlockPackedWriter struct {
	// contains filtered or unexported fields
}

MonotonicBlockPackedWriter A writer for large monotonically increasing sequences of positive longs. The sequence is divided into fixed-size blocks and for each block, values are modeled after a linear function f: x → A × x + B. The block encodes deltas from the expected values computed from this function using as few bits as possible. Format: <BLock>BlockCount BlockCount: ⌈ ValueCount / BlockSize ⌉ Block: <Header, (Ints)> Header: <B, A, BitsPerValue> B: the B from f: x → A × x + B using a zig-zag encoded vLong A: the A from f: x → A × x + B encoded using Float.floatToIntBits(float) on 4 bytes BitsPerValue: a variable-length int Ints: if BitsPerValue is 0, then there is nothing to read and all values perfectly match the result of the function. Otherwise, these are the packed deltas from the expected value (computed from the function) using exactly BitsPerValue bits per value. 请参阅: MonotonicBlockPackedReader lucene.internal

func NewMonotonicBlockPackedWriter

func NewMonotonicBlockPackedWriter(out store.DataOutput, blockSize int) *MonotonicBlockPackedWriter

func (*MonotonicBlockPackedWriter) Add

func (MonotonicBlockPackedWriter) Finish

func (a MonotonicBlockPackedWriter) Finish(ctx context.Context) error

func (*MonotonicBlockPackedWriter) Flush

type MonotonicLongValues

type MonotonicLongValues struct {
	*DeltaPackedLongValues
	// contains filtered or unexported fields
}

func NewMonotonicLongValues

func NewMonotonicLongValues(pageShift int, pageMask uint64,
	values []Reader, mins []int64, averages []float32, size int) *MonotonicLongValues

func (*MonotonicLongValues) Get

func (m *MonotonicLongValues) Get(index int) (uint64, error)

func (*MonotonicLongValues) Iterator

type MonotonicLongValuesBuilder

type MonotonicLongValuesBuilder struct {
	*DeltaPackedLongValuesBuilder
	// contains filtered or unexported fields
}

func NewMonotonicLongValuesBuilder

func NewMonotonicLongValuesBuilder(pageSize int, acceptableOverheadRatio float64) *MonotonicLongValuesBuilder

func (*MonotonicLongValuesBuilder) Add

func (m *MonotonicLongValuesBuilder) Add(value int64) error

func (*MonotonicLongValuesBuilder) Build

type Mutable

type Mutable interface {
	Reader

	// GetBitsPerValue returns the number of bits used to store any given value.
	// Note: This does not imply that memory usage is bitsPerValue * #values as implementations
	// are free to use non-space-optimal packing of bits.
	GetBitsPerValue() int

	// Set the value at the given index in the array.
	// index: where the value should be positioned.
	// value: a value conforming to the constraints set by the array.
	Set(index int, value uint64)

	// SetBulk set at least one and at most len longs starting at off in arr into this mutable,
	// starting at index. Returns the actual number of values that have been set.
	SetBulk(index int, values []uint64) int

	// Fill the mutable from fromIndex (inclusive) to toIndex (exclusive) with val.
	Fill(fromIndex, toIndex int, value uint64)

	// Clear Sets all values to 0.
	Clear()

	// Save this mutable into out. Instantiating a reader from the generated data will return a
	// reader with the same number of bits per value.
	Save(ctx context.Context, out store.DataOutput) error

	GetFormat() Format
}

Mutable A packed integer array that can be modified. lucene.internal

func DefaultGetMutable

func DefaultGetMutable(valueCount, bitsPerValue int, acceptableOverheadRatio float64) Mutable

DefaultGetMutable Create a packed integer array with the given amount of values initialized to 0. the valueCount and the bitsPerValue cannot be changed after creation. All Mutables known by this factory are kept fully in RAM. Positive values of acceptableOverheadRatio will trade space for speed by selecting a faster but potentially less memory-efficient implementation. An acceptableOverheadRatio of COMPACT will make sure that the most memory-efficient implementation is selected whereas FASTEST will make sure that the fastest implementation is selected.

valueCount: the number of elements bitsPerValue: the number of bits available for any given value acceptableOverheadRatio: an acceptable overhead ratio per value

Returns a mutable packed integer array lucene.internal

type NullReader

type NullReader struct {
	// contains filtered or unexported fields
}

NullReader A PackedInts.PackedIntsReader which has all its values equal to 0 (bitsPerValue = 0).

func NewNullReader

func NewNullReader(valueCount int) *NullReader

func (*NullReader) Get

func (n *NullReader) Get(index int) (uint64, error)

func (*NullReader) GetBulk

func (n *NullReader) GetBulk(index int, arr []uint64) int

func (*NullReader) Size

func (n *NullReader) Size() int

type Packed16ThreeBlocks

type Packed16ThreeBlocks struct {
	// contains filtered or unexported fields
}

Packed16ThreeBlocks Packs integers into 3 shorts (48 bits per value). lucene.internal

func NewPacked16ThreeBlocks

func NewPacked16ThreeBlocks(valueCount int) *Packed16ThreeBlocks

func NewPacked16ThreeBlocksV1

func NewPacked16ThreeBlocksV1(ctx context.Context, packedIntsVersion int,
	in store.DataInput, valueCount int) (*Packed16ThreeBlocks, error)

func (*Packed16ThreeBlocks) Clear

func (p *Packed16ThreeBlocks) Clear()

func (*Packed16ThreeBlocks) Fill

func (p *Packed16ThreeBlocks) Fill(fromIndex, toIndex int, value uint64)

func (*Packed16ThreeBlocks) Get

func (p *Packed16ThreeBlocks) Get(index int) (uint64, error)

func (Packed16ThreeBlocks) GetBitsPerValue

func (m Packed16ThreeBlocks) GetBitsPerValue() int

func (*Packed16ThreeBlocks) GetBulk

func (p *Packed16ThreeBlocks) GetBulk(index int, arr []uint64) int

func (*Packed16ThreeBlocks) GetTest

func (p *Packed16ThreeBlocks) GetTest(index int) uint64

func (*Packed16ThreeBlocks) Set

func (p *Packed16ThreeBlocks) Set(index int, value uint64)

func (*Packed16ThreeBlocks) SetBulk

func (p *Packed16ThreeBlocks) SetBulk(index int, arr []uint64) int

func (Packed16ThreeBlocks) Size

func (m Packed16ThreeBlocks) Size() int

type Packed64

type Packed64 struct {
	// contains filtered or unexported fields
}

func NewPacked64

func NewPacked64(valueCount, bitsPerValue int) *Packed64

NewPacked64 Creates an array with the internal structures adjusted for the given limits and initialized to 0. valueCount: the number of elements. bitsPerValue: the number of bits available for any given value.

func NewPacked64V1

func NewPacked64V1(ctx context.Context, packedIntsVersion int, in store.DataInput, valueCount, bitsPerValue int) (*Packed64, error)

NewPacked64V1 Creates an array with content retrieved from the given DataInput. in: a DataInput, positioned at the start of Packed64-content. valueCount: the number of elements. bitsPerValue the number of bits available for any given value.

func (*Packed64) Clear

func (p *Packed64) Clear()

func (*Packed64) Fill

func (p *Packed64) Fill(fromIndex, toIndex int, value uint64)

func (*Packed64) Get

func (p *Packed64) Get(index int) (uint64, error)

func (Packed64) GetBitsPerValue

func (m Packed64) GetBitsPerValue() int

func (*Packed64) GetBulk

func (p *Packed64) GetBulk(index int, buffer []uint64) int

func (*Packed64) GetFormat

func (p *Packed64) GetFormat() Format

func (*Packed64) GetTest

func (p *Packed64) GetTest(index int) uint64

func (*Packed64) Save

func (p *Packed64) Save(ctx context.Context, out store.DataOutput) error

func (*Packed64) Set

func (p *Packed64) Set(index int, value uint64)

func (*Packed64) SetBulk

func (p *Packed64) SetBulk(index int, arr []uint64) int

func (Packed64) Size

func (m Packed64) Size() int

type Packed64SingleBlock

type Packed64SingleBlock struct {
	// contains filtered or unexported fields
}

func NewPacked64SingleBlock

func NewPacked64SingleBlock(valueCount, bitsPerValue int) (*Packed64SingleBlock, error)

func NewPacked64SingleBlockV1

func NewPacked64SingleBlockV1(ctx context.Context, in store.DataInput, valueCount, bitsPerValue int) (*Packed64SingleBlock, error)

func (*Packed64SingleBlock) Clear

func (p *Packed64SingleBlock) Clear()

func (*Packed64SingleBlock) Get

func (p *Packed64SingleBlock) Get(index int) (uint64, error)

func (Packed64SingleBlock) GetBitsPerValue

func (m Packed64SingleBlock) GetBitsPerValue() int

func (*Packed64SingleBlock) Set

func (p *Packed64SingleBlock) Set(index int, value uint64)

func (Packed64SingleBlock) Size

func (m Packed64SingleBlock) Size() int

type Packed8ThreeBlocks

type Packed8ThreeBlocks struct {
	// contains filtered or unexported fields
}

Packed8ThreeBlocks Packs integers into 3 bytes (24 bits per value). lucene.internal

func NewNewPacked8ThreeBlocksV1

func NewNewPacked8ThreeBlocksV1(ctx context.Context, packedIntsVersion int,
	in store.DataInput, valueCount int) (*Packed8ThreeBlocks, error)

func NewPacked8ThreeBlocks

func NewPacked8ThreeBlocks(valueCount int) *Packed8ThreeBlocks

func (*Packed8ThreeBlocks) Clear

func (p *Packed8ThreeBlocks) Clear()

func (*Packed8ThreeBlocks) Fill

func (p *Packed8ThreeBlocks) Fill(fromIndex, toIndex int, value uint64)

func (*Packed8ThreeBlocks) Get

func (p *Packed8ThreeBlocks) Get(index int) (uint64, error)

func (Packed8ThreeBlocks) GetBitsPerValue

func (m Packed8ThreeBlocks) GetBitsPerValue() int

func (*Packed8ThreeBlocks) GetBulk

func (p *Packed8ThreeBlocks) GetBulk(index int, arr []uint64) int

func (*Packed8ThreeBlocks) GetTest

func (p *Packed8ThreeBlocks) GetTest(index int) uint64

func (*Packed8ThreeBlocks) Set

func (p *Packed8ThreeBlocks) Set(index int, value uint64)

func (*Packed8ThreeBlocks) SetBulk

func (p *Packed8ThreeBlocks) SetBulk(index int, arr []uint64) int

func (Packed8ThreeBlocks) Size

func (m Packed8ThreeBlocks) Size() int

type PackedInts

type PackedInts struct {
}

PackedInts Simplistic compression for array of unsigned long values. Each value is >= 0 and <= a specified maximum value. The values are stored as packed ints, with each value consuming a fixed number of bits. lucene.internal

type PackedLongValues

type PackedLongValues struct {
	// contains filtered or unexported fields
}

PackedLongValues compress integers into a PackedLongValues instance.

func NewPackedLongValues

func NewPackedLongValues(values []Reader, pageShift int, pageMask uint64, size int) *PackedLongValues

func (*PackedLongValues) Get

func (p *PackedLongValues) Get(index int) (uint64, error)

func (*PackedLongValues) Iterator

func (*PackedLongValues) Size

func (p *PackedLongValues) Size() int

type PackedLongValuesBuilder

type PackedLongValuesBuilder struct {
	// contains filtered or unexported fields
}

func NewPackedLongValuesBuilder

func NewPackedLongValuesBuilder(pageSize int, acceptableOverheadRatio float64) *PackedLongValuesBuilder

func (*PackedLongValuesBuilder) Add

func (p *PackedLongValuesBuilder) Add(value int64) error

Add a new element to this builder.

func (*PackedLongValuesBuilder) Build

func (*PackedLongValuesBuilder) Size

func (p *PackedLongValuesBuilder) Size() int

type PackedLongValuesIterator

type PackedLongValuesIterator interface {
	HasNext() bool
	Next() (uint64, error)
}

type PackedWriter

type PackedWriter struct {
	*BaseWriter
	// contains filtered or unexported fields
}

func NewPackedWriter

func NewPackedWriter(format Format, out store.DataOutput, valueCount, bitsPerValue, mem int) *PackedWriter

func (*PackedWriter) Add

func (p *PackedWriter) Add(v uint64) error

func (*PackedWriter) Finish

func (p *PackedWriter) Finish() error

func (*PackedWriter) GetFormat

func (p *PackedWriter) GetFormat() Format

func (*PackedWriter) Ord

func (p *PackedWriter) Ord() int

type PagedGrowableWriter

type PagedGrowableWriter struct {
	// contains filtered or unexported fields
}

PagedGrowableWriter A PagedGrowableWriter. This class slices data into fixed-size blocks which have independent numbers of bits per value and grow on-demand. You should use this class instead of the packedLongValues related ones only when you need random write-access. Otherwise this class will likely be slower and less memory-efficient. lucene.internal

func NewPagedGrowableWriter

func NewPagedGrowableWriter(size, pageSize, startBitsPerValue int, acceptableOverheadRatio float64) (*PagedGrowableWriter, error)

func (PagedGrowableWriter) Get

func (a PagedGrowableWriter) Get(index int) (uint64, error)

func (PagedGrowableWriter) GetSubMutableByIndex

func (a PagedGrowableWriter) GetSubMutableByIndex(index int) Mutable

func (PagedGrowableWriter) GetTest

func (a PagedGrowableWriter) GetTest(index int) uint64

func (PagedGrowableWriter) Grow

func (a PagedGrowableWriter) Grow(minSize int) PagedMutable

func (PagedGrowableWriter) GrowOne

func (a PagedGrowableWriter) GrowOne() PagedMutable

GrowOne Similar to ArrayUtil.grow(long[]).

func (*PagedGrowableWriter) NewMutable

func (p *PagedGrowableWriter) NewMutable(valueCount, bitsPerValue int) Mutable

func (*PagedGrowableWriter) NewPagedGrowableWriter

func (p *PagedGrowableWriter) NewPagedGrowableWriter(size, pageSize, startBitsPerValue int,
	acceptableOverheadRatio float64, fillPages bool) (*PagedGrowableWriter, error)

func (*PagedGrowableWriter) NewUnfilledCopy

func (p *PagedGrowableWriter) NewUnfilledCopy(newSize int) (PagedMutable, error)

func (PagedGrowableWriter) Resize

func (a PagedGrowableWriter) Resize(newSize int) PagedMutable

Resize Create a new copy of size newSize based on the content of this buffer. This method is much more efficient than creating a new instance and copying values one by one.

func (PagedGrowableWriter) Set

func (a PagedGrowableWriter) Set(index int, value uint64)

func (PagedGrowableWriter) SetSubMutableByIndex

func (a PagedGrowableWriter) SetSubMutableByIndex(index int, value Mutable)

func (PagedGrowableWriter) Size

func (a PagedGrowableWriter) Size() int

func (PagedGrowableWriter) SubMutables

func (a PagedGrowableWriter) SubMutables() []Mutable

type PagedMutable

type PagedMutable interface {
	Get(index int) (uint64, error)
	GetTest(index int) uint64
	Set(index int, value uint64)
	Resize(newSize int) PagedMutable
	Grow(minSize int) PagedMutable
	GrowOne() PagedMutable
	SubMutables() []Mutable
	GetSubMutableByIndex(index int) Mutable
	SetSubMutableByIndex(index int, value Mutable)
}

type PagedMutableBuilder

type PagedMutableBuilder interface {
	NewMutable(valueCount, bitsPerValue int) Mutable
	NewUnfilledCopy(newSize int) (PagedMutable, error)
}

type Reader

type Reader interface {
	// Get the long at the given index. Behavior is undefined for out-of-range indices.
	Get(index int) (uint64, error)

	// GetBulk Bulk get: read at least one and at most len longs starting from index into
	// arr[off:off+len] and return the actual number of values that have been read.
	GetBulk(index int, arr []uint64) int

	// Size Returns: the number of values.
	Size() int
}

Reader A read-only random access array of positive integers. lucene.internal

type ReaderIterator

type ReaderIterator interface {
}

ReaderIterator Run-once iterator interface, to decode previously saved PackedInts.

type Writer

type Writer interface {
	WriteHeader(ctx context.Context) error

	// GetFormat The format used to serialize values.
	GetFormat() Format

	// Add a value to the stream.
	Add(v uint64) error

	// BitsPerValue The number of bits per value.
	BitsPerValue() int

	// Finish Perform end-of-stream operations.
	Finish() error

	// Ord Returns the current ord in the stream (number of values that have been written so far minus one).
	Ord() int
}

Writer A write-once Writer. lucene.internal

type WriterFormat

type WriterFormat interface {
	GetFormat() Format
}

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL