internal

Documentation

Index

• Constants
• Variables
• func Seek(r io.ReadSeeker, off int64) error
• type ChunkScanner
  • func NewChunkScanner(r io.ReadSeeker, err *errors.Once) *ChunkScanner
  • func (r *ChunkScanner) Block() (MagicBytes, [][]byte)
  • func (r *ChunkScanner) LimitShard(start, limit, nshard int)
  • func (r *ChunkScanner) ReadLastBlock() (MagicBytes, [][]byte)
  • func (r *ChunkScanner) Scan() bool
  • func (r *ChunkScanner) Seek(off int64)
  • func (r *ChunkScanner) Tell() int64
• type ChunkWriter
  • func NewChunkWriter(w io.Writer, err *errors.Once) *ChunkWriter
  • func (w *ChunkWriter) Len() int64
  • func (w *ChunkWriter) Write(magic MagicBytes, payload []byte)
• type MagicBytes

Constants

const (
	// ChunkHeaderSize is the fixed header size for a chunk.
	ChunkHeaderSize = 28

	// ChunkSize is the fixed size of a chunk, including its header.
	ChunkSize = 32 << 10

	// MaxChunkPayloadSize is the maximum size of payload a chunk can carry.
	MaxChunkPayloadSize = ChunkSize - ChunkHeaderSize
)

const NumMagicBytes = 8

NumMagicBytes is the size of a magic header stored at the beginning of every chunk.

Variables

var IEEECRC *crc32.Table

IEEECRC is used to compute IEEE CRC chunk checksums.

var MagicHeader = MagicBytes{0xd9, 0xe1, 0xd9, 0x5c, 0xc2, 0x16, 0x04, 0xf7}

MagicHeader is a constant random 8 bytes used to distinguish the header block of a v2 recordio file.

var MagicInvalid = MagicBytes{0xe4, 0xe7, 0x9a, 0xc1, 0xb3, 0xf6, 0xb7, 0xa2}

MagicInvalid is a sentinel. It is never stored in storage.

var MagicLegacyUnpacked = MagicBytes{0xfc, 0xae, 0x95, 0x31, 0xf0, 0xd9, 0xbd, 0x20}

MagicLegacy is the legacy "unpacked" block header.

var MagicPacked = MagicBytes{0x2e, 0x76, 0x47, 0xeb, 0x34, 0x07, 0x3c, 0x2e}

MagicPacked is the legacy "packed" block header, and the v2 data block header.

var MagicTrailer = MagicBytes{0xfe, 0xba, 0x1a, 0xd7, 0xcb, 0xdf, 0x75, 0x3a}

MagicTrailer is a constant random 8 bytes used to distinguish the trailer block of a v2 recordio file.

var MaxReadRecordSize = uint64(1 << 29)

MaxReadRecordSize defines a max size for a record when reading to avoid crashes for unreasonable requests.

Functions

func Seek

func Seek(r io.ReadSeeker, off int64) error

Seek to "off". Returns nil iff the seek ptr moves to "off".

Types

type ChunkScanner

type ChunkScanner struct {
	// contains filtered or unexported fields
}

ChunkScanner reads a sequence of chunks and reconstructs a logical block. Thread compatible.

func NewChunkScanner

func NewChunkScanner(r io.ReadSeeker, err *errors.Once) *ChunkScanner

NewChunkScanner creates a new chunk scanner. Any error is reported through "err".

func (*ChunkScanner) Block

func (r *ChunkScanner) Block() (MagicBytes, [][]byte)

Block returns the current block contents.

REQUIRES: Last Scan() call returned true.

func (*ChunkScanner) LimitShard

func (r *ChunkScanner) LimitShard(start, limit, nshard int)

LimitShard limits this scanner to scan the blocks belonging to a shard range [start,limit) out of [0, nshard). The shard range begins at the scanner's current offset, which must be on a block boundary. The file (beginning at the current scanner offset) is divided into n shards. Each shard scans blocks until the next segment. If a shard begins in the middle of a block, that block belongs to the previous shard.

func (*ChunkScanner) ReadLastBlock

func (r *ChunkScanner) ReadLastBlock() (MagicBytes, [][]byte)

ReadLastBlock reads the trailer. Sets err if the trailer does not exist, or is corrupt. After the call, the read pointer is at an undefined position so the user must call Seek() explicitly.

func (*ChunkScanner) Scan

func (r *ChunkScanner) Scan() bool

Scan reads the next block. It returns false on EOF or any error. AnB error is reported in r.Err()

func (*ChunkScanner) Seek

func (r *ChunkScanner) Seek(off int64)

Seek moves the read pointer so that next Scan() will move to the block at the given file offset. Any error is reported in r.Err()

func (*ChunkScanner) Tell

func (r *ChunkScanner) Tell() int64

Tell returns the file offset of the next block to be read. Any error is reported in r.Err()

type ChunkWriter

type ChunkWriter struct {
	// contains filtered or unexported fields
}

ChunkWriter implements low-level block-write operations. It takes logical block and stores it as a sequence of chunks. Thread compatible.

func NewChunkWriter

func NewChunkWriter(w io.Writer, err *errors.Once) *ChunkWriter

NewChunkWriter creates a new chunk writer. Any error is reported through "err".

func (*ChunkWriter) Len

func (w *ChunkWriter) Len() int64

Len returns the number of bytes successfully written so far. The value is meaningful only when err.Err()==nil.

func (*ChunkWriter) Write

func (w *ChunkWriter) Write(magic MagicBytes, payload []byte)

Write one block. An error is reported through w.err.

type MagicBytes

type MagicBytes = [NumMagicBytes]byte

MagicBytes is stored in the first 8 bytes of any chunk.