stor

Documentation

Overview

Package stor is used to access physical storage, normally by memory mapped file access.

Storage is chunked. Allocations may not straddle chunks.

Stor has an impl(ementation) of either heapstor or mmapstor. OS dependent parts of mmapstor are in mmap_nonwin.go and mmap_windows.go.

Index

• Constants
• func AppendSmallOffset(buf []byte, offset uint64) []byte
• func LenStr(s string) int
• func LenStrs(ss []string) int
• func ReadSmallOffset(buf []byte) uint64
• func WriteSmallOffset(buf []byte, offset uint64)
• type Mode
• type Offset
• type Reader
  • func NewReader(buf []byte) *Reader
  • func (r *Reader) Get1() int
  • func (r *Reader) Get2() int
  • func (r *Reader) Get3() int
  • func (r *Reader) Get4() int
  • func (r *Reader) Get5() uint64
  • func (r *Reader) GetStr() string
  • func (r *Reader) GetStrs() []string
  • func (r *Reader) Remaining() int
• type SmallOffset
• type Stor
  • func HeapStor(chunksize int) *Stor
  • func MmapStor(filename string, mode Mode) (*Stor, error)
  • func NewStor(impl storage, chunksize uint64, size uint64, chunks [][]byte) *Stor
  • func (s *Stor) Alloc(n int) (Offset, []byte)
  • func (s *Stor) Close(unmap bool, callback ...func(uint64))
  • func (s *Stor) Data(offset Offset) []byte
  • func (s *Stor) FirstOffset(off uint64, str string) uint64
  • func (s *Stor) FlushTo(offset uint64)
  • func (s *Stor) LastOffset(off uint64, str string) uint64
  • func (s *Stor) Reader(off uint64) *Reader
  • func (s *Stor) Size() uint64
  • func (s *Stor) Write(off uint64, data []byte)
• type Writer
  • func NewWriter(buf []byte) *Writer
  • func (w *Writer) Len() int
  • func (w *Writer) Put1(n int) *Writer
  • func (w *Writer) Put2(n int) *Writer
  • func (w *Writer) Put3(n int) *Writer
  • func (w *Writer) Put4(n int) *Writer
  • func (w *Writer) Put5(n uint64) *Writer
  • func (w *Writer) PutStr(s string) *Writer
  • func (w *Writer) PutStrs(ss []string) *Writer

Constants

const MaxSmallOffset = 1<<40 - 1

const SmallOffsetLen = 5

Functions

func AppendSmallOffset

func AppendSmallOffset(buf []byte, offset uint64) []byte

func LenStr

func LenStr(s string) int

LenStr returns the space requirecd by PutStr

func LenStrs

func LenStrs(ss []string) int

LenStrs returns the space requirecd by PutStrs

func ReadSmallOffset

func ReadSmallOffset(buf []byte) uint64

func WriteSmallOffset

func WriteSmallOffset(buf []byte, offset uint64)

Types

type Mode

type Mode int

const (
	Read Mode = iota
	Create
	Update
)

type Offset

type Offset = uint64

Offset is an offset within storage

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

func NewReader

func NewReader(buf []byte) *Reader

func (*Reader) Get1

func (r *Reader) Get1() int

Get1 reads an unsigned byte value

func (*Reader) Get2

func (r *Reader) Get2() int

Get2 reads an unsigned two byte value

func (*Reader) Get3

func (r *Reader) Get3() int

Get3 reads an unsigned three byte value

func (*Reader) Get4

func (r *Reader) Get4() int

Get4 reads an unsigned four byte value

func (*Reader) Get5

func (r *Reader) Get5() uint64

Get5 reads an unsigned five byte value

func (*Reader) GetStr

func (r *Reader) GetStr() string

GetStr reads a string

func (*Reader) GetStrs

func (r *Reader) GetStrs() []string

GetStrs reads a slice of strings

func (*Reader) Remaining

func (r *Reader) Remaining() int

Remaining returns the number of unread bytes left

type SmallOffset

type SmallOffset [SmallOffsetLen]byte

SmallOffset is used to store database offsets in the database file to save space rather than using int64. 5 bytes = 40 bits = 1tb In memory we use int64. Beware of padding or you won't actually save space.

type Stor

type Stor struct {
	// contains filtered or unexported fields
}

Stor is the externally visible storage

func HeapStor

func HeapStor(chunksize int) *Stor

HeapStor returns an empty in-memory stor for testing.

func MmapStor

func MmapStor(filename string, mode Mode) (*Stor, error)

MmapStor returns a memory mapped file stor.

func NewStor

func NewStor(impl storage, chunksize uint64, size uint64, chunks [][]byte) *Stor

func (*Stor) Alloc

func (s *Stor) Alloc(n int) (Offset, []byte)

Alloc allocates n bytes of storage and returns its Offset and byte slice Returning data here allows slicing to the correct length and capacity to prevent erroneously writing too far. If insufficient room in the current chunk, advance to next (allocations may not straddle chunks). Usual case is just an atomic increment, and an atomic load to check if hit the end of the chunk. Locking is only used to extend the storage.

func (*Stor) Close

func (s *Stor) Close(unmap bool, callback ...func(uint64))

func (*Stor) Data

func (s *Stor) Data(offset Offset) []byte

Data returns a byte slice starting at the given offset and extending to the end of the chunk since we don't know the size of the original alloc.

func (*Stor) FirstOffset

func (s *Stor) FirstOffset(off uint64, str string) uint64

FirstOffset searches forewards from a given offset for a given byte slice and returns the offset, or 0 if not found

func (*Stor) FlushTo

func (s *Stor) FlushTo(offset uint64)

Flush writes change to disk in the background. It is called by persist (e.g. once per minute).

func (*Stor) LastOffset

func (s *Stor) LastOffset(off uint64, str string) uint64

LastOffset searches backwards from a given offset for a given byte slice and returns the offset, or 0 if not found. It is used by repair and by asof/history.

func (*Stor) Reader

func (s *Stor) Reader(off uint64) *Reader

func (*Stor) Size

func (s *Stor) Size() uint64

Size returns the current (allocated) size of the data. The actual file size will be rounded up to the next chunk size.

func (*Stor) Write

func (s *Stor) Write(off uint64, data []byte)

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

func NewWriter

func NewWriter(buf []byte) *Writer

NewWriter returns a new Writer on a byte slice

func (*Writer) Len

func (w *Writer) Len() int

Len returns the current position within this writer

func (*Writer) Put1

func (w *Writer) Put1(n int) *Writer

Put1 writes an unsigned byte value

func (*Writer) Put2

func (w *Writer) Put2(n int) *Writer

Put2 writes an unsigned two byte value

func (*Writer) Put3

func (w *Writer) Put3(n int) *Writer

Put3 writes an unsigned three byte value

func (*Writer) Put4

func (w *Writer) Put4(n int) *Writer

Put4 writes an unsigned four byte value

func (*Writer) Put5

func (w *Writer) Put5(n uint64) *Writer

Put5 writes an unsigned five byte value

func (*Writer) PutStr

func (w *Writer) PutStr(s string) *Writer

PutStr writes a string with a maximum length of 64k

func (*Writer) PutStrs

func (w *Writer) PutStrs(ss []string) *Writer

PutStrs writes a slice of up to 64k strings