kar

README

kar - Koru Archive

kar is a package that provides a file format for use in memory mapping. Storage of read-only assets that need to be loaded into usable state quickly. Every file in the archive is individually compressed, while the archive itself is kept raw. Files are compressed with lz4, thus they decompress very quickly on the fly.

Properties of kar

• performant with mmap
• index of files is in the header, it is known prior reading exactly where the files are and how big they are
• non-appendable, intended to be a read-only distributable archive
• safe to use concurrently

Documentation

Overview

Package kar is an api for an lz4 backed file format. It's purpose is to be well suited for resource streaming resources from it. It's designed to be memory mapped, so (unlike tar) it knows where all the files are located before they're read. This nescesitates a bit of an unusual setup, where the archive itself is not compressed in any form, rather every file is individually compressed, so it could be immediately read from it's place and decompressed on the fly. This somewhat compromises space efficiency, but space efficiency is not the primary goal of this package. It instead focuses on getting resources from disk to a usable state as fast as possible. It can be read from concurrently.

Index

• Constants
• Variables
• type Archive
  • func Open(r io.ReaderAt) (*Archive, error)
  • func (a *Archive) GetFileInfo(name string) (IndexEntry, error)
  • func (a *Archive) Open(name string) (*Reader, error)
  • func (a *Archive) ReadAll(name string) ([]byte, error)
• type Builder
  • func NewBuilder(header Header) (*Builder, error)
  • func (b *Builder) Add(name string, r io.Reader) error
  • func (b *Builder) WriteTo(w io.Writer) (int64, error)
• type Header
  • func (h *Header) MaxExpectedSize() int64
• type IndexEntry
• type Reader
  • func (r *Reader) Read(p []byte) (n int, err error)

Constants

const (
	MagicLength            = 4
	HeaderSizeNumberLength = 16
)

Sizes relevant to the header of file

Variables

var (
	ErrFileFormat = errors.New("corrupted or not a kar archive")
	ErrTempFail   = errors.New("temporary folder or file operation failed")
	ErrIOMisc     = errors.New("some unknown error unhandled by the io occured")
)

package errors

Types

type Archive

type Archive struct {
	// contains filtered or unexported fields
}

Archive provides concurrent io for a kar file, and can provide an io.Reader for each file separately to perform actions on.

func Open

func Open(r io.ReaderAt) (*Archive, error)

Open opens the kar archived from r. It will also check if the file is actually a kar archive, will return an error when file incorrect.

func (*Archive) GetFileInfo

func (a *Archive) GetFileInfo(name string) (IndexEntry, error)

GetFileInfo queries for a file with a given name in the archive and returns it's info if found. If not found it will return os.ErrNotExist error.

func (*Archive) Open

func (a *Archive) Open(name string) (*Reader, error)

Open returns a Reader for a file in the Archive

func (*Archive) ReadAll

func (a *Archive) ReadAll(name string) ([]byte, error)

ReadAll returns the entire contents of a file with a given name

type Builder

type Builder struct {
	io.WriterTo
	// contains filtered or unexported fields
}

Builder is the high level builder for the archive format. Arhives are versioned and cannot be appended to, This Builder is the way to create an archive. Whenever Add is called, KarBuilder will create a temporary dir, where it will store compressed files, then finally bundling them togeter and writing them out with WriteTo.

func NewBuilder

func NewBuilder(header Header) (*Builder, error)

NewBuilder creates a new Builder. Do not fill the Index in the header, it will be overwritten anyway.

func (*Builder) Add

func (b *Builder) Add(name string, r io.Reader) error

Add appends data to the builder with a given name. Will block until lz4 finishes compression. Is safe to use concurrently in different goroutines.

func (*Builder) WriteTo

func (b *Builder) WriteTo(w io.Writer) (int64, error)

WriteTo bundles and writes all of the files added to the Builder into a kar archive that is ready to use. This function may block for a long time, cosidering it does a lot of operations. It will keep the mutex locked.

type Header

type Header struct {
	Author      string
	DateCreated int64
	Version     int64
	Index       []IndexEntry
}

Header is the file header for kar files.

func (*Header) MaxExpectedSize

func (h *Header) MaxExpectedSize() int64

MaxExpectedSize calculates the amount of space a Header could take. It's important to know this before writing the header into the file. It only needs to be roughtly correct, offsets will be calculated with consideration for this number

type IndexEntry

type IndexEntry struct {
	Name           string
	Offset         int64
	Size           int64
	CompressedSize int64
}

IndexEntry is info for one file in the file index.

type Reader

type Reader struct {
	io.Reader
	// contains filtered or unexported fields
}

Reader is a reader for a single file in an Archive. Abstracts away the location that needs to be known.

func (*Reader) Read

func (r *Reader) Read(p []byte) (n int, err error)

Read reads already decompressed data