idxfile

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 24, 2021 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Package idxfile implements encoding and decoding of packfile idx files. 

== Original (version 1) pack-*.idx files have the following format:

  - The header consists of 256 4-byte network byte order
    integers.  N-th entry of this table records the number of
    objects in the corresponding pack, the first byte of whose
    object name is less than or equal to N.  This is called the
    'first-level fan-out' table.

  - The header is followed by sorted 24-byte entries, one entry
    per object in the pack.  Each entry is:

   4-byte network byte order integer, recording where the
   object is stored in the packfile as the offset from the
   beginning.

   20-byte object name.

 - The file is concluded with a trailer:

   A copy of the 20-byte SHA1 checksum at the end of
   corresponding packfile.

   20-byte SHA1-checksum of all of the above.

 Pack Idx file:

      --  +--------------------------------+
 fanout   | fanout[0] = 2 (for example)    |-.
 table    +--------------------------------+ |
          | fanout[1]                      | |
          +--------------------------------+ |
          | fanout[2]                      | |
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
          | fanout[255] = total objects    |---.
      --  +--------------------------------+ | |
 main     | offset                         | | |
 index    | object name 00XXXXXXXXXXXXXXXX | | |
 tab      +--------------------------------+ | |
          | offset                         | | |
          | object name 00XXXXXXXXXXXXXXXX | | |
          +--------------------------------+<+ |
        .-| offset                         |   |
        | | object name 01XXXXXXXXXXXXXXXX |   |
        | +--------------------------------+   |
        | | offset                         |   |
        | | object name 01XXXXXXXXXXXXXXXX |   |
        | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~   |
        | | offset                         |   |
        | | object name FFXXXXXXXXXXXXXXXX |   |
      --| +--------------------------------+<--+
trailer | | packfile checksum              |
        | +--------------------------------+
        | | idxfile checksum               |
        | +--------------------------------+
        .---------.
                  |
Pack file entry: <+

   packed object header:
   1-byte size extension bit (MSB)
         type (next 3 bit)
         size0 (lower 4-bit)
       n-byte sizeN (as long as MSB is set, each 7-bit)
       size0..sizeN form 4+7+7+..+7 bit integer, size0
       is the least significant part, and sizeN is the
       most significant part.
   packed object data:
       If it is not DELTA, then deflated bytes (the size above
       is the size before compression).
   If it is REF_DELTA, then
     20-byte base object name SHA1 (the size above is the
       size of the delta data that follows).
         delta data, deflated.
   If it is OFS_DELTA, then
     n-byte offset (see below) interpreted as a negative
       offset from the type-byte of the header of the
       ofs-delta entry (the size above is the size of
       the delta data that follows).
     delta data, deflated.

   offset encoding:
     n bytes with MSB set in all but the last one.
     The offset is then the number constructed by
     concatenating the lower 7 bit of each byte, and
     for n >= 2 adding 2^7 + 2^14 + ... + 2^(7*(n-1))
     to the result.

 == Version 2 pack-*.idx files support packs larger than 4 GiB, and
    have some other reorganizations.  They have the format:

   - A 4-byte magic number '\377tOc' which is an unreasonable
     fanout[0] value.

   - A 4-byte version number (= 2)

   - A 256-entry fan-out table just like v1.

   - A table of sorted 20-byte SHA1 object names.  These are
     packed together without offset values to reduce the cache
     footprint of the binary search for a specific object name.

   - A table of 4-byte CRC32 values of the packed object data.
     This is new in v2 so compressed data can be copied directly
     from pack to pack during repacking without undetected
     data corruption.

   - A table of 4-byte offset values (in network byte order).
     These are usually 31-bit pack file offsets, but large
     offsets are encoded as an index into the next table with
     the msbit set.

   - A table of 8-byte offset entries (empty for pack files less
     than 2 GiB).  Pack files are organized with heavily used
     objects toward the front, so most object references should
     not need to refer to this table.

   - The same trailer as a v1 pack file:

     A copy of the 20-byte SHA1 checksum at the end of
     corresponding packfile.

     20-byte SHA1-checksum of all of the above.

Source: https://www.kernel.org/pub/software/scm/git/docs/v1.7.5/technical/pack-format.txt

Index

Constants

View Source 
const (
	// VersionSupported is the only idx version supported.
	VersionSupported = 2
)

Variables

View Source 
var (
	// ErrUnsupportedVersion is returned by Decode when the idx file version
	// is not supported.
	ErrUnsupportedVersion = errors.New("Unsupported version")
	// ErrMalformedIdxFile is returned by Decode when the idx file is corrupted.
	ErrMalformedIdxFile = errors.New("Malformed IDX file")
)

Functions

This section is empty.

Types

type Decoder 

type Decoder struct {
	*bufio.Reader
}

Decoder reads and decodes idx files from an input stream.

func NewDecoder 

func NewDecoder(r io.Reader) *Decoder

NewDecoder builds a new idx stream decoder, that reads from r.

func (*Decoder) Decode 

func (d *Decoder) Decode(idx *MemoryIndex) error

Decode reads from the stream and decode the content into the MemoryIndex struct.

type Encoder 

type Encoder struct {
	io.Writer
	// contains filtered or unexported fields
}

Encoder writes MemoryIndex structs to an output stream.

func NewEncoder 

func NewEncoder(w io.Writer) *Encoder

NewEncoder returns a new stream encoder that writes to w.

func (*Encoder) Encode 

func (e *Encoder) Encode(idx *MemoryIndex) (int, error)

Encode encodes an MemoryIndex to the encoder writer.

type Entry 

type Entry struct {
	Hash   plumbing.Hash
	CRC32  uint32
	Offset uint64
}

Entry is the in memory representation of an object entry in the idx file.

type EntryIter 

type EntryIter interface {
	// Next returns the next entry in the packfile index.
	Next() (*Entry, error)
	// Close closes the iterator.
	Close() error
}

EntryIter is an iterator that will return the entries in a packfile index.

type Index 

type Index interface {
	// Contains checks whether the given hash is in the index.
	Contains(h plumbing.Hash) (bool, error)
	// FindOffset finds the offset in the packfile for the object with
	// the given hash.
	FindOffset(h plumbing.Hash) (int64, error)
	// FindCRC32 finds the CRC32 of the object with the given hash.
	FindCRC32(h plumbing.Hash) (uint32, error)
	// FindHash finds the hash for the object with the given offset.
	FindHash(o int64) (plumbing.Hash, error)
	// Count returns the number of entries in the index.
	Count() (int64, error)
	// Entries returns an iterator to retrieve all index entries.
	Entries() (EntryIter, error)
	// EntriesByOffset returns an iterator to retrieve all index entries ordered
	// by offset.
	EntriesByOffset() (EntryIter, error)
}

Index represents an index of a packfile.

type MemoryIndex 

type MemoryIndex struct {
	Version uint32
	Fanout  [256]uint32
	// FanoutMapping maps the position in the fanout table to the position
	// in the Names, Offset32 and CRC32 slices. This improves the memory
	// usage by not needing an array with unnecessary empty slots.
	FanoutMapping    [256]int
	Names            [][]byte
	Offset32         [][]byte
	CRC32            [][]byte
	Offset64         []byte
	PackfileChecksum [20]byte
	IdxChecksum      [20]byte
	// contains filtered or unexported fields
}

MemoryIndex is the in memory representation of an idx file.

func NewMemoryIndex 

func NewMemoryIndex() *MemoryIndex

NewMemoryIndex returns an instance of a new MemoryIndex.

func (*MemoryIndex) Contains 

func (idx *MemoryIndex) Contains(h plumbing.Hash) (bool, error)

Contains implements the Index interface.

func (*MemoryIndex) Count 

func (idx *MemoryIndex) Count() (int64, error)

Count implements the Index interface.

func (*MemoryIndex) Entries 

func (idx *MemoryIndex) Entries() (EntryIter, error)

Entries implements the Index interface.

func (*MemoryIndex) EntriesByOffset 

func (idx *MemoryIndex) EntriesByOffset() (EntryIter, error)

EntriesByOffset implements the Index interface.

func (*MemoryIndex) FindCRC32 

func (idx *MemoryIndex) FindCRC32(h plumbing.Hash) (uint32, error)

FindCRC32 implements the Index interface.

func (*MemoryIndex) FindHash 

func (idx *MemoryIndex) FindHash(o int64) (plumbing.Hash, error)

FindHash implements the Index interface.

func (*MemoryIndex) FindOffset 

func (idx *MemoryIndex) FindOffset(h plumbing.Hash) (int64, error)

FindOffset implements the Index interface.

type Writer 

type Writer struct {
	// contains filtered or unexported fields
}

Writer implements a packfile Observer interface and is used to generate indexes.

func (*Writer) Add 

func (w *Writer) Add(h plumbing.Hash, pos uint64, crc uint32)

Add appends new object data.

func (*Writer) Finished 

func (w *Writer) Finished() bool

func (*Writer) Index 

func (w *Writer) Index() (*MemoryIndex, error)

Index returns a previously created MemoryIndex or creates a new one if needed.

func (*Writer) OnFooter 

func (w *Writer) OnFooter(h plumbing.Hash) error

OnFooter implements packfile.Observer interface.

func (*Writer) OnHeader 

func (w *Writer) OnHeader(count uint32) error

OnHeader implements packfile.Observer interface.

func (*Writer) OnInflatedObjectContent 

func (w *Writer) OnInflatedObjectContent(h plumbing.Hash, pos int64, crc uint32, _ []byte) error

OnInflatedObjectContent implements packfile.Observer interface.

func (*Writer) OnInflatedObjectHeader 

func (w *Writer) OnInflatedObjectHeader(t plumbing.ObjectType, objSize int64, pos int64) error

OnInflatedObjectHeader implements packfile.Observer interface.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.