blobpacked

Documentation

Overview

Package blobpacked registers the "blobpacked" blobserver storage type, storing blobs initially as one physical blob per logical blob, but then rearranging little physical blobs into large contiguous blobs organized by how they'll likely be accessed. An index tracks the mapping from logical to physical blobs.

Example low-level config:

"/storage/": {
    "handler": "storage-blobpacked",
    "handlerArgs": {
       "smallBlobs": "/small/",
       "largeBlobs": "/large/",
       "metaIndex": {
          "type": "mysql",
           .....
       }
     }
}

The resulting large blobs are valid zip files. Those blobs may up be up to 16 MB and contain the original contiguous file (or fractions of it), as well as metadata about how the file is cut up. The zip file will have the following structure:

foo.jpg       (or whatever)
camlistore/sha1-beb1df0b75952c7d277905ad14de71ef7ef90c44.json (some file ref)
camlistore/sha1-a0ceb10b04403c9cc1d032e07a9071db5e711c9a.json (some bytes ref)
camlistore/sha1-7b4d9c8529c27d592255c6dfb17188493db96ccc.json (another bytes ref)
camlistore/camlistore-pack-manifest.json

The camlistore-pack-manifest.json is documented on the exported Manifest type. It looks like this:

{
  "wholeRef": "sha1-0e64816d731a56915e8bb4ae4d0ac7485c0b84da",
  "wholeSize": 2962227200, // 2.8GB; so will require ~176-180 16MB chunks
  "wholePartIndex": 17,    // 0-based
  "dataBlobsOrigin": "sha1-355705cf62a56669303d2561f29e0620a676c36e",
  "dataBlobs": [
      {"blob": "sha1-f1d2d2f924e986ac86fdf7b36c94bcdf32beec15", "offset": 0, "size": 273048},
      {"blob": "sha1-e242ed3bffccdf271b7fbaf34ed72d089537b42f", "offset": 273048, "size": 112783},
      {"blob": "sha1-6eadeac2dade6347e87c0d24fd455feffa7069f0", "offset": 385831, ...},
      {"blob": "sha1-9425cca1dde5d8b6eb70cd087db4e356da92396e", "offset": ...},
      {"blob": "sha1-7709559a3c8668c57cc0a2f57c418b1cc3598049", "offset": ...},
      {"blob": "sha1-f62cb5d05cfbf2a7a6c7f8339d0a4bf1dcd0ab6c", "offset": ...}
  ] // raw data blobs of foo.jpg
}

The manifest.json ensures that if the metadata index is lost, all the data can be reconstructed from the raw zip files.

The 'wholeRef' property specifies which large file that this zip is building up. If the file is less than 15.5 MB or so (leaving room for the zip overhead and manifest size), it will probably all be in one zip and the first file in the zip will be the whole thing. Otherwise it'll be cut across multiple zip files, each no larger than 16MB. In that case, each part of the file will have a different 'wholePartIndex' number, starting at index 0. Each will have the same 'wholeSize'.

Index

• func SetRecovery()
• func UnsetRecovery()
• func WipeMeta(s blobserver.Storage) error
• type BlobAndPos
• type Manifest

Functions

func SetRecovery

func SetRecovery()

SetRecovery notes that the user ran the camlistored binary with the --recovery flag. It means that any blobpacked storage subsequently initialized will automatically start with rebuilding its meta index of zip files.

func UnsetRecovery

func UnsetRecovery()

See SetRecovery.

func WipeMeta

func WipeMeta(s blobserver.Storage) error

Types

type BlobAndPos

type BlobAndPos struct {
	blob.SizedRef
	Offset int64 `json:"offset"`
}

A BlobAndPos is a blobref, its size, and where it is located within a larger group of bytes.

type Manifest

type Manifest struct {
	// WholeRef is the blobref of the entire file that this zip is
	// either fully or partially describing.  For files under
	// around 16MB, the WholeRef and DataBlobsOrigin will be
	// the same.
	WholeRef blob.Ref `json:"wholeRef"`

	// WholeSize is the number of bytes in the original file being
	// cut up.
	WholeSize int64 `json:"wholeSize"`

	// WholePartIndex is the chunk number (0-based) of this zip file.
	// If a client has 'n' zip files with the same WholeRef whose
	// WholePartIndexes are contiguous (including 0) and the sum of
	// the DataBlobs equals WholeSize, the client has the entire
	// original file.
	WholePartIndex int `json:"wholePartIndex"`

	// DataBlobsOrigin is the blobref of the contents of the first
	// file in the zip pack file. It is the origin of all the logical data
	// blobs referenced in DataBlobs.
	DataBlobsOrigin blob.Ref `json:"dataBlobsOrigin"`

	// DataBlobs describes all the logical blobs that are
	// concatenated together in the first file in the zip file.
	// The offsets are relative to the beginning of that first
	// file, not the beginning of the zip file itself.
	DataBlobs []BlobAndPos `json:"dataBlobs"`
}

Manifest is the JSON description type representing the "camlistore/camlistore-pack-manifest.json" file found in a blobpack zip file.