devmapper

README

devicemapper - a storage backend based on Device Mapper

Theory of operation

The device mapper graphdriver uses the device mapper thin provisioning module (dm-thinp) to implement CoW snapshots. For each devicemapper graph location (typically /var/lib/docker/devicemapper, $graph below) a thin pool is created based on two block devices, one for data and one for metadata. By default these block devices are created automatically by using loopback mounts of automatically created sparse files.

The default loopback files used are $graph/devicemapper/data and $graph/devicemapper/metadata. Additional metadata required to map from docker entities to the corresponding devicemapper volumes is stored in the $graph/devicemapper/json file (encoded as Json).

In order to support multiple devicemapper graphs on a system, the thin pool will be named something like: docker-0:33-19478248-pool, where the 0:33 part is the minor/major device nr and 19478248 is the inode number of the $graph directory.

On the thin pool, docker automatically creates a base thin device, called something like docker-0:33-19478248-base of a fixed size. This is automatically formatted with an empty filesystem on creation. This device is the base of all docker images and containers. All base images are snapshots of this device and those images are then in turn used as snapshots for other images and eventually containers.

options

The devicemapper backend supports some options that you can specify when starting the docker daemon using the --storage-opt flags. This uses the dm prefix and would be used something like docker -d --storage-opt dm.foo=bar.

Here is the list of supported options:

• dm.basesize

  Specifies the size to use when creating the base device, which limits the size of images and containers. The default value is 10G. Note, thin devices are inherently "sparse", so a 10G device which is mostly empty doesn't use 10 GB of space on the pool. However, the filesystem will use more space for the empty case the larger the device is. Warning: This value affects the system-wide "base" empty filesystem that may already be initialized and inherited by pulled images. Typically, a change to this value will require additional steps to take effect: 1) stop docker -d, 2) rm -rf /var/lib/docker, 3) start docker -d.

  Example use:

  docker -d --storage-opt dm.basesize=20G

• dm.loopdatasize

  Specifies the size to use when creating the loopback file for the "data" device which is used for the thin pool. The default size is 100G. Note that the file is sparse, so it will not initially take up this much space.

  Example use:

  docker -d --storage-opt dm.loopdatasize=200G

• dm.loopmetadatasize

  Specifies the size to use when creating the loopback file for the "metadadata" device which is used for the thin pool. The default size is 2G. Note that the file is sparse, so it will not initially take up this much space.

  Example use:

  docker -d --storage-opt dm.loopmetadatasize=4G

• dm.fs

  Specifies the filesystem type to use for the base device. The supported options are "ext4" and "xfs". The default is "ext4"

  Example use:

  docker -d --storage-opt dm.fs=xfs

• dm.mkfsarg

  Specifies extra mkfs arguments to be used when creating the base device.

  Example use:

  docker -d --storage-opt "dm.mkfsarg=-O ^has_journal"

• dm.mountopt

  Specifies extra mount options used when mounting the thin devices.

  Example use:

  docker -d --storage-opt dm.mountopt=nodiscard

• dm.thinpooldev

  Specifies a custom blockdevice to use for the thin pool.

  If using a block device for device mapper storage, ideally lvm2 would be used to create/manage the thin-pool volume that is then handed to docker to exclusively create/manage the thin and thin snapshot volumes needed for it's containers. Managing the thin-pool outside of docker makes for the most feature-rich method of having docker utilize device mapper thin provisioning as the backing storage for docker's containers. lvm2-based thin-pool management feature highlights include: automatic or interactive thin-pool resize support, dynamically change thin-pool features, automatic thinp metadata checking when lvm2 activates the thin-pool, etc.

  Example use:

  docker -d --storage-opt dm.thinpooldev=/dev/mapper/thin-pool

• dm.datadev

  Specifies a custom blockdevice to use for data for the thin pool.

  If using a block device for device mapper storage, ideally both datadev and metadatadev should be specified to completely avoid using the loopback device.

  Example use:

  docker -d --storage-opt dm.datadev=/dev/sdb1 --storage-opt dm.metadatadev=/dev/sdc1

• dm.metadatadev

  Specifies a custom blockdevice to use for metadata for the thin pool.

  For best performance the metadata should be on a different spindle than the data, or even better on an SSD.

  If setting up a new metadata pool it is required to be valid. This can be achieved by zeroing the first 4k to indicate empty metadata, like this:

  ``dd if=/dev/zero of=$metadata_dev bs=4096 count=1```

  Example use:

  docker -d --storage-opt dm.datadev=/dev/sdb1 --storage-opt dm.metadatadev=/dev/sdc1

• dm.blocksize

  Specifies a custom blocksize to use for the thin pool. The default blocksize is 64K.

  Example use:

  docker -d --storage-opt dm.blocksize=512K

• dm.blkdiscard

  Enables or disables the use of blkdiscard when removing devicemapper devices. This is enabled by default (only) if using loopback devices and is required to res-parsify the loopback file on image/container removal.

  Disabling this on loopback can lead to much faster container removal times, but will make the space used in /var/lib/docker directory not be returned to the system for other use when containers are removed.

  Example use:

  docker -d --storage-opt dm.blkdiscard=false

Documentation

Index

• Variables
• func Init(home string, options []string) (graphdriver.Driver, error)
• func Mounted(mountpoint string) (bool, error)
• func ProbeFsType(device string) (string, error)
• type DevInfo
  • func (info *DevInfo) DevName() string
  • func (info *DevInfo) Name() string
• type DevStatus
• type DeviceSet
  • func NewDeviceSet(root string, doInit bool, options []string) (*DeviceSet, error)
  • func (devices *DeviceSet) AddDevice(hash, baseHash string) error
  • func (devices *DeviceSet) DMLog(level int, file string, line int, dmError int, message string)
  • func (devices *DeviceSet) DeleteDevice(hash string) error
  • func (devices *DeviceSet) GetDeviceStatus(hash string) (*DevStatus, error)
  • func (devices *DeviceSet) HasActivatedDevice(hash string) bool
  • func (devices *DeviceSet) HasDevice(hash string) bool
  • func (devices *DeviceSet) List() []string
  • func (devices *DeviceSet) MountDevice(hash, path, mountLabel string) error
  • func (devices *DeviceSet) ResizePool(size int64) error
  • func (devices *DeviceSet) Shutdown() error
  • func (devices *DeviceSet) Status() *Status
  • func (devices *DeviceSet) UnmountDevice(hash string) error
• type DiskUsage
• type Driver
  • func (d *Driver) Cleanup() error
  • func (d *Driver) Create(id, parent string) error
  • func (d *Driver) Exists(id string) bool
  • func (d *Driver) Get(id, mountLabel string) (string, error)
  • func (d *Driver) Put(id string)
  • func (d *Driver) Remove(id string) error
  • func (d *Driver) Status() [][2]string
  • func (d *Driver) String() string
• type MetaData
• type Status
• type Transaction

Variables

var (
	DefaultDataLoopbackSize     int64  = 100 * 1024 * 1024 * 1024
	DefaultMetaDataLoopbackSize int64  = 2 * 1024 * 1024 * 1024
	DefaultBaseFsSize           uint64 = 10 * 1024 * 1024 * 1024
	DefaultThinpBlockSize       uint32 = 128      // 64K = 128 512b sectors
	MaxDeviceId                 int    = 0xffffff // 24 bit, pool limit
	DeviceIdMapSz               int    = (MaxDeviceId + 1) / 8
)

Functions

func Init

func Init(home string, options []string) (graphdriver.Driver, error)

func Mounted

func Mounted(mountpoint string) (bool, error)

func ProbeFsType

func ProbeFsType(device string) (string, error)

Types

type DevInfo

type DevInfo struct {
	Hash          string `json:"-"`
	DeviceId      int    `json:"device_id"`
	Size          uint64 `json:"size"`
	TransactionId uint64 `json:"transaction_id"`
	Initialized   bool   `json:"initialized"`
	// contains filtered or unexported fields
}

func (*DevInfo) DevName

func (info *DevInfo) DevName() string

func (*DevInfo) Name

func (info *DevInfo) Name() string

type DevStatus

type DevStatus struct {
	DeviceId            int
	Size                uint64
	TransactionId       uint64
	SizeInSectors       uint64
	MappedSectors       uint64
	HighestMappedSector uint64
}

type DeviceSet

type DeviceSet struct {
	MetaData   `json:"-"`
	sync.Mutex `json:"-"` // Protects Devices map and serializes calls into libdevmapper

	TransactionId uint64 `json:"-"`
	NextDeviceId  int    `json:"next_device_id"`

	Transaction `json:"-"`
	// contains filtered or unexported fields
}

func NewDeviceSet

func NewDeviceSet(root string, doInit bool, options []string) (*DeviceSet, error)

func (*DeviceSet) AddDevice

func (devices *DeviceSet) AddDevice(hash, baseHash string) error

func (*DeviceSet) DMLog

func (devices *DeviceSet) DMLog(level int, file string, line int, dmError int, message string)

func (*DeviceSet) DeleteDevice

func (devices *DeviceSet) DeleteDevice(hash string) error

func (*DeviceSet) GetDeviceStatus

func (devices *DeviceSet) GetDeviceStatus(hash string) (*DevStatus, error)

func (*DeviceSet) HasActivatedDevice

func (devices *DeviceSet) HasActivatedDevice(hash string) bool

func (*DeviceSet) HasDevice

func (devices *DeviceSet) HasDevice(hash string) bool

func (*DeviceSet) List

func (devices *DeviceSet) List() []string

func (*DeviceSet) MountDevice

func (devices *DeviceSet) MountDevice(hash, path, mountLabel string) error

func (*DeviceSet) ResizePool

func (devices *DeviceSet) ResizePool(size int64) error

func (*DeviceSet) Shutdown

func (devices *DeviceSet) Shutdown() error

func (*DeviceSet) Status

func (devices *DeviceSet) Status() *Status

func (*DeviceSet) UnmountDevice

func (devices *DeviceSet) UnmountDevice(hash string) error

type DiskUsage

type DiskUsage struct {
	Used  uint64
	Total uint64
}

type Driver

type Driver struct {
	*DeviceSet
	// contains filtered or unexported fields
}

func (*Driver) Cleanup

func (d *Driver) Cleanup() error

func (*Driver) Create

func (d *Driver) Create(id, parent string) error

func (*Driver) Exists

func (d *Driver) Exists(id string) bool

func (*Driver) Get

func (d *Driver) Get(id, mountLabel string) (string, error)

func (*Driver) Put

func (d *Driver) Put(id string)

func (*Driver) Remove

func (d *Driver) Remove(id string) error

func (*Driver) Status

func (d *Driver) Status() [][2]string

func (*Driver) String

func (d *Driver) String() string

type MetaData

type MetaData struct {
	Devices map[string]*DevInfo `json:"Devices"`
	// contains filtered or unexported fields
}

type Status

type Status struct {
	PoolName         string
	DataLoopback     string
	MetadataLoopback string
	Data             DiskUsage
	Metadata         DiskUsage
	SectorSize       uint64
}

type Transaction

type Transaction struct {
	OpenTransactionId uint64 `json:"open_transaction_id"`
	DeviceIdHash      string `json:"device_hash"`
	DeviceId          int    `json:"device_id"`
}