devmapper

package
v1.6.33 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 5, 2024 License: Apache-2.0 Imports: 24 Imported by: 47

README

Devmapper snapshotter

Devmapper is a containerd snapshotter plugin that stores snapshots in ext4-formatted filesystem images in a devicemapper thin pool.

Setup

To make it work you need to prepare thin-pool in advance and update containerd's configuration file. This file is typically located at /etc/containerd/config.toml.

Here's minimal sample entry that can be made in the configuration file:

version = 2

[plugins]
  ...
  [plugins."io.containerd.snapshotter.v1.devmapper"]
    pool_name = "containerd-pool"
    base_image_size = "8192MB"
  ...

The following configuration flags are supported:

  • root_path - a directory where the metadata will be available (if empty default location for containerd plugins will be used)
  • pool_name - a name to use for the devicemapper thin pool. Pool name should be the same as in /dev/mapper/ directory
  • base_image_size - defines how much space to allocate when creating the base device
  • async_remove - flag to async remove device using snapshot GC's cleanup callback
  • discard_blocks - whether to discard blocks when removing a device. This is especially useful for returning disk space to the filesystem when using loopback devices.
  • fs_type - defines the file system to use for snapshot device mount. Valid values are ext4 and xfs. Defaults to ext4 if unspecified.
  • fs_options - optionally defines the file system options. This is currently only applicable to ext4 file system.

Pool name and base image size are required snapshotter parameters.

Run

Give it a try with the following commands:

ctr images pull --snapshotter devmapper docker.io/library/hello-world:latest
ctr run --snapshotter devmapper docker.io/library/hello-world:latest test

Requirements

The devicemapper snapshotter requires dmsetup (>= 1.02.110) command line tool to be installed and available on your computer. On Ubuntu, it can be installed with apt-get install dmsetup command.

How to setup device mapper thin-pool

There are many ways how to configure a devmapper thin-pool depending on your requirements, disk configuration, and environment.

On local dev environment you can utilize loopback devices. This type of configuration is simple and suits well for development and testing (please note that this configuration is slow and not recommended for production uses). Run the following script to create a thin-pool device:

#!/bin/bash
set -ex

DATA_DIR=/var/lib/containerd/devmapper
POOL_NAME=devpool

mkdir -p ${DATA_DIR}

# Create data file
sudo touch "${DATA_DIR}/data"
sudo truncate -s 100G "${DATA_DIR}/data"

# Create metadata file
sudo touch "${DATA_DIR}/meta"
sudo truncate -s 10G "${DATA_DIR}/meta"

# Allocate loop devices
DATA_DEV=$(sudo losetup --find --show "${DATA_DIR}/data")
META_DEV=$(sudo losetup --find --show "${DATA_DIR}/meta")

# Define thin-pool parameters.
# See https://www.kernel.org/doc/Documentation/device-mapper/thin-provisioning.txt for details.
SECTOR_SIZE=512
DATA_SIZE="$(sudo blockdev --getsize64 -q ${DATA_DEV})"
LENGTH_IN_SECTORS=$(bc <<< "${DATA_SIZE}/${SECTOR_SIZE}")
DATA_BLOCK_SIZE=128
LOW_WATER_MARK=32768

# Create a thin-pool device
sudo dmsetup create "${POOL_NAME}" \
    --table "0 ${LENGTH_IN_SECTORS} thin-pool ${META_DEV} ${DATA_DEV} ${DATA_BLOCK_SIZE} ${LOW_WATER_MARK}"

cat << EOF
#
# Add this to your config.toml configuration file and restart containerd daemon
#
[plugins]
  [plugins.devmapper]
    pool_name = "${POOL_NAME}"
    root_path = "${DATA_DIR}"
    base_image_size = "10GB"
    discard_blocks = true
EOF

Use dmsetup to verify that the thin-pool created successfully:

sudo dmsetup ls
devpool	(253:0)

Once configured and restarted containerd, you'll see the following output:

INFO[2020-03-17T20:24:45.532604888Z] loading plugin "io.containerd.snapshotter.v1.devmapper"...  type=io.containerd.snapshotter.v1
INFO[2020-03-17T20:24:45.532672738Z] initializing pool device "dev-pool"

Another way to setup a thin-pool is via container-storage-setup tool (formerly known as docker-storage-setup). It is a script to configure CoW file systems like devicemapper:

#!/bin/bash
set -ex

# Block device to use for devmapper thin-pool
BLOCK_DEV=/dev/sdf
POOL_NAME=devpool
VG_NAME=containerd

# Install container-storage-setup tool
git clone https://github.com/projectatomic/container-storage-setup.git
cd container-storage-setup/
sudo make install-core
echo "Using version $(container-storage-setup -v)"

# Create configuration file
# Refer to `man container-storage-setup` to see available options
sudo tee /etc/sysconfig/docker-storage-setup <<EOF
DEVS=${BLOCK_DEV}
VG=${VG_NAME}
CONTAINER_THINPOOL=${POOL_NAME}
EOF

# Run the script
sudo container-storage-setup

cat << EOF
#
# Add this to your config.toml configuration file and restart containerd daemon
#
[plugins]
  [plugins.devmapper]
    pool_name = "${VG_NAME}-${POOL_NAME}"
    base_image_size = "10GB"
EOF

If successful container-storage-setup will output:

+ echo VG=containerd
+ sudo container-storage-setup
INFO: Volume group backing root filesystem could not be determined
INFO: Writing zeros to first 4MB of device /dev/xvdf
4+0 records in
4+0 records out
4194304 bytes (4.2 MB) copied, 0.0162906 s, 257 MB/s
INFO: Device node /dev/xvdf1 exists.
  Physical volume "/dev/xvdf1" successfully created.
  Volume group "containerd" successfully created
  Rounding up size to full physical extent 12.00 MiB
  Thin pool volume with chunk size 512.00 KiB can address at most 126.50 TiB of data.
  Logical volume "devpool" created.
  Logical volume containerd/devpool changed.
...

And dmsetup will produce the following output:

sudo dmsetup ls
containerd-devpool          (253:2)
containerd-devpool_tdata    (253:1)
containerd-devpool_tmeta    (253:0)

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNotFound represents an error returned when object not found in meta store
	ErrNotFound = errors.New("not found")
	// ErrAlreadyExists represents an error returned when object can't be duplicated in meta store
	ErrAlreadyExists = errors.New("object already exists")
)

Functions

This section is empty.

Types

type Config 

type Config struct {
	// Device snapshotter root directory for metadata
	RootPath string `toml:"root_path"`

	// Name for 'thin-pool' device to be used by snapshotter (without /dev/mapper/ prefix)
	PoolName string `toml:"pool_name"`

	// Defines how much space to allocate when creating base image for container
	BaseImageSize      string `toml:"base_image_size"`
	BaseImageSizeBytes uint64 `toml:"-"`

	// Flag to async remove device using Cleanup() callback in snapshots GC
	AsyncRemove bool `toml:"async_remove"`

	// Whether to discard blocks when removing a thin device.
	DiscardBlocks bool `toml:"discard_blocks"`

	// Defines file system to use for snapshout device mount. Defaults to "ext4"
	FileSystemType fsType `toml:"fs_type"`

	// Defines optional file system options passed through config file
	FsOptions string `toml:"fs_options"`
}

Config represents device mapper configuration loaded from file. Size units can be specified in human-readable string format (like "32KIB", "32GB", "32Tb")

func LoadConfig 

func LoadConfig(path string) (*Config, error)

LoadConfig reads devmapper configuration file from disk in TOML format

func (*Config) Validate 

func (c *Config) Validate() error

Validate makes sure configuration fields are valid

type DeviceInfo 

type DeviceInfo struct {
	// DeviceID is a 24-bit number assigned to a device within thin-pool device
	DeviceID uint32 `json:"device_id"`
	// Size is a thin device size
	Size uint64 `json:"size"`
	// Name is a device name to be used in /dev/mapper/
	Name string `json:"name"`
	// ParentName is a name of parent device (if snapshot)
	ParentName string `json:"parent_name"`
	// State represents current device state
	State DeviceState `json:"state"`
	// Error details if device state change failed
	Error string `json:"error"`
}

DeviceInfo represents metadata for thin device within thin-pool

type DeviceInfoCallback 

type DeviceInfoCallback func(deviceInfo *DeviceInfo) error

DeviceInfoCallback is a callback used for device updates

type DeviceState 

type DeviceState int

DeviceState represents current devmapper device state reflected in meta store 

const (
	// Unknown means that device just allocated and no operations were performed
	Unknown DeviceState = iota
	// Creating means that device is going to be created
	Creating
	// Created means that devices successfully created
	Created
	// Activating means that device is going to be activated
	Activating
	// Activated means that device successfully activated
	Activated
	// Suspending means that device is going to be suspended
	Suspending
	// Suspended means that device successfully suspended
	Suspended
	// Resuming means that device is going to be resumed from suspended state
	Resuming
	// Resumed means that device successfully resumed
	Resumed
	// Deactivating means that device is going to be deactivated
	Deactivating
	// Deactivated means that device successfully deactivated
	Deactivated
	// Removing means that device is going to be removed
	Removing
	// Removed means that device successfully removed but not yet deleted from meta store
	Removed
	// Faulty means that the device is errored and the snapshotter failed to rollback it
	Faulty
)

func (DeviceState) String 

func (s DeviceState) String() string

type PoolDevice 

type PoolDevice struct {
	// contains filtered or unexported fields
}

PoolDevice ties together data and metadata volumes, represents thin-pool and manages volumes, snapshots and device ids.

func NewPoolDevice 

func NewPoolDevice(ctx context.Context, config *Config) (*PoolDevice, error)

NewPoolDevice creates new thin-pool from existing data and metadata volumes. If pool 'poolName' already exists, it'll be reloaded with new parameters.

func (*PoolDevice) Close 

func (p *PoolDevice) Close() error

Close closes pool device (thin-pool will not be removed)

func (*PoolDevice) CreateSnapshotDevice 

func (p *PoolDevice) CreateSnapshotDevice(ctx context.Context, deviceName string, snapshotName string, virtualSizeBytes uint64) (retErr error)

CreateSnapshotDevice creates and activates new thin-device from parent thin-device (makes snapshot)

func (*PoolDevice) CreateThinDevice 

func (p *PoolDevice) CreateThinDevice(ctx context.Context, deviceName string, virtualSizeBytes uint64) (retErr error)

CreateThinDevice creates new devmapper thin-device with given name and size. Device ID for thin-device will be allocated from metadata store. If allocation successful, device will be activated with /dev/mapper/<deviceName>

func (*PoolDevice) DeactivateDevice 

func (p *PoolDevice) DeactivateDevice(ctx context.Context, deviceName string, deferred, withForce bool) error

DeactivateDevice deactivates thin device

func (*PoolDevice) GetUsage 

func (p *PoolDevice) GetUsage(deviceName string) (int64, error)

GetUsage reports total size in bytes consumed by a thin-device. It relies on the number of used blocks reported by 'dmsetup status'. The output looks like: 

device2: 0 204800 thin 17280 204799

Where 17280 is the number of used sectors

func (*PoolDevice) IsActivated 

func (p *PoolDevice) IsActivated(deviceName string) bool

IsActivated returns true if thin-device is activated

func (*PoolDevice) IsLoaded 

func (p *PoolDevice) IsLoaded(deviceName string) bool

IsLoaded returns true if thin-device is visible for dmsetup

func (*PoolDevice) MarkDeviceState added in v1.4.0 

func (p *PoolDevice) MarkDeviceState(ctx context.Context, name string, state DeviceState) error

MarkDeviceState changes the device's state in metastore

func (*PoolDevice) RemoveDevice 

func (p *PoolDevice) RemoveDevice(ctx context.Context, deviceName string) error

RemoveDevice completely wipes out thin device from thin-pool and frees it's device ID

func (*PoolDevice) RemovePool 

func (p *PoolDevice) RemovePool(ctx context.Context) error

RemovePool deactivates all child thin-devices and removes thin-pool device

func (*PoolDevice) ResumeDevice added in v1.3.7 

func (p *PoolDevice) ResumeDevice(ctx context.Context, deviceName string) error

ResumeDevice resumes IO for the given device

func (*PoolDevice) SuspendDevice 

func (p *PoolDevice) SuspendDevice(ctx context.Context, deviceName string) error

SuspendDevice flushes the outstanding IO and blocks the further IO

func (*PoolDevice) WalkDevices added in v1.4.0 

func (p *PoolDevice) WalkDevices(ctx context.Context, cb func(info *DeviceInfo) error) error

WalkDevices iterates all devices in pool metadata

type PoolMetadata 

type PoolMetadata struct {
	// contains filtered or unexported fields
}

PoolMetadata keeps device info for the given thin-pool device, it also responsible for generating next available device ids and tracking devmapper transaction numbers

func NewPoolMetadata 

func NewPoolMetadata(dbfile string) (*PoolMetadata, error)

NewPoolMetadata creates new or open existing pool metadata database

func (*PoolMetadata) AddDevice 

func (m *PoolMetadata) AddDevice(ctx context.Context, info *DeviceInfo) error

AddDevice saves device info to database.

func (*PoolMetadata) ChangeDeviceState added in v1.4.0 

func (m *PoolMetadata) ChangeDeviceState(ctx context.Context, name string, state DeviceState) error

ChangeDeviceState changes the device state given the device name in devices bucket.

func (*PoolMetadata) Close 

func (m *PoolMetadata) Close() error

Close closes metadata store

func (*PoolMetadata) GetDevice 

func (m *PoolMetadata) GetDevice(ctx context.Context, name string) (*DeviceInfo, error)

GetDevice retrieves device info by name from database

func (*PoolMetadata) GetDeviceNames 

func (m *PoolMetadata) GetDeviceNames(ctx context.Context) ([]string, error)

GetDeviceNames retrieves the list of device names currently stored in database

func (*PoolMetadata) MarkFaulty 

func (m *PoolMetadata) MarkFaulty(ctx context.Context, name string) error

MarkFaulty marks the given device and corresponding devmapper device ID as faulty. The snapshotter might attempt to recreate a device in 'Faulty' state with another devmapper ID in subsequent calls, and in case of success it's status will be changed to 'Created' or 'Activated'. The devmapper dev ID will remain in 'deviceFaulty' state until manually handled by a user.

func (*PoolMetadata) RemoveDevice 

func (m *PoolMetadata) RemoveDevice(ctx context.Context, name string) error

RemoveDevice removes device info from store.

func (*PoolMetadata) UpdateDevice 

func (m *PoolMetadata) UpdateDevice(ctx context.Context, name string, fn DeviceInfoCallback) error

UpdateDevice updates device info in metadata store. The callback should be used to indicate whether device info update was successful or not. An error returned from the callback will rollback the update transaction in the database. Name and Device ID are not allowed to change.

func (*PoolMetadata) WalkDevices 

func (m *PoolMetadata) WalkDevices(ctx context.Context, cb func(info *DeviceInfo) error) error

WalkDevices walks all devmapper devices in metadata store and invokes the callback with device info. The provided callback function must not modify the bucket.

type Snapshotter 

type Snapshotter struct {
	// contains filtered or unexported fields
}

Snapshotter implements containerd's snapshotter (https://godoc.org/github.com/containerd/containerd/snapshots#Snapshotter) based on Linux device-mapper targets.

func NewSnapshotter 

func NewSnapshotter(ctx context.Context, config *Config) (*Snapshotter, error)

NewSnapshotter creates new device mapper snapshotter. Internally it creates thin-pool device (or reloads if it's already exists) and initializes a database file for metadata.

func (*Snapshotter) Cleanup added in v1.4.0 

func (s *Snapshotter) Cleanup(ctx context.Context) error

Cleanup cleans up all removed and unused resources

func (*Snapshotter) Close 

func (s *Snapshotter) Close() error

Close releases devmapper snapshotter resources. All subsequent Close calls will be ignored.

func (*Snapshotter) Commit 

func (s *Snapshotter) Commit(ctx context.Context, name, key string, opts ...snapshots.Opt) error

Commit marks an active snapshot as committed in meta store. Block device unmount operation captures snapshot changes by itself, so no additional actions needed within Commit operation.

func (*Snapshotter) Mounts 

func (s *Snapshotter) Mounts(ctx context.Context, key string) ([]mount.Mount, error)

Mounts return the list of mounts for the active or view snapshot

func (*Snapshotter) Prepare 

func (s *Snapshotter) Prepare(ctx context.Context, key, parent string, opts ...snapshots.Opt) ([]mount.Mount, error)

Prepare creates thin device for an active snapshot identified by key

func (*Snapshotter) Remove 

func (s *Snapshotter) Remove(ctx context.Context, key string) error

Remove removes thin device and snapshot metadata by key

func (*Snapshotter) ResetPool 

func (s *Snapshotter) ResetPool(ctx context.Context) error

ResetPool deactivates and deletes all thin devices in thin-pool. Used for cleaning pool after benchmarking.

func (*Snapshotter) Stat 

func (s *Snapshotter) Stat(ctx context.Context, key string) (snapshots.Info, error)

Stat returns the info for an active or committed snapshot from store

func (*Snapshotter) Update 

func (s *Snapshotter) Update(ctx context.Context, info snapshots.Info, fieldpaths ...string) (snapshots.Info, error)

Update updates an existing snapshot info's data

func (*Snapshotter) Usage 

func (s *Snapshotter) Usage(ctx context.Context, key string) (snapshots.Usage, error)

Usage returns the resource usage of an active or committed snapshot excluding the usage of parent snapshots.

func (*Snapshotter) View 

func (s *Snapshotter) View(ctx context.Context, key, parent string, opts ...snapshots.Opt) ([]mount.Mount, error)

View creates readonly thin device for the given snapshot key

func (*Snapshotter) Walk 

func (s *Snapshotter) Walk(ctx context.Context, fn snapshots.WalkFunc, fs ...string) error

Walk iterates through all metadata Info for the stored snapshots and calls the provided function for each.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.