server

package
v0.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 17, 2019 License: Apache-2.0 Imports: 40 Imported by: 0

README

DAOS Server (control-plane)

Workflow

Control plane server (daos_server) instances will open a gRPC channel to listen for requests from control plane client applications. Administrators can perform provisioning operations on network and storage hardware through the control plane dmg management tool. Calling dmg storage format formats persistent storage on the server node, writes the superblock and starts the data plane.

Server format diagram

Running

For instructions on building and running DAOS see the Quickstart guide.

Example output from invoking `daos_server` on multiple hosts with `orterun` (with logging to stdout for illustration purposes) 

[daosuser@boro-45 daos_m]$ orterun -np 2 -H boro-44,boro-45 --report-uri /tmp/urifile --enable-recovery daos_server start -c 1  -o /home/daosuser/projects/daos_m/utils/config/examples/daos_server_sockets.yml
2019/03/28 12:28:07 config.go:85: debug: DAOS config read from /home/daosuser/projects/daos_m/utils/config/examples/daos_server_sockets.yml
2019/03/28 12:28:07 config.go:85: debug: DAOS config read from /home/daosuser/projects/daos_m/utils/config/examples/daos_server_sockets.yml
2019/03/28 12:28:07 main.go:79: debug: Switching control log level to DEBUG
boro-44.boro.hpdd.intel.com 2019/03/28 12:28:07 config.go:121: debug: Active config saved to /home/daosuser/projects/daos_m/utils/config/examples/.daos_server.active.yml (read-only)
Starting SPDK v18.07-pre / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: spdk -c 0x1 --file-prefix=spdk234203216 --base-virtaddr=0x200000000000 --proc-type=auto ]
EAL: Detected 72 lcore(s)
EAL: Auto-detected process type: PRIMARY
2019/03/28 12:28:07 main.go:79: debug: Switching control log level to DEBUG
boro-45.boro.hpdd.intel.com 2019/03/28 12:28:07 config.go:121: debug: Active config saved to /home/daosuser/projects/daos_m/utils/config/examples/.daos_server.active.yml (read-only)
EAL: Detected 72 lcore(s)
Starting SPDK v18.07-pre / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: spdk -c 0x1 --file-prefix=spdk290246766 --base-virtaddr=0x200000000000 --proc-type=auto ]
EAL: Auto-detected process type: PRIMARY
EAL: No free hugepages reported in hugepages-1048576kB
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Multi-process socket /home/daosuser/.spdk234203216_unix
EAL: Probing VFIO support...
EAL: Multi-process socket /home/daosuser/.spdk290246766_unix
EAL: Probing VFIO support...
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL:   probe driver: 8086:2701 spdk_nvme
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL:   probe driver: 8086:2701 spdk_nvme
Starting SPDK v18.07-pre / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: daos -c 0x1 --file-prefix=spdk290246766 --base-virtaddr=0x200000000000 --proc-type=auto ]
EAL: Detected 72 lcore(s)
Starting SPDK v18.07-pre / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: daos -c 0x1 --file-prefix=spdk234203216 --base-virtaddr=0x200000000000 --proc-type=auto ]
EAL: Auto-detected process type: SECONDARY
EAL: Detected 72 lcore(s)
EAL: Auto-detected process type: SECONDARY
EAL: Multi-process socket /home/daosuser/.spdk234203216_unix_141938_a591f56066dd7
EAL: Probing VFIO support...
EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the kernel.
EAL:    This may cause issues with mapping memory into secondary processes
EAL: Multi-process socket /home/daosuser/.spdk290246766_unix_23680_14e5a164a3bd1db
EAL: Probing VFIO support...
EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the kernel.
EAL:    This may cause issues with mapping memory into secondary processes
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL:   probe driver: 8086:2701 spdk_nvme
boro-44.boro.hpdd.intel.com 2019/03/28 12:28:11 main.go:188: debug: DAOS server listening on 0.0.0.0:10001
DAOS I/O server (v0.0.2) process 141938 started on rank 1 (out of 2) with 1 target xstream set(s).
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL:   probe driver: 8086:2701 spdk_nvme
boro-45.boro.hpdd.intel.com 2019/03/28 12:28:11 main.go:188: debug: DAOS server listening on 0.0.0.0:10001
DAOS I/O server (v0.0.2) process 23680 started on rank 0 (out of 2) with 1 target xstream set(s).

Configuration files

A populated daos_server config file is required when starting daos_server process, it's location can be specified on the commandline (-o option) or default location (<daos install dir>/install/etc/daos_server.yml).

Example config files can be found in the examples folder.

Config file parameters will be parsed and populated with defaults as documented in the default daos server config.

Parameters passed to daos_server on the commandline as application options (excluding environment variables) take precedence over values specified in config file.

For convenience, active parsed config values are written to the directory where the server config file was read from or /tmp/ if that fails.

It is strongly recommended to specify all parameters and environment for running DAOS servers in the server config file.

NOTES:

  • some environment variables can only be supplied to daos_io_server instances through the server config file
    • CRT_PHY_ADDR_STR, OFI_INTERFACE, OFI_PORT, D_LOG_MASK, D_LOG_FILE
  • other environment variables can be specified in config file as "key=value" strings in the per-server env_vars list
    • these environment variables will be applied to the daos_io_server environment overriding any specified in the environment used to launch daos_io_server (e.g. using "-x" orterun option)
  • while it is very highly recommended to use the server config file as a means to supply parameters, environment variables not applied through the config file but specified in the calling environment will still be present in the environment used to launch daos_io_server
Logging

Log file and level mask for both data (daos_io_server) and control (daos_server) planes can be set in the server config file. TODO: examples for global (control) and per-server (data) log parameters in config including DAOS specific syntax (D_LOG_MASK=ERR,mgmt=DEBUG).

NVMe/block-device storage

Parameters prefixed with nvme_ in the per-server section of the config file determine how NVMe storage will be assigned for use by DAOS on the storage node. TODO: examples for NVMe and AIO (emulation) classes including config file syntax and results/logging of formatting operations including generated nvme.conf file and SPDK device format through language bindings.

SCM/pmem storage

Parameters prefixed with scm_ in the per-server section of the config file determine how SCM storage will be assigned for use by DAOS on the storage node. TODO: examples for both DCPM and RAM (emulation) SCM classes including config file syntax and results/logging of formatting operations.

Subcommands

daos_server supports various subcommands (see daos_server --help for available subcommands) which will perform stand-alone tasks (e.g. storage prepare|scan) or attempt to bring-up the data plane (start subcommand).

storage prepare --nvme-only

This subcommand requires elevated permissions (sudo).

NVMe access through SPDK as an unprivileged user can be enabled by first running sudo daos_server storage prepare --nvme-only -w 0000:81:00.0 -p 4096 -u bob. This will perform the required setup in order for daos_server to be run by user "bob" who will own the hugepage mountpoint directory and vfio groups as needed in SPDK operations. If the target-user is unspecified (-u short option), the target user will be the issuer of the sudo command (or root if not using sudo). The specification of hugepages (-p short option) defines the number of huge pages to allocate for use by SPDK. The specification of pci-whitelist (-w short option) allows user to optionally specify which PCI devices to unbind from the Kernel driver for use with SPDK, as opposed to unbinding all devices by default. Multiple devices can be specified as a whitespace separated list of full PCI addresses (-w "0000:81:00.0 000:2"). If one of the addresses is non-valid (for example 000:2), then that device will be skipped, unless it is the only address listed in which case all PCI devices will be blacklisted. A use for all device blacklisting could involve the need to only set up hugepages and skip all device unbindings.

The configuration commands that require elevated permissions are in src/control/server/init/setup_spdk.sh (script is installed as install/share/control/setup_spdk.sh).

The sudoers file can be accessed with command visudo and permissions can be granted to a user to execute a specific command pattern (requires prior knowledge of daos_server binary location):

linuxuser ALL=/home/linuxuser/projects/daos_m/install/bin/daos_server prepare --nvme-only*

See daos_server storage prepare --help for usage.

storage scan
List NVMe SSDs and SCM modules locally attached to the host. 

[daosuser@boro-45 daos_m]$ daos_server storage scan
Starting SPDK v18.07-pre / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: spdk -c 0x1 --file-prefix=spdk_pid29193 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Multi-process socket /home/daosuser/.spdk_pid29193_unix
EAL: Probing VFIO support...
Unable to unlink shared memory file: /var/run/.spdk_pid29193_config. Error code: 2
Unable to unlink shared memory file: /var/run/.spdk_pid29193_hugepage_info. Error code: 2
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL:   probe driver: 8086:2701 spdk_nvme
Listing attached storage...
NVMe:
- id: 0
  model: 'INTEL SSDPED1K375GA '
  serial: 'PHKS7335006W375AGN  '
  pciaddr: 0000:81:00.0
  fwrev: E2010420
  namespace:
  - id: 1
    capacity: 375

SCM:
- physicalid: 28
  channel: 0
  channelpos: 1
  memctrlr: 0
  socket: 0
  capacity: 539661172736
- physicalid: 40
  channel: 0
  channelpos: 1
  memctrlr: 1
  socket: 0
  capacity: 539661172736
- physicalid: 50
  channel: 0
  channelpos: 1
  memctrlr: 0
  socket: 1
  capacity: 539661172736
- physicalid: 62
  channel: 0
  channelpos: 1
  memctrlr: 1
  socket: 1
  capacity: 539661172736

See daos_server storage scan --help for usage.

Management Tool (client) Usage

dmg is a management tool which exercises the client api and can be run on login nodes by an unprivileged user. The tool is lightweight and doesn't depend on storage libraries. Implemented as a gRPC client application it connects and interacts with multiple gRPC servers concurrently, connecting to specified ports.

In order to run the tool to perform administrative tasks from non-storage nodes (e.g. login nodes) over an out-of-band management network;

  1. build and run the daos_server as per the quickstart guide on the storage nodes
  2. run dmg from a login node

See dmg --help for usage and here for package details.

Storage management

The DAOS data plane utilises two forms of non-volatile storage, storage class memory (SCM) in the form of persistent memory modules and NVMe in the form of high-performance SSDs.

The DAOS control plane provides capability to provision and manage the non-volatile storage including the allocation of resources to data plane instances.

Storage format is required after other storage management operations have been performed as a precursor to bringing up the DAOS data plane:

Storage format diagram

SCM management capabilities

Operations on SCM persistent memory modules are performed using go-ipmctl bindings to issue commands through the ipmctl native C libraries.

SCM provisioning is to be performed prior to formatting and allocating SCM through the DAOS control plane.

Formatting SCM involves creating an ext4 filesystem on the nvdimm device.

Mounting SCM results in an active mount using the DAX extension enabling direct access without restrictions imposed by legacy HDD hardware.

The DAOS control plane will provide SCM storage management capabilities enabling the discovery, initial burn-in testing, firmware update and allocation of devices to data plane instances.

SCM module discovery

Device details for any discovered (Intel) data-centre persistent memory modules (DCPM modules) on the storage server will be returned when running storage scan subcommand on dmg or daos_server executables.

TODO: return details of AppDirect memory regions

SCM format

Format can be triggered through the management tool at DAOS system installation time, formatting and mounting of SCM device namespace is performed as specified in config file parameters prefixed with scm_.

SCM device format is expected only to be performed when installing DAOS system for the first time and requires daos_server to be started by root user.

If daos_server is run by root user, data plane will be started automatically ONLY if the DAOS superblock exists within the specified scm_mount. Otherwise, the control plane will wait until an administrator calls "storage format" over the client API from the management tool. If storage format call is successful, control plane will continue to start data plane instances when SCM and NVMe have been formatted, SCM mounted and superblock and nvme.conf successfully written to SCM mount.

If daos_server is run by a normal (non-root) user, the data plane will be started automatically if the DAOS superblock exists within the specified scm_mount. If scm_mount IS NOT mounted, control plane will fail to start data plane. If scm_mount IS mounted but no superblock exists, control plane will attempt to create superblock and start data plane.

Example output from invoking `daos_server` on single host when run as a normal user and `scm_mount` IS NOT mounted 

[root@wolf-72 daos_m]# install/bin/orterun -np 1 --hostfile hostfile --enable-recovery --allow-run-as-root install/bin/daos_server start -t 1 -o /root/daos_m/utils/config/examples/daos_server_sockets.yml
2019/04/10 04:04:05 config.go:103: debug: DAOS config read from /root/daos_m/utils/config/examples/daos_server_sockets.yml
2019/04/10 04:04:05 config.go:135: debug: Active config saved to /root/daos_m/utils/config/examples/.daos_server.active.yml (read-only)
2019/04/10 04:04:05 config.go:405: debug: Switching control log level to DEBUG
Starting SPDK v18.07-pre / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: spdk -c 0x1 --file-prefix=spdk1319116020 --base-virtaddr=0x200000000000 --proc-type=auto ]
EAL: Detected 96 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Multi-process socket /var/run/.spdk1319116020_unix
EAL: Probing VFIO support...
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL:   probe driver: 8086:953 spdk_nvme
EAL: PCI device 0000:87:00.0 on NUMA socket 1
EAL:   probe driver: 8086:2701 spdk_nvme
EAL: PCI device 0000:da:00.0 on NUMA socket 1
EAL:   probe driver: 8086:953 spdk_nvme
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:04:09 iosrv.go:104: debug: continuing without storage format on server 0 (format_override set in config)
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:04:09 external.go:54: debug: exec 'mount | grep ' /mnt/daos ''
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:04:09 external.go:54: debug: exec 'mount | grep ' /mnt ''
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:04:09 main.go:129: error: Failed to format servers: server0 scm mount path (/mnt/daos) not mounted: Error running mount | grep ' /mnt ': : exit status 1

Example output from invoking `daos_server` on single host run as a normal user and `scm_mount` IS mounted 

[root@wolf-72 daos_m]# mkdir /mnt/daos
[root@wolf-72 daos_m]# mount -t tmpfs -o size=68719476736 tmpfs /mnt/daos
[root@wolf-72 daos_m]# install/bin/orterun -np 1 --hostfile hostfile --enable-recovery --allow-run-as-root install/bin/daos_server start -t 1 -o /root/daos_m/utils/config/examples/daos_server_sockets.yml
2019/04/10 04:04:50 config.go:103: debug: DAOS config read from /root/daos_m/utils/config/examples/daos_server_sockets.yml
2019/04/10 04:04:50 config.go:135: debug: Active config saved to /root/daos_m/utils/config/examples/.daos_server.active.yml (read-only)
2019/04/10 04:04:50 config.go:405: debug: Switching control log level to DEBUG
Starting SPDK v18.07-pre / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: spdk -c 0x1 --file-prefix=spdk1234882151 --base-virtaddr=0x200000000000 --proc-type=auto ]
EAL: Detected 96 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Multi-process socket /var/run/.spdk1234882151_unix
EAL: Probing VFIO support...
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL:   probe driver: 8086:953 spdk_nvme
EAL: PCI device 0000:87:00.0 on NUMA socket 1
EAL:   probe driver: 8086:2701 spdk_nvme
EAL: PCI device 0000:da:00.0 on NUMA socket 1
EAL:   probe driver: 8086:953 spdk_nvme
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:04:54 iosrv.go:104: debug: continuing without storage format on server 0 (format_override set in config)
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:04:54 external.go:54: debug: exec 'mount | grep ' /mnt/daos ''
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:04:54 iosrv.go:128: debug: format server /mnt/daos (createMS=false bootstrapMS=false)
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:04:54 main.go:156: debug: DAOS server listening on 0.0.0.0:10001
DAOS I/O server (v0.4.0) process 135939 started on rank 0 (out of 1) with 1 target xstream set(s), 2 helper XS per target, firstcore 0.

Example output from invoking `daos_server` on single host when run as root user when superblock already exists 

[root@wolf-72 daos_m]# install/bin/orterun -np 1 --hostfile hostfile --enable-recovery --allow-run-as-root install/bin/daos_server start -t 1 -o /root/daos_m/utils/config/examples/daos_server_sockets.yml
2019/04/10 03:34:22 config.go:103: debug: DAOS config read from /root/daos_m/utils/config/examples/daos_server_sockets.yml
2019/04/10 03:34:22 config.go:135: debug: Active config saved to /root/daos_m/utils/config/examples/.daos_server.active.yml (read-only)
2019/04/10 03:34:22 config.go:405: debug: Switching control log level to DEBUG
Starting SPDK v18.07-pre / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: spdk -c 0x1 --file-prefix=spdk850287469 --base-virtaddr=0x200000000000 --proc-type=auto ]
EAL: Detected 96 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Multi-process socket /var/run/.spdk850287469_unix
EAL: Probing VFIO support...
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL:   probe driver: 8086:953 spdk_nvme
EAL: PCI device 0000:87:00.0 on NUMA socket 1
EAL:   probe driver: 8086:2701 spdk_nvme
EAL: PCI device 0000:da:00.0 on NUMA socket 1
EAL:   probe driver: 8086:953 spdk_nvme
wolf-72.wolf.hpdd.intel.com 2019/04/10 03:34:25 iosrv.go:93: debug: server 0 has already been formatted
wolf-72.wolf.hpdd.intel.com 2019/04/10 03:34:26 main.go:156: debug: DAOS server listening on 0.0.0.0:10001
DAOS I/O server (v0.4.0) process 135671 started on rank 0 (out of 1) with 1 target xstream set(s), 2 helper XS per target, firstcore 0.

Example output from invoking `daos_server` on single host when run as root and `scm_mount` location is not mounted 

[root@wolf-72 daos_m]# install/bin/orterun -np 1 --hostfile hostfile --enable-recovery --allow-run-as-root install/bin/daos_server start -t 1 -o /root/daos_m/utils/config/examples/daos_server_sockets.yml
2019/04/10 03:34:22 config.go:103: debug: DAOS config read from /root/daos_m/utils/config/examples/daos_server_sockets.yml
2019/04/10 03:34:22 config.go:135: debug: Active config saved to /root/daos_m/utils/config/examples/.daos_server.active.yml (read-only)
2019/04/10 03:34:22 config.go:405: debug: Switching control log level to DEBUG
Starting SPDK v18.07-pre / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: spdk -c 0x1 --file-prefix=spdk850287469 --base-virtaddr=0x200000000000 --proc-type=auto ]
EAL: Detected 96 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Multi-process socket /var/run/.spdk850287469_unix
EAL: Probing VFIO support...
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL:   probe driver: 8086:953 spdk_nvme
EAL: PCI device 0000:87:00.0 on NUMA socket 1
EAL:   probe driver: 8086:2701 spdk_nvme
EAL: PCI device 0000:da:00.0 on NUMA socket 1
EAL:   probe driver: 8086:953 spdk_nvme
wolf-72.wolf.hpdd.intel.com 2019/04/10 03:49:19 iosrv.go:108: debug: waiting for storage format on server 0

...after storage format gets called, data plane is started...

wolf-72.wolf.hpdd.intel.com 2019/04/10 04:00:21 mgmt.go:108: debug: performing nvme format, may take several minutes!
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:00:21 mgmt.go:114: debug: performing scm format, should be quick!
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:00:21 external.go:111: debug: calling unmount with /mnt/daos, MNT_DETACH
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:00:21 storage_scm.go:140: debug: wiping all fs identifiers on device /dev/pmem4
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:00:21 external.go:54: debug: exec 'wipefs -a /dev/pmem4'
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:00:21 external.go:54: debug: exec 'mkfs.ext4 /dev/pmem4'
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:00:22 external.go:98: debug: calling mount with /dev/pmem4, /mnt/daos, ext4, 33806, dax
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:00:22 external.go:54: debug: exec 'mount | grep ' /mnt/daos ''
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:00:22 mgmt.go:153: debug: StorageFormat: storage format successful on server 0
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:00:22 iosrv.go:128: debug: format server /mnt/daos (createMS=false bootstrapMS=false)
wolf-72.wolf.hpdd.intel.com 2019/04/10 04:00:23 main.go:156: debug: DAOS server listening on 0.0.0.0:10001
DAOS I/O server (v0.4.0) process 135863 started on rank 0 (out of 1) with 1 target xstream set(s), 2 helper XS per target, firstcore 0.

SCM Provisioning

Provisioning SCM occurs by configuring DCPM modules in AppDirect memory regions (interleaved mode) in groups of modules local to a specific socket (NUMA) and resultant nvdimm namespaces are defined a device identifier (e.g. /dev/pmem0). This can be performed through the ipmctl tool.

TODO: implement set-interleaved command

NVMe management capabilities

Operations on NVMe SSD devices are performed using go-spdk bindings to issue commands over the SPDK framework.

The DAOS data plane utilises NVMe devices as block storage using the SPDK "blobstore" abstraction, creation and management of the blobstore and blobs is performed by the data plane.

The DAOS control plane will provide NVMe storage management capabilities enabling the discovery, format, initial burn-in testing, firmware update and allocation of devices to data plane instances.

NVMe Controller and Namespace Discovery

Device details for any discovered NVMe SSDs accessible through SPDK on the storage server will be returned when running storage scan subcommand on dmg or daos_server executables.

The following animation illustrates starting the control server and using the management shell to view the NVMe Namespaces discovered on a locally available NVMe Controller (assuming the quickstart guide instructions have already been performed):

NVMe Format

Format can be triggered through the management tool at DAOS system installation time.

NVMe device format is expected only to be performed when installing DAOS system for the first time.

In the context of what is required from the control plane to prepare NVMe devices for operation with DAOS data plane, "formatting" refers to the low level reset of storage media which will remove blobstores and remove any filesystem signatures from the SSD controller namespaces.

Formatting will be performed on devices identified by PCI addresses specified in config file parameter bdev_list when bdev_class is equal to nvme.

The SPDK "blobcli" can be used to initiate and manipulate blobstores for testing and verification purposes.

In order to designate NVMe devices to be used by DAOS data plane instances, the control plane will generate an nvme.conf file to be consumed by SPDK which will be written to the scm_mount (persistent) mounted location as a final stage of formatting before the superblock is written, signifying the server has been formatted.

Bootstrapping and DAOS system membership

When starting a data-plane instance, we look at the superblock to determine whether it should be a MS (management service) replica. The daos_server.yml's access_points parameter is used (only during format) to determine whether an instance is to be a MS replica or not.

When the starting instance is identified as an MS replica, it performs bootstrap and starts. If the DAOS system has only one replica (as specified by access_points parameter), the host of the bootstrapped instance is now the MS leader. Whereas if there are multiple replicas, elections will happen in the background and eventually a leader will be elected.

When the starting instance is not identified as an MS replica, the instance's host calls Join on MgmtSvcClient over gRPC including the instance's host ControlAddress (address that the gRPC server is listening on) in the request addressed to the MS leader.

MgmtSvc running on the MS leader handles the Join request received by gRPC server and forwards request over dRPC to the MS leader instance. If the Join request is successful then the MS leader MgmtSvc records the address contained in the request as a new system member.

Documentation

Overview

(C) Copyright 2019 Intel Corporation.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

GOVERNMENT LICENSE RIGHTS-OPEN SOURCE SOFTWARE The Government's rights to use, modify, reproduce, release, perform, display, or disclose this software are subject to the terms of the Apache License as provided in Contract No. 8F-30005. Any reproduction of computer software, computer software documentation, or portions thereof marked with this legend must also reproduce the markings.

(C) Copyright 2019 Intel Corporation.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

GOVERNMENT LICENSE RIGHTS-OPEN SOURCE SOFTWARE The Government's rights to use, modify, reproduce, release, perform, display, or disclose this software are subject to the terms of the Apache License as provided in Contract No. 8F-30005. Any reproduction of computer software, computer software documentation, or portions thereof marked with this legend must also reproduce the markings.

Index

Constants

View Source 
const (
	ControlLogLevelDebug = ControlLogLevel(logging.LogLevelDebug)
	ControlLogLevelInfo  = ControlLogLevel(logging.LogLevelInfo)
	ControlLogLevelError = ControlLogLevel(logging.LogLevelError)
)

TODO(mjmac): Evaluate whether or not this layer of indirection adds any value.

Variables

This section is empty.

Functions

func CheckReplica 

func CheckReplica(lis net.Listener, accessPoints []string, srv *exec.Cmd) (msReplicaCheck string, err error)

CheckReplica verifies if this server is supposed to host an MS replica, only performing the check and printing the result for now.

func Start 

func Start(log *logging.LeveledLogger, cfg *Configuration) error

Start is the entry point for a daos_server instance.

func WriteSuperblock 

func WriteSuperblock(sbPath string, sb *Superblock) error

WriteSuperblock writes a Superblock to storage.

Types

type Configuration 

type Configuration struct {
	// control-specific
	ControlPort         int                       `yaml:"port"`
	TransportConfig     *security.TransportConfig `yaml:"transport_config"`
	Servers             []*ioserver.Config        `yaml:"servers"`
	BdevInclude         []string                  `yaml:"bdev_include,omitempty"`
	BdevExclude         []string                  `yaml:"bdev_exclude,omitempty"`
	NrHugepages         int                       `yaml:"nr_hugepages"`
	ControlLogMask      ControlLogLevel           `yaml:"control_log_mask"`
	ControlLogFile      string                    `yaml:"control_log_file"`
	ControlLogJSON      bool                      `yaml:"control_log_json,omitempty"`
	HelperLogFile       string                    `yaml:"helper_log_file"`
	UserName            string                    `yaml:"user_name"`
	GroupName           string                    `yaml:"group_name"`
	RecreateSuperblocks bool                      `yaml:"recreate_superblocks"`

	// duplicated in ioserver.Config
	SystemName string                `yaml:"name"`
	SocketDir  string                `yaml:"socket_dir"`
	Fabric     ioserver.FabricConfig `yaml:",inline"`
	Modules    string

	AccessPoints []string `yaml:"access_points"`

	// unused (?)
	FaultPath    string `yaml:"fault_path"`
	FaultCb      string `yaml:"fault_cb"`
	Hyperthreads bool   `yaml:"hyperthreads"`

	Path string // path to config file

	// Shared memory segment ID to enable SPDK multiprocess mode,
	// SPDK application processes can then access the same shared
	// memory and therefore NVMe controllers.
	// TODO: Is it also necessary to provide distinct coremask args?
	NvmeShmID int
	// contains filtered or unexported fields
}

Configuration describes options for DAOS control plane. See utils/config/daos_server.yml for parameter descriptions.

func NewConfiguration 

func NewConfiguration() *Configuration

NewConfiguration creates a new instance of configuration struct populated with defaults and default external interface.

func (*Configuration) Load 

func (c *Configuration) Load() error

Load reads the serialized configuration from disk and validates it.

func (*Configuration) SaveToFile 

func (c *Configuration) SaveToFile(filename string) error

SaveToFile serializes the configuration and saves it to the specified filename.

func (*Configuration) SetNvmeShmID 

func (c *Configuration) SetNvmeShmID(base string)

func (*Configuration) SetPath 

func (c *Configuration) SetPath(path string) error

SetPath sets the default path to the configuration file.

func (*Configuration) Validate 

func (c *Configuration) Validate() (err error)

Validate asserts that config meets minimum requirements.

func (*Configuration) WithAccessPoints 

func (c *Configuration) WithAccessPoints(aps ...string) *Configuration

WithAccessPoints sets the access point list.

func (*Configuration) WithBdevExclude 

func (c *Configuration) WithBdevExclude(bList ...string) *Configuration

WithBdevExclude sets the block device exclude list.

func (*Configuration) WithBdevInclude 

func (c *Configuration) WithBdevInclude(bList ...string) *Configuration

WithBdevInclude sets the block device include list.

func (*Configuration) WithControlLogFile 

func (c *Configuration) WithControlLogFile(filePath string) *Configuration

WithControlLogFile sets the path to the daos_server logfile.

func (*Configuration) WithControlLogJSON 

func (c *Configuration) WithControlLogJSON(enabled bool) *Configuration

WithControlLogJSON enables or disables JSON output.

func (*Configuration) WithControlLogMask 

func (c *Configuration) WithControlLogMask(lvl ControlLogLevel) *Configuration

WithControlLogMask sets the daos_server log level.

func (*Configuration) WithControlPort 

func (c *Configuration) WithControlPort(port int) *Configuration

WithControlPort sets the gRPC listener port.

func (*Configuration) WithFabricProvider 

func (c *Configuration) WithFabricProvider(provider string) *Configuration

WithFabricProvider sets the top-level fabric provider.

func (*Configuration) WithFaultCb 

func (c *Configuration) WithFaultCb(cb string) *Configuration

WithFaultCb sets the path to the fault callback script.

func (*Configuration) WithFaultPath 

func (c *Configuration) WithFaultPath(fp string) *Configuration

WithFaultPath sets the fault path (identification string e.g. rack/shelf/node).

func (*Configuration) WithGroupName 

func (c *Configuration) WithGroupName(name string) *Configuration

WithGroupName sets the group to run as.

func (*Configuration) WithHelperLogFile 

func (c *Configuration) WithHelperLogFile(filePath string) *Configuration

WithHelperLogFile sets the path to the daos_admin logfile.

func (*Configuration) WithHyperthreads 

func (c *Configuration) WithHyperthreads(enabled bool) *Configuration

WithHyperthreads enables or disables hyperthread support.

func (*Configuration) WithModules 

func (c *Configuration) WithModules(mList string) *Configuration

WithModules sets a list of server modules to load.

func (*Configuration) WithNUMAValidator 

func (c *Configuration) WithNUMAValidator(fn networkNUMAValidation) *Configuration

WithNUMAValidator is used for unit testing configurations that are not necessarily valid on the test machine. We use the stub function ValidateNetworkConfigStub to avoid unnecessary failures in those tests that are not concerned with testing a truly valid configuration for the test system.

func (*Configuration) WithNrHugePages 

func (c *Configuration) WithNrHugePages(nr int) *Configuration

WithNrHugePages sets the number of huge pages to be used.

func (*Configuration) WithNvmeShmID 

func (c *Configuration) WithNvmeShmID(id int) *Configuration

WithNvmeShmID sets the common shmID used for SPDK multiprocess mode.

func (*Configuration) WithProviderValidator 

func (c *Configuration) WithProviderValidator(fn networkProviderValidation) *Configuration

WithProviderValidator is used for unit testing configurations that are not necessarily valid on the test machine. We use the stub function ValidateNetworkConfigStub to avoid unnecessary failures in those tests that are not concerned with testing a truly valid configuration for the test system.

func (*Configuration) WithRecreateSuperblocks 

func (c *Configuration) WithRecreateSuperblocks() *Configuration

WithRecreateSuperblocks indicates that a missing superblock should not be treated as an error. The server will create new superblocks as necessary.

func (*Configuration) WithScmMountPoint deprecated 

func (c *Configuration) WithScmMountPoint(mp string) *Configuration

WithScmMountPoint sets the SCM mountpoint for the first I/O Server.

Deprecated: This function exists to ease transition away from specifying the SCM mountpoint via daos_server CLI flag. Future versions will require the mountpoint to be set via configuration.

func (*Configuration) WithServers 

func (c *Configuration) WithServers(srvList ...*ioserver.Config) *Configuration

WithServers sets the list of IOServer configurations.

func (*Configuration) WithSocketDir 

func (c *Configuration) WithSocketDir(sockDir string) *Configuration

WithSocketDir sets the default socket directory.

func (*Configuration) WithSystemName 

func (c *Configuration) WithSystemName(name string) *Configuration

WithSystemName sets the system name.

func (*Configuration) WithTransportConfig 

func (c *Configuration) WithTransportConfig(cfg *security.TransportConfig) *Configuration

WithTransportConfig sets the gRPC transport configuration.

func (*Configuration) WithUserName 

func (c *Configuration) WithUserName(name string) *Configuration

WithUserName sets the user to run as.

type ControlLogLevel 

type ControlLogLevel logging.LogLevel

ControlLogLevel is a type that specifies log levels

func (ControlLogLevel) MarshalYAML 

func (c ControlLogLevel) MarshalYAML() (interface{}, error)

func (ControlLogLevel) String 

func (c ControlLogLevel) String() string

func (*ControlLogLevel) UnmarshalYAML 

func (c *ControlLogLevel) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements yaml.Unmarshaler on ControlLogMask struct

type ControlService 

type ControlService struct {
	StorageControlService
	// contains filtered or unexported fields
}

ctlpb.MgmtCtlServer, and is the data container for the service.

func NewControlService 

func NewControlService(l logging.Logger, h *IOServerHarness,
	bp *bdev.Provider, sp *scm.Provider,
	cfg *Configuration, m *common.Membership) (*ControlService, error)

NewControlService returns ControlService to be used as gRPC control service datastore. Initialised with sensible defaults and provided components.

func (*ControlService) NetworkListProviders 

func (c *ControlService) NetworkListProviders(ctx context.Context, in *pb.ProviderListRequest) (*pb.ProviderListReply, error)

func (*ControlService) NetworkScanDevices 

func (c *ControlService) NetworkScanDevices(in *pb.DeviceScanRequest, stream pb.MgmtCtl_NetworkScanDevicesServer) error

func (*ControlService) StorageFormat 

func (c *ControlService) StorageFormat(req *ctlpb.StorageFormatReq, stream ctlpb.MgmtCtl_StorageFormatServer) error

StorageFormat delegates to Storage implementation's Format methods to prepare storage for use by DAOS data plane.

Errors returned will stop other servers from formatting, non-fatal errors specific to particular device should be reported within resp results instead.

Send response containing multiple results of format operations on scm mounts and nvme controllers.

func (*ControlService) SystemMemberQuery 

func (svc *ControlService) SystemMemberQuery(ctx context.Context, req *ctlpb.SystemMemberQueryReq) (*ctlpb.SystemMemberQueryResp, error)

SystemMemberQuery implements the method defined for the Management Service.

Return system membership list including member state.

func (*ControlService) SystemStop 

func (svc *ControlService) SystemStop(ctx context.Context, req *ctlpb.SystemStopReq) (*ctlpb.SystemStopResp, error)

SystemStop implements the method defined for the Management Service.

Initiate controlled shutdown of DAOS system.

type External 

type External interface {
	// contains filtered or unexported methods
}

External interface provides methods to support various os operations.

type IOServerHarness 

type IOServerHarness struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

IOServerHarness is responsible for managing IOServer instances

func NewIOServerHarness 

func NewIOServerHarness(log logging.Logger) *IOServerHarness

NewHarness returns an initialized *IOServerHarness

func (*IOServerHarness) AddInstance 

func (h *IOServerHarness) AddInstance(srv *IOServerInstance) error

AddInstance adds a new IOServer instance to be managed.

func (*IOServerHarness) AwaitStorageReady 

func (h *IOServerHarness) AwaitStorageReady(ctx context.Context, skipMissingSuperblock bool) error

AwaitStorageReady blocks until all managed IOServer instances have storage available and ready to be used.

func (*IOServerHarness) CreateSuperblocks 

func (h *IOServerHarness) CreateSuperblocks(recreate bool) error

CreateSuperblocks creates instance superblocks as needed.

func (*IOServerHarness) GetMSLeaderInstance 

func (h *IOServerHarness) GetMSLeaderInstance() (*IOServerInstance, error)

GetMSLeaderInstance returns a managed IO Server instance to be used as a management target and fails if selected instance is not MS Leader.

func (*IOServerHarness) Instances 

func (h *IOServerHarness) Instances() []*IOServerInstance

func (*IOServerHarness) IsStarted 

func (h *IOServerHarness) IsStarted() bool

func (*IOServerHarness) Start 

func (h *IOServerHarness) Start(parent context.Context) error

Start starts all configured instances and the management service, then waits for them to exit.

func (*IOServerHarness) StartManagementService 

func (h *IOServerHarness) StartManagementService(ctx context.Context) error

StartManagementService starts the DAOS management service on this node.

type IOServerInstance 

type IOServerInstance struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

IOServerInstance encapsulates control-plane specific configuration and functionality for managed I/O server instances. The distinction between this structure and what's in the ioserver package is that the ioserver package is only concerned with configuring and executing a single daos_io_server instance. IOServerInstance is intended to be used with IOServerHarness to manage and monitor multiple instances per node.

func NewIOServerInstance 

func NewIOServerInstance(log logging.Logger,
	bcp *bdev.ClassProvider, sp *scm.Provider,
	msc *mgmtSvcClient, r IOServerStarter) *IOServerInstance

NewIOServerInstance returns an *IOServerInstance initialized with its dependencies.

func (*IOServerInstance) AwaitReady 

func (srv *IOServerInstance) AwaitReady() chan *srvpb.NotifyReadyReq

AwaitReady returns a channel which receives a ready message when the started IOServer instance indicates that it is ready to receive dRPC messages.

func (*IOServerInstance) AwaitStorageReady 

func (srv *IOServerInstance) AwaitStorageReady(ctx context.Context)

AwaitStorageReady blocks until the IOServer's storage is ready.

func (*IOServerInstance) BioErrorNotify added in v0.8.0 

func (srv *IOServerInstance) BioErrorNotify(bio *srvpb.BioErrorReq)

func (*IOServerInstance) CallDrpc 

func (srv *IOServerInstance) CallDrpc(module, method int32, body proto.Message) (*drpc.Response, error)

CallDrpc makes the supplied dRPC call via this instance's dRPC client.

func (*IOServerInstance) CreateSuperblock 

func (srv *IOServerInstance) CreateSuperblock(msInfo *mgmtInfo) error

CreateSuperblock creates the superblock for this instance.

func (*IOServerInstance) Index 

func (srv *IOServerInstance) Index() uint32

Index returns the server index assigned by the harness.

func (*IOServerInstance) IsMSReplica 

func (srv *IOServerInstance) IsMSReplica() bool

IsMSReplica indicates whether or not this instance is a management service replica.

func (*IOServerInstance) LoadModules added in v0.8.0 

func (srv *IOServerInstance) LoadModules() error

LoadModules initiates the I/O server startup sequence.

func (*IOServerInstance) MountScmDevice 

func (srv *IOServerInstance) MountScmDevice() error

MountScmDevice mounts the configured SCM device (DCPM or ramdisk emulation) at the mountpoint specified in the configuration. If the device is already mounted, the function returns nil, indicating success.

func (*IOServerInstance) NeedsScmFormat 

func (srv *IOServerInstance) NeedsScmFormat() (bool, error)

NeedsScmFormat probes the configured instance storage and determines whether or not it requires a format operation before it can be used.

func (*IOServerInstance) NeedsSuperblock 

func (srv *IOServerInstance) NeedsSuperblock() (bool, error)

NeedsSuperblock indicates whether or not the instance appears to need a superblock to be created in order to start.

func (*IOServerInstance) NotifyReady 

func (srv *IOServerInstance) NotifyReady(msg *srvpb.NotifyReadyReq)

NotifyReady receives a ready message from the running IOServer instance.

func (*IOServerInstance) NotifyStorageReady 

func (srv *IOServerInstance) NotifyStorageReady()

NotifyStorageReady releases any blocks on AwaitStorageReady().

func (*IOServerInstance) ReadSuperblock 

func (srv *IOServerInstance) ReadSuperblock() error

ReadSuperblock reads the instance's superblock from storage.

func (*IOServerInstance) SetIndex 

func (srv *IOServerInstance) SetIndex(idx uint32)

SetIndex sets the server index assigned by the harness.

func (*IOServerInstance) SetRank 

func (srv *IOServerInstance) SetRank(ctx context.Context, ready *srvpb.NotifyReadyReq) error

SetRank determines the instance rank and sends a SetRank dRPC request to the IOServer.

func (*IOServerInstance) Start 

func (srv *IOServerInstance) Start(ctx context.Context, errChan chan<- error) error

Start checks to make sure that the instance has a valid superblock before performing any required NVMe preparation steps and launching a managed daos_io_server instance.

func (*IOServerInstance) StartManagementService 

func (srv *IOServerInstance) StartManagementService() error

StartManagementService starts the DAOS management service replica associated with this instance. If no replica is associated with this instance, this function is a no-op.

func (*IOServerInstance) WriteSuperblock 

func (srv *IOServerInstance) WriteSuperblock() error

WriteSuperblock writes the instance's superblock to storage.

type IOServerStarter 

type IOServerStarter interface {
	Start(context.Context, chan<- error) error
	GetConfig() *ioserver.Config
}

IOServerStarter defines an interface for starting the actual daos_io_server.

type SecurityModule 

type SecurityModule struct {
	// contains filtered or unexported fields
}

SecurityModule is the security drpc module struct

func NewSecurityModule 

func NewSecurityModule(tc *security.TransportConfig) *SecurityModule

NewSecurityModule creates a new security module with a transport config

func (*SecurityModule) HandleCall 

func (m *SecurityModule) HandleCall(session *drpc.Session, method int32, body []byte) ([]byte, error)

HandleCall is the handler for calls to the SecurityModule

func (*SecurityModule) ID 

func (m *SecurityModule) ID() int32

ID will return Security module ID

type StorageControlService 

type StorageControlService struct {
	// contains filtered or unexported fields
}

StorageControlService encapsulates the storage part of the control service

func DefaultStorageControlService 

func DefaultStorageControlService(log logging.Logger, cfg *Configuration) (*StorageControlService, error)

DefaultStorageControlService returns a initialized *StorageControlService with default behaviour

func NewStorageControlService 

func NewStorageControlService(log logging.Logger, bdev *bdev.Provider, scm *scm.Provider, srvCfgs []*ioserver.Config) *StorageControlService

NewStorageControlService returns an initialized *StorageControlService

func (*StorageControlService) GetScmState 

func (c *StorageControlService) GetScmState() (storage.ScmState, error)

GetScmState performs required initialisation and returns current state of SCM module preparation.

func (*StorageControlService) NvmePrepare 

func (c *StorageControlService) NvmePrepare(req bdev.PrepareRequest) (*bdev.PrepareResponse, error)

NvmePrepare preps locally attached SSDs and returns error.

Suitable for commands invoked directly on server, not over gRPC.

func (*StorageControlService) NvmeScan 

func (c *StorageControlService) NvmeScan() (*bdev.ScanResponse, error)

NvmeScan scans locally attached SSDs and returns list directly.

Suitable for commands invoked directly on server, not over gRPC.

func (*StorageControlService) ScmPrepare 

func (c *StorageControlService) ScmPrepare(req scm.PrepareRequest) (*scm.PrepareResponse, error)

ScmPrepare preps locally attached modules and returns need to reboot message, list of pmem device files and error directly.

Suitable for commands invoked directly on server, not over gRPC.

func (*StorageControlService) ScmScan 

func (c *StorageControlService) ScmScan() (*scm.ScanResponse, error)

ScmScan scans locally attached modules, namespaces and state of DCPM config.

Suitable for commands invoked directly on server, not over gRPC.

func (*StorageControlService) Setup 

func (c *StorageControlService) Setup() error

Setup delegates to Storage implementation's Setup methods.

func (*StorageControlService) StoragePrepare 

func (c *StorageControlService) StoragePrepare(ctx context.Context, req *ctlpb.StoragePrepareReq) (
	*ctlpb.StoragePrepareResp, error)

StoragePrepare configures SSDs for user specific access with SPDK and groups SCM modules in AppDirect/interleaved mode as kernel "pmem" devices.

func (*StorageControlService) StorageScan 

func (c *StorageControlService) StorageScan(ctx context.Context, req *ctlpb.StorageScanReq) (*ctlpb.StorageScanResp, error)

StorageScan discovers non-volatile storage hardware on node.

type Superblock 

type Superblock struct {
	Version     uint8
	UUID        string
	System      string
	Rank        *ioserver.Rank
	ValidRank   bool
	MS          bool
	CreateMS    bool
	BootstrapMS bool
}

Superblock is the per-Instance superblock

func ReadSuperblock 

func ReadSuperblock(sbPath string) (*Superblock, error)

ReadSuperblock reads a Superblock from storage.

func (*Superblock) Marshal 

func (sb *Superblock) Marshal() ([]byte, error)

Marshal transforms the Superblock into a storable representation

func (*Superblock) Unmarshal 

func (sb *Superblock) Unmarshal(raw []byte) error

Unmarshal reconstitutes a Superblock from a Marshaled representation

Source Files

View all Source files

Directories

Path Synopsis
(C) Copyright 2019 Intel Corporation.
 (C) Copyright 2019 Intel Corporation.
(C) Copyright 2019 Intel Corporation.
 (C) Copyright 2019 Intel Corporation.
bdev
(C) Copyright 2019 Intel Corporation.
 (C) Copyright 2019 Intel Corporation.
scm
(C) Copyright 2019 Intel Corporation.
 (C) Copyright 2019 Intel Corporation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.