setup

README

NOTE: The following document describes an experimental, work-in-progress feature. To enable the cscli setup command, set the environment variable CROWDSEC_FEATURE_CSCLI_SETUP=true or add the line " - cscli_setup" to /etc/crowdsec/feature.yaml. Any feedback is welcome.

---

cscli setup

The "cscli setup" command can configure a crowdsec instance based on the services that are installed or running on the server.

There are three main subcommands:

• cscli setup detect: detect the services, the OS family, version or the Linux distribution
• cscli setup install-hub: install the recommended collections, parsers, etc. based on the detection result
• cscli setup datasources: generate the appropriate acquisition rules

The setup command is used in the wizard.sh script, but can also be invoked by hand or customized via a configuration file by adding new services, log locations and detection rules.

Detection and installation are performed as separate steps, as you can see in the following diagram:

 +-------------+
 |             |
 | detect.yaml |
 |             |
 +-------------+
        |
        v
  setup detect
        |
        v
 +--------------+
 |              +---> setup install-hub     +-----------------------+
 |  setup.yaml  |                           |                       |
 |              +---> setup datasources --->| etc/crowdsec/acquis.d |
 +--------------+                           |                       |
                                            +-----------------------+

You can inspect and customize the intermediary file (setup.yaml), which is useful in case of many instances, deployment automation or unusual setups.

A subcommand can be used to check your changes in this case:

• cscli setup validate: validate or report errors on a setup file

Basic usage

Identify the existing services and write out what was detected:

# cscli setup detect > setup.yaml

See what was found.

# cscli setup install-hub setup.yaml --dry-run
dry-run: would install collection crowdsecurity/apache2
dry-run: would install collection crowdsecurity/linux
dry-run: would install collection crowdsecurity/pgsql
dry-run: would install parser crowdsecurity/whitelists

Install the objects (parsers, scenarios...) required to support the detected services:

# cscli setup install-hub setup.yaml
INFO[29-06-2022 03:16:14 PM] crowdsecurity/apache2-logs : OK              
INFO[29-06-2022 03:16:14 PM] Enabled parsers : crowdsecurity/apache2-logs 
INFO[29-06-2022 03:16:14 PM] crowdsecurity/http-logs : OK             
[...]
INFO[29-06-2022 03:16:18 PM] Enabled crowdsecurity/linux      

Generate the datasource configuration:

# cscli setup datasources setup.yaml --to-dir /etc/crowdsec/acquis.d

With the above command, each detected service gets a corresponding file in the acquis.d directory. Running cscli setup again may add more services as they are detected, but datasource files or hub items are never removed automatically.

The detect.yaml file

A detect.yaml file is downloaded when you first install crowdsec, and is updated by the cscli hub update command.

NOTE: XXX XXX - this is currently not the case, the file is distributed in the crowdsec repository, but it should change.

You can see the default location with cscli setup detect --help | grep detect-config

The YAML file contains a version number (always 1.0) and a list of sections, one per supported service.

Each service defines its detection rules, the recommended hub items and recommended datasources. The same software can be defined in multiple service sections: for example, apache on debian and fedora have different detection rules and different datasources so it requires two sections to support both platforms.

The following are minimal detect.yaml examples just to show a few concepts.

version: 1.0

services:

  apache2:
    when:
      - ProcessRunning("apache2")
    install:
      collections:
        - crowdsecurity/apache2
    datasources:
      source: file
      labels:
        type: apache2
      filenames:
        - /var/log/apache2/*.log
        - /var/log/httpd/*.log

• ProcessRunning() matches the process name of a running application. The when: clause can contain any number of expressions, they are all evaluated and must all return true for a service to be detected (implied and clause, no short-circuit). A missing or empty when: section is evaluated as true. The expression engine is the same one used by CrowdSec parser filters. You can force the detection of a process by using the cscli setup detect... --force-process <processname> flag. It will always behave as if <processname> was running.

The install: section can contain any number of collections, parsers, scenarios and postoverflows. In practices, it's most often a single collection.

The datasource: section is copied as-is in the acquisition file.

NOTE: XXX TODO - the current version does not validate the datasource: mapping. Bad content is written to acquis.d until crowdsec chokes on it.

Detecting a running process may seem a good idea, but if a process manager like systemd is available it's better to ask it for the information we want.

version: 1.0

services:

  apache2-systemd:
    when:
      - UnitFound("apache2.service")
      - OS.ID != "centos"
    install:
      collections:
        - crowdsecurity/apache2
    datasource:
      source: file
      labels:
        type: syslog
      filenames:
        - /var/log/apache2/*.log

  apache2-systemd-centos:
    when:
      - UnitFound("httpd.service")
      - OS.ID == "centos"
    install:
      collections:
        - crowdsecurity/apache2
    datasource:
      source: file
      labels:
        type: syslog
      filenames:
        - /var/log/httpd/*.log

Here we see two more detection methods:

• UnitFound() matches the name of systemd units, if the are in state enabled, generated or static. You can see here that CentOS is using a different unit name for Apache so it must have its own service section. You can force the detection of a unit by using the cscli setup detect... --force-unit <unitname> flag.

• OS.Family, OS.ID and OS.RawVersion are read from /etc/os-release in case of Linux, and detected by other methods for FreeBSD and Windows. Under FreeBSD and Windows, the value of OS.ID is the same as OS.Family. If OS detection fails, it can be overridden with the flags --force-os-family, --force-os-id and --force-os-version.

If you want to ignore one or more services (i.e. not install anything and not generate acquisition rules) you can specify it with cscli setup detect... --skip-service <servicename>. For example, --skip-service apache2-systemd. If you want to disable systemd unit detection, use cscli setup detect... --snub-systemd.

If you used the --force-process or --force-unit flags, but none of the defined services is looking for them, you'll have an error like "detecting services: process(es) forced but not supported".

NOTE: XXX XXX - having an error for this is maybe too much, but can tell that a configuration is outdated. Could this be a warning with optional flag to make it an error?

We used the OS.ID value to check for the linux distribution, but since the same configuration is required for CentOS and the other RedHat derivatives, it's better to check for the existence of a file that is known to exist in all of them:

version: 1.0

services:

  apache2-systemd-deb:
    when:
      - UnitFound("apache2.service")
      - PathExists("/etc/debian_version")
    install:
    # [...]

  apache2-systemd-rpm:
    when:
      - UnitFound("httpd.service")
      - PathExists("/etc/redhat-release")
    install:
    # [...]

• PathExists() evaluates to true if a file, directory or link exists at the given path. It does not check for broken links.

Rules can be used to detect operating systems and environments:

version: 1.0

services:

  linux:
    when:
      - OS.Family == "linux"
    install:
      collections:
        - crowdsecurity/linux
    datasource:
      type: file
      labels:
        type: syslog
      log_files:
      - /var/log/syslog
      - /var/log/kern.log
      - /var/log/messages

  freebsd:
    when:
      - OS.Family == "freebsd"
    install:
      collections:
        - crowdsecurity/freebsd

  windows:
    when:
      - OS.Family == "windows"
    install:
      collections:
        - crowdsecurity/windows

The OS object contains a methods to check for version numbers: OS.VersionCheck("<constraint>"). It uses the Masterminds/semver package and accepts a variety of operators.

Instead of: OS.RawVersion == "1.2.3" you should use OS.VersionCheck("~1"), OS.VersionCheck("~1.2") depending if you want to match the major or the minor version. It's unlikely that you need to match the exact patch level.

Leading zeroes are permitted, to allow comparison of Ubuntu versions: strict semver rules would treat "22.04" as invalid.

The setup.yaml file

This file does not actually have a specific name, as it's usually written to standard output.

For example, on a Debian system running Apache under systemd you can execute:

$ cscli setup detect --yaml
setup:
  - detected_service: apache2-systemd-deb
    install:
      collections:
        - crowdsecurity/apache2
    datasource:
      filenames:
        - /var/log/apache2/*.log
      labels:
        type: apache2
  - detected_service: linux
    install:
      collections:
        - crowdsecurity/linux
    datasource:
      filenames:
        - /var/log/syslog
        - /var/log/kern.log
        - /var/log/messages
      labels:
        type: syslog
  - detected_service: whitelists
    install:
      parsers:
        - crowdsecurity/whitelists

The default output format is JSON, which is compatible with YAML but less readable to humans.

• detected_service: used to generate a name for the files written to acquis.d
• install: can contain collections, parsers, scenarios, postoverflows
• datasource: copied to acquis.d

$ cscli setup datasources --help
generate datasource (acquisition) configuration from a setup file

Usage:
  cscli setup datasources [setup_file] [flags]

Flags:
  -h, --help            help for datasources
      --to-dir string   write the configuration to a directory, in multiple files
[...]

If the --to-dir option is not specified, a single monolithic acquis.yaml is printed to the standard output.

Documentation

Index

• Variables
• func DataSources(input []byte, toDir string) (string, error)
• func InstallHubItems(hub *cwhub.Hub, input []byte, dryRun bool) error
• func ListSupported(detectConfig io.Reader) ([]string, error)
• func Validate(input []byte) error
• type AcquisDocument
• type DataSourceItem
• type DetectConfig
• type DetectOptions
• type ExprEnvironment
  • func NewExprEnvironment(opts DetectOptions, os ExprOS) ExprEnvironment
  • func (e ExprEnvironment) PathExists(path string) bool
  • func (e ExprEnvironment) ProcessRunning(processName string) (bool, error)
  • func (e ExprEnvironment) UnitFound(unitName string) (bool, error)
• type ExprOS
  • func (os ExprOS) VersionAtLeast(constraint string) (bool, error)
  • func (os ExprOS) VersionCheck(constraint string) (bool, error)
  • func (os ExprOS) VersionIsLower(version string) (bool, error)
• type ExprServiceState
• type ExprState
• type HubItems
• type Service
• type ServiceSetup
• type Setup
  • func Detect(detectReader io.Reader, opts DetectOptions) (Setup, error)

Variables

var ExecCommand = exec.Command

ExecCommand can be replaced with a mock during tests.

Functions

func DataSources

func DataSources(input []byte, toDir string) (string, error)

DataSources generates the acquisition documents from a setup file.

func InstallHubItems

func InstallHubItems(hub *cwhub.Hub, input []byte, dryRun bool) error

InstallHubItems installs the objects recommended in a setup file.

func ListSupported

func ListSupported(detectConfig io.Reader) ([]string, error)

ListSupported parses the configuration file and outputs a list of the supported services.

func Validate

func Validate(input []byte) error

Validate checks the validity of a setup file.

Types

type AcquisDocument

type AcquisDocument struct {
	AcquisFilename string
	DataSource     map[string]interface{}
}

AcquisDocument is created from a SetupItem. It represents a single YAML document, and can be part of a multi-document file.

type DataSourceItem

type DataSourceItem map[string]interface{}

type DetectConfig

type DetectConfig struct {
	Version string             `yaml:"version"`
	Detect  map[string]Service `yaml:"detect"`
}

DetectConfig is the container of all detection rules (detect.yaml).

type DetectOptions

type DetectOptions struct {
	// slice of unit names that we want to force-detect
	ForcedUnits []string
	// slice of process names that we want to force-detect
	ForcedProcesses []string
	ForcedOS        ExprOS
	SkipServices    []string
	SnubSystemd     bool
}

DetectOptions contains parameters for the Detect function.

type ExprEnvironment

type ExprEnvironment struct {
	OS ExprOS
	// contains filtered or unexported fields
}

ExprEnvironment is used to expose functions and values to the rule engine. It can cache the results of service detection commands, like systemctl etc.

func NewExprEnvironment

func NewExprEnvironment(opts DetectOptions, os ExprOS) ExprEnvironment

NewExprEnvironment creates an environment object for the rule engine.

func (ExprEnvironment) PathExists

func (e ExprEnvironment) PathExists(path string) bool

PathExists returns true if the given path exists.

func (ExprEnvironment) ProcessRunning

func (e ExprEnvironment) ProcessRunning(processName string) (bool, error)

ProcessRunning returns true if there is a running process with the given name.

func (ExprEnvironment) UnitFound

func (e ExprEnvironment) UnitFound(unitName string) (bool, error)

UnitFound returns true if the unit is listed in the systemctl output. Whether a disabled or failed unit is considered found or not, depends on the systemctl parameters used.

type ExprOS

type ExprOS struct {
	Family     string
	ID         string
	RawVersion string
}

ExprOS contains the detected (or forced) OS fields available to the rule engine.

func (ExprOS) VersionAtLeast

func (os ExprOS) VersionAtLeast(constraint string) (bool, error)

VersionAtLeast returns true if the version of the OS is at least the given version.

func (ExprOS) VersionCheck

func (os ExprOS) VersionCheck(constraint string) (bool, error)

VersionCheck returns true if the version of the OS matches the given constraint

func (ExprOS) VersionIsLower

func (os ExprOS) VersionIsLower(version string) (bool, error)

VersionIsLower returns true if the version of the OS is lower than the given version.

type ExprServiceState

type ExprServiceState struct {
	// contains filtered or unexported fields
}

ExprServiceState keep a local state during the detection of a single service. It is reset before each service rules' evaluation.

type ExprState

type ExprState struct {
	// contains filtered or unexported fields
}

ExprState keeps a global state for the duration of the service detection (cache etc.)

type HubItems

type HubItems struct {
	Collections   []string `yaml:"collections,omitempty"`
	Parsers       []string `yaml:"parsers,omitempty"`
	Scenarios     []string `yaml:"scenarios,omitempty"`
	PostOverflows []string `yaml:"postoverflows,omitempty"`
}

HubItems contains the objects that are recommended to support a service.

type Service

type Service struct {
	When       []string       `yaml:"when"`
	Install    *HubItems      `yaml:"install,omitempty"`
	DataSource DataSourceItem `yaml:"datasource,omitempty"`
}

Service describes the rules for detecting a service and its recommended items.

type ServiceSetup

type ServiceSetup struct {
	DetectedService string         `yaml:"detected_service"`
	Install         *HubItems      `yaml:"install,omitempty"`
	DataSource      DataSourceItem `yaml:"datasource,omitempty"`
}

ServiceSetup describes the recommendations (hub objects and datasources) for a detected service.

type Setup

type Setup struct {
	Setup []ServiceSetup `yaml:"setup"`
}

Setup is a container for a list of ServiceSetup objects, allowing for future extensions.

func Detect

func Detect(detectReader io.Reader, opts DetectOptions) (Setup, error)

Detect performs the service detection from a given configuration. It outputs a setup file that can be used as input to "cscli setup install-hub" or "cscli setup datasources".