cwhub

package
v1.6.5-rc5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 5, 2025 License: MIT Imports: 26 Imported by: 1

Documentation

Overview

Package cwhub is responsible for providing the state of the local hub to the security engine and cscli command. Installation, upgrade and removal of items or data files has been moved to pkg/hubops.

Definitions

  • A hub ITEM is a file that defines a parser, a scenario, a collection... in the case of a collection, it has dependencies on other hub items.
  • The hub INDEX is a JSON file that contains a tree of available hub items.
  • A REMOTE HUB is an HTTP server that hosts the hub index and the hub items. It can serve from several branches, usually linked to the CrowdSec version.
  • A LOCAL HUB is a directory that contains a copy of the hub index and the downloaded hub items.

Once downloaded, hub items can be installed by linking to them from the configuration directory. If an item is present in the configuration directory but it's not a link to the local hub, it is considered as a LOCAL ITEM and won't be removed or upgraded.

Directory Structure

A typical directory layout is the following:

For the local hub (HubDir = /etc/crowdsec/hub):

  • /etc/crowdsec/hub/.index.json
  • /etc/crowdsec/hub/parsers/{stage}/{author}/{parser-name}.yaml
  • /etc/crowdsec/hub/scenarios/{author}/{scenario-name}.yaml

For the configuration directory (InstallDir = /etc/crowdsec):

  • /etc/crowdsec/parsers/{stage}/{parser-name.yaml} -> /etc/crowdsec/hub/parsers/{stage}/{author}/{parser-name}.yaml
  • /etc/crowdsec/scenarios/{scenario-name.yaml} -> /etc/crowdsec/hub/scenarios/{author}/{scenario-name}.yaml
  • /etc/crowdsec/scenarios/local-scenario.yaml

Note that installed items are not grouped by author, this may change in the future if we want to support items with the same name from different authors.

Only parsers and postoverflows have the concept of stage.

Additionally, an item can reference a DATA SET that is installed in a different location than the item itself. These files are stored in the data directory (InstallDataDir = /var/lib/crowdsec/data).

  • /var/lib/crowdsec/data/http_path_traversal.txt
  • /var/lib/crowdsec/data/jira_cve_2021-26086.txt
  • /var/lib/crowdsec/data/log4j2_cve_2021_44228.txt
  • /var/lib/crowdsec/data/sensitive_data.txt

Using the package

The main entry point is the Hub struct. You can create a new instance with NewHub(). This constructor takes three parameters, but only the LOCAL HUB configuration is required: 

import (
	"fmt"
	"github.com/crowdsecurity/crowdsec/pkg/csconfig"
	"github.com/crowdsecurity/crowdsec/pkg/cwhub"
)

localHub := csconfig.LocalHubCfg{
	HubIndexFile:	"/etc/crowdsec/hub/.index.json",
	HubDir:		"/etc/crowdsec/hub",
	InstallDir:	"/etc/crowdsec",
	InstallDataDir: "/var/lib/crowdsec/data",
}

hub, err := cwhub.NewHub(localHub, nil, logger)
if err != nil {
	return fmt.Errorf("unable to initialize hub: %w", err)
}

If the logger is nil, the item-by-item messages will be discarded, including warnings. After configuring the hub, you must sync its state with items on disk. 

err := hub.Load()
if err != nil {
	return fmt.Errorf("unable to load hub: %w", err)
}

Now you can use the hub object to access the existing items: 

// list all the parsers
for _, parser := range hub.GetItemsByType(cwhub.PARSERS, false) {
	fmt.Printf("parser: %s\n", parser.Name)
}

// retrieve a specific collection
coll := hub.GetItem(cwhub.COLLECTIONS, "crowdsecurity/linux")
if coll == nil {
	return fmt.Errorf("collection not found")
}

Some commands require an object to provide the hub index, or contents: 

indexProvider := cwhub.Downloader{
	URLTemplate: "https://cdn-hub.crowdsec.net/crowdsecurity/%s/%s",
	Branch: "master",
}

The URLTemplate is a string that will be used to build the URL of the remote hub. It must contain two placeholders: the branch and the file path (it will be an index or an item).

Before calling hub.Load(), you can update the index file by calling the Update() method: 

err := hub.Update(context.Background(), indexProvider)
if err != nil {
	return fmt.Errorf("unable to update hub index: %w", err)
}

Note that the command will fail if the hub has already been synced. If you want to do it (ex. after a configuration change the application is notified with SIGHUP) you have to instantiate a new hub object and dispose of the old one.

Index

Constants

View Source 
const (
	// managed item types.
	COLLECTIONS    = "collections"
	PARSERS        = "parsers"
	POSTOVERFLOWS  = "postoverflows"
	SCENARIOS      = "scenarios"
	CONTEXTS       = "contexts"
	APPSEC_CONFIGS = "appsec-configs"
	APPSEC_RULES   = "appsec-rules"
)

Variables

View Source 
var ErrSkipPath = errors.New("sentinel")

A sentinel to skip regular files because "nil, nil" is ambiguous. Returning SkipDir with files would skip the rest of the directory.

View Source 
var ErrUpdateAfterSync = errors.New("cannot update hub index after load/sync")
View Source 
var HubClient = &http.Client{
	Timeout:   120 * time.Second,
	Transport: &hubTransport{http.DefaultTransport},
}

HubClient is the HTTP client used to communicate with the CrowdSec Hub.

View Source 
var ItemTypes = []string{PARSERS, POSTOVERFLOWS, SCENARIOS, CONTEXTS, APPSEC_CONFIGS, APPSEC_RULES, COLLECTIONS}

The order is important, as it is used to range over sub-items in collections.

Functions

func SafePath added in v1.6.5 

func SafePath(dir, filePath string) (string, error)

SafePath returns a joined path and ensures that it does not escape the base directory.

Types

type ContentProvider added in v1.6.5 

type ContentProvider interface {
	FetchContent(ctx context.Context, remotePath, destPath, wantHash string, logger *logrus.Logger) (bool, string, error)
}

ContentProvider retrieves and writes the YAML files with the item content.

type Dependencies added in v1.6.5 

type Dependencies struct {
	Parsers       []string `json:"parsers,omitempty"        yaml:"parsers,omitempty"`
	PostOverflows []string `json:"postoverflows,omitempty"  yaml:"postoverflows,omitempty"`
	Scenarios     []string `json:"scenarios,omitempty"      yaml:"scenarios,omitempty"`
	Collections   []string `json:"collections,omitempty"    yaml:"collections,omitempty"`
	Contexts      []string `json:"contexts,omitempty"       yaml:"contexts,omitempty"`
	AppsecConfigs []string `json:"appsec-configs,omitempty" yaml:"appsec-configs,omitempty"`
	AppsecRules   []string `json:"appsec-rules,omitempty"   yaml:"appsec-rules,omitempty"`
}

func (Dependencies) SubItems added in v1.6.5 

func (d Dependencies) SubItems(hub *Hub) func(func(*Item) bool)

SubItems iterates over the sub-items in the struct, excluding the ones that were not found in the hub.

type Downloader added in v1.6.5 

type Downloader struct {
	Branch      string
	URLTemplate string
}

Downloader is used to retrieve index and items from a remote hub, with cache control.

func (*Downloader) FetchContent added in v1.6.5 

func (d *Downloader) FetchContent(ctx context.Context, remotePath, destPath, wantHash string, logger *logrus.Logger) (bool, string, error)

FetchContent downloads the content to the specified path, through a temporary file to avoid partial downloads. If the hash does not match, it will not overwrite and log a warning.

func (*Downloader) FetchIndex added in v1.6.5 

func (d *Downloader) FetchIndex(ctx context.Context, destPath string, withContent bool, logger *logrus.Logger) (bool, error)

FetchIndex downloads the index from the hub and writes it to the filesystem. It uses a temporary file to avoid partial downloads, and won't overwrite the original if it has not changed.

type Hub added in v1.6.0 

type Hub struct {
	Warnings []string // Warnings encountered during sync
	// contains filtered or unexported fields
}

Hub is the main structure for the package.

func NewHub added in v1.6.0 

func NewHub(local *csconfig.LocalHubCfg, logger *logrus.Logger) (*Hub, error)

NewHub returns a new Hub instance with local and (optionally) remote configuration. The hub is not synced automatically. Load() must be called to read the index, sync the local state, and check for unmanaged items.

func (*Hub) GetDataDir added in v1.6.0 

func (h *Hub) GetDataDir() string

GetDataDir returns the data directory, where data sets are installed.

func (*Hub) GetInstalledByType added in v1.6.3 

func (h *Hub) GetInstalledByType(itemType string, sorted bool) []*Item

GetInstalledByType returns a slice of all the installed items of a given type, optionally sorted by case-insensitive name. A non-existent type will silently return an empty slice.

func (*Hub) GetInstalledListForAPI added in v1.6.3 

func (h *Hub) GetInstalledListForAPI() []string

GetInstalledListForAPI returns a slice of names of all the installed scenarios and appsec-rules. The returned list is sorted by type (scenarios first) and case-insensitive name.

func (*Hub) GetItem added in v1.6.0 

func (h *Hub) GetItem(itemType string, itemName string) *Item

GetItem returns an item from hub based on its type and full name (author/name).

func (*Hub) GetItemByPath added in v1.6.0 

func (h *Hub) GetItemByPath(itemPath string) *Item

GetItemByPath returns an item from hub based on its (absolute) local path.

func (*Hub) GetItemFQ added in v1.6.0 

func (h *Hub) GetItemFQ(itemFQName string) (*Item, error)

GetItemFQ returns an item from hub based on its type and name (type:author/name).

func (*Hub) GetItemMap added in v1.6.0 

func (h *Hub) GetItemMap(itemType string) map[string]*Item

GetItemMap returns the map of items for a given type.

func (*Hub) GetItemsByType added in v1.6.2 

func (h *Hub) GetItemsByType(itemType string, sorted bool) []*Item

GetItemsByType returns a slice of all the items of a given type, installed or not, optionally sorted by case-insensitive name. A non-existent type will silently return an empty slice.

func (*Hub) ItemStats added in v1.6.0 

func (h *Hub) ItemStats() []string

ItemStats returns total counts of the hub items, including local and tainted.

func (*Hub) Load added in v1.6.3 

func (h *Hub) Load() error

Load reads the state of the items on disk.

func (*Hub) Update added in v1.6.3 

func (h *Hub) Update(ctx context.Context, indexProvider IndexProvider, withContent bool) error

Update downloads the latest version of the index and writes it to disk if it changed. It cannot be called after Load() unless the index was completely empty.

type HubItems added in v1.6.0 

type HubItems map[string]map[string]*Item

type IndexProvider added in v1.6.5 

type IndexProvider interface {
	FetchIndex(ctx context.Context, indexFile string, withContent bool, logger *logrus.Logger) (bool, error)
}

IndexProvider retrieves and writes .index.json

type Item 

type Item struct {
	State ItemState `json:"-" yaml:"-"` // local state, not stored in the index

	Type        string   `json:"type,omitempty"        yaml:"type,omitempty"`
	Stage       string   `json:"stage,omitempty"       yaml:"stage,omitempty"`     // Stage for parser|postoverflow: s00-raw/s01-...
	Name        string   `json:"name,omitempty"        yaml:"name,omitempty"`      // usually "author/name"
	FileName    string   `json:"file_name,omitempty"   yaml:"file_name,omitempty"` // eg. apache2-logs.yaml
	Description string   `json:"description,omitempty" yaml:"description,omitempty"`
	Content     string   `json:"content,omitempty"     yaml:"-"`
	References  []string `json:"references,omitempty"  yaml:"references,omitempty"`

	// NOTE: RemotePath could be derived from the other fields
	RemotePath string                 `json:"path,omitempty" yaml:"path,omitempty"`       // path relative to the base URL eg. /parsers/stage/author/file.yaml
	Version    string                 `json:"version,omitempty" yaml:"version,omitempty"` // the last available version
	Versions   map[string]ItemVersion `json:"versions,omitempty"  yaml:"-"`               // all the known versions

	// The index contains the dependencies of the "latest" version (collections only)
	Dependencies
	// contains filtered or unexported fields
}

Item is created from an index file and enriched with local info.

func (*Item) Ancestors added in v1.6.0 

func (i *Item) Ancestors() []*Item

Ancestors returns a slice of items (typically collections) that have this item as a direct or indirect dependency.

func (*Item) CurrentDependencies added in v1.6.5 

func (i *Item) CurrentDependencies() Dependencies

CurrentSubItems returns a slice of sub-items of the installed version, excluding the ones that were not found. The list comes from the content file if parseable, otherwise from the index (same as LatestDependencies).

func (*Item) DownloadPath added in v1.6.5 

func (i *Item) DownloadPath() (string, error)

DownloadPath returns the location of the actual config file in the hub (eg. /etc/crowdsec/hub/collections/author/xyz.yaml). Raises an error if the path goes outside of the hub dir.

func (*Item) FQName added in v1.6.0 

func (i *Item) FQName() string

FQName returns the fully qualified name of the item (ie. parsers:crowdsecurity/apache2-logs).

func (*Item) FetchContentTo added in v1.6.2 

func (i *Item) FetchContentTo(ctx context.Context, contentProvider ContentProvider, destPath string) (bool, string, error)

FetchContentTo writes the last version of the item's YAML file to the specified path. If the file is embedded in the index file, it will be written directly without downloads. Returns whether the file was downloaded (to inform if the security engine needs reloading) and the remote url for feedback purposes.

func (*Item) HasSubItems added in v1.6.0 

func (i *Item) HasSubItems() bool

HasSubItems returns true if items of this type can have sub-items. Currently only collections.

func (*Item) InstallPath added in v1.6.5 

func (i *Item) InstallPath() (string, error)

InstallPath returns the location of the symlink to the item in the hub, or the path of the item itself if it's local (eg. /etc/crowdsec/collections/xyz.yaml). Raises an error if the path goes outside of the install dir.

func (*Item) LatestDependencies added in v1.6.5 

func (i *Item) LatestDependencies() Dependencies

LatestDependencies returns a slice of sub-items of the "latest" available version of the item, as opposed to the version that is actually installed. The information comes from the index.

func (Item) MarshalJSON added in v1.6.0 

func (i Item) MarshalJSON() ([]byte, error)

MarshalJSON is used to prepare the output for "cscli ... inspect -o json". It must not use a pointer receiver.

func (Item) MarshalYAML added in v1.6.0 

func (i Item) MarshalYAML() (interface{}, error)

MarshalYAML is used to prepare the output for "cscli ... inspect -o raw". It must not use a pointer receiver.

func (*Item) SafeToRemoveDeps added in v1.6.5 

func (i *Item) SafeToRemoveDeps() ([]*Item, error)

SafeToRemoveDeps returns a slice of dependencies that can be safely removed when this item is removed. The returned slice can contain items that are not installed, or not downloaded.

type ItemState added in v1.6.0 

type ItemState struct {
	LocalPath    string `json:"local_path,omitempty" yaml:"local_path,omitempty"`
	LocalVersion string `json:"local_version,omitempty" yaml:"local_version,omitempty"`
	LocalHash    string `json:"local_hash,omitempty" yaml:"local_hash,omitempty"`
	Installed    bool   `json:"installed"`

	Downloaded           bool     `json:"downloaded"`
	UpToDate             bool     `json:"up_to_date"`
	Tainted              bool     `json:"tainted"`
	TaintedBy            []string `json:"tainted_by,omitempty" yaml:"tainted_by,omitempty"`
	BelongsToCollections []string `json:"belongs_to_collections,omitempty" yaml:"belongs_to_collections,omitempty"`
	// contains filtered or unexported fields
}

ItemState is used to keep the local state (i.e. at runtime) of an item. This data is not stored in the index, but is displayed with "cscli ... inspect".

func (*ItemState) Emoji added in v1.6.0 

func (s *ItemState) Emoji() string

Emoji returns the status of the item as an emoji (eg. emoji.Warning).

func (*ItemState) IsLocal added in v1.6.0 

func (s *ItemState) IsLocal() bool

IsLocal returns true if the item has been create by a user (not downloaded from the hub).

func (*ItemState) Text added in v1.6.0 

func (s *ItemState) Text() string

Text returns the status of the item as a string (eg. "enabled,update-available").

type ItemVersion 

type ItemVersion struct {
	Digest     string `json:"digest,omitempty" yaml:"digest,omitempty"`
	Deprecated bool   `json:"deprecated,omitempty" yaml:"deprecated,omitempty"`
}

ItemVersion is used to detect the version of a given item by comparing the hash of each version to the local file. If the item does not match any known version, it is considered tainted (modified).

type NotFoundError added in v1.6.5 

type NotFoundError = downloader.NotFoundError

no need to import the lib package to use this

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.