cwhub

package
v1.6.4-rc3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 13, 2024 License: MIT Imports: 26 Imported by: 1

Documentation

Overview

Package cwhub is responsible for installing and upgrading the local hub files for CrowdSec.

Definitions

  • A hub ITEM is a file that defines a parser, a scenario, a collection... in the case of a collection, it has dependencies on other hub items.
  • The hub INDEX is a JSON file that contains a tree of available hub items.
  • A REMOTE HUB is an HTTP server that hosts the hub index and the hub items. It can serve from several branches, usually linked to the CrowdSec version.
  • A LOCAL HUB is a directory that contains a copy of the hub index and the downloaded hub items.

Once downloaded, hub items can be installed by linking to them from the configuration directory. If an item is present in the configuration directory but it's not a link to the local hub, it is considered as a LOCAL ITEM and won't be removed or upgraded.

Directory Structure

A typical directory layout is the following:

For the local hub (HubDir = /etc/crowdsec/hub):

  • /etc/crowdsec/hub/.index.json
  • /etc/crowdsec/hub/parsers/{stage}/{author}/{parser-name}.yaml
  • /etc/crowdsec/hub/scenarios/{author}/{scenario-name}.yaml

For the configuration directory (InstallDir = /etc/crowdsec):

  • /etc/crowdsec/parsers/{stage}/{parser-name.yaml} -> /etc/crowdsec/hub/parsers/{stage}/{author}/{parser-name}.yaml
  • /etc/crowdsec/scenarios/{scenario-name.yaml} -> /etc/crowdsec/hub/scenarios/{author}/{scenario-name}.yaml
  • /etc/crowdsec/scenarios/local-scenario.yaml

Note that installed items are not grouped by author, this may change in the future if we want to support items with the same name from different authors.

Only parsers and postoverflows have the concept of stage.

Additionally, an item can reference a DATA SET that is installed in a different location than the item itself. These files are stored in the data directory (InstallDataDir = /var/lib/crowdsec/data).

  • /var/lib/crowdsec/data/http_path_traversal.txt
  • /var/lib/crowdsec/data/jira_cve_2021-26086.txt
  • /var/lib/crowdsec/data/log4j2_cve_2021_44228.txt
  • /var/lib/crowdsec/data/sensitive_data.txt

Using the package

The main entry point is the Hub struct. You can create a new instance with NewHub(). This constructor takes three parameters, but only the LOCAL HUB configuration is required: 

import (
	"fmt"
	"github.com/crowdsecurity/crowdsec/pkg/csconfig"
	"github.com/crowdsecurity/crowdsec/pkg/cwhub"
)

localHub := csconfig.LocalHubCfg{
	HubIndexFile:	"/etc/crowdsec/hub/.index.json",
	HubDir:		"/etc/crowdsec/hub",
	InstallDir:	"/etc/crowdsec",
	InstallDataDir: "/var/lib/crowdsec/data",
}

hub, err := cwhub.NewHub(localHub, nil, logger)
if err != nil {
	return fmt.Errorf("unable to initialize hub: %w", err)
}

If the logger is nil, the item-by-item messages will be discarded, including warnings. After configuring the hub, you must sync its state with items on disk. 

err := hub.Load()
if err != nil {
	return fmt.Errorf("unable to load hub: %w", err)
}

Now you can use the hub object to access the existing items: 

// list all the parsers
for _, parser := range hub.GetItemsByType(cwhub.PARSERS, false) {
	fmt.Printf("parser: %s\n", parser.Name)
}

// retrieve a specific collection
coll := hub.GetItem(cwhub.COLLECTIONS, "crowdsecurity/linux")
if coll == nil {
	return fmt.Errorf("collection not found")
}

You can also install items if they have already been downloaded: 

// install a parser
force := false
downloadOnly := false
err := parser.Install(force, downloadOnly)
if err != nil {
	return fmt.Errorf("unable to install parser: %w", err)
}

As soon as you try to install an item that is not downloaded or is not up-to-date (meaning its computed hash does not correspond to the latest version available in the index), a download will be attempted and you'll get the error "remote hub configuration is not provided".

To provide the remote hub configuration, use the second parameter of NewHub(): 

remoteHub := cwhub.RemoteHubCfg{
	URLTemplate: "https://cdn-hub.crowdsec.net/crowdsecurity/%s/%s",
	Branch: "master",
	IndexPath: ".index.json",
}

hub, err := cwhub.NewHub(localHub, remoteHub, logger)
if err != nil {
	return fmt.Errorf("unable to initialize hub: %w", err)
}

The URLTemplate is a string that will be used to build the URL of the remote hub. It must contain two placeholders: the branch and the file path (it will be an index or an item).

Before calling hub.Load(), you can update the index file by calling the Update() method: 

err := hub.Update(context.Background())
if err != nil {
	return fmt.Errorf("unable to update hub index: %w", err)
}

Note that the command will fail if the hub has already been synced. If you want to do it (ex. after a configuration change the application is notified with SIGHUP) you have to instantiate a new hub object and dispose of the old one.

Index

Constants

View Source 
const (
	// managed item types.
	COLLECTIONS    = "collections"
	PARSERS        = "parsers"
	POSTOVERFLOWS  = "postoverflows"
	SCENARIOS      = "scenarios"
	CONTEXTS       = "contexts"
	APPSEC_CONFIGS = "appsec-configs"
	APPSEC_RULES   = "appsec-rules"
)

Variables

View Source 
var ErrNilRemoteHub = errors.New("remote hub configuration is not provided. Please report this issue to the developers")

ErrNilRemoteHub is returned when trying to download with a local-only configuration.

View Source 
var ItemTypes = []string{PARSERS, POSTOVERFLOWS, SCENARIOS, CONTEXTS, APPSEC_CONFIGS, APPSEC_RULES, COLLECTIONS}

The order is important, as it is used to range over sub-items in collections.

Functions

This section is empty.

Types

type DataSet added in v1.5.3 

type DataSet struct {
	Data []types.DataSource `yaml:"data,omitempty"`
}

The DataSet is a list of data sources required by an item (built from the data: section in the yaml).

type Hub added in v1.6.0 

type Hub struct {
	Warnings []string // Warnings encountered during sync
	// contains filtered or unexported fields
}

Hub is the main structure for the package.

func NewHub added in v1.6.0 

func NewHub(local *csconfig.LocalHubCfg, remote *RemoteHubCfg, logger *logrus.Logger) (*Hub, error)

NewHub returns a new Hub instance with local and (optionally) remote configuration. The hub is not synced automatically. Load() must be called to read the index, sync the local state, and check for unmanaged items. All download operations (including updateIndex) return ErrNilRemoteHub if the remote configuration is not set.

func (*Hub) GetDataDir added in v1.6.0 

func (h *Hub) GetDataDir() string

GetDataDir returns the data directory, where data sets are installed.

func (*Hub) GetInstalledByType added in v1.6.3 

func (h *Hub) GetInstalledByType(itemType string, sorted bool) []*Item

GetInstalledByType returns a slice of all the installed items of a given type, optionally sorted by case-insensitive name. A non-existent type will silently return an empty slice.

func (*Hub) GetInstalledListForAPI added in v1.6.3 

func (h *Hub) GetInstalledListForAPI() []string

GetInstalledListForAPI returns a slice of names of all the installed scenarios and appsec-rules. The returned list is sorted by type (scenarios first) and case-insensitive name.

func (*Hub) GetItem added in v1.6.0 

func (h *Hub) GetItem(itemType string, itemName string) *Item

GetItem returns an item from hub based on its type and full name (author/name).

func (*Hub) GetItemByPath added in v1.6.0 

func (h *Hub) GetItemByPath(itemPath string) *Item

GetItemByPath returns an item from hub based on its (absolute) local path.

func (*Hub) GetItemFQ added in v1.6.0 

func (h *Hub) GetItemFQ(itemFQName string) (*Item, error)

GetItemFQ returns an item from hub based on its type and name (type:author/name).

func (*Hub) GetItemMap added in v1.6.0 

func (h *Hub) GetItemMap(itemType string) map[string]*Item

GetItemMap returns the map of items for a given type.

func (*Hub) GetItemsByType added in v1.6.2 

func (h *Hub) GetItemsByType(itemType string, sorted bool) []*Item

GetItemsByType returns a slice of all the items of a given type, installed or not, optionally sorted by case-insensitive name. A non-existent type will silently return an empty slice.

func (*Hub) ItemStats added in v1.6.0 

func (h *Hub) ItemStats() []string

ItemStats returns total counts of the hub items, including local and tainted.

func (*Hub) Load added in v1.6.3 

func (h *Hub) Load() error

Load reads the state of the items on disk.

func (*Hub) Update added in v1.6.3 

func (h *Hub) Update(ctx context.Context) error

Update downloads the latest version of the index and writes it to disk if it changed. It cannot be called after Load() unless the hub is completely empty.

type HubItems added in v1.6.0 

type HubItems map[string]map[string]*Item

type IndexNotFoundError added in v1.6.0 

type IndexNotFoundError struct {
	URL    string
	Branch string
}

IndexNotFoundError is returned when the remote hub index is not found.

func (IndexNotFoundError) Error added in v1.6.0 

func (e IndexNotFoundError) Error() string

type Item 

type Item struct {
	State ItemState `json:"-" yaml:"-"` // local state, not stored in the index

	Type        string   `json:"type,omitempty" yaml:"type,omitempty"`           // one of the ItemTypes
	Stage       string   `json:"stage,omitempty" yaml:"stage,omitempty"`         // Stage for parser|postoverflow: s00-raw/s01-...
	Name        string   `json:"name,omitempty" yaml:"name,omitempty"`           // usually "author/name"
	FileName    string   `json:"file_name,omitempty" yaml:"file_name,omitempty"` // eg. apache2-logs.yaml
	Description string   `json:"description,omitempty" yaml:"description,omitempty"`
	Content     string   `json:"content,omitempty" yaml:"-"`
	Author      string   `json:"author,omitempty" yaml:"author,omitempty"`
	References  []string `json:"references,omitempty" yaml:"references,omitempty"`

	RemotePath string                 `json:"path,omitempty" yaml:"path,omitempty"`       // path relative to the base URL eg. /parsers/stage/author/file.yaml
	Version    string                 `json:"version,omitempty" yaml:"version,omitempty"` // the last available version
	Versions   map[string]ItemVersion `json:"versions,omitempty"  yaml:"-"`               // all the known versions

	// if it's a collection, it can have sub items
	Parsers       []string `json:"parsers,omitempty" yaml:"parsers,omitempty"`
	PostOverflows []string `json:"postoverflows,omitempty" yaml:"postoverflows,omitempty"`
	Scenarios     []string `json:"scenarios,omitempty" yaml:"scenarios,omitempty"`
	Collections   []string `json:"collections,omitempty" yaml:"collections,omitempty"`
	Contexts      []string `json:"contexts,omitempty" yaml:"contexts,omitempty"`
	AppsecConfigs []string `json:"appsec-configs,omitempty"   yaml:"appsec-configs,omitempty"`
	AppsecRules   []string `json:"appsec-rules,omitempty"   yaml:"appsec-rules,omitempty"`
	// contains filtered or unexported fields
}

Item is created from an index file and enriched with local info.

func (*Item) Ancestors added in v1.6.0 

func (i *Item) Ancestors() []*Item

Ancestors returns a slice of items (typically collections) that have this item as a direct or indirect dependency.

func (*Item) DownloadDataIfNeeded added in v1.6.0 

func (i *Item) DownloadDataIfNeeded(ctx context.Context, force bool) error

DownloadDataIfNeeded downloads the data set for the item.

func (*Item) FQName added in v1.6.0 

func (i *Item) FQName() string

FQName returns the fully qualified name of the item (ie. parsers:crowdsecurity/apache2-logs).

func (*Item) FetchContentTo added in v1.6.2 

func (i *Item) FetchContentTo(ctx context.Context, destPath string) (bool, string, error)

FetchContentTo downloads the last version of the item's YAML file to the specified path.

func (*Item) HasSubItems added in v1.6.0 

func (i *Item) HasSubItems() bool

HasSubItems returns true if items of this type can have sub-items. Currently only collections.

func (*Item) Install added in v1.6.0 

func (i *Item) Install(ctx context.Context, force bool, downloadOnly bool) error

Install installs the item from the hub, downloading it if needed.

func (Item) MarshalJSON added in v1.6.0 

func (i Item) MarshalJSON() ([]byte, error)

MarshalJSON is used to prepare the output for "cscli ... inspect -o json". It must not use a pointer receiver.

func (Item) MarshalYAML added in v1.6.0 

func (i Item) MarshalYAML() (interface{}, error)

MarshalYAML is used to prepare the output for "cscli ... inspect -o raw". It must not use a pointer receiver.

func (*Item) Remove added in v1.6.0 

func (i *Item) Remove(purge bool, force bool) (bool, error)

Remove disables the item, optionally removing the downloaded content.

func (*Item) SubItems added in v1.6.0 

func (i *Item) SubItems() []*Item

SubItems returns a slice of sub-items, excluding the ones that were not found.

func (*Item) Upgrade added in v1.6.0 

func (i *Item) Upgrade(ctx context.Context, force bool) (bool, error)

Upgrade downloads and applies the last version of the item from the hub.

type ItemState added in v1.6.0 

type ItemState struct {
	LocalPath            string   `json:"local_path,omitempty" yaml:"local_path,omitempty"`
	LocalVersion         string   `json:"local_version,omitempty" yaml:"local_version,omitempty"`
	LocalHash            string   `json:"local_hash,omitempty" yaml:"local_hash,omitempty"`
	Installed            bool     `json:"installed"`
	Downloaded           bool     `json:"downloaded"`
	UpToDate             bool     `json:"up_to_date"`
	Tainted              bool     `json:"tainted"`
	TaintedBy            []string `json:"tainted_by,omitempty" yaml:"tainted_by,omitempty"`
	BelongsToCollections []string `json:"belongs_to_collections,omitempty" yaml:"belongs_to_collections,omitempty"`
}

ItemState is used to keep the local state (i.e. at runtime) of an item. This data is not stored in the index, but is displayed with "cscli ... inspect".

func (*ItemState) Emoji added in v1.6.0 

func (s *ItemState) Emoji() string

Emoji returns the status of the item as an emoji (eg. emoji.Warning).

func (*ItemState) IsLocal added in v1.6.0 

func (s *ItemState) IsLocal() bool

IsLocal returns true if the item has been create by a user (not downloaded from the hub).

func (*ItemState) Text added in v1.6.0 

func (s *ItemState) Text() string

Text returns the status of the item as a string (eg. "enabled,update-available").

type ItemVersion 

type ItemVersion struct {
	Digest     string `json:"digest,omitempty" yaml:"digest,omitempty"`
	Deprecated bool   `json:"deprecated,omitempty" yaml:"deprecated,omitempty"`
}

ItemVersion is used to detect the version of a given item by comparing the hash of each version to the local file. If the item does not match any known version, it is considered tainted (modified).

type RemoteHubCfg added in v1.6.0 

type RemoteHubCfg struct {
	Branch           string
	URLTemplate      string
	IndexPath        string
	EmbedItemContent bool
}

RemoteHubCfg is used to retrieve index and items from the remote hub.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.