cwhub

package
v1.6.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 11, 2024 License: MIT Imports: 24 Imported by: 1

Documentation

Overview

Package cwhub is responsible for installing and upgrading the local hub files for CrowdSec.

Definitions

  • A hub ITEM is a file that defines a parser, a scenario, a collection... in the case of a collection, it has dependencies on other hub items.
  • The hub INDEX is a JSON file that contains a tree of available hub items.
  • A REMOTE HUB is an HTTP server that hosts the hub index and the hub items. It can serve from several branches, usually linked to the CrowdSec version.
  • A LOCAL HUB is a directory that contains a copy of the hub index and the downloaded hub items.

Once downloaded, hub items can be installed by linking to them from the configuration directory. If an item is present in the configuration directory but it's not a link to the local hub, it is considered as a LOCAL ITEM and won't be removed or upgraded.

Directory Structure

A typical directory layout is the following:

For the local hub (HubDir = /etc/crowdsec/hub):

  • /etc/crowdsec/hub/.index.json
  • /etc/crowdsec/hub/parsers/{stage}/{author}/{parser-name}.yaml
  • /etc/crowdsec/hub/scenarios/{author}/{scenario-name}.yaml

For the configuration directory (InstallDir = /etc/crowdsec):

  • /etc/crowdsec/parsers/{stage}/{parser-name.yaml} -> /etc/crowdsec/hub/parsers/{stage}/{author}/{parser-name}.yaml
  • /etc/crowdsec/scenarios/{scenario-name.yaml} -> /etc/crowdsec/hub/scenarios/{author}/{scenario-name}.yaml
  • /etc/crowdsec/scenarios/local-scenario.yaml

Note that installed items are not grouped by author, this may change in the future if we want to support items with the same name from different authors.

Only parsers and postoverflows have the concept of stage.

Additionally, an item can reference a DATA SET that is installed in a different location than the item itself. These files are stored in the data directory (InstallDataDir = /var/lib/crowdsec/data).

  • /var/lib/crowdsec/data/http_path_traversal.txt
  • /var/lib/crowdsec/data/jira_cve_2021-26086.txt
  • /var/lib/crowdsec/data/log4j2_cve_2021_44228.txt
  • /var/lib/crowdsec/data/sensitive_data.txt

Using the package

The main entry point is the Hub struct. You can create a new instance with NewHub(). This constructor takes three parameters, but only the LOCAL HUB configuration is required: 

import (
	"fmt"
	"github.com/crowdsecurity/crowdsec/pkg/csconfig"
	"github.com/crowdsecurity/crowdsec/pkg/cwhub"
)

localHub := csconfig.LocalHubCfg{
	HubIndexFile:	"/etc/crowdsec/hub/.index.json",
	HubDir:		"/etc/crowdsec/hub",
	InstallDir:	"/etc/crowdsec",
	InstallDataDir: "/var/lib/crowdsec/data",
}
hub, err := cwhub.NewHub(localHub, nil, false)
if err != nil {
	return fmt.Errorf("unable to initialize hub: %w", err)
}

Now you can use the hub to access the existing items: 

// list all the parsers
for _, parser := range hub.GetItemMap(cwhub.PARSERS) {
	fmt.Printf("parser: %s\n", parser.Name)
}

// retrieve a specific collection
coll := hub.GetItem(cwhub.COLLECTIONS, "crowdsecurity/linux")
if coll == nil {
	return fmt.Errorf("collection not found")
}

You can also install items if they have already been downloaded: 

// install a parser
force := false
downloadOnly := false
err := parser.Install(force, downloadOnly)
if err != nil {
	return fmt.Errorf("unable to install parser: %w", err)
}

As soon as you try to install an item that is not downloaded or is not up-to-date (meaning its computed hash does not correspond to the latest version available in the index), a download will be attempted and you'll get the error "remote hub configuration is not provided".

To provide the remote hub configuration, use the second parameter of NewHub(): 

remoteHub := cwhub.RemoteHubCfg{
	URLTemplate: "https://hub-cdn.crowdsec.net/%s/%s",
	Branch: "master",
	IndexPath: ".index.json",
}
updateIndex := false
hub, err := cwhub.NewHub(localHub, remoteHub, updateIndex)
if err != nil {
	return fmt.Errorf("unable to initialize hub: %w", err)
}

The URLTemplate is a string that will be used to build the URL of the remote hub. It must contain two placeholders: the branch and the file path (it will be an index or an item).

Setting the third parameter to true will download the latest version of the index, if available on the specified branch. There is no exported method to update the index once the hub struct is created.

Index

Constants

View Source 
const (
	// managed item types.
	COLLECTIONS    = "collections"
	PARSERS        = "parsers"
	POSTOVERFLOWS  = "postoverflows"
	SCENARIOS      = "scenarios"
	CONTEXTS       = "contexts"
	APPSEC_CONFIGS = "appsec-configs"
	APPSEC_RULES   = "appsec-rules"
)

Variables

View Source 
var (
	// ErrNilRemoteHub is returned when trying to download with a local-only configuration.
	ErrNilRemoteHub = errors.New("remote hub configuration is not provided. Please report this issue to the developers")
)
View Source 
var (
	// The order is important, as it is used to range over sub-items in collections.
	ItemTypes = []string{PARSERS, POSTOVERFLOWS, SCENARIOS, CONTEXTS, APPSEC_CONFIGS, APPSEC_RULES, COLLECTIONS}
)

Functions

func SortItemSlice added in v1.6.0 

func SortItemSlice(items []*Item)

SortItemSlice sorts a slice of items by name, case insensitive.

Types

type DataSet added in v1.5.3 

type DataSet struct {
	Data []types.DataSource `yaml:"data,omitempty"`
}

The DataSet is a list of data sources required by an item (built from the data: section in the yaml).

type Hub added in v1.6.0 

type Hub struct {
	Warnings []string // Warnings encountered during sync
	// contains filtered or unexported fields
}

Hub is the main structure for the package.

func NewHub added in v1.6.0 

func NewHub(local *csconfig.LocalHubCfg, remote *RemoteHubCfg, updateIndex bool, logger *logrus.Logger) (*Hub, error)

NewHub returns a new Hub instance with local and (optionally) remote configuration, and syncs the local state. If updateIndex is true, the local index file is updated from the remote before reading the state of the items. All download operations (including updateIndex) return ErrNilRemoteHub if the remote configuration is not set.

func (*Hub) GetAllItems added in v1.6.0 

func (h *Hub) GetAllItems(itemType string) ([]*Item, error)

GetAllItems returns a slice of all the items of a given type, installed or not.

func (*Hub) GetDataDir added in v1.6.0 

func (h *Hub) GetDataDir() string

GetDataDir returns the data directory, where data sets are installed.

func (*Hub) GetInstalledItemNames added in v1.6.0 

func (h *Hub) GetInstalledItemNames(itemType string) ([]string, error)

GetInstalledItemNames returns the names of the installed items of a given type.

func (*Hub) GetInstalledItems added in v1.6.0 

func (h *Hub) GetInstalledItems(itemType string) ([]*Item, error)

GetInstalledItems returns a slice of the installed items of a given type.

func (*Hub) GetItem added in v1.6.0 

func (h *Hub) GetItem(itemType string, itemName string) *Item

GetItem returns an item from hub based on its type and full name (author/name).

func (*Hub) GetItemByPath added in v1.6.0 

func (h *Hub) GetItemByPath(itemType string, itemPath string) (*Item, error)

GetItemByPath retrieves an item from the hub index based on its local path.

func (*Hub) GetItemFQ added in v1.6.0 

func (h *Hub) GetItemFQ(itemFQName string) (*Item, error)

GetItemFQ returns an item from hub based on its type and name (type:author/name).

func (*Hub) GetItemMap added in v1.6.0 

func (h *Hub) GetItemMap(itemType string) map[string]*Item

GetItemMap returns the map of items for a given type.

func (*Hub) GetItemNames added in v1.6.0 

func (h *Hub) GetItemNames(itemType string) []string

GetItemNames returns a slice of (full) item names for a given type (eg. for collections: crowdsecurity/apache2 crowdsecurity/nginx).

func (*Hub) ItemStats added in v1.6.0 

func (h *Hub) ItemStats() []string

ItemStats returns total counts of the hub items, including local and tainted.

type HubItems added in v1.6.0 

type HubItems map[string]map[string]*Item

type IndexNotFoundError added in v1.6.0 

type IndexNotFoundError struct {
	URL    string
	Branch string
}

IndexNotFoundError is returned when the remote hub index is not found.

func (IndexNotFoundError) Error added in v1.6.0 

func (e IndexNotFoundError) Error() string

type Item 

type Item struct {
	State ItemState `json:"-" yaml:"-"` // local state, not stored in the index

	Type        string   `json:"type,omitempty" yaml:"type,omitempty"`           // one of the ItemTypes
	Stage       string   `json:"stage,omitempty" yaml:"stage,omitempty"`         // Stage for parser|postoverflow: s00-raw/s01-...
	Name        string   `json:"name,omitempty" yaml:"name,omitempty"`           // usually "author/name"
	FileName    string   `json:"file_name,omitempty" yaml:"file_name,omitempty"` // eg. apache2-logs.yaml
	Description string   `json:"description,omitempty" yaml:"description,omitempty"`
	Author      string   `json:"author,omitempty" yaml:"author,omitempty"`
	References  []string `json:"references,omitempty" yaml:"references,omitempty"`

	RemotePath string                 `json:"path,omitempty" yaml:"path,omitempty"`       // path relative to the base URL eg. /parsers/stage/author/file.yaml
	Version    string                 `json:"version,omitempty" yaml:"version,omitempty"` // the last available version
	Versions   map[string]ItemVersion `json:"versions,omitempty"  yaml:"-"`               // all the known versions

	// if it's a collection, it can have sub items
	Parsers       []string `json:"parsers,omitempty" yaml:"parsers,omitempty"`
	PostOverflows []string `json:"postoverflows,omitempty" yaml:"postoverflows,omitempty"`
	Scenarios     []string `json:"scenarios,omitempty" yaml:"scenarios,omitempty"`
	Collections   []string `json:"collections,omitempty" yaml:"collections,omitempty"`
	Contexts      []string `json:"contexts,omitempty" yaml:"contexts,omitempty"`
	AppsecConfigs []string `json:"appsec-configs,omitempty"   yaml:"appsec-configs,omitempty"`
	AppsecRules   []string `json:"appsec-rules,omitempty"   yaml:"appsec-rules,omitempty"`
	// contains filtered or unexported fields
}

Item is created from an index file and enriched with local info.

func (*Item) Ancestors added in v1.6.0 

func (i *Item) Ancestors() []*Item

Ancestors returns a slice of items (typically collections) that have this item as a direct or indirect dependency.

func (*Item) DownloadDataIfNeeded added in v1.6.0 

func (i *Item) DownloadDataIfNeeded(force bool) error

DownloadDataIfNeeded downloads the data set for the item.

func (*Item) FQName added in v1.6.0 

func (i *Item) FQName() string

FQName returns the fully qualified name of the item (ie. parsers:crowdsecurity/apache2-logs).

func (*Item) FetchLatest added in v1.6.0 

func (i *Item) FetchLatest() ([]byte, string, error)

FetchLatest downloads the latest item from the hub, verifies the hash and returns the content and the used url.

func (*Item) HasSubItems added in v1.6.0 

func (i *Item) HasSubItems() bool

HasSubItems returns true if items of this type can have sub-items. Currently only collections.

func (*Item) Install added in v1.6.0 

func (i *Item) Install(force bool, downloadOnly bool) error

Install installs the item from the hub, downloading it if needed.

func (Item) MarshalJSON added in v1.6.0 

func (i Item) MarshalJSON() ([]byte, error)

MarshalJSON is used to prepare the output for "cscli ... inspect -o json". It must not use a pointer receiver.

func (Item) MarshalYAML added in v1.6.0 

func (i Item) MarshalYAML() (interface{}, error)

MarshalYAML is used to prepare the output for "cscli ... inspect -o raw". It must not use a pointer receiver.

func (*Item) Remove added in v1.6.0 

func (i *Item) Remove(purge bool, force bool) (bool, error)

Remove disables the item, optionally removing the downloaded content.

func (*Item) SubItems added in v1.6.0 

func (i *Item) SubItems() []*Item

SubItems returns a slice of sub-items, excluding the ones that were not found.

func (*Item) Upgrade added in v1.6.0 

func (i *Item) Upgrade(force bool) (bool, error)

Upgrade downloads and applies the last version of the item from the hub.

type ItemState added in v1.6.0 

type ItemState struct {
	LocalPath            string   `json:"local_path,omitempty" yaml:"local_path,omitempty"`
	LocalVersion         string   `json:"local_version,omitempty" yaml:"local_version,omitempty"`
	LocalHash            string   `json:"local_hash,omitempty" yaml:"local_hash,omitempty"`
	Installed            bool     `json:"installed"`
	Downloaded           bool     `json:"downloaded"`
	UpToDate             bool     `json:"up_to_date"`
	Tainted              bool     `json:"tainted"`
	TaintedBy            []string `json:"tainted_by,omitempty" yaml:"tainted_by,omitempty"`
	BelongsToCollections []string `json:"belongs_to_collections,omitempty" yaml:"belongs_to_collections,omitempty"`
}

ItemState is used to keep the local state (i.e. at runtime) of an item. This data is not stored in the index, but is displayed with "cscli ... inspect".

func (*ItemState) Emoji added in v1.6.0 

func (s *ItemState) Emoji() string

Emoji returns the status of the item as an emoji (eg. emoji.Warning).

func (*ItemState) IsLocal added in v1.6.0 

func (s *ItemState) IsLocal() bool

IsLocal returns true if the item has been create by a user (not downloaded from the hub).

func (*ItemState) Text added in v1.6.0 

func (s *ItemState) Text() string

Text returns the status of the item as a string (eg. "enabled,update-available").

type ItemVersion 

type ItemVersion struct {
	Digest     string `json:"digest,omitempty" yaml:"digest,omitempty"`
	Deprecated bool   `json:"deprecated,omitempty" yaml:"deprecated,omitempty"`
}

ItemVersion is used to detect the version of a given item by comparing the hash of each version to the local file. If the item does not match any known version, it is considered tainted (modified).

type RemoteHubCfg added in v1.6.0 

type RemoteHubCfg struct {
	Branch      string
	URLTemplate string
	IndexPath   string
}

RemoteHubCfg is used to retrieve index and items from the remote hub.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.