db

package
v0.0.0-...-b42f91c Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 28, 2024 License: MIT Imports: 13 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// DefaultTimeout is the default timeout for each db operation
	DefaultTimeout = 10 * time.Second

	// GetSizeUnlimited is the constant to indicate that there's no size limit for db return record size.
	GetSizeUnlimited = -1
)
View Source 
const (
	MongoDBID = "mongo"

	// DBNvd is the database name of nvd service
	DBNvd = "nvd"

	// CltCve and CltCpe are collection name in 'DBNvd'
	CltCve = "cve"
	CltCpe = "cpe"

	MongoTLSTemplate = "mongodb://%s:%s@%s/nvd?tls=true&replicaSet=rs0&retryWrites=false"
)

Variables

View Source 
var (
	// ErrNotFound is general not found error for all structs that implement Nvd[Cve|Cpe]DB interface
	// It is expected to return this error if the input is not found in the table
	ErrNotFound = errors.New("not found")

	// ErrPageEnd is the error to indicate that the start index is larger than the total number of entries
	ErrPageEnd = errors.New("page end")
)
View Source 
var (
	// DefaultGetMongoURI is the url template to connect to local testing mongo db,
	// query string of URI might be different when connecting to db with different authenication method
	DefaultGetMongoURI = func(user, pwd, endpoint string) string {
		return fmt.Sprintf("mongodb://%s:%s@%s/nvd?authSource=admin", user, pwd, endpoint)
	}
)

Functions

func IsNotFound 

func IsNotFound(err error) bool

IsNotFound checks whether the error belongs to data not found in database

func IsPageEnd 

func IsPageEnd(err error) bool

IsPageEnd checks whether the error belongs to page end

func IsUnexpectedError 

func IsUnexpectedError(err error) bool

IsUnexpectedError checkes whether the error belongs to unexpected error in database

Types

type Config 

type Config struct {
	Timeout time.Duration
	Batch   int
	Logger  *zap.Logger
}

Config is the general(default) config to initialize the collection or control the db logic.

func NewConfig 

func NewConfig() Config

NewConfig initializes config with default value

func (Config) Validate 

func (cfg Config) Validate() error

Validate validates config

type DB 

type DB interface {
	ID() string
	Init(ctx context.Context) error
	Connect(ctx context.Context, uri string) error
	Disconnect(ctx context.Context) error
	IsConnected(ctx context.Context) error
}

DB is the interface to record information and handle general operations such as connect and disconnect to database.

type Keyword 

type Keyword struct {
	// Val is the content to match
	Val string

	// ExactMatch decides whether matching all the words includes space or not.
	// E.g.,
	// * Val="Hello World" and `ExactMatch=true`: matches with "Hello World"
	// * Val="Hello World" and `ExactMatch=false`: matches with "Hello" AND "World"
	ExactMatch bool
}

Keyword defines how a keyword matches with the description of CVE or CPE

type Mongo 

type Mongo struct {
	// contains filtered or unexported fields
}

Mongo is the db client to handle mongo db operations

func NewMongo 

func NewMongo(user, pwd, endpoint string, opts ...MongoOptions) (*Mongo, error)

NewMongo initializes db client to interact with mongo db

func (*Mongo) Connect 

func (mdb *Mongo) Connect(ctx context.Context, uri string) (err error)

Connect connects to mongo db with provided uri and options

func (*Mongo) Disconnect 

func (mdb *Mongo) Disconnect(ctx context.Context) error

Disconnect closes connection to database

func (Mongo) GetCPEByKeyword 

func (mdb Mongo) GetCPEByKeyword(ctx context.Context, keyword Keyword, opts ...QueryOptions) (*Result[schema.Cpe], error)

GetCPEByKeyword searchs with text index in 'nvd.cpe' ('titles.title' and 'refs.ref') to find match keyword It is the same as using mongo query 

db.cpe.find({ $text: { $search: "<keyword>" } });

Usage: 

// find cpes that contains "Hello" AND "World" in Titles or References
db.GetCPEByKeyword(ctx, Keyword{Val: "Hello World", ExactMatch: false})

// find cpes that contains "Hello World" in Titles or References
db.GetCPEByKeyword(ctx, Keyword{Val: "Hello World", ExactMatch: true})

func (Mongo) GetCPEByMatchString 

func (mdb Mongo) GetCPEByMatchString(ctx context.Context, cpeNameMatchString string, opts ...QueryOptions) (*Result[schema.Cpe], error)

GetCPEByMatchString get CPEs by CPEMatchString which is prefix of full CPE 2.3 Name. 

E.g., cpe:2.3:*:Microsoft, cpe:2.3:o:microsoft:windows_10:1511

It matches the value in db based on parsed CPE (schema.CPEParsed) which store each part in different fields. For the incoming cpeNameMatchString, we try to match the parts that contains meaningful values ('*' and '-' is excluded), and returns with CPEs that matches all of the values. 

E.g. 1, cpeNameMatchString=cpe:2.3:*:Microsoft, matches CPEs with vendor=Microsoft
E.g. 2, cpeNameMatchString=cpe:2.3:o:microsoft:windows_10, matches CPEs with part=o, vendor=microsoft and product=windows_10

func (Mongo) GetCPEByName 

func (mdb Mongo) GetCPEByName(ctx context.Context, cpeName string) (*schema.Cpe, error)

GetCPEByName get CPE by CPEName (E.g., `cpe:2.3:a:rusqlite_project:rusqlite:0.18.0:*:*:*:*:*:*:*`)

  • When CPE is found in `nvd.cpe`: return the detail information of cpe
  • When CPE is not found in `nvd.cpe`: return ErrNotFound

func (Mongo) GetCVEByCPE 

func (mdb Mongo) GetCVEByCPE(ctx context.Context, cpeName string, opts ...QueryOptions) (*Result[schema.Cve], error)

GetCVEByCPE returns CVE with cpe name To query with $regex to the index 'nodes.cpeMatch.criteria' which has relatively poor performance 

{
    "configurations": {
        "$elemMatch": {
            "nodes.cpeMatch.criteria": {
                "$regex": "^<cpe name product prefix>"
            }
        }
    }
}

To leverage the index 'nodes.cpeMatch.criteriaProductPrefix' generated from us, the query becomes 

{
    "configurations": {
        "$elemMatch": {
            "nodes.cpeMatch.criteriaProductPrefix": {
                "$eq": "<cpe name product prefix>"
            }
        }
    }
}

func (Mongo) GetCVEByID 

func (mdb Mongo) GetCVEByID(ctx context.Context, cveId string) (*schema.Cve, error)

GetCVEByID get CVE by CVEID (E.g., CVE-2001-0131)

  • When CVE is found in `nvd.cve`: return the detail information of cve
  • When CVE is not found in `nvd.cve`: return ErrNotFound

func (Mongo) GetCVEByKeyword 

func (mdb Mongo) GetCVEByKeyword(ctx context.Context, keyword Keyword, opts ...QueryOptions) (*Result[schema.Cve], error)

GetCVEByKeyword searchs with text index in 'nvd.cve' ('descriptions.value') to find match keyword. It is the same as using mongo query 

db.cve.find({ $text: { $search: "<keyword>" } });

Usage: 

// find cves that contains "Hello" AND "World" in Descriptions
db.GetCVEByKeyword(ctx, Keyword{Val: "Hello World", ExactMatch: false})

// find cves that contains "Hello World" in Descriptions
db.GetCVEByKeyword(ctx, Keyword{Val: "Hello World", ExactMatch: true})

func (Mongo) ID 

func (Mongo) ID() string

func (Mongo) Init 

func (mdb Mongo) Init(ctx context.Context) error

Init creates collections and TTL indexes and text indexes for NVD service. It should be only used when initializing environment.

The same mongo command as below: 

$ use nvd;
$ db.cve.createIndex({ "cveId": 1 }, { unique: true, name: "cveId" });
$ db.cve.createIndex({ "configurations.nodes.cpeMatch.criteria": 1 }, { name: "cpeName" });
$ db.cve.createIndex({ "configurations.nodes.cpeMatch.criteriaProductPrefix": 1 }, { name: "cpeNameProductPrefix" });
$ db.cve.createIndex({ "descriptions.value": "text" }, { name: "keyword" });
$ db.cpe.createIndex({ "cpeName": 1 }, { unique: true, name: "cpeName" });
$ db.cpe.createIndex({ "titles.title": "text", "refs.ref": "text" }, { name: "keyword" });

func (Mongo) IsConnected 

func (mdb Mongo) IsConnected(ctx context.Context) error

IsConnected checks if mongo db is reachable, which is expected to be used for health check endpoint

func (Mongo) UpsertCPE 

func (mdb Mongo) UpsertCPE(ctx context.Context, cpe schema.Cpe) (upserted bool, err error)

UpsertCPE inserts or replaces an CPE record in 'nvd.cpe' collection. It Replaces when the incoming CPE record contains latest 'lastModified'. Since mitre does not contain 'lastModified', it might be empty for some stale record.

It use string comparison since all the value in "lastModified" is converted to same time format, and empty 'lastModified' is less than 'lastModified' that contains any value when using string comparison, which is reasonable when sorting 'lastModified' in descending order.

The first return value shows whether the the cve has been upserted after this function.

func (Mongo) UpsertCPEs 

func (mdb Mongo) UpsertCPEs(ctx context.Context, cpes []schema.Cpe) (upsertCnt int, err error)

UpsertCPEs upserts CPEs to 'nvd.cpe' collection in batch

func (Mongo) UpsertCVE 

func (mdb Mongo) UpsertCVE(ctx context.Context, cve schema.Cve) (upserted bool, err error)

UpsertCVE inserts or replaces an CVE record in 'nvd.cve' collection. It replaces when the incoming CVE record contains latest 'lastModified'.

It use string comparison since all the value in "lastModified" is converted to same time format, and the first return value shows whether the the cve has been upserted after this function

func (Mongo) UpsertCVEs 

func (mdb Mongo) UpsertCVEs(ctx context.Context, cves []schema.Cve) (upsertedCnt int, err error)

UpsertCVEs upserts CVEs to 'nvd.cve' collection in batch

type MongoOptions 

type MongoOptions func(*Mongo) error

MongoOptions is the options of Mongo client

func MongoClientOptions 

func MongoClientOptions(opts ...*options.ClientOptions) MongoOptions

MongoClientOptions appends additional mongo options to client

func MongoConfig 

func MongoConfig(cfg Config) MongoOptions

MongoConfig is expected to use only in testing to mock the client

func MongoGetURI 

func MongoGetURI(fn func(user, pwd, endpoint string) string) MongoOptions

MongoGetURI overwrites default function to get uri to connect to database

func MongoTestClient 

func MongoTestClient(client *mongo.Client) MongoOptions

MongoTestClient is expected to use only in testing to mock the client

type NVDUnit 

type NVDUnit interface {
	schema.Cve | schema.Cpe
}

NVDUnit is the interface to define the possible type of response, which is either CVE or CPE

type NvdCpeDB 

type NvdCpeDB interface {
	GetCPEByName(ctx context.Context, cpeName string) (*schema.Cpe, error)
	GetCPEByMatchString(ctx context.Context, cpeNameMatchString string, opts ...QueryOptions) (*Result[schema.Cpe], error)
	GetCPEByKeyword(ctx context.Context, keyword Keyword, opts ...QueryOptions) (*Result[schema.Cpe], error)
}

NvdCpeDB is the interface to define the operations related to NVD API for CPE

type NvdCveDB 

type NvdCveDB interface {
	GetCVEByID(ctx context.Context, cveId string) (*schema.Cve, error)
	GetCVEByCPE(ctx context.Context, cpeName string, opts ...QueryOptions) (*Result[schema.Cve], error)
	GetCVEByKeyword(ctx context.Context, keywords Keyword, opts ...QueryOptions) (*Result[schema.Cve], error)
}

NvdCveDB is the interface to define the operations related to NVD API for CVE

type QueryConfig 

type QueryConfig struct {
	Start int
	Size  int
}

QueryConfig is the config for each DB query Use options to stay flexible for future extension

func GetQueryConfig 

func GetQueryConfig(opts ...QueryOptions) (QueryConfig, error)

GetQueryConfig gets the query config with given options, use default value in Config if not specified

type QueryOptions 

type QueryOptions func(*QueryConfig) error

func Size 

func Size(size int) QueryOptions

func StartWith 

func StartWith(start int) QueryOptions

type Result 

type Result[T NVDUnit] struct {
	Total, Start, Size int
	Entries            []T // result entries from start to start+size
}

Result is the result of a query, including the total number of entries, the start index, the size of the result. Since the result is paginated, the entries are the entries from start to start+size. While total is the total number before pagination.

func (Result[T]) Filter 

func (r Result[T]) Filter(fn func(item T) bool) Result[T]

Filter filters the entries with the given function, which is used for complex conditions which is implemented by the caller with code after getting the result from database. It is expected to be invoke before pagination.

func (Result[T]) GetPaginated 

func (r Result[T]) GetPaginated(start, size int) Result[T]

GetPaginated gets the paginated result from the result based on given start and size. start and size should be checked before calling this function. * start should be 0 or positive number * size should be positive number

func (Result[T]) IsPageEnd 

func (r Result[T]) IsPageEnd() bool

func (*Result[T]) Sort 

func (r *Result[T]) Sort(sortFn func(i, j int) bool)

Sort sorts the entries with the given function

Source Files

View all Source files

Directories

Path Synopsis
cmd
initdb

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.