engine

package
v3.62.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 30, 2023 License: AGPL-3.0 Imports: 804 Imported by: 1

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DefaultDetectorTypesImplementing added in v3.33.0 

func DefaultDetectorTypesImplementing[T any]() map[detectorspb.DetectorType]struct{}

func DefaultDetectors 

func DefaultDetectors() []detectors.Detector
func FragmentFirstLineAndLink(chunk *sources.Chunk) (int64, *int64, string)

FragmentFirstLineAndLink extracts the first line number and the link from the chunk metadata. It returns:

  • The first line number of the fragment.
  • A pointer to the line number, facilitating direct updates.
  • The link associated with the fragment. This link may be updated in the chunk metadata if there's a change in the line number.

func FragmentLineOffset added in v3.4.3 

func FragmentLineOffset(chunk *sources.Chunk, result *detectors.Result) (int64, bool)

FragmentLineOffset sets the line number for a provided source chunk with a given detector result.

func SetResultLineNumber added in v3.19.0 

func SetResultLineNumber(chunk *sources.Chunk, result *detectors.Result, fragStart int64, mdLine *int64) bool

SetResultLineNumber sets the line number in the provided result.

func SupportsLineNumbers added in v3.25.0 

func SupportsLineNumbers(sourceType sourcespb.SourceType) bool

SupportsLineNumbers determines if a line number can be found for a source type. 

func UpdateLink(ctx context.Context, metadata *source_metadatapb.MetaData, link string, line int64) error

UpdateLink updates the link of the provided source metadata.

Types

type AhoCorasickCore added in v3.60.2 

type AhoCorasickCore struct {
	// contains filtered or unexported fields
}

AhoCorasickCore encapsulates the operations and data structures used for keyword matching via the Aho-Corasick algorithm. It is responsible for constructing and managing the trie for efficient substring searches, as well as mapping keywords to their associated detectors for rapid lookups.

func NewAhoCorasickCore added in v3.60.2 

func NewAhoCorasickCore(allDetectors []detectors.Detector) *AhoCorasickCore

NewAhoCorasickCore allocates and initializes a new instance of AhoCorasickCore. It uses the provided detector slice to create a map from keywords to detectors and build the Aho-Corasick prefilter trie.

func (*AhoCorasickCore) MatchString added in v3.60.2 

func (ac *AhoCorasickCore) MatchString(input string) []*ahocorasick.Match

MatchString performs a string match using the Aho-Corasick algorithm, returning an array of matches. Designed for internal use within the AhoCorasickCore component.

func (*AhoCorasickCore) PopulateDetectorsByMatch added in v3.60.2 

func (ac *AhoCorasickCore) PopulateDetectorsByMatch(match *ahocorasick.Match, detectors map[detectorspb.DetectorType]detectors.Detector) bool

PopulateDetectorsByMatch populates the given detectorMap based on the Aho-Corasick match results. This method is designed to reuse the same map for performance optimization, reducing the need for repeated allocations within each detector worker in the engine.

type Engine 

type Engine struct {
	WgNotifier sync.WaitGroup
	// contains filtered or unexported fields
}

func Start 

func Start(ctx context.Context, options ...Option) (*Engine, error)

Start initializes and activates the engine's processing pipeline. It sets up various default configurations, prepares lookup structures for detectors, conducts basic sanity checks, and kickstarts all necessary workers. Once started, the engine begins processing input data to identify secrets.

func (*Engine) ChunksChan 

func (e *Engine) ChunksChan() <-chan *sources.Chunk

func (*Engine) DetectorAvgTime 

func (e *Engine) DetectorAvgTime() map[string][]time.Duration

DetectorAvgTime returns the average time taken by each detector.

func (*Engine) Finish added in v3.6.1 

func (e *Engine) Finish(ctx context.Context) error

Finish waits for running sources to complete and workers to finish scanning chunks before closing their respective channels. Once Finish is called, no more sources may be scanned by the engine.

func (*Engine) GetDetectorsMetrics added in v3.46.0 

func (e *Engine) GetDetectorsMetrics() map[string]time.Duration

GetDetectorsMetrics returns a copy of the average time taken by each detector.

func (*Engine) GetMetrics added in v3.46.0 

func (e *Engine) GetMetrics() Metrics

GetMetrics returns a copy of Metrics. It's safe for concurrent use, and the caller can't modify the original data.

func (*Engine) HasFoundResults added in v3.46.0 

func (e *Engine) HasFoundResults() bool

HasFoundResults returns true if any results are found.

func (*Engine) ResultsChan 

func (e *Engine) ResultsChan() chan detectors.ResultWithMetadata

func (*Engine) ScanChunk added in v3.51.0 

func (e *Engine) ScanChunk(chunk *sources.Chunk)

ScanChunk injects a chunk into the output stream of chunks to be scanned. This method should rarely be used. TODO: Remove when dependencies no longer rely on this functionality.

func (*Engine) ScanCircleCI added in v3.23.0 

func (e *Engine) ScanCircleCI(ctx context.Context, token string) error

ScanCircleCI scans CircleCI logs.

func (*Engine) ScanDocker added in v3.41.0 

func (e *Engine) ScanDocker(ctx context.Context, conn *anypb.Any) error

ScanDocker scans a given docker connection.

func (*Engine) ScanFileSystem 

func (e *Engine) ScanFileSystem(ctx context.Context, c sources.FilesystemConfig) error

ScanFileSystem scans a given file system.

func (*Engine) ScanGCS added in v3.29.0 

func (e *Engine) ScanGCS(ctx context.Context, c sources.GCSConfig) error

ScanGCS with the provided options.

func (*Engine) ScanGit 

func (e *Engine) ScanGit(ctx context.Context, c sources.GitConfig) error

ScanGit scans any git source.

func (*Engine) ScanGitHub 

func (e *Engine) ScanGitHub(ctx context.Context, c sources.GithubConfig) error

ScanGitHub scans Github with the provided options.

func (*Engine) ScanGitLab 

func (e *Engine) ScanGitLab(ctx context.Context, c sources.GitlabConfig) error

ScanGitLab scans GitLab with the provided configuration.

func (*Engine) ScanS3 

func (e *Engine) ScanS3(ctx context.Context, c sources.S3Config) error

ScanS3 scans S3 buckets.

func (*Engine) ScanSyslog added in v3.4.3 

func (e *Engine) ScanSyslog(ctx context.Context, c sources.SyslogConfig) error

ScanSyslog is a source that scans syslog files.

func (*Engine) ScanTravisCI added in v3.62.0 

func (e *Engine) ScanTravisCI(ctx context.Context, token string) error

ScanTravisCI scans TravisCI logs.

type Metrics added in v3.46.0 

type Metrics struct {
	BytesScanned           uint64
	ChunksScanned          uint64
	VerifiedSecretsFound   uint64
	UnverifiedSecretsFound uint64
	AvgDetectorTime        map[string]time.Duration

	ScanDuration time.Duration
	// contains filtered or unexported fields
}

Metrics for the scan engine for external consumption.

type Option added in v3.60.2 

type Option func(*Engine)

Option is used to configure the engine during initialization using functional options.

func WithConcurrency 

func WithConcurrency(concurrency uint8) Option

func WithDecoders 

func WithDecoders(decoders ...decoders.Decoder) Option

func WithDetectors 

func WithDetectors(d ...detectors.Detector) Option

func WithFilterDetectors added in v3.28.3 

func WithFilterDetectors(filterFunc func(detectors.Detector) bool) Option

WithFilterDetectors applies a filter to the configured list of detectors. If the filterFunc returns true, the detector will be included for scanning. This option applies to the existing list of detectors configured, so the order this option appears matters. All filtering happens before scanning.

func WithFilterEntropy added in v3.60.0 

func WithFilterEntropy(entropy float64) Option

WithFilterEntropy filters out unverified results using Shannon entropy.

func WithFilterUnverified added in v3.16.2 

func WithFilterUnverified(filter bool) Option

WithFilterUnverified sets the filterUnverified flag on the engine. If set to true, the engine will only return the first unverified result for a chunk for a detector.

func WithOnlyVerified added in v3.46.0 

func WithOnlyVerified(onlyVerified bool) Option

WithOnlyVerified sets the onlyVerified flag on the engine. If set to true, the engine will only print verified results.

func WithPrintAvgDetectorTime added in v3.46.0 

func WithPrintAvgDetectorTime(printAvgDetectorTime bool) Option

WithPrintAvgDetectorTime sets the printAvgDetectorTime flag on the engine. If set to true, the engine will print the average time taken by each detector. This option allows us to measure the time taken for each detector ONLY if the engine is configured to print the results. Calculating the average time taken by each detector is an expensive operation and should be avoided unless specified by the user.

func WithPrinter added in v3.46.0 

func WithPrinter(printer Printer) Option

WithPrinter sets the Printer on the engine.

func WithVerify added in v3.62.0 

func WithVerify(verify bool) Option

WithVerify configures whether the scanner will verify candidate secrets.

type Printer added in v3.46.0 

type Printer interface {
	Print(ctx context.Context, r *detectors.ResultWithMetadata) error
}

Printer is used to format found results and output them to the user. Ex JSON, plain text, etc. Please note printer implementations SHOULD BE thread safe.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.