Documentation ¶
Index ¶
- func DefaultDetectorTypesImplementing[T any]() map[detectorspb.DetectorType]struct{}
- func DefaultDetectors() []detectors.Detector
- func FragmentFirstLineAndLink(chunk *sources.Chunk) (int64, *int64, string)
- func FragmentLineOffset(chunk *sources.Chunk, result *detectors.Result) (int64, bool)
- func SetResultLineNumber(chunk *sources.Chunk, result *detectors.Result, fragStart int64, mdLine *int64) bool
- func SupportsLineNumbers(sourceType sourcespb.SourceType) bool
- func UpdateLink(ctx context.Context, metadata *source_metadatapb.MetaData, link string, ...) error
- type AhoCorasickCore
- type DetectorInfo
- type Engine
- func (e *Engine) ChunksChan() <-chan *sources.Chunk
- func (e *Engine) DetectorAvgTime() map[string][]time.Duration
- func (e *Engine) Finish(ctx context.Context) error
- func (e *Engine) GetDetectorsMetrics() map[string]time.Duration
- func (e *Engine) GetMetrics() Metrics
- func (e *Engine) HasFoundResults() bool
- func (e *Engine) ResultsChan() chan detectors.ResultWithMetadata
- func (e *Engine) ScanChunk(chunk *sources.Chunk)
- func (e *Engine) ScanCircleCI(ctx context.Context, token string) error
- func (e *Engine) ScanDocker(ctx context.Context, conn *anypb.Any) error
- func (e *Engine) ScanFileSystem(ctx context.Context, c sources.FilesystemConfig) error
- func (e *Engine) ScanGCS(ctx context.Context, c sources.GCSConfig) error
- func (e *Engine) ScanGit(ctx context.Context, c sources.GitConfig) error
- func (e *Engine) ScanGitHub(ctx context.Context, c sources.GithubConfig) error
- func (e *Engine) ScanGitLab(ctx context.Context, c sources.GitlabConfig) error
- func (e *Engine) ScanS3(ctx context.Context, c sources.S3Config) error
- func (e *Engine) ScanSyslog(ctx context.Context, c sources.SyslogConfig) error
- type Metrics
- type Option
- func WithConcurrency(concurrency uint8) Option
- func WithDecoders(decoders ...decoders.Decoder) Option
- func WithDetectors(verify bool, d ...detectors.Detector) Option
- func WithFilterDetectors(filterFunc func(detectors.Detector) bool) Option
- func WithFilterEntropy(entropy float64) Option
- func WithFilterUnverified(filter bool) Option
- func WithOnlyVerified(onlyVerified bool) Option
- func WithPrintAvgDetectorTime(printAvgDetectorTime bool) Option
- func WithPrinter(printer Printer) Option
- type Printer
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DefaultDetectorTypesImplementing ¶ added in v3.33.0
func DefaultDetectorTypesImplementing[T any]() map[detectorspb.DetectorType]struct{}
func DefaultDetectors ¶
func FragmentFirstLineAndLink ¶ added in v3.57.0
FragmentFirstLineAndLink extracts the first line number and the link from the chunk metadata. It returns:
- The first line number of the fragment.
- A pointer to the line number, facilitating direct updates.
- The link associated with the fragment. This link may be updated in the chunk metadata if there's a change in the line number.
func FragmentLineOffset ¶ added in v3.4.3
FragmentLineOffset sets the line number for a provided source chunk with a given detector result.
func SetResultLineNumber ¶ added in v3.19.0
func SetResultLineNumber(chunk *sources.Chunk, result *detectors.Result, fragStart int64, mdLine *int64) bool
SetResultLineNumber sets the line number in the provided result.
func SupportsLineNumbers ¶ added in v3.25.0
func SupportsLineNumbers(sourceType sourcespb.SourceType) bool
SupportsLineNumbers determines if a line number can be found for a source type.
func UpdateLink ¶ added in v3.57.0
func UpdateLink(ctx context.Context, metadata *source_metadatapb.MetaData, link string, line int64) error
UpdateLink updates the link of the provided source metadata.
Types ¶
type AhoCorasickCore ¶ added in v3.60.2
type AhoCorasickCore struct {
// contains filtered or unexported fields
}
AhoCorasickCore encapsulates the operations and data structures used for keyword matching via the Aho-Corasick algorithm. It is responsible for constructing and managing the trie for efficient substring searches, as well as mapping keywords to their associated detectors for rapid lookups.
func NewAhoCorasickCore ¶ added in v3.60.2
func NewAhoCorasickCore(detectors map[bool][]detectors.Detector) *AhoCorasickCore
NewAhoCorasickCore allocates and initializes a new instance of AhoCorasickCore. It creates an empty keyword-to-detectors map for future string matching operations. The map detectorTypeToDetectorInfo is pre-allocated based on the size of detectors provided, for efficient storage and lookup of detector information.
func (*AhoCorasickCore) MatchString ¶ added in v3.60.2
func (ac *AhoCorasickCore) MatchString(input string) []*ahocorasick.Match
MatchString performs a string match using the Aho-Corasick algorithm, returning an array of matches. Designed for internal use within the AhoCorasickCore component.
func (*AhoCorasickCore) PopulateDetectorsByMatch ¶ added in v3.60.2
func (ac *AhoCorasickCore) PopulateDetectorsByMatch(match *ahocorasick.Match, detectors map[detectorspb.DetectorType]DetectorInfo) bool
PopulateDetectorsByMatch populates the given detectorMap based on the Aho-Corasick match results. This method is designed to reuse the same map for performance optimization, reducing the need for repeated allocations within each detector worker in the engine.
func (*AhoCorasickCore) Setup ¶ added in v3.60.2
func (ac *AhoCorasickCore) Setup(ctx context.Context)
Setup initializes the internal state of AhoCorasickCore to prepare it for keyword matching. This involves pre-filtering setup and lookup optimization, critical for the engine's performance.
type DetectorInfo ¶ added in v3.60.2
DetectorInfo is used to store a detector and whether it should be verified.
type Engine ¶
func Start ¶
Start initializes and activates the engine's processing pipeline. It sets up various default configurations, prepares lookup structures for detectors, conducts basic sanity checks, and kickstarts all necessary workers. Once started, the engine begins processing input data to identify secrets.
func (*Engine) ChunksChan ¶
func (*Engine) DetectorAvgTime ¶
DetectorAvgTime returns the average time taken by each detector.
func (*Engine) Finish ¶ added in v3.6.1
Finish waits for running sources to complete and workers to finish scanning chunks before closing their respective channels. Once Finish is called, no more sources may be scanned by the engine.
func (*Engine) GetDetectorsMetrics ¶ added in v3.46.0
GetDetectorsMetrics returns a copy of the average time taken by each detector.
func (*Engine) GetMetrics ¶ added in v3.46.0
GetMetrics returns a copy of Metrics. It's safe for concurrent use, and the caller can't modify the original data.
func (*Engine) HasFoundResults ¶ added in v3.46.0
HasFoundResults returns true if any results are found.
func (*Engine) ResultsChan ¶
func (e *Engine) ResultsChan() chan detectors.ResultWithMetadata
func (*Engine) ScanChunk ¶ added in v3.51.0
ScanChunk injects a chunk into the output stream of chunks to be scanned. This method should rarely be used. TODO: Remove when dependencies no longer rely on this functionality.
func (*Engine) ScanCircleCI ¶ added in v3.23.0
ScanCircleCI scans CircleCI logs.
func (*Engine) ScanDocker ¶ added in v3.41.0
ScanDocker scans a given docker connection.
func (*Engine) ScanFileSystem ¶
ScanFileSystem scans a given file system.
func (*Engine) ScanGitHub ¶
ScanGitHub scans Github with the provided options.
func (*Engine) ScanGitLab ¶
ScanGitLab scans GitLab with the provided configuration.
func (*Engine) ScanSyslog ¶ added in v3.4.3
ScanSyslog is a source that scans syslog files.
type Metrics ¶ added in v3.46.0
type Metrics struct { BytesScanned uint64 ChunksScanned uint64 VerifiedSecretsFound uint64 UnverifiedSecretsFound uint64 AvgDetectorTime map[string]time.Duration ScanDuration time.Duration // contains filtered or unexported fields }
Metrics for the scan engine for external consumption.
type Option ¶ added in v3.60.2
type Option func(*Engine)
Option is used to configure the engine during initialization using functional options.
func WithConcurrency ¶
func WithDecoders ¶
func WithFilterDetectors ¶ added in v3.28.3
WithFilterDetectors applies a filter to the configured list of detectors. If the filterFunc returns true, the detector will be included for scanning. This option applies to the existing list of detectors configured, so the order this option appears matters. All filtering happens before scanning.
func WithFilterEntropy ¶ added in v3.60.0
WithFilterEntropy filters out unverified results using Shannon entropy.
func WithFilterUnverified ¶ added in v3.16.2
WithFilterUnverified sets the filterUnverified flag on the engine. If set to true, the engine will only return the first unverified result for a chunk for a detector.
func WithOnlyVerified ¶ added in v3.46.0
WithOnlyVerified sets the onlyVerified flag on the engine. If set to true, the engine will only print verified results.
func WithPrintAvgDetectorTime ¶ added in v3.46.0
WithPrintAvgDetectorTime sets the printAvgDetectorTime flag on the engine. If set to true, the engine will print the average time taken by each detector. This option allows us to measure the time taken for each detector ONLY if the engine is configured to print the results. Calculating the average time taken by each detector is an expensive operation and should be avoided unless specified by the user.
func WithPrinter ¶ added in v3.46.0
WithPrinter sets the Printer on the engine.