detectors

Documentation

Index

• Variables
• func GetFalsePositiveCheck(detector Detector) func(Result) (bool, string)
• func HasDigit(key string) bool
• func IsKnownFalsePositive(match string, falsePositives []FalsePositive, wordCheck bool) (bool, string)
• func KeyIsRandom(key string) bool
• func MustGetBenchmarkData() map[string][]byte
• func PrefixRegex(keywords []string) string
• func RedactURL(u url.URL) string
• func StringShannonEntropy(input string) float64
• type CustomFalsePositiveChecker
• type CustomMultiPartCredentialProvider
  • func NewCustomMultiPartCredentialProvider(maxCredentialSpan int64) *CustomMultiPartCredentialProvider
  • func (d CustomMultiPartCredentialProvider) MaxCredentialSpan() int64
• type DefaultMultiPartCredentialProvider
  • func (d DefaultMultiPartCredentialProvider) MaxCredentialSpan() int64
• type Detector
• type EndpointCustomizer
• type EndpointSetter
  • func (e *EndpointSetter) Endpoints(defaultEndpoint string) []string
  • func (e *EndpointSetter) SetEndpoints(endpoints ...string) error
• type FalsePositive
• type MaxSecretSizeProvider
• type MultiPartCredentialProvider
• type Result
  • func CleanResults(results []Result) []Result
  • func FilterKnownFalsePositives(ctx context.Context, detector Detector, results []Result) []Result
  • func FilterResultsWithEntropy(ctx context.Context, results []Result, entropy float64, shouldLog bool) []Result
  • func (r *Result) SetVerificationError(err error, secrets ...string)
  • func (r *Result) VerificationError() error
• type ResultWithMetadata
  • func CopyMetadata(chunk *sources.Chunk, result Result) ResultWithMetadata
• type StartOffsetProvider
• type Versioner

Variables

var DefaultFalsePositives = []FalsePositive{"example", "xxxxxx", "aaaaaa", "abcde", "00000", "sample", "*****"}

Functions

func GetFalsePositiveCheck

func GetFalsePositiveCheck(detector Detector) func(Result) (bool, string)

func HasDigit

func HasDigit(key string) bool

func IsKnownFalsePositive

func IsKnownFalsePositive(match string, falsePositives []FalsePositive, wordCheck bool) (bool, string)

IsKnownFalsePositive returns whether a finding is (likely) a known false positive, and the reason for the detection.

Currently, this includes: english word in key or matches common example patterns. Only the secret key material should be passed into this function

func KeyIsRandom

func KeyIsRandom(key string) bool

KeyIsRandom is a Low cost check to make sure that 'keys' include a number to reduce FPs. Golang doesn't support regex lookaheads, so must be done in separate calls. TODO improve checks. Shannon entropy did not work well.

func MustGetBenchmarkData

func MustGetBenchmarkData() map[string][]byte

func PrefixRegex

func PrefixRegex(keywords []string) string

PrefixRegex ensures that at least one of the given keywords is within 40 characters of the capturing group that follows. This can help prevent false positives.

func RedactURL

func RedactURL(u url.URL) string

func StringShannonEntropy

func StringShannonEntropy(input string) float64

Types

type CustomFalsePositiveChecker

type CustomFalsePositiveChecker interface {
	// IsFalsePositive returns two values:
	// 1. Whether the result is a false positive.
	// 2. If #1 is `true`, the reason why.
	IsFalsePositive(result Result) (bool, string)
}

type CustomMultiPartCredentialProvider

type CustomMultiPartCredentialProvider struct {
	// contains filtered or unexported fields
}

func NewCustomMultiPartCredentialProvider

func NewCustomMultiPartCredentialProvider(maxCredentialSpan int64) *CustomMultiPartCredentialProvider

NewCustomMultiPartCredentialProvider creates a new instance of CustomMultiPartCredentialProvider with the specified maximum credential span.

func (CustomMultiPartCredentialProvider) MaxCredentialSpan

func (d CustomMultiPartCredentialProvider) MaxCredentialSpan() int64

MaxCredentialSpan returns the custom maximum credential span specified during the creation of the CustomMultiPartCredentialProvider.

type DefaultMultiPartCredentialProvider

type DefaultMultiPartCredentialProvider struct{}

func (DefaultMultiPartCredentialProvider) MaxCredentialSpan

func (d DefaultMultiPartCredentialProvider) MaxCredentialSpan() int64

MaxCredentialSpan returns the default maximum credential span of 1024 for the DefaultMultiPartCredentialProvider.

type Detector

type Detector interface {
	// FromData will scan bytes for results, and optionally verify them.
	FromData(ctx context.Context, verify bool, data []byte) ([]Result, error)
	// Keywords are used for efficiently pre-filtering chunks using substring operations.
	// Use unique identifiers that are part of the secret if you can, or the provider name.
	Keywords() []string
	// Type returns the DetectorType number from detectors.proto for the given detector.
	Type() detectorspb.DetectorType
}

Detector defines an interface for scanning for and verifying secrets.

type EndpointCustomizer

type EndpointCustomizer interface {
	SetEndpoints(...string) error
	DefaultEndpoint() string
}

EndpointCustomizer is an optional interface that a detector can implement to support verifying against user-supplied endpoints.

type EndpointSetter

type EndpointSetter struct {
	// contains filtered or unexported fields
}

EndpointSetter implements a sensible default for the SetEndpoints function of the EndpointCustomizer interface. A detector can embed this struct to gain the functionality.

func (*EndpointSetter) Endpoints

func (e *EndpointSetter) Endpoints(defaultEndpoint string) []string

func (*EndpointSetter) SetEndpoints

func (e *EndpointSetter) SetEndpoints(endpoints ...string) error

type FalsePositive

type FalsePositive string

type MaxSecretSizeProvider

type MaxSecretSizeProvider interface {
	MaxSecretSize() int64
}

MaxSecretSizeProvider is an optional interface that a detector can implement to provide a custom max size for the secret it finds.

type MultiPartCredentialProvider

type MultiPartCredentialProvider interface {
	// MaxCredentialSpan returns the maximum span or range of characters that the
	// detector should consider when searching for a multi-part credential.
	MaxCredentialSpan() int64
}

MultiPartCredentialProvider is an optional interface that a detector can implement to indicate its compatibility with multi-part credentials and provide the maximum secret size for the credential it finds.

type Result

type Result struct {
	// DetectorType is the type of Detector.
	DetectorType detectorspb.DetectorType
	// DetectorName is the name of the Detector. Used for custom detectors.
	DetectorName string
	// DecoderType is the type of Decoder.
	DecoderType detectorspb.DecoderType
	Verified    bool
	// Raw contains the raw secret identifier data. Prefer IDs over secrets since it is used for deduping after hashing.
	Raw []byte
	// RawV2 contains the raw secret identifier that is a combination of both the ID and the secret.
	// This is used for secrets that are multi part and could have the same ID. Ex: AWS credentials
	RawV2 []byte
	// Redacted contains the redacted version of the raw secret identification data for display purposes.
	// A secret ID should be used if available.
	Redacted       string
	ExtraData      map[string]string
	StructuredData *detectorspb.StructuredData
	// contains filtered or unexported fields
}

func CleanResults

func CleanResults(results []Result) []Result

CleanResults returns all verified secrets, and if there are no verified secrets, just one unverified secret if there are any.

func FilterKnownFalsePositives

func FilterKnownFalsePositives(ctx context.Context, detector Detector, results []Result) []Result

FilterKnownFalsePositives filters out known false positives from the results.

func FilterResultsWithEntropy

func FilterResultsWithEntropy(ctx context.Context, results []Result, entropy float64, shouldLog bool) []Result

FilterResultsWithEntropy filters out determinately unverified results that have a shannon entropy below the given value.

func (*Result) SetVerificationError

func (r *Result) SetVerificationError(err error, secrets ...string)

SetVerificationError is the only way to set a verification error. Any sensitive values should be passed-in as secrets to be redacted.

func (*Result) VerificationError

func (r *Result) VerificationError() error

Public accessors for the fields could also be provided if needed.

type ResultWithMetadata

type ResultWithMetadata struct {
	// IsWordlistFalsePositive indicates whether this secret was flagged as a false positive based on a wordlist check
	IsWordlistFalsePositive bool
	// SourceMetadata contains source-specific contextual information.
	SourceMetadata *source_metadatapb.MetaData
	// SourceID is the ID of the source that the API uses to map secrets to specific sources.
	SourceID sources.SourceID
	// JobID is the ID of the job that the API uses to map secrets to specific jobs.
	JobID sources.JobID
	// SecretID is the ID of the secret, if it exists.
	// Only secrets that are being reverified will have a SecretID.
	SecretID int64
	// SourceType is the type of Source.
	SourceType sourcespb.SourceType
	// SourceName is the name of the Source.
	SourceName string
	Result
	// Data from the sources.Chunk which this result was emitted for
	Data []byte
}

func CopyMetadata

func CopyMetadata(chunk *sources.Chunk, result Result) ResultWithMetadata

CopyMetadata returns a detector result with included metadata from the source chunk.

type StartOffsetProvider

type StartOffsetProvider interface {
	StartOffset() int64
}

StartOffsetProvider is an optional interface that a detector can implement to provide a custom start offset for the secret it finds.

type Versioner

type Versioner interface {
	Version() int
}

Versioner is an optional interface that a detector can implement to differentiate instances of the same detector type.