sources

Documentation

Index

• func DecodeResumeInfo(resumeInfo string) []string
• func EncodeResumeInfo(resumeInfoSlice []string) string
• func FilterReposToResume(repos []string, resumeInfo string) (reposToScan []string, progressOffsetCount int)
• func RemoveRepoFromResumeInfo(resumeRepos []string, repoURL string) []string
• type Chunk
• type Progress
  • func (p *Progress) GetProgress() *Progress
  • func (p *Progress) SetProgressComplete(i, scope int, message, encodedResumeInfo string)
• type Source

Functions

func DecodeResumeInfo

func DecodeResumeInfo(resumeInfo string) []string

func EncodeResumeInfo

func EncodeResumeInfo(resumeInfoSlice []string) string

func FilterReposToResume

func FilterReposToResume(repos []string, resumeInfo string) (reposToScan []string, progressOffsetCount int)

FilterReposToResume filters the existing repos down to those that are included in the encoded resume info. It returns the new slice of repos to be scanned. It also returns the difference between the original length of the repos and the new length to use for progress reporting. It is required that both the resumeInfo repos and the existing repos are sorted.

func RemoveRepoFromResumeInfo

func RemoveRepoFromResumeInfo(resumeRepos []string, repoURL string) []string

RemoveRepoFromResumeInfo removes the repoURL from the resume info.

Types

type Chunk

type Chunk struct {
	// SourceName is the name of the Source that produced the chunk.
	SourceName string
	// SourceID is the ID of the source that the Chunk originated from.
	SourceID int64
	// SourceType is the type of Source that produced the chunk.
	SourceType sourcespb.SourceType
	// SourceMetadata holds the context of where the Chunk was found.
	SourceMetadata *source_metadatapb.MetaData

	// Data is the data to decode and scan.
	Data []byte
	// Verify specifies whether any secrets in the Chunk should be verified.
	Verify bool
}

Chunk contains data to be decoded and scanned along with context on where it came from.

type Progress

type Progress struct {
	PercentComplete   int64
	Message           string
	EncodedResumeInfo string
	SectionsCompleted int32
	SectionsRemaining int32
	// contains filtered or unexported fields
}

PercentComplete is used to update job completion percentages across sources

func (*Progress) GetProgress

func (p *Progress) GetProgress() *Progress

GetProgressComplete gets job completion percentage for metrics reporting

func (*Progress) SetProgressComplete

func (p *Progress) SetProgressComplete(i, scope int, message, encodedResumeInfo string)

SetProgressComplete sets job progress information for a running job based on the highest level objects in the source. i is the current iteration in the loop of target scope scope should be the len(scopedItems) message is the public facing user information about the current progress encodedResumeInfo is an optional string representing any information necessary to resume the job if interrupted

type Source

type Source interface {
	// Type returns the source type, used for matching against configuration and jobs.
	Type() sourcespb.SourceType
	// SourceID returns the initialized source ID used for tracking relationships in the DB.
	SourceID() int64
	// JobID returns the initialized job ID used for tracking relationships in the DB.
	JobID() int64
	// Init initializes the source.
	Init(aCtx context.Context, name string, jobId, sourceId int64, verify bool, connection *anypb.Any, concurrency int) error
	// Chunks emits data over a channel that is decoded and scanned for secrets.
	Chunks(ctx context.Context, chunksChan chan *Chunk) error
	// Completion Percentage for Scanned Source
	GetProgress() *Progress
}

Source defines the interface required to implement a source chunker.