sources

Documentation

Index

• Constants
• Variables
• func Chunker(originalChunk *Chunk) chan *Chunk
• func DecodeResumeInfo(resumeInfo string) []string
• func EncodeResumeInfo(resumeInfoSlice []string) string
• func FilterReposToResume(repos []string, resumeInfo string) (reposToScan []string, progressOffsetCount int)
• func HandleTestChannel(chunksCh chan *Chunk, cf ChunkFunc) error
• func RemoveRepoFromResumeInfo(resumeRepos []string, repoURL string) []string
• type Chunk
• type ChunkFunc
• type Config
  • func NewConfig(opts ...func(*Config)) Config
• type Progress
  • func (p *Progress) GetProgress() *Progress
  • func (p *Progress) SetProgressComplete(i, scope int, message, encodedResumeInfo string)
• type Source

Constants

const (
	// ChunkSize is the maximum size of a chunk.
	ChunkSize = 10 * 1024
	// PeekSize is the size of the peek into the previous chunk.
	PeekSize = 3 * 1024
)

Variables

var MatchError = errors.New("chunk doesn't match")

Functions

func Chunker

func Chunker(originalChunk *Chunk) chan *Chunk

Chunker takes a chunk and splits it into chunks of ChunkSize.

func DecodeResumeInfo

func DecodeResumeInfo(resumeInfo string) []string

func EncodeResumeInfo

func EncodeResumeInfo(resumeInfoSlice []string) string

func FilterReposToResume

func FilterReposToResume(repos []string, resumeInfo string) (reposToScan []string, progressOffsetCount int)

FilterReposToResume filters the existing repos down to those that are included in the encoded resume info. It returns the new slice of repos to be scanned. It also returns the difference between the original length of the repos and the new length to use for progress reporting. It is required that both the resumeInfo repos and the existing repos are sorted.

func HandleTestChannel

func HandleTestChannel(chunksCh chan *Chunk, cf ChunkFunc) error

func RemoveRepoFromResumeInfo

func RemoveRepoFromResumeInfo(resumeRepos []string, repoURL string) []string

RemoveRepoFromResumeInfo removes the repoURL from the resume info.

Types

type Chunk

type Chunk struct {
	// SourceName is the name of the Source that produced the chunk.
	SourceName string
	// SourceID is the ID of the source that the Chunk originated from.
	SourceID int64
	// SourceType is the type of Source that produced the chunk.
	SourceType sourcespb.SourceType
	// SourceMetadata holds the context of where the Chunk was found.
	SourceMetadata *source_metadatapb.MetaData

	// Data is the data to decode and scan.
	Data []byte
	// Verify specifies whether any secrets in the Chunk should be verified.
	Verify bool
}

Chunk contains data to be decoded and scanned along with context on where it came from.

type ChunkFunc

type ChunkFunc func(chunk *Chunk) error

type Config

type Config struct {
	// Endpoint is the endpoint of the source.
	Endpoint,

	Repo,

	Token,

	Key,

	Secret,

	Address,

	Protocol,

	CertPath,

	KeyPath,

	Format,

	RepoPath,

	HeadRef,

	BaseRef string
	// Concurrency is the number of concurrent workers to use to scan the source.
	Concurrency,

	MaxDepth int
	// IncludeForks indicates whether to include forks in the scan.
	IncludeForks,

	IncludeMembers,

	CloudCred bool
	// Repos is the list of repositories to scan.
	Repos,

	Orgs,

	Buckets,

	ExcludeRepos,

	IncludeRepos,

	Directories []string
	// Filter is the filter to use to scan the source.
	Filter *common.Filter
}

Config defines the optional configuration for a source.

func NewConfig

func NewConfig(opts ...func(*Config)) Config

NewConfig returns a new Config with optional values.

type Progress

type Progress struct {
	PercentComplete   int64
	Message           string
	EncodedResumeInfo string
	SectionsCompleted int32
	SectionsRemaining int32
	// contains filtered or unexported fields
}

Progress is used to update job completion progress across sources.

func (*Progress) GetProgress

func (p *Progress) GetProgress() *Progress

GetProgress gets job completion percentage for metrics reporting.

func (*Progress) SetProgressComplete

func (p *Progress) SetProgressComplete(i, scope int, message, encodedResumeInfo string)

SetProgressComplete sets job progress information for a running job based on the highest level objects in the source. i is the current iteration in the loop of target scope scope should be the len(scopedItems) message is the public facing user information about the current progress encodedResumeInfo is an optional string representing any information necessary to resume the job if interrupted

type Source

type Source interface {
	// Type returns the source type, used for matching against configuration and jobs.
	Type() sourcespb.SourceType
	// SourceID returns the initialized source ID used for tracking relationships in the DB.
	SourceID() int64
	// JobID returns the initialized job ID used for tracking relationships in the DB.
	JobID() int64
	// Init initializes the source.
	Init(aCtx context.Context, name string, jobId, sourceId int64, verify bool, connection *anypb.Any, concurrency int) error
	// Chunks emits data over a channel that is decoded and scanned for secrets.
	Chunks(ctx context.Context, chunksChan chan *Chunk) error
	// GetProgress is the completion progress (percentage) for Scanned Source.
	GetProgress() *Progress
}

Source defines the interface required to implement a source chunker.