collection_state

package
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 10, 2025 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Index

Constants

View Source 
const MinArtifactGranularity = time.Hour * 24

Variables

This section is empty.

Functions

This section is empty.

Types

type ArtifactCollectionState 

type ArtifactCollectionState[T parse.Config] interface {
	CollectionState[T]
	RegisterPath(path string, metadata map[string]string)
}

type ArtifactCollectionStateImpl 

type ArtifactCollectionStateImpl[T artifact_source_config.ArtifactSourceConfig] struct {
	// map of trunk paths to collection state for that trunk
	// a trunk is a path segment that does not contain any time metadata
	// for example if the path is s3://bucket/folder1/folder2/2021/01/01/file.txt then the trunk is s3://bucket/folder1/folder2
	TrunkStates map[string]*TimeRangeCollectionStateImpl `json:"trunk_states,omitempty"`

	// the time the last artifact was collected
	// TACTICAL: this is used in GetEndTime called by RowSourceImpl.setFromTime
	// if there is no timing information in the files, we use this to determine the end time
	// which we pass to the CLI to use as the --from time (if one has not been passed)
	// NOTE: this assumes forward collection
	LastModifiedTime time.Time `json:"last_modified_time,omitempty"`
	// contains filtered or unexported fields
}

func (*ArtifactCollectionStateImpl[T]) Clear 

func (s *ArtifactCollectionStateImpl[T]) Clear()

func (*ArtifactCollectionStateImpl[T]) GetEndTime 

func (s *ArtifactCollectionStateImpl[T]) GetEndTime() time.Time

func (*ArtifactCollectionStateImpl[T]) GetGranularity 

func (s *ArtifactCollectionStateImpl[T]) GetGranularity() time.Duration

GetGranularity returns the granularity of the collection state

func (*ArtifactCollectionStateImpl[T]) GetStartTime 

func (s *ArtifactCollectionStateImpl[T]) GetStartTime() time.Time

func (*ArtifactCollectionStateImpl[T]) Init 

func (s *ArtifactCollectionStateImpl[T]) Init(_ T, path string) error

Init sets the filepath of the collection state and loads the state from the file if it exists

func (*ArtifactCollectionStateImpl[T]) IsEmpty 

func (s *ArtifactCollectionStateImpl[T]) IsEmpty() bool

IsEmpty returns whether the collection state is empty

func (*ArtifactCollectionStateImpl[T]) OnCollected 

func (s *ArtifactCollectionStateImpl[T]) OnCollected(id string, timestamp time.Time) error

OnCollected is called when an object has been collected - update our end time and end objects if needed

func (*ArtifactCollectionStateImpl[T]) RegisterPath 

func (s *ArtifactCollectionStateImpl[T]) RegisterPath(path string, metadata map[string]string)

RegisterPath registers a path with the collection state - we determine whether this is a potential trunk (i.e. a path segment with no time metadata for which we need to track collection state separately) and if so, add it to the map of trunk states

func (*ArtifactCollectionStateImpl[T]) Save 

func (s *ArtifactCollectionStateImpl[T]) Save() error

Save serialises the collection state to a JSON file

func (*ArtifactCollectionStateImpl[T]) SetEndTime 

func (s *ArtifactCollectionStateImpl[T]) SetEndTime(newEndTime time.Time)

SetEndTime sets the end time for the collection state - update all trunk states This is called when we are using the --from flag to force recollection

func (*ArtifactCollectionStateImpl[T]) SetGranularity 

func (s *ArtifactCollectionStateImpl[T]) SetGranularity(granularity time.Duration)

SetGranularity sets the granularity of the collection state - this is determined by the file layout and the granularity of the time metadata it contains

func (*ArtifactCollectionStateImpl[T]) ShouldCollect 

func (s *ArtifactCollectionStateImpl[T]) ShouldCollect(id string, timestamp time.Time) bool

ShouldCollect returns whether the object should be collected, based on the time metadata in the object

type CollectionOrder 

type CollectionOrder int
const (
	CollectionOrderChronological CollectionOrder = iota
	CollectionOrderReverse
)

type CollectionState 

type CollectionState[T parse.Config] interface {
	IsEmpty() bool
	Init(config T, path string) error
	Save() error
	SetGranularity(time.Duration)
	ShouldCollect(id string, timestamp time.Time) bool
	OnCollected(id string, timestamp time.Time) error
	GetGranularity() time.Duration
	GetStartTime() time.Time
	GetEndTime() time.Time
	Clear()
	SetEndTime(time.Time)
}

func NewArtifactCollectionStateImpl 

func NewArtifactCollectionStateImpl[T artifact_source_config.ArtifactSourceConfig]() CollectionState[T]

func NewReverseOrderCollectionState 

func NewReverseOrderCollectionState[T parse.Config]() CollectionState[T]

func NewTimeRangeCollectionState 

func NewTimeRangeCollectionState[T parse.Config]() CollectionState[T]

type CollectionStateImplDeprecated 

type CollectionStateImplDeprecated[T config.Config] struct {
	Mut sync.RWMutex `json:"-"`
	// the time range of the data in the bucket
	StartTime time.Time `json:"start_time,omitempty"`
	EndTime   time.Time `json:"end_time,omitempty"`

	// for  end boundary we store the metadata
	// each time the end time changes, we must clear the map
	EndObjects map[string]struct{} `json:"end_objects,omitempty"`

	// the granularity of the file naming scheme - so we must keep track of object metadata
	// this will depend on the template used to name the files
	Granularity time.Duration `json:"granularity,omitempty"`
	// contains filtered or unexported fields
}

func (*CollectionStateImplDeprecated[T]) GetEndTime 

func (s *CollectionStateImplDeprecated[T]) GetEndTime() time.Time

func (*CollectionStateImplDeprecated[T]) GetGranularity 

func (s *CollectionStateImplDeprecated[T]) GetGranularity() time.Duration

func (*CollectionStateImplDeprecated[T]) GetStartTime 

func (s *CollectionStateImplDeprecated[T]) GetStartTime() time.Time

func (*CollectionStateImplDeprecated[T]) Init 

func (s *CollectionStateImplDeprecated[T]) Init(_ T, path string) error

Init sets the filepath of the collection state and loads the state from the file if it exists

func (*CollectionStateImplDeprecated[T]) IsEmpty 

func (s *CollectionStateImplDeprecated[T]) IsEmpty() bool

func (*CollectionStateImplDeprecated[T]) OnCollected 

func (s *CollectionStateImplDeprecated[T]) OnCollected(id string, itemTimestamp time.Time) error

OnCollected is called when an object has been collected - update the end time and end objects if needed Note: the object name is the full path to the object

func (*CollectionStateImplDeprecated[T]) Save 

func (s *CollectionStateImplDeprecated[T]) Save() error

func (*CollectionStateImplDeprecated[T]) SetEndTime 

func (s *CollectionStateImplDeprecated[T]) SetEndTime(t time.Time)

SetEndTime overrides the base implementation to also clear the end objects

func (*CollectionStateImplDeprecated[T]) SetGranularity 

func (s *CollectionStateImplDeprecated[T]) SetGranularity(granularity time.Duration)

func (*CollectionStateImplDeprecated[T]) SetJSONPath 

func (s *CollectionStateImplDeprecated[T]) SetJSONPath(jsonPath string)

func (*CollectionStateImplDeprecated[T]) ShouldCollect 

func (s *CollectionStateImplDeprecated[T]) ShouldCollect(id string, timestamp time.Time) bool

ShouldCollect returns whether the object should be collected

type ReverseOrderCollectionState 

type ReverseOrderCollectionState[T parse.Config] struct {
	// collection of time ranges ordered by time
	TimeRanges []*TimeRangeCollectionStateImpl `json:"time_ranges"`
	// contains filtered or unexported fields
}

func (*ReverseOrderCollectionState[T]) Clear 

func (s *ReverseOrderCollectionState[T]) Clear()

func (*ReverseOrderCollectionState[T]) End 

func (s *ReverseOrderCollectionState[T]) End()

func (*ReverseOrderCollectionState[T]) GetEndTime 

func (s *ReverseOrderCollectionState[T]) GetEndTime() time.Time

func (*ReverseOrderCollectionState[T]) GetGranularity 

func (s *ReverseOrderCollectionState[T]) GetGranularity() time.Duration

GetGranularity returns the granularity of the collection state

func (*ReverseOrderCollectionState[T]) GetStartTime 

func (s *ReverseOrderCollectionState[T]) GetStartTime() time.Time

func (*ReverseOrderCollectionState[T]) Init 

func (s *ReverseOrderCollectionState[T]) Init(_ T, path string) error

Init sets the filepath of the collection state and loads the state from the file if it exists

func (*ReverseOrderCollectionState[T]) IsEmpty 

func (s *ReverseOrderCollectionState[T]) IsEmpty() bool

IsEmpty returns whether the collection state is empty

func (*ReverseOrderCollectionState[T]) OnCollected 

func (s *ReverseOrderCollectionState[T]) OnCollected(id string, timestamp time.Time) error

OnCollected is called when an object has been collected - update our end time and end objects if needed

func (*ReverseOrderCollectionState[T]) Save 

func (s *ReverseOrderCollectionState[T]) Save() error

Save serialises the collection state to a JSON file

func (*ReverseOrderCollectionState[T]) SetEndTime 

func (s *ReverseOrderCollectionState[T]) SetEndTime(newEndTime time.Time)

SetEndTime sets the end time for the collection state - update all trunk states This is called when we are using the --from flag to force recollection

func (*ReverseOrderCollectionState[T]) SetGranularity 

func (s *ReverseOrderCollectionState[T]) SetGranularity(granularity time.Duration)

SetGranularity sets the granularity of the collection state - this is determined by the file layout and the granularity of the time metadata it contains

func (*ReverseOrderCollectionState[T]) ShouldCollect 

func (s *ReverseOrderCollectionState[T]) ShouldCollect(id string, timestamp time.Time) bool

ShouldCollect returns whether the object should be collected, based on the time metadata in the object

func (*ReverseOrderCollectionState[T]) Start 

func (s *ReverseOrderCollectionState[T]) Start()

type TimeRangeCollectionState 

type TimeRangeCollectionState[T parse.Config] struct {
	TimeRangeCollectionStateImpl
	// contains filtered or unexported fields
}

func (*TimeRangeCollectionState[T]) Clear 

func (s *TimeRangeCollectionState[T]) Clear()

func (*TimeRangeCollectionState[T]) Init 

func (s *TimeRangeCollectionState[T]) Init(_ T, path string) error

Init sets the filepath of the collection state and loads the state from the file if it exists

func (*TimeRangeCollectionState[T]) OnCollected 

func (s *TimeRangeCollectionState[T]) OnCollected(id string, timestamp time.Time) error

OnCollected is called when an object has been collected - update our end time and end objects if needed

func (*TimeRangeCollectionState[T]) Save 

func (s *TimeRangeCollectionState[T]) Save() error

Save serialises the collection state to a JSON file

func (*TimeRangeCollectionState[T]) SetEndTime 

func (s *TimeRangeCollectionState[T]) SetEndTime(newEndTime time.Time)

SetEndTime sets the end time for the collection state It may be called to set the end time to earlier than the current end time if a --from flag is used to force recollection

func (*TimeRangeCollectionState[T]) ShouldCollect 

func (s *TimeRangeCollectionState[T]) ShouldCollect(id string, timestamp time.Time) bool

ShouldCollect returns whether the object should be collected, based on the time metadata in the object

type TimeRangeCollectionStateImpl 

type TimeRangeCollectionStateImpl struct {

	// for end boundary (i.e. the end granularity) we store the metadata
	// whenever the end time changes, we must clear the map
	EndObjects map[string]struct{} `json:"end_objects"`

	// the granularity of the file naming scheme - so we must keep track of object metadata
	// this will depend on the template used to name the files
	Granularity time.Duration `json:"granularity,omitempty"`

	// are we collecting forwards (the default) or backwards
	CollectionOrder CollectionOrder `json:"collection_order,omitempty"`
	// contains filtered or unexported fields
}

TimeRangeCollectionStateImpl is a struct that tracks time ranges and objects that have been collected it is used by the ArtifactCollectionStateImpl and TimeRangeCollectionState NOTE: we do not implement mutex locking here - it is assumed that the caller will lock the state before calling

func NewTimeRangeCollectionStateImpl 

func NewTimeRangeCollectionStateImpl(order CollectionOrder) *TimeRangeCollectionStateImpl

func (*TimeRangeCollectionStateImpl) Clear 

func (s *TimeRangeCollectionStateImpl) Clear()

func (*TimeRangeCollectionStateImpl) GetEndTime 

func (s *TimeRangeCollectionStateImpl) GetEndTime() time.Time

func (*TimeRangeCollectionStateImpl) GetGranularity 

func (s *TimeRangeCollectionStateImpl) GetGranularity() time.Duration

GetGranularity returns the granularity of the collection state

func (*TimeRangeCollectionStateImpl) GetStartTime 

func (s *TimeRangeCollectionStateImpl) GetStartTime() time.Time

func (*TimeRangeCollectionStateImpl) IsEmpty 

func (s *TimeRangeCollectionStateImpl) IsEmpty() bool

func (*TimeRangeCollectionStateImpl) MarshalJSON 

func (s *TimeRangeCollectionStateImpl) MarshalJSON() ([]byte, error)

TODO we do not want to serialise start and end time if they are zero until go 1.24 comes out, we manage this by having separate fields to serialise https://github.com/turbot/tailpipe-plugin-sdk/issues/84

func (*TimeRangeCollectionStateImpl) OnCollected 

func (s *TimeRangeCollectionStateImpl) OnCollected(id string, timestamp time.Time) error

OnCollected is called when an object has been collected - update the end time and end objects if needed Note: the object name is the full path to the object

func (*TimeRangeCollectionStateImpl) SetEndTime 

func (s *TimeRangeCollectionStateImpl) SetEndTime(newEndTime time.Time)

SetEndTime sets the end time for the collection state

func (*TimeRangeCollectionStateImpl) SetGranularity 

func (s *TimeRangeCollectionStateImpl) SetGranularity(granularity time.Duration)

SetGranularity sets the granularity of the collection state - this is determined by the file layout and the granularity of the time metadata it contains

func (*TimeRangeCollectionStateImpl) ShouldCollect 

func (s *TimeRangeCollectionStateImpl) ShouldCollect(id string, timestamp time.Time) bool

ShouldCollect returns whether the object should be collected

func (*TimeRangeCollectionStateImpl) UnmarshalJSON 

func (s *TimeRangeCollectionStateImpl) UnmarshalJSON(data []byte) error

UnmarshalJSON override unmashal to handle the special case of the start and end time

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.