stats

Documentation

Index

• type APTBucketReaderStats
  • func APTBucketReaderStatsLoadFromFile(pathToFile string) (*APTBucketReaderStats, error)
  • func NewAPTBucketReaderStats() *APTBucketReaderStats
  • func (stats *APTBucketReaderStats) AddError(message string)
  • func (stats *APTBucketReaderStats) AddS3Item(bucketAndKey string)
  • func (stats *APTBucketReaderStats) AddToInstitutionsCached(inst *models.Institution)
  • func (stats *APTBucketReaderStats) AddWarning(message string)
  • func (stats *APTBucketReaderStats) AddWorkItem(listName string, item *models.WorkItem) error
  • func (stats *APTBucketReaderStats) DumpToFile(pathToFile string) error
  • func (stats *APTBucketReaderStats) FindWorkItemById(listName string, id int) (*models.WorkItem, error)
  • func (stats *APTBucketReaderStats) FindWorkItemByNameAndEtag(listName, name, etag string) (*models.WorkItem, error)
  • func (stats *APTBucketReaderStats) HasErrors() bool
  • func (stats *APTBucketReaderStats) HasWarnings() bool
  • func (stats *APTBucketReaderStats) InstitutionByIdentifier(identifier string) *models.Institution
  • func (stats *APTBucketReaderStats) InstitutionsCachedContains(identifier string) bool
  • func (stats *APTBucketReaderStats) S3ItemWasFound(bucketAndKey string) bool
• type APTQueueStats
  • func APTQueueStatsLoadFromFile(pathToFile string) (*APTQueueStats, error)
  • func NewAPTQueueStats() *APTQueueStats
  • func (stats *APTQueueStats) AddError(message string)
  • func (stats *APTQueueStats) AddItemMarkedAsQueued(item *models.WorkItem)
  • func (stats *APTQueueStats) AddWarning(message string)
  • func (stats *APTQueueStats) AddWorkItem(topic string, item *models.WorkItem)
  • func (stats *APTQueueStats) DumpToFile(pathToFile string) error
  • func (stats *APTQueueStats) FindMarkedItemByName(name string) *models.WorkItem
  • func (stats *APTQueueStats) FindQueuedItemByName(name string) (*models.WorkItem, string)
  • func (stats *APTQueueStats) HasErrors() bool
  • func (stats *APTQueueStats) HasWarnings() bool

Types

type APTBucketReaderStats

type APTBucketReaderStats struct {
	InstitutionsCached      []*models.Institution
	WorkItemsCached         []*models.WorkItem
	WorkItemsFetched        []*models.WorkItem
	WorkItemsCreated        []*models.WorkItem
	WorkItemsQueued         []*models.WorkItem
	WorkItemsMarkedAsQueued []*models.WorkItem
	S3Items                 []string
	Errors                  []string
	Warnings                []string
}

APTBucketReaderStats records stats for the apt_bucket_reader worker. It's not meant for regular production, because it can use a lot of member when the bucket reader encounters thousands of objects. It's primarily useful for integration tests when we want the bucket reader to tell us what it did, so we can compare that to what actually happened. It's also useful periodicall in production, to diagnose problems in smaller runs.

func APTBucketReaderStatsLoadFromFile

func APTBucketReaderStatsLoadFromFile(pathToFile string) (*APTBucketReaderStats, error)

Loads bucket reader stats from a JSON file like the ones that APTBucketReaderStats.DumpToFile dumps out.

func NewAPTBucketReaderStats

func NewAPTBucketReaderStats() *APTBucketReaderStats

Creates a new, empty APTBucketReaderStats object.

func (*APTBucketReaderStats) AddError

func (stats *APTBucketReaderStats) AddError(message string)

Adds an error message to the stats.

func (*APTBucketReaderStats) AddS3Item

func (stats *APTBucketReaderStats) AddS3Item(bucketAndKey string)

Adds an item to the list of files that the bucket reader found in the S3 receiving buckets. Param bucketAndKey should be something like "aptrust.receiving.virginia.edu/virginia.edu_12345678.tar"

func (*APTBucketReaderStats) AddToInstitutionsCached

func (stats *APTBucketReaderStats) AddToInstitutionsCached(inst *models.Institution)

Adds an institution to the list of cached institutions.

func (*APTBucketReaderStats) AddWarning

func (stats *APTBucketReaderStats) AddWarning(message string)

Adds a warning to the stats.

func (*APTBucketReaderStats) AddWorkItem

func (stats *APTBucketReaderStats) AddWorkItem(listName string, item *models.WorkItem) error

Adds a WorkItem to a list.

func (*APTBucketReaderStats) DumpToFile

func (stats *APTBucketReaderStats) DumpToFile(pathToFile string) error

Dumps a JSON representation of this object to a file at the specified path. This will overwrite the existing file, if the existing file has a .json extension. Note that converting the stats object to JSON can use a lot of memory, if you're working with a lot of data. This is safe for integration testing, and it dumps out human-readable formatted JSON. See also APTBucketReaderStatsLoadFromFile.

func (*APTBucketReaderStats) FindWorkItemById

func (stats *APTBucketReaderStats) FindWorkItemById(listName string, id int) (*models.WorkItem, error)

Returns the WorkItem with the matching ID, or nil.

func (*APTBucketReaderStats) FindWorkItemByNameAndEtag

func (stats *APTBucketReaderStats) FindWorkItemByNameAndEtag(listName, name, etag string) (*models.WorkItem, error)

Does what it says.

func (*APTBucketReaderStats) HasErrors

func (stats *APTBucketReaderStats) HasErrors() bool

Returns true if this object contains any errors

func (*APTBucketReaderStats) HasWarnings

func (stats *APTBucketReaderStats) HasWarnings() bool

Returns true if this object contains any warnings

func (*APTBucketReaderStats) InstitutionByIdentifier

func (stats *APTBucketReaderStats) InstitutionByIdentifier(identifier string) *models.Institution

Finds an Institution in the cache by identifier. Returns nil if not found.

func (*APTBucketReaderStats) InstitutionsCachedContains

func (stats *APTBucketReaderStats) InstitutionsCachedContains(identifier string) bool

Returns true if the Institution with the specified identifier is in the Institutions cache.

func (*APTBucketReaderStats) S3ItemWasFound

func (stats *APTBucketReaderStats) S3ItemWasFound(bucketAndKey string) bool

Returns true if the specified bucketAndKey was found in S3

type APTQueueStats

type APTQueueStats struct {
	ItemsQueued         map[string][]*models.WorkItem
	ItemsMarkedAsQueued []*models.WorkItem
	Errors              []string
	Warnings            []string
}

APTQueueStats records information about what apt_queue did.

func APTQueueStatsLoadFromFile

func APTQueueStatsLoadFromFile(pathToFile string) (*APTQueueStats, error)

APTQueueStatsLoadFromFile loads APTQueueStats from a JSON file.

func NewAPTQueueStats

func NewAPTQueueStats() *APTQueueStats

NewAPTQueueStats creates a new, empty APTQueueStats object.

func (*APTQueueStats) AddError

func (stats *APTQueueStats) AddError(message string)

Adds an error message to the stats.

func (*APTQueueStats) AddItemMarkedAsQueued

func (stats *APTQueueStats) AddItemMarkedAsQueued(item *models.WorkItem)

AddItemMarkedAsQueued adds a WorkItem to the items marked as queued. Marked means apt_queue told Pharos that the item has been added to the appropriate NSQ topic.

func (*APTQueueStats) AddWarning

func (stats *APTQueueStats) AddWarning(message string)

Adds a warning to the stats.

func (*APTQueueStats) AddWorkItem

func (stats *APTQueueStats) AddWorkItem(topic string, item *models.WorkItem)

AddWorkItem adds a WorkItem to the specified topic. This is to record the fact that apt_queue added the specified WorkItem id to the specified NSQ topic.

func (*APTQueueStats) DumpToFile

func (stats *APTQueueStats) DumpToFile(pathToFile string) error

DumpToFile dumps a JSON representation of this object to a file at the specified path. This will overwrite the existing file, if the existing file has a .json extension. Note that converting the stats object to JSON can use a lot of memory, if you're working with a lot of data. This is safe for integration testing, and it dumps out human-readable formatted JSON. See also APTQueueStatsLoadFromFile.

func (*APTQueueStats) FindMarkedItemByName

func (stats *APTQueueStats) FindMarkedItemByName(name string) *models.WorkItem

FindMarkedItemByName returns the "marked as queued" item with the specified name. Name is the bag name as it appears in the receiving bucket, with the tar extension, e.g. "my_bag.tar".

func (*APTQueueStats) FindQueuedItemByName

func (stats *APTQueueStats) FindQueuedItemByName(name string) (*models.WorkItem, string)

FindQueuedItemByName returns the queued WorkItem with the specified name, or nil. It also returns the name of the topic the WorkItem was added to. Name is the bag name as it appears in the receiving bucket, with the tar extension, e.g. "my_bag.tar".

func (*APTQueueStats) HasErrors

func (stats *APTQueueStats) HasErrors() bool

Returns true if this object contains any errors

func (*APTQueueStats) HasWarnings

func (stats *APTQueueStats) HasWarnings() bool

Returns true if this object contains any warnings