info

Documentation

Overview

Package info provides utilities for summarizing the contents of a kzip.

Index

• func KzipInfo(f kzip.File, fileSize int64, scanOpts ...kzip.ScanOption) (*apb.KzipInfo, error)
• func KzipInfoTotalCount(infos []*apb.KzipInfo) *apb.KzipInfo_CorpusInfo
• func MergeKzipInfo(infos []*apb.KzipInfo) *apb.KzipInfo
• type Accumulator
  • func NewAccumulator(fileSize int64) *Accumulator
  • func (a *Accumulator) Accumulate(u *kzip.Unit)
  • func (a *Accumulator) Get() *apb.KzipInfo

Functions

func KzipInfo

func KzipInfo(f kzip.File, fileSize int64, scanOpts ...kzip.ScanOption) (*apb.KzipInfo, error)

KzipInfo scans the kzip in f and counts contained files and units, giving a breakdown by corpus and language. It also records the size (in bytes) of the kzip specified by fileSize in the returned KzipInfo. This is a convenience method and thin wrapper over the Accumulator. If you need to do more than just calculate KzipInfo while doing a kzip.Scan(), you should use the Accumulator directly.

func KzipInfoTotalCount

func KzipInfoTotalCount(infos []*apb.KzipInfo) *apb.KzipInfo_CorpusInfo

KzipInfoTotalCount returns the total CompilationUnits counts for infos split apart by language.

func MergeKzipInfo

func MergeKzipInfo(infos []*apb.KzipInfo) *apb.KzipInfo

MergeKzipInfo combines the counts from multiple KzipInfos.

Types

type Accumulator

type Accumulator struct {
	KzipInfo *apb.KzipInfo
}

Accumulator is used to build a summary of a collection of compilation units. Usage:

a := NewAccumulator(fileSize)
a.Accumulate(unit) // call for each compilation unit
info := a.Get()    // get the resulting KzipInfo

func NewAccumulator

func NewAccumulator(fileSize int64) *Accumulator

NewAccumulator creates a new Accumulator instance given the kzip fileSize (in bytes).

func (*Accumulator) Accumulate

func (a *Accumulator) Accumulate(u *kzip.Unit)

Accumulate should be called for each unit in the kzip so its counts can be recorded.

func (*Accumulator) Get

func (a *Accumulator) Get() *apb.KzipInfo

Get returns the final KzipInfo after info from each unit in the kzip has been accumulated.