store

package
v0.0.38 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 14, 2015 License: BSD-3-Clause, MIT Imports: 30 Imported by: 0

Documentation

Overview

Package store handles srclib data access, querying, and storage.

CONVENTION - ZERO VALUES FOR FIELDS OUTSIDE OF A STORE'S SCOPE

We adopt the convention that all items in a store contain zero values in fields pertaining to things outside the store's scope. For example, remember that all items in a TreeStore are, by definition, at the same commit. So, the TreeStore need not be aware of commits; commits are out of a TreeStore's scope. Thus all items in a TreeStore or returned by its methods have their CommitID fields set to "", but when a higher-level store (MultiRepoStore or RepoStore) receives data from a TreeStore, it sets those CommitID fields to the TreeStore's commit ID.

Why? This lets us avoid reduce allocations and assignments and save storage space.

The following demonstrates why this approach does not lead to incorrect results from filters. There are 2 ways filters can be applied. Filters can either:

  • Narrow the scope of the search to only certain stores, excluding those that we're certain can't contain what we're looking for (e.g., when we're looking for a specific repo, only that repo's RepoStore needs to be opened). If the filters implement any of the ByXyzFilter types, this occurs. OR
  • Filter each item individually by having their SelectXyz methods called on each item.

If a filter is used to narrow the scope (the first filter application method above), then when it is called to filter items in lower-level stores, the data in those lower-level stores will have the zero value for the field by which we narrowed the scope. We could set the field value on each item after narrowing the scope but before calling our filter on each item, but that would be both time-consuming and unnecessary.

E.g., Suppose we have a ByReposFilter and immediately narrow our scope to a single RepoStore. All items in a RepoStore by definition are from the same repo, so it's unnecessary for any of their Repo fields to be set. Therefore their Repo fields are "", but we know that they have already satisfied the filter.

Filters should still apply criteria in their SelectXyz funcs even if they also implement corresponding ByXyzFilter interfaces. This allows the filters to be combined arbitrarily (even if the AND of their ByXyzFilters can't be used to narrow the scope) and allows scope narrowing to use techniques that may yield false positives (e.g., bloom filters).

Index

Constants

View Source
const SrclibStoreDir = ".srclib-store"

SrclibStoreDir is the name of the directory under which a RepoStore's data is stored.

Variables

View Source
var Codec codec = ProtobufCodec{}

Codec is the codec used by all file-backed stores. It should only be set at init time or when you can guarantee that no stores will be using the codec.

View Source
var DefaultRepoPaths defaultRepoPaths

DefaultRepoPaths is the default repo path configuration for FS-backed multi-repo stores. It stores repos underneath the "${REPO}/.srclib-store" dir.

View Source
var MaxIndexParallel = 1
View Source
var NoSourceUnit = &unit.ID2{}

NoSourceUnit can be used as a value for IndexCriteria.Unit to indicate that source unit indexes should not be selected.

Functions

func ByCommitIDs added in v0.0.36

func ByCommitIDs(commitIDs ...string) interface {
	DefFilter
	RefFilter
	UnitFilter
	VersionFilter
	ByCommitIDsFilter
}

ByCommitIDs creates a new filter by commit IDs. It panics if any commit ID is empty.

func ByDefKey

ByDefKey returns a filter by a def key. It panics if the def path is not set. If you pass a ByDefKey filter to a store that's scoped to a specific repo/version/unit, then it will match all items in that repo/version/unit even if the key.Repo/key.CommitID/key.UnitType/key.Unit fields do not match (because stores do not "know" the repo/version/unit they store data for, and therefore they can't apply filter criteria for the level above them).

func ByDefPath

func ByDefPath(defPath string) interface {
	DefFilter
	ByDefPathFilter
}

ByDefPath returns a filter by def path. It panics if defPath is empty.

func ByDefQuery

func ByDefQuery(q string) interface {
	DefFilter
	ByDefQueryFilter
}

ByDefQuery returns a filter by def query. It panics if q is empty.

func ByFiles

func ByFiles(files ...string) interface {
	DefFilter
	RefFilter
	UnitFilter
	ByFilesFilter
}

ByFiles returns a filter that selects objects that are defined in or contain any of the listed files. It panics if any file path is empty, or if the file path has not been cleaned (i.e., if file != path.Clean(file)).

func ByRepoCommitIDs added in v0.0.36

ByRepoCommitIDs creates a new filter by a set of repository versions. It panics if repo is empty.

func ByRepos added in v0.0.36

ByRepos creates a new filter by a set of repositories. It panics if repo is empty.

func ByUnitKey

ByUnitKey returns a filter by a source unit key. It panics if any fields on the unit key are not set. To filter by only source unit name and type, use ByUnits.

func ByUnits

func ByUnits(units ...unit.ID2) interface {
	DefFilter
	RefFilter
	UnitFilter
	ByUnitsFilter
}

ByUnits creates a new filter that matches objects in any of the given source units. It panics if any of the unit IDs' names or types are empty.

func Limit

func Limit(limit, offset int) interface {
	DefFilter
	RefFilter
}

Limit is an EXPERIMENTAL filter for limiting the number of results. It is not correct because it assumes that if it is called on an object, it gets to decide whether that object appears in the final results. In reality, other filters could reject the object, and then those would count toward the limit for this filter but would never get returned. We could guarantee that Limit always runs last (after all other filters have accepted something).

func LimitRemaining

func LimitRemaining(filters interface{}) (remaining int, moreOK bool)

LimitRemaining returns how many more results may be added before the limit is exceeded (specified by the Limit filter, for example). If additional results may be added (either because the limit has not been reached, or there is no limit), moreOK is true.

Types

type ByCommitIDsFilter added in v0.0.36

type ByCommitIDsFilter interface {
	ByCommitIDs() []string
}

ByCommitIDsFilter is implemented by filters that restrict their selection to items at specific commit IDs. It allows the store to optimize calls by skipping data that it knows is not at any of the specified commits.

type ByDefPathFilter

type ByDefPathFilter interface {
	ByDefPath() string
}

ByDefPathFilter is implemented by filters that restrict their selection to defs with a specific def path.

type ByDefQueryFilter

type ByDefQueryFilter interface {
	ByDefQuery() string
}

ByDefQueryFilter is implemented by filters that restrict their selection to defs whose names match the query.

type ByFilesFilter

type ByFilesFilter interface {
	ByFiles() []string
}

ByFilesFilter is implemented by filters that restrict their selection to defs, refs, etc., that exist in any file in a set, or source units that contain any of the files in the set.

type ByRefDefFilter

type ByRefDefFilter interface {
	ByDefRepo() string
	ByDefUnitType() string
	ByDefUnit() string
	ByDefPath() string
	// contains filtered or unexported methods
}

ByRefDefFilter is implemented by filters that restrict their selection to refs with a specific target definition.

type ByRepoCommitIDsFilter added in v0.0.36

type ByRepoCommitIDsFilter interface {
	ByRepoCommitIDs() []Version
}

ByRepoCommitIDsFilter is implemented by filters that restrict their selections to items in a set of repositories (and in each repository, to a specific version). It allows the store to optimize calls by skipping data that it knows is not in any of the specified repository versions.

type ByReposFilter added in v0.0.36

type ByReposFilter interface {
	ByRepos() []string
}

ByReposFilter is implemented by filters that restrict their selections to items in a set of repository. It allows the store to optimize calls by skipping data that it knows is not in any of the specified repositories.

type ByUnitsFilter

type ByUnitsFilter interface {
	ByUnits() []unit.ID2
}

ByUnitsFilter is implemented by filters that restrict their selections to items that are in a set of source units. It allows the store to optimize calls by skipping data that it knows is not any of the the specified source units.

type DefFilter

type DefFilter interface {
	SelectDef(*graph.Def) bool
}

A DefFilter filters a set of defs to only those for which SelectDef returns true.

type DefFilterFunc

type DefFilterFunc func(*graph.Def) bool

A DefFilterFunc is a DefFilter that selects only those defs for which the func returns true.

func (DefFilterFunc) SelectDef

func (f DefFilterFunc) SelectDef(def *graph.Def) bool

SelectDef calls f(def).

func (DefFilterFunc) String added in v0.0.36

func (f DefFilterFunc) String() string

type DefsSortByKey added in v0.0.36

type DefsSortByKey struct{}

func (DefsSortByKey) DefsSort added in v0.0.36

func (ds DefsSortByKey) DefsSort(defs []*graph.Def)

func (DefsSortByKey) SelectDef added in v0.0.36

func (ds DefsSortByKey) SelectDef(def *graph.Def) bool

func (DefsSortByKey) String added in v0.0.36

func (ds DefsSortByKey) String() string

type DefsSortByName

type DefsSortByName struct{}

func (DefsSortByName) DefsSort

func (ds DefsSortByName) DefsSort(defs []*graph.Def)

func (DefsSortByName) SelectDef

func (ds DefsSortByName) SelectDef(def *graph.Def) bool

func (DefsSortByName) String

func (ds DefsSortByName) String() string

type DefsSorter

type DefsSorter interface {
	DefsSort(defs []*graph.Def)
}

type FSMultiRepoStoreConf

type FSMultiRepoStoreConf struct {
	// RepoPathConfig specifies where the multi-repo store stores
	// repository data. If nil, DefaultRepoPaths is used, which stores
	// repos at "${REPO}/.srclib-store".
	RepoPaths
}

FSMultiRepoStoreConf configures an FS-backed multi-repo store. Pass it to NewFSMultiRepoStore to construct a new store with the specified options.

type Index

type Index interface {
	// Ready indicates whether the index is ready to be
	// queried. Persisted indexes typically become ready after their
	// Read method is called and returns.
	Ready() bool

	// Covers returns the number of filters that this index
	// covers. Indexes with greater coverage are selected over others
	// with lesser coverage.
	Covers(filters interface{}) int
}

An Index enables efficient store queries using filters that the index covers. An index may be in one of 3 states:

  • Not built: the index neither exists in memory nor is it persisted. It can't be used.

  • Persisted but not ready: the index has been built and persisted (e.g., to disk) but has not been loaded into memory and therefore can't be used.

  • Ready: the index is loaded into memory (either because it was just built in memory, or because it was read from its persisted form) and can be used.

type IndexCriteria

type IndexCriteria struct {
	Repo     string
	CommitID string
	Unit     *unit.ID2
	Name     string
	Type     string
	Stale    *bool

	ReposLimit  int
	ReposOffset int
}

IndexCriteria restricts a set of indexes to only those that match the criteria. Non-empty conditions are ANDed together.

type IndexStatus

type IndexStatus struct {
	// Repo is the ID of the repository this index pertains to. If it
	// pertains to all repositories in a MultiRepoStore, or if it
	// pertains to the current (and only) repository in a RepoStore or
	// lower-level store, the Repo field is empty.
	Repo string `json:",omitempty"`

	// CommitID is the commit ID of the version this index pertains to. If it
	// pertains to all commits in a RepoStore, or if it
	// pertains to the current (and only) commit in a TreeStore or
	// lower-level store, the CommitID field is empty.
	CommitID string `json:",omitempty"`

	// Unit is the commit ID of the version this index pertains to. If
	// it pertains to all units in a TreeStore, or if it pertains to
	// the current (and only) source unit in a UnitStore, the Unit
	// field is empty.
	Unit *unit.ID2 `json:",omitempty"`

	// Stale is a boolean value indicating whether the index needs to
	// be (re)built.
	Stale bool

	// Name is the name of the index.
	Name string

	// Type is the type of the index.
	Type string

	// Size is the length in bytes of the index if it is a regular
	// file.
	Size int64 `json:",omitempty"`

	// Error is the error encountered while determining this index's
	// status, if any.
	Error string `json:",omitempty"`

	// DependsOnChildren is true if this index needs its child indexes
	// to be built first before it can be built.
	DependsOnChildren bool `json:",omitempty"`

	// BuildError is the error encountered while building this index,
	// if any. It is only returned by BuildIndexes (not Indexes).
	BuildError string `json:",omitempty"`

	// BuildDuration is how long it took to build the index. It is
	// only returned by BuildIndexes (not Indexes).
	BuildDuration time.Duration `json:",omitempty"`
	// contains filtered or unexported fields
}

IndexStatus describes an index and its status (whether it exists, etc.).

func BuildIndexes

func BuildIndexes(store interface{}, c IndexCriteria, indexChan chan<- IndexStatus) ([]IndexStatus, error)

BuildIndexes builds all indexes on store and its lower-level stores that match the specified criteria. It returns the status of each index that was built (or rebuilt).

func Indexes

func Indexes(store interface{}, c IndexCriteria, indexChan chan<- IndexStatus) ([]IndexStatus, error)

Indexes returns a list of indexes and their statuses for store and its lower-level stores. Only indexes matching the criteria are returned. If indexChan is non-nil, it receives indexes as soon as they are found; when all matching indexes have been found, the func returns and all indexes are included in the returned slice.

The caller is responsible for closing indexChan after Indexes returns (if desired).

func (IndexStatus) Fprint added in v0.0.36

func (s IndexStatus) Fprint(w io.Writer) error

Fprint prints a representation of s's index's contents to w.

type JSONCodec

type JSONCodec struct{}

func (JSONCodec) NewDecoder

func (JSONCodec) NewDecoder(r io.Reader) decoder

func (JSONCodec) NewEncoder

func (JSONCodec) NewEncoder(w io.Writer) encoder

type MockMultiRepoStore

type MockMultiRepoStore struct {
	Repos_    func(...RepoFilter) ([]string, error)
	Versions_ func(...VersionFilter) ([]*Version, error)
	Units_    func(...UnitFilter) ([]*unit.SourceUnit, error)
	Defs_     func(...DefFilter) ([]*graph.Def, error)
	Refs_     func(...RefFilter) ([]*graph.Ref, error)
}

func (MockMultiRepoStore) Defs

func (m MockMultiRepoStore) Defs(f ...DefFilter) ([]*graph.Def, error)

func (MockMultiRepoStore) Refs

func (m MockMultiRepoStore) Refs(f ...RefFilter) ([]*graph.Ref, error)

func (MockMultiRepoStore) Repos

func (m MockMultiRepoStore) Repos(f ...RepoFilter) ([]string, error)

func (MockMultiRepoStore) Units

func (m MockMultiRepoStore) Units(f ...UnitFilter) ([]*unit.SourceUnit, error)

func (MockMultiRepoStore) Versions

func (m MockMultiRepoStore) Versions(f ...VersionFilter) ([]*Version, error)

type MockRepoStore

type MockRepoStore struct {
	Versions_ func(...VersionFilter) ([]*Version, error)
	MockTreeStore
}

func (MockRepoStore) Versions

func (m MockRepoStore) Versions(f ...VersionFilter) ([]*Version, error)

type MockTreeStore

type MockTreeStore struct {
	Units_ func(...UnitFilter) ([]*unit.SourceUnit, error)
	MockUnitStore
}

func (MockTreeStore) Units

func (m MockTreeStore) Units(f ...UnitFilter) ([]*unit.SourceUnit, error)

type MockUnitStore

type MockUnitStore struct {
	Defs_ func(...DefFilter) ([]*graph.Def, error)
	Refs_ func(...RefFilter) ([]*graph.Ref, error)
}

func (MockUnitStore) Defs

func (m MockUnitStore) Defs(f ...DefFilter) ([]*graph.Def, error)

func (MockUnitStore) Refs

func (m MockUnitStore) Refs(f ...RefFilter) ([]*graph.Ref, error)

type MultiRepoImporter

type MultiRepoImporter interface {
	// Import imports srclib build data for a source unit at a
	// specific version into the store.
	Import(repo, commitID string, unit *unit.SourceUnit, data graph.Output) error
}

A MultiRepoImporter imports srclib build data for a repository's source unit at a specific version into a RepoStore.

type MultiRepoIndexer

type MultiRepoIndexer interface {
	// Index builds indexes for the store.
	Index(repo, commitID string) error
}

type MultiRepoStore

type MultiRepoStore interface {
	// Repos returns all repositories that match the RepoFilter.
	Repos(...RepoFilter) ([]string, error)

	// RepoStore's methods call the corresponding methods on the
	// RepoStore of each repository contained within this multi-repo
	// store. The combined results are returned (in undefined order).
	RepoStore
}

MultiRepoStore provides access to RepoStores for multiple repositories.

Using this interface instead of directly accessing a single RepoStore allows aliasing repository URIs and supporting both ID and URI lookups.

type MultiRepoStoreImporter

type MultiRepoStoreImporter interface {
	MultiRepoStore
	MultiRepoImporter
}

A MultiRepoStoreImporter implements both MultiRepoStore and MultiRepoImporter.

func NewFSMultiRepoStore

NewFSMultiRepoStore creates a new repository store (that can be imported into) that is backed by files on a filesystem.

type ProtobufCodec

type ProtobufCodec struct{}

func (ProtobufCodec) NewDecoder

func (ProtobufCodec) NewDecoder(r io.Reader) decoder

func (ProtobufCodec) NewEncoder

func (ProtobufCodec) NewEncoder(w io.Writer) encoder

type RefFilter

type RefFilter interface {
	SelectRef(*graph.Ref) bool
}

A RefFilter filters a set of refs to only those for which SelectRef returns true.

func AbsRefFilterFunc

func AbsRefFilterFunc(f RefFilterFunc) RefFilter

An AbsRefFilterFunc creates a RefFilter that selects only those refs for which the func returns true. Unlike RefFilterFunc, the ref's Def{Repo,UnitType,Unit,Path}, Repo, and CommitID fields are populated.

AbsRefFilterFunc is less efficient than RefFilterFunc because it must make a copy of each ref before passing it to the func. If you don't need to access any of the fields it sets, use a RefFilterFunc.

func ByRefDef

func ByRefDef(def graph.RefDefKey) RefFilter

ByRefDef returns a filter by ref target def. It panics if def.DefPath is empty. If other fields are empty, they are assumed to match any value.

type RefFilterFunc

type RefFilterFunc func(*graph.Ref) bool

A RefFilterFunc is a RefFilter that selects only those refs for which the func returns true.

func (RefFilterFunc) SelectRef

func (f RefFilterFunc) SelectRef(ref *graph.Ref) bool

SelectRef calls f(ref).

func (RefFilterFunc) String added in v0.0.36

func (f RefFilterFunc) String() string

type RepoFilter

type RepoFilter interface {
	SelectRepo(string) bool
}

A RepoFilter filters a set of repos to only those for which SelectRepo returns true.

type RepoFilterFunc

type RepoFilterFunc func(string) bool

A RepoFilterFunc is a RepoFilter that selects only those repos for which the func returns true.

func (RepoFilterFunc) SelectRepo

func (f RepoFilterFunc) SelectRepo(repo string) bool

SelectRepo calls f(repo).

func (RepoFilterFunc) String added in v0.0.36

func (f RepoFilterFunc) String() string

type RepoImporter

type RepoImporter interface {
	// Import imports srclib build data for a source unit at a
	// specific version into the store.
	Import(commitID string, unit *unit.SourceUnit, data graph.Output) error
}

A RepoImporter imports srclib build data for a source unit at a specific version into a RepoStore.

type RepoIndexer

type RepoIndexer interface {
	// Index builds indexes for the store.
	Index(commitID string) error
}

type RepoPaths

type RepoPaths interface {
	// RepoToPath takes a repo identifier (URI) and returns the path
	// components of the path under which its data should be stored in
	// the multi-repo store.
	//
	// The path components are joined using the VFS's Join method to
	// construct the full subpath. Using the VFS's Join method ensures
	// that the RepoToPath func uses the filesystem's path separator
	// (which usually but not always '/').
	RepoToPath(string) []string

	// PathToRepo is the inverse of RepoToPath.
	PathToRepo(path []string) string

	// ListRepoPaths returns a lexicographically sorted list of repo
	// subpaths (originally created using the RepoToPath func). Only
	// those that sort lexicographically after the "after" arg are
	// returned. If "after" is empty, all keys are returned (up to the
	// max).
	ListRepoPaths(vfs rwvfs.WalkableFileSystem, after string, max int) ([][]string, error)
}

RepoPaths specifies how to generate and list repos in a multi-repo store.

type RepoStore

type RepoStore interface {
	// Versions returns all commits that match the VersionFilter.
	Versions(...VersionFilter) ([]*Version, error)

	// TreeStore's methods call the corresponding methods on the
	// TreeStore of each version contained within this repository. The
	// combined results are returned (in undefined order).
	TreeStore
}

A RepoStore stores and accesses srclib build data for a repository (consisting of any number of commits, each of which have any number of source units).

type RepoStoreImporter

type RepoStoreImporter interface {
	RepoStore
	RepoImporter
}

A RepoStoreImporter implements both RepoStore and RepoImporter.

func NewFSRepoStore

func NewFSRepoStore(fs rwvfs.FileSystem) RepoStoreImporter

NewFSRepoStore creates a new repository store (that can be imported into) that is backed by files on a filesystem.

type TreeImporter

type TreeImporter interface {
	// Import imports a source unit and its graph data into the
	// store. If Import is called with a nil SourceUnit and output
	// data, the importer considers the tree to have no source units
	// until others are imported in the future (this makes it possible
	// to distinguish between a tree that has no source units and a
	// tree whose source units simply haven't been imported yet).
	Import(*unit.SourceUnit, graph.Output) error
}

A TreeImporter imports srclib build data for a source unit into a TreeStore.

type TreeIndexer

type TreeIndexer interface {
	// Index builds indexes for the store, which may include data from
	// multiple source units in the tree.
	Index() error
}

type TreeStore

type TreeStore interface {
	// Units returns all units that match the filter.
	Units(...UnitFilter) ([]*unit.SourceUnit, error)

	// UnitStore's methods call the corresponding methods on the
	// UnitStore of each source unit contained within this tree. The
	// combined results are returned (in undefined order).
	UnitStore
}

A TreeStore stores and accesses srclib build data for an arbitrary source tree (consisting of any number of source units).

type TreeStoreImporter

type TreeStoreImporter interface {
	TreeStore
	TreeImporter
}

A TreeStoreImporter implements both TreeStore and TreeImporter.

type UnitFilter

type UnitFilter interface {
	SelectUnit(*unit.SourceUnit) bool
}

A UnitFilter filters a set of units to only those for which Select returns true.

type UnitFilterFunc

type UnitFilterFunc func(*unit.SourceUnit) bool

A UnitFilterFunc is a UnitFilter that selects only those units for which the func returns true.

func (UnitFilterFunc) SelectUnit

func (f UnitFilterFunc) SelectUnit(unit *unit.SourceUnit) bool

SelectUnit calls f(unit).

func (UnitFilterFunc) String added in v0.0.36

func (f UnitFilterFunc) String() string

type UnitImporter

type UnitImporter interface {
	// Import imports defs, refs, etc., into the store. It overwrites
	// all existing data for this source unit (and at the commit, if
	// applicable).
	Import(graph.Output) error
}

A UnitImporter imports srclib build data for a single source unit into a UnitStore.

type UnitStore

type UnitStore interface {
	// Defs returns all defs that match the filter.
	Defs(...DefFilter) ([]*graph.Def, error)

	// Refs returns all refs that match the filter.
	Refs(...RefFilter) ([]*graph.Ref, error)
}

A UnitStore stores and accesses srclib build data for a single source unit.

type UnitStoreImporter

type UnitStoreImporter interface {
	UnitStore
	UnitImporter
}

A UnitStoreImporter implements both UnitStore and UnitImporter.

type Version

type Version struct {
	// Repo is the URI of the repository that contains this commit.
	Repo string

	// CommitID is the commit ID of the VCS revision that this version
	// represents. If blank, then this version refers to the current
	// workspace.
	CommitID string
}

A Version represents a revision (i.e., commit) of a repository.

func (Version) IsCurrentWorkspace

func (v Version) IsCurrentWorkspace() bool

IsCurrentWorkspace returns a boolean indicating whether this version represents the current workspace, as opposed to a specific VCS commit.

type VersionFilter

type VersionFilter interface {
	SelectVersion(*Version) bool
}

A VersionFilter filters a set of versions to only those for which SelectVersion returns true.

type VersionFilterFunc

type VersionFilterFunc func(*Version) bool

A VersionFilterFunc is a VersionFilter that selects only those versions for which the func returns true.

func (VersionFilterFunc) SelectVersion

func (f VersionFilterFunc) SelectVersion(version *Version) bool

SelectVersion calls f(version).

func (VersionFilterFunc) String added in v0.0.36

func (f VersionFilterFunc) String() string

type VersionKey

type VersionKey struct {
	// Repo is the URI of the commit's repository.
	Repo string

	// CommitID is the commit ID of the commit.
	CommitID string
}

A VersionKey is a unique identifier for a version across all repositories.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL