goref

README

Goref

Goref is a Go package that analyzes a set of Go programs, starting from one or more main packages, and computes the inverse of the cross-package identifier usage graph. In other words, it indexes your Go code and tells you where an exported identifier is used. It can answer questions such as:

• Where is this type instantiated?
• Where is this function called?
• Where are all the references to this identifier?
• What types implement this interface?
• What interfaces are implemented by this type?

Usage

Goref, unlike guru relies in pre-indexing code to make searches faster. Guru is very convenient when developing in a modest-sized codebase, but calls like guru callers -scope ... are very slow and use a lot of memory. Instead, goref indexes code once and lets you search it quickly.

As a library

Goref can be used as a library, see its godoc for usage information.

With ElasticSearch

Goref can also be used to index code into ElasticSearch. This is currently a Work-In-Progress. The binary for this is at elasticsearch/main/main for lack of a good name until now. Usage is:

./main
  --version 42    # version of the code being indexed
  --include_tests # or --noinclude_tests to avoid indexing [XTests](https://godoc.org/golang.org/x/tools/go/loader#hdr-CONCEPTS_AND_TERMINOLOGY)
  --elasticsearch_url http://localhost:9200
  --elasticsearch_user user
  --elasticsearch_password hunter2
  github.com/korfuri/goref
  your/awesome/pacakge

This always imports dependencies recursively.

Code versioning

When code is indexed, the concept of "version" is critical. Since code is a living thing, indexes of code must be versioned. Since the code we index lives in many repositories, we can't use the repositories' history as a versioning tool.

So versions have to be provided to the index externally. It's recommended to keep a global, monotonically increasing counter for this. Every time you go get one or more packages, that counter should be incremented. The counter is an int64, so an elegant way to do this is to use the time.Time() at which you last sync'd your entire Go tree.

Note that the goref PackageGraph is not version-aware: it will only load one version of any given package. The version is purely meant for later indexing after the graph has been initialized and references have been created.

If you'll be doing operations to a completely immutable tree of packages (typically, your PackageGraph remains in memory and is never serialized to disk, and you don't go get or git pull while loading packages), you can just set the version to a fixed number and ignore that.

Goref is (obviously) not safe to use if you concurrently update the code while it's analyzing it.

Vendoring and goref

Vendored packages are treated as separate packages in goref. There is no support to deduplicate github.com/foo/bar and github.com/baz/qux/vendor/foo/bar. This follows the go tool's philosophy on that question.

Similarly, code mirrors such as gopkg.in are not considered in any specific way by goref. github.com/foo/bar and gopkg.in/foo/bar.v42 are treated as different packages even if they are identical.

Types of references

Currently, goref provides the following kinds of references, defined in reftype.go:

• Import represents an import of a package by a file. Since a package doesn't exist in a single position, there will be a Ref from the importing file to each file in the imported package.
• Call represents a call of a function by another function.
• Instantiation are generated for composite literals.
• Implementation represent a reference from a type implementing an interface to that interface.
• Extension represent a reference from interface A to interface B if interface A is a superset of interface B.
• Reference is the default enum value, used if goref can't figure out what kind of reference is used but detects that a package depends on an identifier in another package.

Documentation

Overview

Package goref provides tools to analyze a set of Go packages, starting from one or more `main` packages, and produce a graph of relations between identifiers in these packages. In other words, it indexes your Go code and tells you where an exported identifier is used. It can answer questions such as:

• Where is this type instantiated?
• Where is this function called?
• Where are all the references to this identifier?
• What types implement this interface?
• What interfaces are implemented by this type?

Index

• Constants
• func CandidatePaths(loadpath, parent string) []string
• func CleanImportSpec(spec *ast.ImportSpec) string
• func ConstantVersion(v int64) func(loader.Program, loader.PackageInfo) (int64, error)
• func FileMTimeVersion(prog loader.Program, pi loader.PackageInfo) (int64, error)
• func FilterPass(loadpath string, version int64) bool
• type Corpus
  • func DefaultCorpora() []Corpus
  • func NewCorpus(basepath string) (Corpus, error)
  • func (c Corpus) Abs(rel string) string
  • func (c Corpus) Contains(fpath string) bool
  • func (c Corpus) ContainsRel(rel string) bool
  • func (c Corpus) Pkg(rel string) string
  • func (c Corpus) Rel(fpath string) string
• type Package
  • func (p Package) DocumentID() string
  • func (p Package) MarshalJSON() ([]byte, error)
  • func (p *Package) String() string
• type PackageGraph
  • func NewPackageGraph(versionF func(loader.Program, loader.PackageInfo) (int64, error)) *PackageGraph
  • func (pg *PackageGraph) ComputeInterfaceImplementationMatrix()
  • func (pg *PackageGraph) LoadPackages(packages []string, includeTests bool) error
  • func (pg *PackageGraph) SetFilterF(f func(string, int64) bool)
• type Position
  • func NewPosition(corpus Corpus, fset *token.FileSet, pos, end token.Pos) Position
  • func (p Position) MarshalJSON() ([]byte, error)
  • func (p Position) String() string
  • func (p Position) ToProto() *pb.Position
• type Ref
  • func (r Ref) MarshalJSON() ([]byte, error)
  • func (r *Ref) String() string
  • func (r Ref) ToProto() *pb.Ref
• type RefType
  • func (rt RefType) MarshalJSON() ([]byte, error)
  • func (rt RefType) String() string

Examples

• CandidatePaths
• CleanImportSpec

Constants

const (
	// Instantiation of a type in another package.
	Instantiation = iota

	// Call of a function in another package.
	Call

	// Implementation of an interface by a type.
	Implementation

	// Extension of an interface by another interface.
	Extension

	// Import is the import of a package by another. `fromIdent`
	// may differ from the name of the target package in the case
	// of named imports. For dot-imports, `fromIdent` is ".".
	Import

	// Reference is the default, used when we can't determine the
	// type of reference.
	Reference
)

These are the possible types of edges in a graph.

const (
	// NoPos represents a mising position. It has the same
	// semantics as token.NoPos. It may be used in place of an
	// "end" Pos if the end of an identifier isn't known.
	NoPos = token.NoPos
)

Functions

func CandidatePaths

func CandidatePaths(loadpath, parent string) []string

CandidatePaths returns a slice enumerating all the possible import paths for a package. This means inserting the possible "vendor" directory location from the load path of the importing package.

If package a/b imports c/d, the following paths are candidates: a/b/vendor/c/d a/vendor/c/d vendor/c/d c/d

Order matters, as the most-specific vendored package is selected. Note that multi-level vendoring works, as PackageGraph will consider the full import path, including path/to/vendor/, as the package path when building the graph. In this sense we follow the go tool's convention to not try to detect when two packages loaded through different paths are the same package.

Example

package main

import (
	"fmt"

	"github.com/korfuri/goref"
)

func main() {
	for _, p := range goref.CandidatePaths("lib/util", "program/bin") {
		fmt.Println(p)
	}
}

Output:

program/bin/vendor/lib/util
program/vendor/lib/util
vendor/lib/util
lib/util

func CleanImportSpec

func CleanImportSpec(spec *ast.ImportSpec) string

CleanImportSpec takes an ast.ImportSpec and cleans the Path component by trimming the quotes (") that surround it.

Example

package main

import (
	"fmt"
	"go/ast"

	"github.com/korfuri/goref"
)

func main() {
	fmt.Println(goref.CleanImportSpec(&ast.ImportSpec{Path: &ast.BasicLit{Value: "\"foo/bar/baz\""}}))
}

Output:

foo/bar/baz

func ConstantVersion

func ConstantVersion(v int64) func(loader.Program, loader.PackageInfo) (int64, error)

ConstantVersion returns a versionF function that always replies with a constant version. Useful for experimenting, or for graphs who load from an immutable snapshot of the Go universe.

func FileMTimeVersion

func FileMTimeVersion(prog loader.Program, pi loader.PackageInfo) (int64, error)

FileMTimeVersion is a versionF function that processes all files in the provided PackageInfo and returns the newest mtime's second as a time.Time-compatible int64.

func FilterPass

func FilterPass(loadpath string, version int64) bool

FilterPass is a filterF function that always says yes.

Types

type Corpus

type Corpus string

A Corpus represents a prefix from which Go packages may be loaded. Default corpora are $GOROOT/src and each of $GOPATH/src

func DefaultCorpora

func DefaultCorpora() []Corpus

DefaultCorpora returns the set of default corpora based on GOROOT and GOPATH.

func NewCorpus

func NewCorpus(basepath string) (Corpus, error)

NewCorpus creates a new Corpus.

func (Corpus) Abs

func (c Corpus) Abs(rel string) string

Abs returns the absolute path of a file within a Corpus.

func (Corpus) Contains

func (c Corpus) Contains(fpath string) bool

Contains returns whether the provided filepath exists under this Corpus.

func (Corpus) ContainsRel

func (c Corpus) ContainsRel(rel string) bool

ContainsRel returns whether the provided relpath exists under this Corpus.

func (Corpus) Pkg

func (c Corpus) Pkg(rel string) string

Pkg returns the package containing this file.

func (Corpus) Rel

func (c Corpus) Rel(fpath string) string

Rel returns the relative path of a file within a Corpus. If the string does not belong to the corpus, it returns fpath.

type Package

type Package struct {
	// Name of the package
	Name string `json:"-"`

	// Files in this package
	Files []string `json:"-"`

	// OutRefs and InRefs are slices of references. For OutRefs
	// the Ref is to an identifier in another package. For InRefs
	// the Ref is to an identifier within this package.  Most
	// RefTypes are not indexed if the ToPackage and the
	// FromPackage are the same, but some do such as
	// Implementation. This means that a ref can exist in both
	// OutRefs and InRefs of the same package.
	OutRefs []*Ref `json:"-"`
	InRefs  []*Ref `json:"-"`

	// Interfaces is the list of interface types in this package.
	//
	// This is used to compute the interface-implementation matrix.
	//
	// Only named interfaces matter, because an unnamed interface
	// can't be exported.
	//
	// Interfaces equivalent to interface{} are excluded.
	Interfaces []*types.Named `json:"-"`

	// Impls is the list of non-interface types in this package.
	//
	// This is used to compute the interface-implementation matrix.
	//
	// Only named types matter, because an unnamed type can't have
	// methods.
	Impls []*types.Named `json:"-"`

	// Fset is a reference to the token.FileSet that loaded this
	// package.
	Fset *token.FileSet `json:"-"`

	// Version is the version of the package that was loaded.
	Version int64 `json:"version"`

	// Path is the package's load path
	Path string `json:"loadpath"`

	// Corpus is the corpus that contains this package
	Corpus `json:"-"`
}

Package represents a Go Package, including its dependencies.

func (Package) DocumentID

func (p Package) DocumentID() string

DocumentID returns a consistent id for this package at this version. This can be used to index the package e.g. in ElasticSearch. The ID contains the document version and path.

func (Package) MarshalJSON

func (p Package) MarshalJSON() ([]byte, error)

MarshalJSON implements encoding/json.Marshaler interface

func (*Package) String

func (p *Package) String() string

String implements the Stringer interface

type PackageGraph

type PackageGraph struct {
	// Map of package load-path to Package objects.
	Packages map[string]*Package

	// Slice of corpora that files may be loaded from
	Corpora []Corpus
	// contains filtered or unexported fields
}

PackageGraph represents a collection of Go packages and their mutual dependencies. All dependencies of a Package in the PackageGraph are also part of the PackageGraph.

func NewPackageGraph

func NewPackageGraph(versionF func(loader.Program, loader.PackageInfo) (int64, error)) *PackageGraph

NewPackageGraph returns a new, empty PackageGraph.

func (*PackageGraph) ComputeInterfaceImplementationMatrix

func (pg *PackageGraph) ComputeInterfaceImplementationMatrix()

ComputeInterfaceImplementationMatrix processes all loaded types and adds cross-package and intra-package Refs for Implementation and Extension edges of the graph.

func (*PackageGraph) LoadPackages

func (pg *PackageGraph) LoadPackages(packages []string, includeTests bool) error

LoadPackages loads the specified packages and their transitive dependencies, as well as XTests (as defined by go/loader) if includeTests is true. It may be called multiple times to load multiple package sets in the PackageGraph.

func (*PackageGraph) SetFilterF

func (pg *PackageGraph) SetFilterF(f func(string, int64) bool)

SetFilterF sets the filterF for this PackageGraph

type Position

type Position struct {
	File string `json:"filename"`
	PosL int    `json:"start_line"`
	PosC int    `json:"start_col"`
	EndL int    `json:"end_line"`
	EndC int    `json:"end_col"`
}

A Position is similar to token.Position in that it gives an absolute position within a file, but it may also denote the Pos + End concept of token.Pos.

The End is optional. If NoPos is used as the End, Position only contains file:line:column.

The Pos is not optional and must resolve to file:line:column.

func NewPosition

func NewPosition(corpus Corpus, fset *token.FileSet, pos, end token.Pos) Position

NewPosition creates a Position from a token.FileSet and a pair of Pos in that FileSet. It will panic if both Pos are not from the same Filename.

func (Position) MarshalJSON

func (p Position) MarshalJSON() ([]byte, error)

MarshalJSON implements encoding/json.Marshaler interface

func (Position) String

func (p Position) String() string

func (Position) ToProto

func (p Position) ToProto() *pb.Position

ToProto marshals a Position as a pb.Position

type Ref

type Ref struct {
	// Type of reference
	RefType

	// Where this reference points from, i.e. where the identifier
	// was used in another package.
	FromPosition Position

	// Where this reference points to, i.e. where the definition
	// is
	ToPosition Position

	// What identifier points to this Ref
	FromIdent string

	// What identifier this Ref points to
	ToIdent string

	// What package contains what the identifier refers to
	ToPackage *Package

	// What package the ref is from, i.e. what foreign package was
	// this identifier used in.
	FromPackage *Package
}

A Ref is a reference to an identifier whose definition lives in another package.

func (Ref) MarshalJSON

func (r Ref) MarshalJSON() ([]byte, error)

MarshalJSON implements encoding/json.Marshaler interface

func (*Ref) String

func (r *Ref) String() string

func (Ref) ToProto

func (r Ref) ToProto() *pb.Ref

ToProto marshals a Ref as a pb.Ref

type RefType

type RefType int

RefType is an enum of the various ways a package can reference an identifier in another package (e.g. as a call, an instantiation, etc.)

func (RefType) MarshalJSON

func (rt RefType) MarshalJSON() ([]byte, error)

MarshalJSON implements encoding/json.Marshaler interface

func (RefType) String

func (rt RefType) String() string