loader

Documentation

Overview

Package loader loads a complete Go program from source code, parsing and type-checking the initial packages plus their transitive closure of dependencies. The ASTs and the derived facts are retained for later use.

THIS INTERFACE IS EXPERIMENTAL AND IS LIKELY TO CHANGE.

The package defines two primary types: Config, which specifies a set of initial packages to load and various other options; and Program, which is the result of successfully loading the packages specified by a configuration.

The configuration can be set directly, but *Config provides various convenience methods to simplify the common cases, each of which can be called any number of times. Finally, these are followed by a call to Load() to actually load and type-check the program.

var conf loader.Config

// Use the command-line arguments to specify
// a set of initial packages to load from source.
// See FromArgsUsage for help.
rest, err := conf.FromArgs(os.Args[1:], wantTests)

// Parse the specified files and create an ad hoc package with path "foo".
// All files must have the same 'package' declaration.
conf.CreateFromFilenames("foo", "foo.go", "bar.go")

// Create an ad hoc package with path "foo" from
// the specified already-parsed files.
// All ASTs must have the same 'package' declaration.
conf.CreateFromFiles("foo", parsedFiles)

// Add "runtime" to the set of packages to be loaded.
conf.Import("runtime")

// Adds "fmt" and "fmt_test" to the set of packages
// to be loaded.  "fmt" will include *_test.go files.
conf.ImportWithTests("fmt")

// Finally, load all the packages specified by the configuration.
prog, err := conf.Load()

See examples_test.go for examples of API usage.

CONCEPTS AND TERMINOLOGY

The WORKSPACE is the set of packages accessible to the loader. The workspace is defined by Config.Build, a *build.Context. The default context treats subdirectories of $GOROOT and $GOPATH as packages, but this behavior may be overridden.

An AD HOC package is one specified as a set of source files on the command line. In the simplest case, it may consist of a single file such as $GOROOT/src/net/http/triv.go.

EXTERNAL TEST packages are those comprised of a set of *_test.go files all with the same 'package foo_test' declaration, all in the same directory. (go/build.Package calls these files XTestFiles.)

An IMPORTABLE package is one that can be referred to by some import spec. Every importable package is uniquely identified by its PACKAGE PATH or just PATH, a string such as "fmt", "encoding/json", or "cmd/vendor/golang.org/x/arch/x86/x86asm". A package path typically denotes a subdirectory of the workspace.

An import declaration uses an IMPORT PATH to refer to a package. Most import declarations use the package path as the import path.

Due to VENDORING (https://golang.org/s/go15vendor), the interpretation of an import path may depend on the directory in which it appears. To resolve an import path to a package path, go/build must search the enclosing directories for a subdirectory named "vendor".

ad hoc packages and external test packages are NON-IMPORTABLE. The path of an ad hoc package is inferred from the package declarations of its files and is therefore not a unique package key. For example, Config.CreatePkgs may specify two initial ad hoc packages, both with path "main".

An AUGMENTED package is an importable package P plus all the *_test.go files with same 'package foo' declaration as P. (go/build.Package calls these files TestFiles.)

The INITIAL packages are those specified in the configuration. A DEPENDENCY is a package loaded to satisfy an import in an initial package or another dependency.

Index

• Constants
• type Config
  • func (conf *Config) CreateFromFilenames(path string, filenames ...string)
  • func (conf *Config) CreateFromFiles(path string, files ...*ast.File)
  • func (conf *Config) FromArgs(args []string, xtest bool) ([]string, error)
  • func (conf *Config) Import(path string)
  • func (conf *Config) ImportWithTests(path string)
  • func (conf *Config) Load() (*Program, error)
  • func (conf *Config) ParseFile(filename string, src interface{}) (*ast.File, error)
• type PackageInfo
  • func (info *PackageInfo) String() string
• type PkgSpec
• type Program
  • func (prog *Program) InitialPackages() []*PackageInfo
  • func (prog *Program) Package(path string) *PackageInfo
  • func (prog *Program) PathEnclosingInterval(start, end token.Pos) (pkg *PackageInfo, path []ast.Node, exact bool)

Examples

• Config.CreateFromFilenames
• Config.CreateFromFiles
• Config.FromArgs
• Config.Import

Constants

const FromArgsUsage = `` /* 1067-byte string literal not displayed */

FromArgsUsage is a partial usage message that applications calling FromArgs may wish to include in their -help output.

Types

type Config

type Config struct {
	// Fset is the file set for the parser to use when loading the
	// program.  If nil, it may be lazily initialized by any
	// method of Config.
	Fset *token.FileSet

	// ParserMode specifies the mode to be used by the parser when
	// loading source packages.
	ParserMode parser.Mode

	// TypeChecker contains options relating to the type checker.
	//
	// The supplied IgnoreFuncBodies is not used; the effective
	// value comes from the TypeCheckFuncBodies func below.
	// The supplied Import function is not used either.
	TypeChecker types.Config

	// TypeCheckFuncBodies is a predicate over package paths.
	// A package for which the predicate is false will
	// have its package-level declarations type checked, but not
	// its function bodies; this can be used to quickly load
	// dependencies from source.  If nil, all func bodies are type
	// checked.
	TypeCheckFuncBodies func(path string) bool

	// If Build is non-nil, it is used to locate source packages.
	// Otherwise &build.Default is used.
	//
	// By default, cgo is invoked to preprocess Go files that
	// import the fake package "C".  This behaviour can be
	// disabled by setting CGO_ENABLED=0 in the environment prior
	// to startup, or by setting Build.CgoEnabled=false.
	Build *build.Context

	// The current directory, used for resolving relative package
	// references such as "./go/loader".  If empty, os.Getwd will be
	// used instead.
	Cwd string

	// If DisplayPath is non-nil, it is used to transform each
	// file name obtained from Build.Import().  This can be used
	// to prevent a virtualized build.Config's file names from
	// leaking into the user interface.
	DisplayPath func(path string) string

	// If AllowErrors is true, Load will return a Program even
	// if some of the its packages contained I/O, parser or type
	// errors; such errors are accessible via PackageInfo.Errors.  If
	// false, Load will fail if any package had an error.
	AllowErrors bool

	// CreatePkgs specifies a list of non-importable initial
	// packages to create.  The resulting packages will appear in
	// the corresponding elements of the Program.Created slice.
	CreatePkgs []PkgSpec

	// ImportPkgs specifies a set of initial packages to load.
	// The map keys are package paths.
	//
	// The map value indicates whether to load tests.  If true, Load
	// will add and type-check two lists of files to the package:
	// non-test files followed by in-package *_test.go files.  In
	// addition, it will append the external test package (if any)
	// to Program.Created.
	ImportPkgs map[string]bool

	// FindPackage is called during Load to create the build.Package
	// for a given import path from a given directory.
	// If FindPackage is nil, (*build.Context).Import is used.
	// A client may use this hook to adapt to a proprietary build
	// system that does not follow the "go build" layout
	// conventions, for example.
	//
	// It must be safe to call concurrently from multiple goroutines.
	FindPackage func(ctxt *build.Context, fromDir, importPath string, mode build.ImportMode) (*build.Package, error)
}

Config specifies the configuration for loading a whole program from Go source code. The zero value for Config is a ready-to-use default configuration.

func (*Config) CreateFromFilenames

func (conf *Config) CreateFromFilenames(path string, filenames ...string)

CreateFromFilenames is a convenience function that adds a conf.CreatePkgs entry to create a package of the specified *.go files.

Example

This example creates and type-checks a single package (without tests) from a list of filenames, and loads all of its dependencies.

package main

import (
	"fmt"
	"log"
	"path/filepath"
	"runtime"
	"sort"
	"strings"

	"golang.org/x/tools/go/loader"
)

func printProgram(prog *loader.Program) {

	var names []string
	for _, info := range prog.Created {
		names = append(names, info.Pkg.Path())
	}
	fmt.Printf("created: %s\n", names)

	names = nil
	for _, info := range prog.Imported {
		if strings.Contains(info.Pkg.Path(), "internal") {
			continue
		}
		names = append(names, info.Pkg.Path())
	}
	sort.Strings(names)
	fmt.Printf("imported: %s\n", names)

	names = nil
	for _, info := range prog.InitialPackages() {
		names = append(names, info.Pkg.Path())
	}
	sort.Strings(names)
	fmt.Printf("initial: %s\n", names)

	names = nil
	for pkg := range prog.AllPackages {
		names = append(names, pkg.Path())
	}
	sort.Strings(names)
	fmt.Printf("all: %s\n", names)
}

func main() {
	var conf loader.Config
	filename := filepath.Join(runtime.GOROOT(), "src/container/heap/heap.go")
	conf.CreateFromFilenames("container/heap", filename)
	prog, err := conf.Load()
	if err != nil {
		log.Fatal(err)
	}

	printProgram(prog)
}

Output:

created: [container/heap]
imported: []
initial: [container/heap]
all: [container/heap sort]

func (*Config) CreateFromFiles

func (conf *Config) CreateFromFiles(path string, files ...*ast.File)

CreateFromFiles is a convenience function that adds a conf.CreatePkgs entry to create package of the specified path and parsed files.

Example

This example creates and type-checks a package from a list of already-parsed files, and loads all its dependencies.

package main

import (
	"fmt"
	"go/token"
	"log"
	"path/filepath"
	"sort"
	"strings"

	"golang.org/x/tools/go/loader"
)

func printProgram(prog *loader.Program) {

	var names []string
	for _, info := range prog.Created {
		names = append(names, info.Pkg.Path())
	}
	fmt.Printf("created: %s\n", names)

	names = nil
	for _, info := range prog.Imported {
		if strings.Contains(info.Pkg.Path(), "internal") {
			continue
		}
		names = append(names, info.Pkg.Path())
	}
	sort.Strings(names)
	fmt.Printf("imported: %s\n", names)

	names = nil
	for _, info := range prog.InitialPackages() {
		names = append(names, info.Pkg.Path())
	}
	sort.Strings(names)
	fmt.Printf("initial: %s\n", names)

	names = nil
	for pkg := range prog.AllPackages {
		names = append(names, pkg.Path())
	}
	sort.Strings(names)
	fmt.Printf("all: %s\n", names)
}

func printFilenames(fset *token.FileSet, info *loader.PackageInfo) {
	var names []string
	for _, f := range info.Files {
		names = append(names, filepath.Base(fset.File(f.Pos()).Name()))
	}
	fmt.Printf("%s.Files: %s\n", info.Pkg.Path(), names)
}

const hello = `package main

import "fmt"

func main() {
	fmt.Println("Hello, world.")
}
`

func main() {
	var conf loader.Config
	f, err := conf.ParseFile("hello.go", hello)
	if err != nil {
		log.Fatal(err)
	}
	conf.CreateFromFiles("hello", f)
	prog, err := conf.Load()
	if err != nil {
		log.Fatal(err)
	}

	printProgram(prog)
	printFilenames(prog.Fset, prog.Package("strconv"))
}

Output:

created: [hello]
imported: []
initial: [hello]
all: [errors fmt hello internal/race io math os reflect runtime runtime/internal/atomic runtime/internal/sys strconv sync sync/atomic syscall time unicode/utf8 unsafe]
strconv.Files: [atob.go atof.go atoi.go decimal.go doc.go extfloat.go ftoa.go isprint.go itoa.go quote.go]

func (*Config) FromArgs

func (conf *Config) FromArgs(args []string, xtest bool) ([]string, error)

FromArgs interprets args as a set of initial packages to load from source and updates the configuration. It returns the list of unconsumed arguments.

It is intended for use in command-line interfaces that require a set of initial packages to be specified; see FromArgsUsage message for details.

Only superficial errors are reported at this stage; errors dependent on I/O are detected during Load.

Example

This example loads a set of packages and all of their dependencies from a typical command-line. FromArgs parses a command line and makes calls to the other methods of Config shown in the examples that follow.

package main

import (
	"fmt"
	"log"
	"sort"
	"strings"

	"golang.org/x/tools/go/loader"
)

func printProgram(prog *loader.Program) {

	var names []string
	for _, info := range prog.Created {
		names = append(names, info.Pkg.Path())
	}
	fmt.Printf("created: %s\n", names)

	names = nil
	for _, info := range prog.Imported {
		if strings.Contains(info.Pkg.Path(), "internal") {
			continue
		}
		names = append(names, info.Pkg.Path())
	}
	sort.Strings(names)
	fmt.Printf("imported: %s\n", names)

	names = nil
	for _, info := range prog.InitialPackages() {
		names = append(names, info.Pkg.Path())
	}
	sort.Strings(names)
	fmt.Printf("initial: %s\n", names)

	names = nil
	for pkg := range prog.AllPackages {
		names = append(names, pkg.Path())
	}
	sort.Strings(names)
	fmt.Printf("all: %s\n", names)
}

func main() {
	args := []string{"mytool", "unicode/utf8", "errors", "runtime", "--", "foo", "bar"}
	const wantTests = false

	var conf loader.Config
	rest, err := conf.FromArgs(args[1:], wantTests)
	prog, err := conf.Load()
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("rest: %s\n", rest)
	printProgram(prog)
}

Output:

rest: [foo bar]
created: []
imported: [errors runtime unicode/utf8]
initial: [errors runtime unicode/utf8]
all: [errors runtime runtime/internal/atomic runtime/internal/sys unicode/utf8 unsafe]

func (*Config) Import

func (conf *Config) Import(path string)

Import is a convenience function that adds path to ImportPkgs, the set of initial packages that will be imported from source.

Example

This example imports three packages, including the tests for one of them, and loads all their dependencies.

package main

import (
	"fmt"
	"go/token"
	"log"
	"path/filepath"
	"sort"
	"strings"

	"golang.org/x/tools/go/loader"
)

func printProgram(prog *loader.Program) {

	var names []string
	for _, info := range prog.Created {
		names = append(names, info.Pkg.Path())
	}
	fmt.Printf("created: %s\n", names)

	names = nil
	for _, info := range prog.Imported {
		if strings.Contains(info.Pkg.Path(), "internal") {
			continue
		}
		names = append(names, info.Pkg.Path())
	}
	sort.Strings(names)
	fmt.Printf("imported: %s\n", names)

	names = nil
	for _, info := range prog.InitialPackages() {
		names = append(names, info.Pkg.Path())
	}
	sort.Strings(names)
	fmt.Printf("initial: %s\n", names)

	names = nil
	for pkg := range prog.AllPackages {
		names = append(names, pkg.Path())
	}
	sort.Strings(names)
	fmt.Printf("all: %s\n", names)
}

func printFilenames(fset *token.FileSet, info *loader.PackageInfo) {
	var names []string
	for _, f := range info.Files {
		names = append(names, filepath.Base(fset.File(f.Pos()).Name()))
	}
	fmt.Printf("%s.Files: %s\n", info.Pkg.Path(), names)
}

func main() {
	// ImportWithTest("strconv") causes strconv to include
	// internal_test.go, and creates an external test package,
	// strconv_test.
	// (Compare with the example of CreateFromFiles.)

	var conf loader.Config
	conf.Import("unicode/utf8")
	conf.Import("errors")
	conf.ImportWithTests("strconv")
	prog, err := conf.Load()
	if err != nil {
		log.Fatal(err)
	}

	printProgram(prog)
	printFilenames(prog.Fset, prog.Package("strconv"))
	printFilenames(prog.Fset, prog.Package("strconv_test"))
}

Output:

created: [strconv_test]
imported: [errors strconv unicode/utf8]
initial: [errors strconv strconv_test unicode/utf8]
all: [bufio bytes errors flag fmt internal/race io log math math/rand os reflect runtime runtime/debug runtime/internal/atomic runtime/internal/sys runtime/pprof runtime/trace sort strconv strconv_test strings sync sync/atomic syscall testing text/tabwriter time unicode unicode/utf8 unsafe]
strconv.Files: [atob.go atof.go atoi.go decimal.go doc.go extfloat.go ftoa.go isprint.go itoa.go quote.go internal_test.go]
strconv_test.Files: [atob_test.go atof_test.go atoi_test.go decimal_test.go example_test.go fp_test.go ftoa_test.go itoa_test.go quote_test.go strconv_test.go]

func (*Config) ImportWithTests

func (conf *Config) ImportWithTests(path string)

ImportWithTests is a convenience function that adds path to ImportPkgs, the set of initial source packages located relative to $GOPATH. The package will be augmented by any *_test.go files in its directory that contain a "package x" (not "package x_test") declaration.

In addition, if any *_test.go files contain a "package x_test" declaration, an additional package comprising just those files will be added to CreatePkgs.

func (*Config) Load

func (conf *Config) Load() (*Program, error)

Load creates the initial packages specified by conf.{Create,Import}Pkgs, loading their dependencies packages as needed.

On success, Load returns a Program containing a PackageInfo for each package. On failure, it returns an error.

If AllowErrors is true, Load will return a Program even if some packages contained I/O, parser or type errors, or if dependencies were missing. (Such errors are accessible via PackageInfo.Errors. If false, Load will fail if any package had an error.

It is an error if no packages were loaded.

func (*Config) ParseFile

func (conf *Config) ParseFile(filename string, src interface{}) (*ast.File, error)

ParseFile is a convenience function (intended for testing) that invokes the parser using the Config's FileSet, which is initialized if nil.

src specifies the parser input as a string, []byte, or io.Reader, and filename is its apparent name. If src is nil, the contents of filename are read from the file system.

type PackageInfo

type PackageInfo struct {
	Pkg                   *types.Package
	Importable            bool        // true if 'import "Pkg.Path()"' would resolve to this
	TransitivelyErrorFree bool        // true if Pkg and all its dependencies are free of errors
	Files                 []*ast.File // syntax trees for the package's files
	Errors                []error     // non-nil if the package had errors
	types.Info                        // type-checker deductions.
	// contains filtered or unexported fields
}

PackageInfo holds the ASTs and facts derived by the type-checker for a single package.

Not mutated once exposed via the API.

func (*PackageInfo) String

func (info *PackageInfo) String() string

type PkgSpec

type PkgSpec struct {
	Path      string      // package path ("" => use package declaration)
	Files     []*ast.File // ASTs of already-parsed files
	Filenames []string    // names of files to be parsed
}

A PkgSpec specifies a non-importable package to be created by Load. Files are processed first, but typically only one of Files and Filenames is provided. The path needn't be globally unique.

type Program

type Program struct {
	Fset *token.FileSet // the file set for this program

	// Created[i] contains the initial package whose ASTs or
	// filenames were supplied by Config.CreatePkgs[i], followed by
	// the external test package, if any, of each package in
	// Config.ImportPkgs ordered by ImportPath.
	//
	// NOTE: these files must not import "C".  Cgo preprocessing is
	// only performed on imported packages, not ad hoc packages.
	//
	// TODO(adonovan): we need to copy and adapt the logic of
	// goFilesPackage (from $GOROOT/src/cmd/go/build.go) and make
	// Config.Import and Config.Create methods return the same kind
	// of entity, essentially a build.Package.
	// Perhaps we can even reuse that type directly.
	Created []*PackageInfo

	// Imported contains the initially imported packages,
	// as specified by Config.ImportPkgs.
	Imported map[string]*PackageInfo

	// AllPackages contains the PackageInfo of every package
	// encountered by Load: all initial packages and all
	// dependencies, including incomplete ones.
	AllPackages map[*types.Package]*PackageInfo
	// contains filtered or unexported fields
}

A Program is a Go program loaded from source as specified by a Config.

func (*Program) InitialPackages

func (prog *Program) InitialPackages() []*PackageInfo

InitialPackages returns a new slice containing the set of initial packages (Created + Imported) in unspecified order.

func (*Program) Package

func (prog *Program) Package(path string) *PackageInfo

Package returns the ASTs and results of type checking for the specified package.

func (*Program) PathEnclosingInterval

func (prog *Program) PathEnclosingInterval(start, end token.Pos) (pkg *PackageInfo, path []ast.Node, exact bool)

PathEnclosingInterval returns the PackageInfo and ast.Node that contain source interval [start, end), and all the node's ancestors up to the AST root. It searches all ast.Files of all packages in prog. exact is defined as for astutil.PathEnclosingInterval.

The zero value is returned if not found.