gcimporter

Documentation

Overview

Package gcimporter provides various functions for reading gc-generated object files that can be used to implement the Importer interface defined by the Go 1.5 standard library package.

The encoding is deterministic: if the encoder is applied twice to the same types.Package data structure, both encodings are equal. This property may be important to avoid spurious changes in applications such as build systems.

However, the encoder is not necessarily idempotent. Importing an exported package may yield a types.Package that, while it represents the same set of Go types as the original, may differ in the details of its internal representation. Because of these differences, re-encoding the imported package may yield a different, but equally valid, encoding of the package.

Index

• func FindExportData(r *bufio.Reader) (size int64, err error)
• func FindPackageDefinition(r *bufio.Reader) (size int, err error)
• func FindPkg(path, srcDir string) (filename, id string, err error)
• func IExportBundle(out io.Writer, fset *token.FileSet, pkgs []*types.Package) error
• func IExportData(out io.Writer, fset *token.FileSet, pkg *types.Package) error
• func IExportShallow(fset *token.FileSet, pkg *types.Package, reportf ReportFunc) ([]byte, error)
• func IImportBundle(fset *token.FileSet, imports map[string]*types.Package, data []byte) ([]*types.Package, error)
• func IImportData(fset *token.FileSet, imports map[string]*types.Package, data []byte, ...) (int, *types.Package, error)
• func IImportShallow(fset *token.FileSet, getPackages GetPackagesFunc, data []byte, path string, ...) (*types.Package, error)
• func Import(fset *token.FileSet, packages map[string]*types.Package, path, srcDir string, ...) (pkg *types.Package, err error)
• func ReadExportDataHeader(r *bufio.Reader) (n int, err error)
• func ReadObjectHeaders(r *bufio.Reader) (objapi string, headers []string, err error)
• func ReadUnified(r *bufio.Reader) (data []byte, err error)
• func UImportData(fset *token.FileSet, imports map[string]*types.Package, data []byte, ...) (_ int, pkg *types.Package, err error)
• type GetPackagesFunc
  • func GetPackagesFromMap(m map[string]*types.Package) GetPackagesFunc
• type GetPackagesItem
• type ReportFunc

Functions

func FindExportData

func FindExportData(r *bufio.Reader) (size int64, err error)

FindExportData positions the reader r at the beginning of the export data section of an underlying cmd/compile created archive file by reading from it. The reader must be positioned at the start of the file before calling this function. This returns the length of the export data in bytes.

This function is needed by [gcexportdata.Read], which must accept inputs produced by the last two releases of cmd/compile, plus tip.

func FindPackageDefinition

func FindPackageDefinition(r *bufio.Reader) (size int, err error)

FindPackageDefinition positions the reader r at the beginning of a package definition file ("__.PKGDEF") within a GC-created archive by reading from it, and returns the size of the package definition file in the archive.

The reader must be positioned at the start of the archive file before calling this function, and "__.PKGDEF" is assumed to be the first file in the archive.

See cmd/internal/archive for details on the archive format.

func FindPkg

func FindPkg(path, srcDir string) (filename, id string, err error)

FindPkg returns the filename and unique package id for an import path based on package information provided by build.Import (using the build.Default build.Context). A relative srcDir is interpreted relative to the current working directory.

FindPkg is only used in tests within x/tools.

func IExportBundle

func IExportBundle(out io.Writer, fset *token.FileSet, pkgs []*types.Package) error

IExportBundle writes an indexed export bundle for pkgs to out.

func IExportData

func IExportData(out io.Writer, fset *token.FileSet, pkg *types.Package) error

IExportData writes indexed export data for pkg to out.

If no file set is provided, position info will be missing. The package path of the top-level package will not be recorded, so that calls to IImportData can override with a provided package path.

func IExportShallow

func IExportShallow(fset *token.FileSet, pkg *types.Package, reportf ReportFunc) ([]byte, error)

IExportShallow encodes "shallow" export data for the specified package.

For types, we use "shallow" export data. Historically, the Go compiler always produced a summary of the types for a given package that included types from other packages that it indirectly referenced: "deep" export data. This had the advantage that the compiler (and analogous tools such as gopls) need only load one file per direct import. However, it meant that the files tended to get larger based on the level of the package in the import graph. For example, higher-level packages in the kubernetes module have over 1MB of "deep" export data, even when they have almost no content of their own, merely because they mention a major type that references many others. In pathological cases the export data was 300x larger than the source for a package due to this quadratic growth.

"Shallow" export data means that the serialized types describe only a single package. If those types mention types from other packages, the type checker may need to request additional packages beyond just the direct imports. Type information for the entire transitive closure of imports is provided (lazily) by the DAG.

No promises are made about the encoding other than that it can be decoded by the same version of IIExportShallow. If you plan to save export data in the file system, be sure to include a cryptographic digest of the executable in the key to avoid version skew.

If the provided reportf func is non-nil, it will be used for reporting bugs encountered during export. TODO(rfindley): remove reportf when we are confident enough in the new objectpath encoding.

func IImportBundle

func IImportBundle(fset *token.FileSet, imports map[string]*types.Package, data []byte) ([]*types.Package, error)

IImportBundle imports a set of packages from the serialized package bundle.

func IImportData

func IImportData(fset *token.FileSet, imports map[string]*types.Package, data []byte, path string) (int, *types.Package, error)

IImportData imports a package from the serialized package data and returns 0 and a reference to the package. If the export data version is not recognized or the format is otherwise compromised, an error is returned.

func IImportShallow

func IImportShallow(fset *token.FileSet, getPackages GetPackagesFunc, data []byte, path string, reportf ReportFunc) (*types.Package, error)

IImportShallow decodes "shallow" types.Package data encoded by IExportShallow in the same executable. This function cannot import data from cmd/compile or gcexportdata.Write.

The importer calls getPackages to obtain package symbols for all packages mentioned in the export data, including the one being decoded.

If the provided reportf func is non-nil, it will be used for reporting bugs encountered during import. TODO(rfindley): remove reportf when we are confident enough in the new objectpath encoding.

func Import

func Import(fset *token.FileSet, packages map[string]*types.Package, path, srcDir string, lookup func(path string) (io.ReadCloser, error)) (pkg *types.Package, err error)

Import imports a gc-generated package given its import path and srcDir, adds the corresponding package object to the packages map, and returns the object. The packages map must contain all packages already imported.

Import is only used in tests.

func ReadExportDataHeader

func ReadExportDataHeader(r *bufio.Reader) (n int, err error)

ReadExportDataHeader reads the export data header and format from r. It returns the number of bytes read, or an error if the format is no longer supported or it failed to read.

The only currently supported format is binary export data in the unified export format.

func ReadObjectHeaders

func ReadObjectHeaders(r *bufio.Reader) (objapi string, headers []string, err error)

ReadObjectHeaders reads object headers from the reader. Object headers are lines that do not start with an end-of-section marker "$$". The first header is the objabi header. On success, the reader will be positioned at the beginning of the end-of-section marker.

It returns an error if any header does not fit in r.Size() bytes.

func ReadUnified

func ReadUnified(r *bufio.Reader) (data []byte, err error)

ReadUnified reads the contents of the unified export data from a reader r that contains the contents of a GC-created archive file.

On success, the reader will be positioned after the end-of-section marker "\n$$\n".

Supported GC-created archive files have 4 layers of nesting:

• An archive file containing a package definition file.
• The package definition file contains headers followed by a data section. Headers are lines (≤ 4kb) that do not start with "$$".
• The data section starts with "$$B\n" followed by export data followed by an end of section marker "\n$$\n". (The section start "$$\n" is no longer supported.)
• The export data starts with a format byte ('u') followed by the <data> in the given format. (See ReadExportDataHeader for older formats.)

Putting this together, the bytes in a GC-created archive files are expected to look like the following. See cmd/internal/archive for more details on ar file headers.

| <!arch>\n | ar file signature | __.PKGDEF...size...\n | ar header for __.PKGDEF including size. | go object <...>\n | objabi header | <optional headers>\n | other headers such as build id | $$B\n | binary format marker | u<data>\n | unified export <data> | $$\n | end-of-section marker | [optional padding] | padding byte (0x0A) if size is odd | [ar file header] | other ar files | [ar file data] |

func UImportData

func UImportData(fset *token.FileSet, imports map[string]*types.Package, data []byte, path string) (_ int, pkg *types.Package, err error)

Types

type GetPackagesFunc

type GetPackagesFunc = func(items []GetPackagesItem) error

A GetPackagesFunc function obtains the non-nil symbols for a set of packages, creating and recursively importing them as needed. An implementation should store each package symbol is in the Pkg field of the items array.

Any error causes importing to fail. This can be used to quickly read the import manifest of an export data file without fully decoding it.

func GetPackagesFromMap

func GetPackagesFromMap(m map[string]*types.Package) GetPackagesFunc

GetPackagesFromMap returns a GetPackagesFunc that retrieves packages from the given map of package path to package.

The returned function may mutate m: each requested package that is not found is created with types.NewPackage and inserted into m.

type GetPackagesItem

type GetPackagesItem struct {
	Name, Path string
	Pkg        *types.Package // to be filled in by GetPackagesFunc call
	// contains filtered or unexported fields
}

A GetPackagesItem is a request from the importer for the package symbol of the specified name and path.

type ReportFunc

type ReportFunc = func(string, ...interface{})

ReportFunc is the type of a function used to report formatted bugs.