modfetch

Documentation

Index

• Variables
• func CachePath(m module.Version, suffix string) (string, error)
• func Download(mod module.Version) (dir string, err error)
• func DownloadDir(m module.Version) (string, error)
• func DownloadZip(mod module.Version) (zipfile string, err error)
• func GoMod(path, rev string) ([]byte, error)
• func GoModFile(path, version string) (string, error)
• func GoModSum(path, version string) (string, error)
• func ImportRepoRev(path, rev string) (Repo, *RevInfo, error)
• func InfoFile(path, version string) (string, error)
• func IsPseudoVersion(v string) bool
• func PseudoVersion(major, older string, t time.Time, rev string) string
• func PseudoVersionBase(v string) (string, error)
• func PseudoVersionRev(v string) (rev string, err error)
• func PseudoVersionTime(v string) (time.Time, error)
• func RemoveAll(dir string) error
• func SideLock() (unlock func(), err error)
• func SortVersions(list []string)
• func Sum(mod module.Version) string
• func TrimGoSum(keep map[module.Version]bool)
• func TryProxies(f func(proxy string) error) error
• func WriteGoSum()
• type Repo
  • func Lookup(proxy, path string) (Repo, error)
• type RevInfo
  • func Stat(proxy, path, rev string) (*RevInfo, error)

Variables

var GoSumFile string // path to go.sum; set by package modload

var HelpGoproxy = &base.Command{
	UsageLine: "goproxy",
	Short:     "module proxy protocol",
	Long: `
A Go module proxy is any web server that can respond to GET requests for
URLs of a specified form. The requests have no query parameters, so even
a site serving from a fixed file system (including a file:/// URL)
can be a module proxy.

The GET requests sent to a Go module proxy are:

GET $GOPROXY/<module>/@v/list returns a list of known versions of the given
module, one per line.

GET $GOPROXY/<module>/@v/<version>.info returns JSON-formatted metadata
about that version of the given module.

GET $GOPROXY/<module>/@v/<version>.mod returns the go.mod file
for that version of the given module.

GET $GOPROXY/<module>/@v/<version>.zip returns the zip archive
for that version of the given module.

GET $GOPROXY/<module>/@latest returns JSON-formatted metadata about the
latest known version of the given module in the same format as
<module>/@v/<version>.info. The latest version should be the version of
the module the go command may use if <module>/@v/list is empty or no
listed version is suitable. <module>/@latest is optional and may not
be implemented by a module proxy.

When resolving the latest version of a module, the go command will request
<module>/@v/list, then, if no suitable versions are found, <module>/@latest.
The go command prefers, in order: the semantically highest release version,
the semantically highest pre-release version, and the chronologically
most recent pseudo-version. In Go 1.12 and earlier, the go command considered
pseudo-versions in <module>/@v/list to be pre-release versions, but this is
no longer true since Go 1.13.

To avoid problems when serving from case-sensitive file systems,
the <module> and <version> elements are case-encoded, replacing every
uppercase letter with an exclamation mark followed by the corresponding
lower-case letter: github.com/Azure encodes as github.com/!azure.

The JSON-formatted metadata about a given module corresponds to
this Go data structure, which may be expanded in the future:

    type Info struct {
        Version string    // version string
        Time    time.Time // commit time
    }

The zip archive for a specific version of a given module is a
standard zip file that contains the file tree corresponding
to the module's source code and related files. The archive uses
slash-separated paths, and every file path in the archive must
begin with <module>@<version>/, where the module and version are
substituted directly, not case-encoded. The root of the module
file tree corresponds to the <module>@<version>/ prefix in the
archive.

Even when downloading directly from version control systems,
the go command synthesizes explicit info, mod, and zip files
and stores them in its local cache, $GOPATH/pkg/mod/cache/download,
the same as if it had downloaded them directly from a proxy.
The cache layout is the same as the proxy URL space, so
serving $GOPATH/pkg/mod/cache/download at (or copying it to)
https://example.com/proxy would let other users access those
cached module versions with GOPROXY=https://example.com/proxy.
`,
}

var HelpModuleAuth = &base.Command{
	UsageLine: "module-auth",
	Short:     "module authentication using go.sum",
	Long: `
The go command tries to authenticate every downloaded module,
checking that the bits downloaded for a specific module version today
match bits downloaded yesterday. This ensures repeatable builds
and detects introduction of unexpected changes, malicious or not.

In each module's root, alongside go.mod, the go command maintains
a file named go.sum containing the cryptographic checksums of the
module's dependencies.

The form of each line in go.sum is three fields:

	<module> <version>[/go.mod] <hash>

Each known module version results in two lines in the go.sum file.
The first line gives the hash of the module version's file tree.
The second line appends "/go.mod" to the version and gives the hash
of only the module version's (possibly synthesized) go.mod file.
The go.mod-only hash allows downloading and authenticating a
module version's go.mod file, which is needed to compute the
dependency graph, without also downloading all the module's source code.

The hash begins with an algorithm prefix of the form "h<N>:".
The only defined algorithm prefix is "h1:", which uses SHA-256.

Module authentication failures

The go command maintains a cache of downloaded packages and computes
and records the cryptographic checksum of each package at download time.
In normal operation, the go command checks the main module's go.sum file
against these precomputed checksums instead of recomputing them on
each command invocation. The 'go mod verify' command checks that
the cached copies of module downloads still match both their recorded
checksums and the entries in go.sum.

In day-to-day development, the checksum of a given module version
should never change. Each time a dependency is used by a given main
module, the go command checks its local cached copy, freshly
downloaded or not, against the main module's go.sum. If the checksums
don't match, the go command reports the mismatch as a security error
and refuses to run the build. When this happens, proceed with caution:
code changing unexpectedly means today's build will not match
yesterday's, and the unexpected change may not be beneficial.

If the go command reports a mismatch in go.sum, the downloaded code
for the reported module version does not match the one used in a
previous build of the main module. It is important at that point
to find out what the right checksum should be, to decide whether
go.sum is wrong or the downloaded code is wrong. Usually go.sum is right:
you want to use the same code you used yesterday.

If a downloaded module is not yet included in go.sum and it is a publicly
available module, the go command consults the Go checksum database to fetch
the expected go.sum lines. If the downloaded code does not match those
lines, the go command reports the mismatch and exits. Note that the
database is not consulted for module versions already listed in go.sum.

If a go.sum mismatch is reported, it is always worth investigating why
the code downloaded today differs from what was downloaded yesterday.

The GOSUMDB environment variable identifies the name of checksum database
to use and optionally its public key and URL, as in:

	GOSUMDB="sum.golang.org"
	GOSUMDB="sum.golang.org+<publickey>"
	GOSUMDB="sum.golang.org+<publickey> https://sum.golang.org"

The go command knows the public key of sum.golang.org, and also that the name
sum.golang.google.cn (available inside mainland China) connects to the
sum.golang.org checksum database; use of any other database requires giving
the public key explicitly.
The URL defaults to "https://" followed by the database name.

GOSUMDB defaults to "sum.golang.org", the Go checksum database run by Google.
See https://sum.golang.org/privacy for the service's privacy policy.

If GOSUMDB is set to "off", or if "go get" is invoked with the -insecure flag,
the checksum database is not consulted, and all unrecognized modules are
accepted, at the cost of giving up the security guarantee of verified repeatable
downloads for all modules. A better way to bypass the checksum database
for specific modules is to use the GOPRIVATE or GONOSUMDB environment
variables. See 'go help module-private' for details.

The 'go env -w' command (see 'go help env') can be used to set these variables
for future go command invocations.
`,
}

var HelpModulePrivate = &base.Command{
	UsageLine: "module-private",
	Short:     "module configuration for non-public modules",
	Long: `
The go command defaults to downloading modules from the public Go module
mirror at proxy.golang.org. It also defaults to validating downloaded modules,
regardless of source, against the public Go checksum database at sum.golang.org.
These defaults work well for publicly available source code.

The GOPRIVATE environment variable controls which modules the go command
considers to be private (not available publicly) and should therefore not use the
proxy or checksum database. The variable is a comma-separated list of
glob patterns (in the syntax of Go's path.Match) of module path prefixes.
For example,

	GOPRIVATE=*.corp.example.com,rsc.io/private

causes the go command to treat as private any module with a path prefix
matching either pattern, including git.corp.example.com/xyzzy, rsc.io/private,
and rsc.io/private/quux.

The GOPRIVATE environment variable may be used by other tools as well to
identify non-public modules. For example, an editor could use GOPRIVATE
to decide whether to hyperlink a package import to a godoc.org page.

For fine-grained control over module download and validation, the GONOPROXY
and GONOSUMDB environment variables accept the same kind of glob list
and override GOPRIVATE for the specific decision of whether to use the proxy
and checksum database, respectively.

For example, if a company ran a module proxy serving private modules,
users would configure go using:

	GOPRIVATE=*.corp.example.com
	GOPROXY=proxy.example.com
	GONOPROXY=none

This would tell the go command and other tools that modules beginning with
a corp.example.com subdomain are private but that the company proxy should
be used for downloading both public and private modules, because
GONOPROXY has been set to a pattern that won't match any modules,
overriding GOPRIVATE.

The 'go env -w' command (see 'go help env') can be used to set these variables
for future go command invocations.
`,
}

var PkgMod string // $GOPATH/pkg/mod; set by package modload

Functions

func CachePath

func CachePath(m module.Version, suffix string) (string, error)

func Download

func Download(mod module.Version) (dir string, err error)

Download downloads the specific module version to the local download cache and returns the name of the directory corresponding to the root of the module's file tree.

func DownloadDir

func DownloadDir(m module.Version) (string, error)

DownloadDir returns the directory to which m should be downloaded. Note that the directory may not yet exist.

func DownloadZip

func DownloadZip(mod module.Version) (zipfile string, err error)

DownloadZip downloads the specific module version to the local zip cache and returns the name of the zip file.

func GoMod

func GoMod(path, rev string) ([]byte, error)

GoMod is like Lookup(path).GoMod(rev) but avoids the repository path resolution in Lookup if the result is already cached on local disk.

func GoModFile

func GoModFile(path, version string) (string, error)

GoModFile is like GoMod but returns the name of the file containing the cached information.

func GoModSum

func GoModSum(path, version string) (string, error)

GoModSum returns the go.sum entry for the module version's go.mod file. (That is, it returns the entry listed in go.sum as "path version/go.mod".)

func ImportRepoRev

func ImportRepoRev(path, rev string) (Repo, *RevInfo, error)

ImportRepoRev returns the module and version to use to access the given import path loaded from the source code repository that the original "go get" would have used, at the specific repository revision (typically a commit hash, but possibly also a source control tag).

func InfoFile

func InfoFile(path, version string) (string, error)

InfoFile is like Stat but returns the name of the file containing the cached information.

func IsPseudoVersion

func IsPseudoVersion(v string) bool

IsPseudoVersion reports whether v is a pseudo-version.

func PseudoVersion

func PseudoVersion(major, older string, t time.Time, rev string) string

PseudoVersion returns a pseudo-version for the given major version ("v1") preexisting older tagged version ("" or "v1.2.3" or "v1.2.3-pre"), revision time, and revision identifier (usually a 12-byte commit hash prefix).

func PseudoVersionBase

func PseudoVersionBase(v string) (string, error)

PseudoVersionBase returns the canonical parent version, if any, upon which the pseudo-version v is based.

If v has no parent version (that is, if it is "vX.0.0-[…]"), PseudoVersionBase returns the empty string and a nil error.

func PseudoVersionRev

func PseudoVersionRev(v string) (rev string, err error)

PseudoVersionRev returns the revision identifier of the pseudo-version v. It returns an error if v is not a pseudo-version.

func PseudoVersionTime

func PseudoVersionTime(v string) (time.Time, error)

PseudoVersionTime returns the time stamp of the pseudo-version v. It returns an error if v is not a pseudo-version or if the time stamp embedded in the pseudo-version is not a valid time.

func RemoveAll

func RemoveAll(dir string) error

RemoveAll removes a directory written by Download or Unzip, first applying any permission changes needed to do so.

func SideLock

func SideLock() (unlock func(), err error)

SideLock locks a file within the module cache that that previously guarded edits to files outside the cache, such as go.sum and go.mod files in the user's working directory. If err is nil, the caller MUST eventually call the unlock function.

func SortVersions

func SortVersions(list []string)

func Sum

func Sum(mod module.Version) string

Sum returns the checksum for the downloaded copy of the given module, if present in the download cache.

func TrimGoSum

func TrimGoSum(keep map[module.Version]bool)

TrimGoSum trims go.sum to contain only the modules for which keep[m] is true.

func TryProxies

func TryProxies(f func(proxy string) error) error

TryProxies iterates f over each configured proxy (including "noproxy" and "direct" if applicable) until f returns an error that is not equivalent to os.ErrNotExist.

TryProxies then returns that final error.

If GOPROXY is set to "off", TryProxies invokes f once with the argument "off".

func WriteGoSum

func WriteGoSum()

WriteGoSum writes the go.sum file if it needs to be updated.

Types

type Repo

type Repo interface {
	// ModulePath returns the module path.
	ModulePath() string

	// Versions lists all known versions with the given prefix.
	// Pseudo-versions are not included.
	// Versions should be returned sorted in semver order
	// (implementations can use SortVersions).
	Versions(prefix string) ([]string, error)

	// Stat returns information about the revision rev.
	// A revision can be any identifier known to the underlying service:
	// commit hash, branch, tag, and so on.
	Stat(rev string) (*RevInfo, error)

	// Latest returns the latest revision on the default branch,
	// whatever that means in the underlying source code repository.
	// It is only used when there are no tagged versions.
	Latest() (*RevInfo, error)

	// GoMod returns the go.mod file for the given version.
	GoMod(version string) (data []byte, err error)

	// Zip writes a zip file for the given version to dst.
	Zip(dst io.Writer, version string) error
}

A Repo represents a repository storing all versions of a single module. It must be safe for simultaneous use by multiple goroutines.

func Lookup

func Lookup(proxy, path string) (Repo, error)

Lookup returns the module with the given module path, fetched through the given proxy.

The distinguished proxy "direct" indicates that the path should be fetched from its origin, and "noproxy" indicates that the patch should be fetched directly only if GONOPROXY matches the given path.

For the distinguished proxy "off", Lookup always returns a non-nil error.

A successful return does not guarantee that the module has any defined versions.

type RevInfo

type RevInfo struct {
	Version string    // suggested version string for this revision
	Time    time.Time // commit time

	// These fields are used for Stat of arbitrary rev,
	// but they are not recorded when talking about module versions.
	Name  string `json:"-"` // complete ID in underlying repository
	Short string `json:"-"` // shortened ID, for use in pseudo-version
}

A Rev describes a single revision in a module repository.

func Stat

func Stat(proxy, path, rev string) (*RevInfo, error)

Stat is like Lookup(path).Stat(rev) but avoids the repository path resolution in Lookup if the result is already cached on local disk.