getter

package module
v2.2.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 23, 2024 License: MPL-2.0 Imports: 43 Imported by: 56

README

go-getter

CircleCI Build status Go Documentation

go-getter is a library for Go (golang) for downloading files or directories from various sources using a URL as the primary form of input.

The power of this library is being flexible in being able to download from a number of different sources (file paths, Git, HTTP, Mercurial, etc.) using a single string as input. This removes the burden of knowing how to download from a variety of sources from the implementer.

The concept of a detector automatically turns invalid URLs into proper URLs. For example: "github.com/hashicorp/go-getter" would turn into a Git URL. Or "./foo" would turn into a file URL. These are extensible.

This library is used by Terraform for downloading modules, Packer for downloading binaries, and Nomad for downloading binaries.

Installation and Usage

Package documentation can be found on GoDoc.

Installation can be done with a normal go get:

$ go get github.com/hashicorp/go-getter/v2

go-getter also has a command you can use to test URL strings:

$ go install github.com/hashicorp/go-getter/cmd/go-getter
...

$ go-getter github.com/foo/bar ./foo
...

The command is useful for verifying URL structures.

Security

Fetching resources from user-supplied URLs is an inherently dangerous operation and may leave your application vulnerable to server side request forgery, path traversal, denial of service or other security flaws.

go-getter contains mitigations for some of these security issues, but should still be used with caution in security-critical contexts. See the available security options that can be configured to mitigate some of these risks.

go-getter may return values that contain caller-provided query parameters that can contain sensitive data. Context around what parameters are and are not sensitive is known only by the caller of go-getter, and specific to each use case. We recommend the caller ensure that go-getter's return values (e.g., error messages) are properly handled and sanitized to ensure sensitive data is not persisted to logs.

URL Format

go-getter uses a single string URL as input to download from a variety of protocols. go-getter has various "tricks" with this URL to do certain things. This section documents the URL format.

Supported Protocols and Detectors

Protocols are used to download files/directories using a specific mechanism. Example protocols are Git and HTTP.

Detectors are used to transform a valid or invalid URL into another URL if it matches a certain pattern. Example: "github.com/user/repo" is automatically transformed into a fully valid Git URL. This allows go-getter to be very user friendly.

go-getter out of the box supports the following protocols. Additional protocols can be augmented at runtime by implementing the Getter interface.

  • Local files
  • Git
  • Mercurial
  • HTTP
  • Amazon S3
  • Google GCP
  • SMB

In addition to the above protocols, go-getter has what are called "detectors." These take a URL and attempt to automatically choose the best protocol for it, which might involve even changing the protocol. The following detection is built-in by default:

  • File paths such as "./foo" are automatically changed to absolute file URLs.
  • GitHub URLs, such as "github.com/mitchellh/vagrant" are automatically changed to Git protocol over HTTP.
  • GitLab URLs, such as "gitlab.com/inkscape/inkscape" are automatically changed to Git protocol over HTTP.
  • BitBucket URLs, such as "bitbucket.org/mitchellh/vagrant" are automatically changed to a Git or mercurial protocol using the BitBucket API.
Forced Protocol

In some cases, the protocol to use is ambiguous depending on the source URL. For example, "http://github.com/mitchellh/vagrant.git" could reference an HTTP URL or a Git URL. Forced protocol syntax is used to disambiguate this URL.

Forced protocol can be done by prefixing the URL with the protocol followed by double colons. For example: git::http://github.com/mitchellh/vagrant.git would download the given HTTP URL using the Git protocol.

Forced protocols will also override any detectors.

In the absence of a forced protocol, detectors may be run on the URL, transforming the protocol anyways. The above example would've used the Git protocol either way since the Git detector would've detected it was a GitHub URL.

Protocol-Specific Options

Each protocol can support protocol-specific options to configure that protocol. For example, the git protocol supports specifying a ref query parameter that tells it what ref to checkout for that Git repository.

The options are specified as query parameters on the URL (or URL-like string) given to go-getter. Using the Git example above, the URL below is a valid input to go-getter:

github.com/hashicorp/go-getter?ref=abcd1234

The protocol-specific options are documented below the URL format section. But because they are part of the URL, we point it out here so you know they exist.

Subdirectories

If you want to download only a specific subdirectory from a downloaded directory, you can specify a subdirectory after a double-slash //. go-getter will first download the URL specified before the double-slash (as if you didn't specify a double-slash), but will then copy the path after the double slash into the target directory.

For example, if you're downloading this GitHub repository, but you only want to download the testdata directory, you can do the following:

https://github.com/hashicorp/go-getter.git//testdata

If you downloaded this to the /tmp directory, then the file /tmp/archive.gz would exist. Notice that this file is in the testdata directory in this repository, but because we specified a subdirectory, go-getter automatically copied only that directory contents.

Subdirectory paths may also use filesystem glob patterns. The path must match exactly one entry or go-getter will return an error. This is useful if you're not sure the exact directory name but it follows a predictable naming structure.

For example, the following URL would also work:

https://github.com/hashicorp/go-getter.git//test-*
Checksumming

For file downloads of any protocol, go-getter can automatically verify a checksum for you. Note that checksumming only works for downloading files, not directories, but checksumming will work for any protocol.

To checksum a file, append a checksum query parameter to the URL. go-getter will parse out this query parameter automatically and use it to verify the checksum. The parameter value can be in the format of type:value or just value, where type is "md5", "sha1", "sha256", "sha512" or "file" . The "value" should be the actual checksum value or download URL for "file". When type part is omitted, type will be guessed based on the length of the checksum string. Examples:

./foo.txt?checksum=md5:b7d96c89d09d9e204f5fedc4d5d55b21

./foo.txt?checksum=b7d96c89d09d9e204f5fedc4d5d55b21

./foo.txt?checksum=file:./foo.txt.sha256sum

When checksumming from a file - ex: with checksum=file:url - go-getter will get the file linked in the URL after file: using the same configuration. For example, in file:http://releases.ubuntu.com/cosmic/MD5SUMS go-getter will download a checksum file under the aforementioned url using the http protocol. All protocols supported by go-getter can be used. The checksum file will be downloaded in a temporary file then parsed. The destination of the temporary file can be changed by setting system specific environment variables: TMPDIR for unix; TMP, TEMP or USERPROFILE on windows. Read godoc of os.TempDir for more information on the temporary directory selection. Content of files are expected to be BSD or GNU style. Once go-getter is done with the checksum file; it is deleted.

The checksum query parameter is never sent to the backend protocol implementation. It is used at a higher level by go-getter itself.

If the destination file exists and the checksums match: download will be skipped.

Unarchiving

go-getter will automatically unarchive files into a file or directory based on the extension of the file being requested (over any protocol). This works for both file and directory downloads.

go-getter looks for an archive query parameter to specify the format of the archive. If this isn't specified, go-getter will use the extension of the path to see if it appears archived. Unarchiving can be explicitly disabled by setting the archive query parameter to false.

The following archive formats are supported:

  • tar.gz and tgz
  • tar.bz2 and tbz2
  • tar.xz and txz
  • zip
  • gz
  • bz2
  • xz

For example, an example URL is shown below:

./foo.zip

This will automatically be inferred to be a ZIP file and will be extracted. You can also be explicit about the archive type:

./some/other/path?archive=zip

And finally, you can disable archiving completely:

./some/path?archive=false

You can combine unarchiving with the other features of go-getter such as checksumming. The special archive query parameter will be removed from the URL before going to the final protocol downloader.

Protocol-Specific Options

This section documents the protocol-specific options that can be specified for go-getter. These options should be appended to the input as normal query parameters (HTTP headers are an exception to this, however). Depending on the usage of go-getter, applications may provide alternate ways of inputting options. For example, Nomad provides a nice options block for specifying options rather than in the URL.

General (All Protocols)

The options below are available to all protocols:

  • archive - The archive format to use to unarchive this file, or "" (empty string) to disable unarchiving. For more details, see the complete section on archive support above.

  • checksum - Checksum to verify the downloaded file or archive. See the entire section on checksumming above for format and more details.

  • filename - When in file download mode, allows specifying the name of the downloaded file on disk. Has no effect in directory mode.

Local Files (file)

None

Git (git)

  • ref - The Git ref to checkout. This is a ref, so it can point to a commit SHA, a branch name, etc. If it is a named ref such as a branch name, go-getter will update it to the latest on each get.

  • sshkey - An SSH private key to use during clones. The provided key must be a base64-encoded string. For example, to generate a suitable sshkey from a private key file on disk, you would run base64 -w0 <file>.

    Note: Git 2.3+ is required to use this feature.

  • depth - The Git clone depth. The provided number specifies the last n revisions to clone from the repository.

The git getter accepts both URL-style SSH addresses like git::ssh://git@example.com/foo/bar, and "scp-style" addresses like git::git@example.com/foo/bar. In the latter case, omitting the git:: force prefix is allowed if the username prefix is exactly git@.

The "scp-style" addresses cannot be used in conjunction with the ssh:// scheme prefix, because in that case the colon is used to mark an optional port number to connect on, rather than to delimit the path from the host.

Mercurial (hg)
  • rev - The Mercurial revision to checkout.
HTTP (http)
Basic Authentication

To use HTTP basic authentication with go-getter, simply prepend username:password@ to the hostname in the URL such as https://Aladdin:OpenSesame@www.example.com/index.html. All special characters, including the username and password, must be URL encoded.

Headers

Optional request headers can be added by supplying them in a custom HttpGetter (not as query parameters like most other options). These headers will be sent out on every request the getter in question makes.

S3 (s3)

S3 takes various access configurations in the URL. Note that it will also read these from standard AWS environment variables if they're set. S3 compliant servers like Minio are also supported. If the query parameters are present, these take priority.

  • aws_access_key_id - AWS access key.
  • aws_access_key_secret - AWS access key secret.
  • aws_access_token - AWS access token if this is being used.
  • aws_profile - Use this profile from local ~/.aws/ config. Takes priority over the other three.
Using IAM Instance Profiles with S3

If you use go-getter and want to use an EC2 IAM Instance Profile to avoid using credentials, then just omit these and the profile, if available will be used automatically.

Using S3 with Minio

If you use go-gitter for Minio support, you must consider the following:

  • aws_access_key_id (required) - Minio access key.
  • aws_access_key_secret (required) - Minio access key secret.
  • region (optional - defaults to us-east-1) - Region identifier to use.
  • version (optional - defaults to Minio default) - Configuration file format.
S3 Bucket Examples

S3 has several addressing schemes used to reference your bucket. These are listed here: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro

Some examples for these addressing schemes:

GCS (gcs)
GCS Authentication

In order to access to GCS, authentication credentials should be provided. More information can be found here

GCS Bucket Examples
GCS Testing

The tests for get_gcs.go require you to have GCP credentials set in your environment. These credentials can have any level of permissions to any project, they just need to exist. This means setting GOOGLE_APPLICATION_CREDENTIALS="~/path/to/credentials.json" or GOOGLE_CREDENTIALS="{stringified-credentials-json}". Due to this configuration, get_gcs_test.go will fail for external contributors in CircleCI.

SMB (smb)

There are two options that go-getter will use to download a file in a smb shared folder. The first option uses smbclient and the second one uses the file system to look for a file in a local mount of the shared folder in the OS specific volume folder. go-getter will try to download files from a smb shared folder whenever the url is prefixed with smb://.

⚠️ The smbclient command is available only for Linux. This is the ONLY option for a Linux user and therefore the client must be installed.

The smbclient cli is not available for Windows and MacOS. The go-getter will try to get files using the file system, when this happens the getter uses the FileGetter implementation.

When connecting to a smb server, the OS creates a local mount in a system specific volume folder, and go-getter will try to access the following folders when looking for local mounts.

  • MacOS: /Volumes/<shared_path>
  • Windows: \\<host>\<shared_path>

The following examples work for all the OSes:

  • smb://host/shared/dir (downloads directory content)
  • smb://host/shared/dir/file (downloads file)

The following examples work for Linux:

  • smb://username:password@host/shared/dir (downloads directory content)
  • smb://username@host/shared/dir
  • smb://username:password@host/shared/dir/file (downloads file)
  • smb://username@host/shared/dir/file

⚠️ The above examples also work on the other OSes but the authentication is not used to access the file system.

SMB Testing

The test for get_smb.go requires a smb server running which can be started inside a docker container by running make start-smb. Once the container is up the shared folder can be accessed via smb://<ip|name>/public/<dir|file> or smb://user:password@<ip|name>/private/<dir|file> by another container or machine in the same network.

To run the tests inside get_smb_test.go and client_test.go, prepare the environment with make smbtests-prepare. On prepare some mock files and directories will be added to the shared folder and a go-getter container will start together with the samba server. Once the environment for testing is prepared, run make smbtests to run the tests.

Security Options

Disable Symlinks

In your getter client config, we recommend using the DisableSymlinks option, which prevents writing through or copying from symlinks (which may point outside the directory).

client := getter.Client{
    // This will prevent copying or writing files through symlinks
    DisableSymlinks: true,
}

Disable or Limit X-Terraform-Get

Go-Getter supports arbitrary redirects via the X-Terraform-Get header. This functionality exists to support Terraform use cases, but is likely not needed in most applications.

For code that uses the HttpGetter, add the following configuration options:

var httpGetter = &getter.HttpGetter{
    // Most clients should disable X-Terraform-Get
    // See the note below
    XTerraformGetDisabled: true,
    // Your software probably doesn’t rely on X-Terraform-Get, but
    // if it does, you should set the above field to false, plus
    // set XTerraformGet Limit to prevent endless redirects
    // XTerraformGetLimit: 10,
}

Enforce Timeouts

The HttpGetter supports timeouts and other resource-constraining configuration options. The GitGetter and HgGetter only support timeouts.

Configuration for the HttpGetter:

var httpGetter = &getter.HttpGetter{
    // Disable pre-fetch HEAD requests
    DoNotCheckHeadFirst: true,

    // As an alternative to the above setting, you can
    // set a reasonable timeout for HEAD requests
    // HeadFirstTimeout: 10 * time.Second,
    // Read timeout for HTTP operations
    ReadTimeout: 30 * time.Second,
    // Set the maximum number of bytes
    // that can be read by the getter
    MaxBytes: 500000000, // 500 MB
}

For code that uses the GitGetter or HgGetter, set the Timeout option:

var gitGetter = &getter.GitGetter{
    // Set a reasonable timeout for git operations
    Timeout: 5 * time.Minute,
}

var hgGetter = &getter.HgGetter{
    // Set a reasonable timeout for hg operations
    Timeout: 5 * time.Minute,
}

Documentation

Overview

getter is a package for downloading files or directories from a variety of protocols.

getter is unique in its ability to download both directories and files. It also detects certain source strings to be protocol-specific URLs. For example, "github.com/hashicorp/go-getter/v2" would turn into a Git URL and use the Git protocol.

Protocols and detectors are extensible.

To get started, see Client.

Index

Constants

This section is empty.

Variables

View Source 
var Decompressors = LimitedDecompressors(noFilesLimit, noFileSizeLimit)

Decompressors is the mapping of extension to the Decompressor implementation configured with default settings that will decompress that extension/type.

Note: these decompressors by default do not limit the number of files or the maximum file size created by the decompressed payload.

View Source 
var DefaultClient = &Client{
	Getters:       Getters,
	Decompressors: Decompressors,
}
View Source 
var ErrSymlinkCopy = errors.New("copying of symlinks has been disabled")

ErrSymlinkCopy means that a copy of a symlink was encountered on a request with DisableSymlinks enabled.

View Source 
var ErrUnauthorized = os.ErrPermission
View Source 
var Getters []Getter

Getters is the mapping of scheme to the Getter implementation that will be used to get a dependency.

View Source 
var SymlinkAny = os.Symlink

Functions

func Copy 

func Copy(ctx context.Context, dst io.Writer, src io.Reader) (int64, error)

Copy is a io.Copy cancellable by context

func Detect 

func Detect(req *Request, getter Getter) (bool, error)

Detect is a method used to detect if a Getter is a candidate for downloading an artifact by calling the Getter.Detect(*Request) method

func LimitedDecompressors added in v2.2.1 

func LimitedDecompressors(filesLimit int, fileSizeLimit int64) map[string]Decompressor

LimitedDecompressors creates the set of Decompressors, but with each compressor configured with the given filesLimit and/or fileSizeLimit where applicable.

func NewContextWithClient added in v2.1.0 

func NewContextWithClient(ctx context.Context, client *Client) context.Context

func SourceDirSubdir 

func SourceDirSubdir(src string) (string, string)

SourceDirSubdir takes a source URL and returns a tuple of the URL without the subdir and the subdir.

ex: 

dom.com/path/?q=p               => dom.com/path/?q=p, ""
proto://dom.com/path//*?q=p     => proto://dom.com/path?q=p, "*"
proto://dom.com/path//path2?q=p => proto://dom.com/path?q=p, "path2"

func SubdirGlob 

func SubdirGlob(dst, subDir string) (string, error)

SubdirGlob returns the actual subdir with globbing processed.

dst should be a destination directory that is already populated (the download is complete) and subDir should be the set subDir. If subDir is an empty string, this returns an empty string.

The returned path is the full absolute path.

func TestDecompressor 

func TestDecompressor(t testing.T, d Decompressor, cases []TestDecompressCase)

TestDecompressor is a helper function for testing generic decompressors.

Types

type BitBucketDetector 

type BitBucketDetector struct{}

BitBucketDetector implements Detector to detect BitBucket URLs and turn them into URLs that the Git or Hg Getter can understand.

func (*BitBucketDetector) Detect 

func (d *BitBucketDetector) Detect(src, _ string) (string, bool, error)

type Bzip2Decompressor 

type Bzip2Decompressor struct {
	// FileSizeLimit limits the size of a decompressed file.
	//
	// The zero value means no limit.
	FileSizeLimit int64
}

Bzip2Decompressor is an implementation of Decompressor that can decompress bz2 files.

func (*Bzip2Decompressor) Decompress 

func (d *Bzip2Decompressor) Decompress(dst, src string, dir bool, umask os.FileMode) error

type ChecksumError 

type ChecksumError struct {
	Hash     hash.Hash
	Actual   []byte
	Expected []byte
	File     string
}

A ChecksumError is returned when a checksum differs

func (*ChecksumError) Error 

func (cerr *ChecksumError) Error() string

type Client 

type Client struct {
	// Decompressors is the map of decompressors supported by this client.
	// If this is nil, then the default value is the Decompressors global.
	Decompressors map[string]Decompressor

	// Getters is the list of protocols supported by this client. If this
	// is nil, then the default Getters variable will be used.
	Getters []Getter

	// Disable symlinks is used to prevent copying or writing files through symlinks for Get requests.
	// When set to true any copying or writing through symlinks will result in a ErrSymlinkCopy error.
	DisableSymlinks bool
}

Client is a client for downloading things.

Top-level functions such as Get are shortcuts for interacting with a client. Using a client directly allows more fine-grained control over how downloading is done, as well as customizing the protocols supported.

func ClientFromContext added in v2.1.0 

func ClientFromContext(ctx context.Context) *Client

func (*Client) Get 

func (c *Client) Get(ctx context.Context, req *Request) (*GetResult, error)

Get downloads the configured source to the destination.

func (*Client) GetChecksum 

func (c *Client) GetChecksum(ctx context.Context, req *Request) (*FileChecksum, error)

GetChecksum extracts the checksum from the `checksum` parameter of the src of the Request ex: 

http://hashicorp.com/terraform?checksum=<checksumValue>
http://hashicorp.com/terraform?checksum=<checksumType>:<checksumValue>
http://hashicorp.com/terraform?checksum=file:<checksum_url>

when the checksum is in a file, GetChecksum will first client.Get it in a temporary directory, parse the content of the file and finally delete it. The content of a checksum file is expected to be BSD style or GNU style. For security reasons GetChecksum does not try to get the current working directory and as a result, relative files will only be found when Request.Pwd is set.

BSD-style checksum: 

MD5 (file1) = <checksum>
MD5 (file2) = <checksum>

GNU-style: 

<checksum>  file1
<checksum> *file2

type Decompressor 

type Decompressor interface {
	// Decompress should decompress src to dst. dir specifies whether dst
	// is a directory or single file. src is guaranteed to be a single file
	// that exists. dst is not guaranteed to exist already.
	Decompress(dst, src string, dir bool, umask os.FileMode) error
}

Decompressor defines the interface that must be implemented to add support for decompressing a type.

Important: if you're implementing a decompressor, please use the containsDotDot helper in this file to ensure that files can't be decompressed outside of the specified directory.

type Detector 

type Detector interface {
	// Detect will detect whether the string matches a known pattern to
	// turn it into a proper URL.
	Detect(string, string) (string, bool, error)
}

Detector defines the interface that an invalid URL or a URL with a blank scheme is passed through in order to determine if its shorthand for something else well-known.

type FileChecksum 

type FileChecksum struct {
	Type     string
	Hash     hash.Hash
	Value    []byte
	Filename string
}

FileChecksum helps verifying the checksum for a file.

func (*FileChecksum) Checksum 

func (c *FileChecksum) Checksum(filePath string) error

Checksum computes the Checksum for filePath using the hashing algorithm from c.Hash and compares it to c.Value. If those values differ a ChecksumError will be returned.

func (*FileChecksum) String 

func (c *FileChecksum) String() string

String returns the hash type and the hash separated by a colon, for example: 

"md5:090992ba9fd140077b0661cb75f7ce13"
"sha1:ebfb681885ddf1234c18094a45bbeafd91467911"

type FileDetector 

type FileDetector struct{}

FileDetector implements Detector to detect file paths.

func (*FileDetector) Detect 

func (d *FileDetector) Detect(src, pwd string) (string, bool, error)

type FileGetter 

type FileGetter struct{}

FileGetter is a Getter implementation that will download a module from a file scheme.

func (*FileGetter) Detect 

func (g *FileGetter) Detect(req *Request) (bool, error)

func (*FileGetter) Get 

func (g *FileGetter) Get(ctx context.Context, req *Request) error

func (*FileGetter) GetFile 

func (g *FileGetter) GetFile(ctx context.Context, req *Request) error

func (*FileGetter) Mode 

func (g *FileGetter) Mode(ctx context.Context, u *url.URL) (Mode, error)

type FolderStorage 

type FolderStorage struct {
	// StorageDir is the directory where the modules will be stored.
	StorageDir string
}

FolderStorage is an implementation of the Storage interface that manages modules on the disk.

func (*FolderStorage) Dir 

func (s *FolderStorage) Dir(key string) (d string, e bool, err error)

Dir implements Storage.Dir

func (*FolderStorage) Get 

func (s *FolderStorage) Get(ctx context.Context, key string, source string, update bool) error

Get implements Storage.Get

type GetResult 

type GetResult struct {
	// Local destination of the gotten object.
	Dst string
}

GetResult is the result of a Client.Get

func Get 

func Get(ctx context.Context, dst, src string) (*GetResult, error)

Get downloads the directory specified by src into the folder specified by dst. If dst already exists, Get will attempt to update it.

src is a URL, whereas dst is always just a file path to a folder. This folder doesn't need to exist. It will be created if it doesn't exist.

func GetAny 

func GetAny(ctx context.Context, dst, src string) (*GetResult, error)

GetAny downloads a URL into the given destination. Unlike Get or GetFile, both directories and files are supported.

dst must be a directory. If src is a file, it will be downloaded into dst with the basename of the URL. If src is a directory or archive, it will be unpacked directly into dst.

func GetFile 

func GetFile(ctx context.Context, dst, src string) (*GetResult, error)

GetFile downloads the file specified by src into the path specified by dst.

type Getter 

type Getter interface {
	// Get downloads the given URL into the given directory. This always
	// assumes that we're updating and gets the latest version that it can.
	//
	// The directory may already exist (if we're updating). If it is in a
	// format that isn't understood, an error should be returned. Get shouldn't
	// simply nuke the directory.
	Get(context.Context, *Request) error

	// GetFile downloads the give URL into the given path. The URL must
	// reference a single file. If possible, the Getter should check if
	// the remote end contains the same file and no-op this operation.
	GetFile(context.Context, *Request) error

	// Mode returns the mode based on the given URL. This is used to
	// allow clients to let the getters decide which mode to use.
	Mode(context.Context, *url.URL) (Mode, error)

	// Detect detects whether the Request.Src matches a known pattern to
	// turn it into a proper URL, and also transforms and update Request.Src
	// when necessary.
	// The Getter must validate if the Request.Src is a valid URL
	// with a valid scheme for the Getter, and also check if the
	// current Getter is the forced one and return true if that's the case.
	Detect(*Request) (bool, error)
}

Getter defines the interface that schemes must implement to download things.

type GitDetector 

type GitDetector struct{}

GitDetector implements Detector to detect Git SSH URLs such as git@host.com:dir1/dir2 and converts them to proper URLs.

func (*GitDetector) Detect 

func (d *GitDetector) Detect(src, _ string) (string, bool, error)

type GitGetter 

type GitGetter struct {
	Detectors []Detector

	// Timeout sets a deadline which all git CLI operations should
	// complete within. Defaults to zero which means no timeout.
	Timeout time.Duration
}

GitGetter is a Getter implementation that will download a module from a git repository.

func (*GitGetter) Detect 

func (g *GitGetter) Detect(req *Request) (bool, error)

func (*GitGetter) Get 

func (g *GitGetter) Get(ctx context.Context, req *Request) error

func (*GitGetter) GetFile 

func (g *GitGetter) GetFile(ctx context.Context, req *Request) error

GetFile for Git doesn't support updating at this time. It will download the file every time.

func (*GitGetter) Mode 

func (g *GitGetter) Mode(_ context.Context, u *url.URL) (Mode, error)

type GitHubDetector 

type GitHubDetector struct{}

GitHubDetector implements Detector to detect GitHub URLs and turn them into URLs that the Git Getter can understand.

func (*GitHubDetector) Detect 

func (d *GitHubDetector) Detect(src, _ string) (string, bool, error)

type GitLabDetector 

type GitLabDetector struct{}

GitLabDetector implements Detector to detect GitLab URLs and turn them into URLs that the Git Getter can understand.

func (*GitLabDetector) Detect 

func (d *GitLabDetector) Detect(src, _ string) (string, bool, error)

type GzipDecompressor 

type GzipDecompressor struct {
	// FileSizeLimit limits the size of a decompressed file.
	//
	// The zero value means no limit.
	FileSizeLimit int64
}

GzipDecompressor is an implementation of Decompressor that can decompress gzip files.

func (*GzipDecompressor) Decompress 

func (d *GzipDecompressor) Decompress(dst, src string, dir bool, umask os.FileMode) error

type HgGetter 

type HgGetter struct {

	// Timeout sets a deadline which all hg CLI operations should
	// complete within. Defaults to zero which means no timeout.
	Timeout time.Duration
}

HgGetter is a Getter implementation that will download a module from a Mercurial repository.

func (*HgGetter) Detect 

func (g *HgGetter) Detect(req *Request) (bool, error)

func (*HgGetter) Get 

func (g *HgGetter) Get(ctx context.Context, req *Request) error

func (*HgGetter) GetFile 

func (g *HgGetter) GetFile(ctx context.Context, req *Request) error

GetFile for Hg doesn't support updating at this time. It will download the file every time.

func (*HgGetter) Mode 

func (g *HgGetter) Mode(ctx context.Context, _ *url.URL) (Mode, error)

type HttpGetter 

type HttpGetter struct {

	// Netrc, if true, will lookup and use auth information found
	// in the user's netrc file if available.
	Netrc bool

	// Client is the http.Client to use for Get requests.
	// This defaults to a cleanhttp.DefaultClient if left unset.
	Client *http.Client

	// Header contains optional request header fields that should be included
	// with every HTTP request. Note that the zero value of this field is nil,
	// and as such it needs to be initialized before use, via something like
	// make(http.Header).
	Header http.Header
	// DoNotCheckHeadFirst configures the client to NOT check if the server
	// supports HEAD requests.
	DoNotCheckHeadFirst bool

	// HeadFirstTimeout configures the client to enforce a timeout when
	// the server supports HEAD requests.
	//
	// The zero value means no timeout.
	HeadFirstTimeout time.Duration

	// ReadTimeout configures the client to enforce a timeout when
	// making a request to an HTTP server and reading its response body.
	//
	// The zero value means no timeout.
	ReadTimeout time.Duration

	// MaxBytes limits the number of bytes that will be ready from an HTTP
	// response body returned from a server. The zero value means no limit.
	MaxBytes int64

	// XTerraformGetLimit configures how many times the client with follow
	// the " X-Terraform-Get" header value.
	//
	// The zero value means no limit.
	XTerraformGetLimit int

	// XTerraformGetDisabled disables the client's usage of the "X-Terraform-Get"
	// header value.
	XTerraformGetDisabled bool
}

HttpGetter is a Getter implementation that will download from an HTTP endpoint.

For file downloads, HTTP is used directly.

The protocol for downloading a directory from an HTTP endpoint is as follows:

An HTTP GET request is made to the URL with the additional GET parameter "terraform-get=1". This lets you handle that scenario specially if you wish. The response must be a 2xx.

First, a header is looked for "X-Terraform-Get" which should contain a source URL to download. This source must use one of the configured protocols and getters for the client, or "http"/"https" if using the HttpGetter directly.

If the header is not present, then a meta tag is searched for named "terraform-get" and the content should be a source URL.

The source URL, whether from the header or meta tag, must be a fully formed URL. The shorthand syntax of "github.com/foo/bar" or relative paths are not allowed.

func (*HttpGetter) Detect 

func (g *HttpGetter) Detect(req *Request) (bool, error)

func (*HttpGetter) Get 

func (g *HttpGetter) Get(ctx context.Context, req *Request) error

func (*HttpGetter) GetFile 

func (g *HttpGetter) GetFile(ctx context.Context, req *Request) error

GetFile fetches the file from src and stores it at dst. If the server supports Accept-Range, HttpGetter will attempt a range request. This means it is the caller's responsibility to ensure that an older version of the destination file does not exist, else it will be either falsely identified as being replaced, or corrupted with extra bytes appended.

func (*HttpGetter) Mode 

func (g *HttpGetter) Mode(ctx context.Context, u *url.URL) (Mode, error)

type MockGetter 

type MockGetter struct {

	// Proxy, if set, will be called after recording the calls below.
	// If it isn't set, then the *Err values will be returned.
	Proxy Getter

	GetCalled bool
	GetDst    string
	GetURL    *url.URL
	GetErr    error

	GetFileCalled bool
	GetFileDst    string
	GetFileURL    *url.URL
	GetFileErr    error
}

MockGetter is an implementation of Getter that can be used for tests.

func (*MockGetter) Detect 

func (g *MockGetter) Detect(req *Request) (bool, error)

func (*MockGetter) Get 

func (g *MockGetter) Get(ctx context.Context, req *Request) error

func (*MockGetter) GetFile 

func (g *MockGetter) GetFile(ctx context.Context, req *Request) error

func (*MockGetter) Mode 

func (g *MockGetter) Mode(ctx context.Context, u *url.URL) (Mode, error)

type Mode 

type Mode uint

Mode is the mode that the client operates in. 

const (
	ModeInvalid Mode = iota

	// ModeAny downloads anything it can. In this mode, dst must
	// be a directory. If src is a file, it is saved into the directory
	// with the basename of the URL. If src is a directory or archive,
	// it is unpacked directly into dst.
	ModeAny

	// ModeFile downloads a single file. In this mode, dst must
	// be a file path (doesn't have to exist). src must point to a single
	// file. It is saved as dst.
	ModeFile

	// ModeDir downloads a directory. In this mode, dst must be
	// a directory path (doesn't have to exist). src must point to an
	// archive or directory (such as in s3).
	ModeDir
)

type ProgressTracker 

type ProgressTracker interface {
	// TrackProgress should be called when
	// a new object is being downloaded.
	// src is the location the file is
	// downloaded from.
	// currentSize is the current size of
	// the file in case it is a partial
	// download.
	// totalSize is the total size in bytes,
	// size can be zero if the file size
	// is not known.
	// stream is the file being downloaded, every
	// written byte will add up to processed size.
	//
	// TrackProgress returns a ReadCloser that wraps the
	// download in progress ( stream ).
	// When the download is finished, body shall be closed.
	TrackProgress(src string, currentSize, totalSize int64, stream io.ReadCloser) (body io.ReadCloser)
}

ProgressTracker allows to track the progress of downloads.

type Request 

type Request struct {
	// Src is the source URL to get.
	//
	// Dst is the path to save the downloaded thing as. If Dir is set to
	// true, then this should be a directory. If the directory doesn't exist,
	// it will be created for you.
	//
	// Pwd is the working directory for detection. If this isn't set, some
	// detection may fail. Client will not default pwd to the current
	// working directory for security reasons.
	Src string
	Dst string
	Pwd string

	// Forced is the forced getter detected in the Src string during the
	// Getter detection. Forcing a getter means that go-getter will try
	// to download the artifact only with the Getter that is being forced.
	//
	// For example:
	//
	// Request.Src                                          Forced
	// git::ssh://git@git.example.com:2222/foo/bar.git      git
	//
	// This field is used by the Getters to validate when they are forced to download
	// the artifact.
	// If both Request.Src and Forced contains a forced getter, the one in the Request.Src will
	// be considered and will override the value of this field.
	Forced string

	// Umask is used to mask file permissions when storing local files or
	// decompressing an archive
	Umask os.FileMode

	// GetMode is the method of download the client will use. See Mode for
	// documentation.
	GetMode Mode

	// Copy, in local file mode if set to true, will copy data instead of using
	// a symlink. If false, attempts to symlink to speed up the operation and
	// to lower the disk space usage. If the symlink fails, may attempt to copy
	// on windows.
	Copy bool

	// Inplace, in local file mode if set to true, do nothing and the returned
	// operation will simply contain the source file path. Inplace has precedence
	// over Copy.
	Inplace bool

	// ProgressListener allows to track file downloads.
	// By default a no op progress listener is used.
	ProgressListener ProgressTracker

	// Disable symlinks is used to prevent copying or writing files through symlinks.
	// When set to true any copying or writing through symlinks will result in a ErrSymlinkCopy error.
	DisableSymlinks bool
	// contains filtered or unexported fields
}

func (*Request) CopyReader 

func (req *Request) CopyReader(dst string, src io.Reader, fmode os.FileMode, fileSizeLimit int64) error

func (*Request) Mode 

func (req *Request) Mode(m os.FileMode) os.FileMode

Mode returns file Mode umasked by the Request umask

func (*Request) URL 

func (req *Request) URL() *url.URL

type SmbClientGetter 

type SmbClientGetter struct {

	// Timeout in seconds sets a deadline which all smb client CLI operations should
	// complete within. Defaults to zero which means to use the default client timeout of 20 seconds.
	Timeout int
}

SmbClientGetter is a Getter implementation that will download a module from a shared folder using smbclient cli.

func (*SmbClientGetter) Detect 

func (g *SmbClientGetter) Detect(req *Request) (bool, error)

func (*SmbClientGetter) Get 

func (g *SmbClientGetter) Get(ctx context.Context, req *Request) error

func (*SmbClientGetter) GetFile 

func (g *SmbClientGetter) GetFile(ctx context.Context, req *Request) error

func (*SmbClientGetter) Mode 

func (g *SmbClientGetter) Mode(ctx context.Context, u *url.URL) (Mode, error)

type SmbMountGetter 

type SmbMountGetter struct{}

SmbMountGetter is a Getter implementation that will download an artifact from a shared folder using the file system using FileGetter implementation. For Unix and MacOS users, the Getter will look for usual system specific mount paths such as: /Volumes/ for MacOS /run/user/1000/gvfs/smb-share:server=<hostIP>,share=<path> for Unix

func (*SmbMountGetter) Detect 

func (g *SmbMountGetter) Detect(req *Request) (bool, error)

func (*SmbMountGetter) Get 

func (g *SmbMountGetter) Get(ctx context.Context, req *Request) error

func (*SmbMountGetter) GetFile 

func (g *SmbMountGetter) GetFile(ctx context.Context, req *Request) error

func (*SmbMountGetter) Mode 

func (g *SmbMountGetter) Mode(ctx context.Context, u *url.URL) (Mode, error)

type Storage 

type Storage interface {
	// Dir returns the directory on local disk where the directory source
	// can be loaded from.
	Dir(string) (string, bool, error)

	// Get will download and optionally update the given directory.
	Get(context.Context, string, string, bool) error
}

Storage is an interface that knows how to lookup downloaded directories as well as download and update directories from their sources into the proper location.

type TarBzip2Decompressor 

type TarBzip2Decompressor struct {
	// FileSizeLimit limits the total size of all
	// decompressed files.
	//
	// The zero value means no limit.
	FileSizeLimit int64

	// FilesLimit limits the number of files that are
	// allowed to be decompressed.
	//
	// The zero value means no limit.
	FilesLimit int
}

TarBzip2Decompressor is an implementation of Decompressor that can decompress tar.bz2 files.

func (*TarBzip2Decompressor) Decompress 

func (d *TarBzip2Decompressor) Decompress(dst, src string, dir bool, umask os.FileMode) error

type TarDecompressor added in v2.0.1 

type TarDecompressor struct {
	// FileSizeLimit limits the total size of all
	// decompressed files.
	//
	// The zero value means no limit.
	FileSizeLimit int64

	// FilesLimit limits the number of files that are
	// allowed to be decompressed.
	//
	// The zero value means no limit.
	FilesLimit int
}

TarDecompressor is an implementation of Decompressor that can unpack tar files.

func (*TarDecompressor) Decompress added in v2.0.1 

func (d *TarDecompressor) Decompress(dst, src string, dir bool, umask os.FileMode) error

type TarGzipDecompressor 

type TarGzipDecompressor struct {
	// FileSizeLimit limits the total size of all
	// decompressed files.
	//
	// The zero value means no limit.
	FileSizeLimit int64

	// FilesLimit limits the number of files that are
	// allowed to be decompressed.
	//
	// The zero value means no limit.
	FilesLimit int
}

TarGzipDecompressor is an implementation of Decompressor that can decompress tar.gzip files.

func (*TarGzipDecompressor) Decompress 

func (d *TarGzipDecompressor) Decompress(dst, src string, dir bool, umask os.FileMode) error

type TarXzDecompressor 

type TarXzDecompressor struct {
	// FileSizeLimit limits the total size of all
	// decompressed files.
	//
	// The zero value means no limit.
	FileSizeLimit int64

	// FilesLimit limits the number of files that are
	// allowed to be decompressed.
	//
	// The zero value means no limit.
	FilesLimit int
}

TarXzDecompressor is an implementation of Decompressor that can decompress tar.xz files.

func (*TarXzDecompressor) Decompress 

func (d *TarXzDecompressor) Decompress(dst, src string, dir bool, umask os.FileMode) error

type TarZstdDecompressor added in v2.2.0 

type TarZstdDecompressor struct {
	// FileSizeLimit limits the total size of all
	// decompressed files.
	//
	// The zero value means no limit.
	FileSizeLimit int64

	// FilesLimit limits the number of files that are
	// allowed to be decompressed.
	//
	// The zero value means no limit.
	FilesLimit int
}

TarZstdDecompressor is an implementation of Decompressor that can decompress tar.zstd files.

func (*TarZstdDecompressor) Decompress added in v2.2.0 

func (d *TarZstdDecompressor) Decompress(dst, src string, dir bool, umask os.FileMode) error

type TestDecompressCase 

type TestDecompressCase struct {
	Input   string     // Input is the complete path to the input file
	Dir     bool       // Dir is whether or not we're testing directory mode
	Err     bool       // Err is whether we expect an error or not
	DirList []string   // DirList is the list of files for Dir mode
	FileMD5 string     // FileMD5 is the expected MD5 for a single file
	Mtime   *time.Time // Mtime is the optionally expected mtime for a single file (or all files if in Dir mode)
}

TestDecompressCase is a single test case for testing decompressors

type XzDecompressor 

type XzDecompressor struct {
	// FileSizeLimit limits the size of a decompressed file.
	//
	// The zero value means no limit.
	FileSizeLimit int64
}

XzDecompressor is an implementation of Decompressor that can decompress xz files.

func (*XzDecompressor) Decompress 

func (d *XzDecompressor) Decompress(dst, src string, dir bool, umask os.FileMode) error

type ZipDecompressor 

type ZipDecompressor struct {
	// FileSizeLimit limits the total size of all
	// decompressed files.
	//
	// The zero value means no limit.
	FileSizeLimit int64

	// FilesLimit limits the number of files that are
	// allowed to be decompressed.
	//
	// The zero value means no limit.
	FilesLimit int
}

ZipDecompressor is an implementation of Decompressor that can decompress zip files.

func (*ZipDecompressor) Decompress 

func (d *ZipDecompressor) Decompress(dst, src string, dir bool, umask os.FileMode) error

type ZstdDecompressor added in v2.2.0 

type ZstdDecompressor struct {
	// FileSizeLimit limits the size of a decompressed file.
	//
	// The zero value means no limit.
	FileSizeLimit int64
}

ZstdDecompressor is an implementation of Decompressor that can decompress .zst files.

func (*ZstdDecompressor) Decompress added in v2.2.0 

func (d *ZstdDecompressor) Decompress(dst, src string, dir bool, umask os.FileMode) error

Source Files

View all Source files

Directories

Path Synopsis
helper
testing
url

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.