stashcp

package module
v6.12.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 6, 2023 License: Apache-2.0 Imports: 36 Imported by: 0

README

Open Science Data Federation (OSDF) Client

Version Go Mod Builds

The OSDF Client downloads files from the Open Science Data Federation through a series of caches. The OSDF is used by multiple organizations to effeciently transfer input and output data.

The client consists of two parts: a command-line tool named StashCP, and an HTCondor file transfer plugin.

When configured, this plugin will allow the user to specify transfer_input_files with the stash:// protocol which will be downloaded through the OSDF caches. An example of a submit file:

...
transfer_input_files = stash:///osgconnect/public/dweitzel/blast/queries/query1
...

Note: This repo is the continuation of opensciencegrid/stashcp; visit that repo for old issues and releases.

Building

Building is performed with the goreleaser tool. To build a snapshot (not release):

$ goreleaser --clean --snapshot

The binaries will be located in ./dist directory.

Testing and Usage

Run this simple command to download a test file

$ ./stashcp /osgconnect/public/dweitzel/blast/queries/query1 ./

Configuration

stashcp is affected by the environment variables:

Environment Variable Description
OSG_DISABLE_PROXY_FALLBACK Do not disable using proxies. By default, stashcp will try to use an HTTP proxy when connecting to a cache. If this environment variable is set (no value necessary, only if it's set), then stashcp will not fallback to no proxy if the proxy download fails.
STASHCP_MINIMUM_DOWNLOAD_SPEED The lower limit a download will be cancelled, in bytes per second
STASH_NAMESPACE_URL The URL to download the namespace and cache information. Default: https://topology.opensciencegrid.org/stashcache/namespaces

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var CacheOverride bool

CacheOverride

View Source 
var CachesJsonLocation string
View Source 
var CachesToTry int = 3

Number of caches to attempt to use in any invocation

View Source 
var NearestCache string

Nearest cache

View Source 
var NearestCacheList []string

List of caches, in order from closest to furthest

View Source 
var UploadClient = &http.Client{}

Functions

func AcquireToken added in v6.10.0 

func AcquireToken(destination *url.URL, namespace Namespace, isWrite bool) (string, error)

Given a URL and a piece of the namespace, attempt to acquire a valid token for that URL.

func AddError 

func AddError(err error) bool

AddError will add an accumulated error to the error stack

func ByteCountSI 

func ByteCountSI(b int64) string

func CanDisableProxy added in v6.10.0 

func CanDisableProxy() bool

Determine whether we are allowed to skip the proxy as a fallback

func ClearErrors added in v6.10.0 

func ClearErrors()

func DoStashCPSingle 

func DoStashCPSingle(sourceFile string, destination string, methods []string, recursive bool) (bytesTransferred int64, err error)

Start the transfer, whether read or write back

func DownloadHTTP 

func DownloadHTTP(transfer TransferDetails, dest string, token string) (int64, error)

DownloadHTTP - Perform the actual download of the file

func ErrorsRetryable 

func ErrorsRetryable() bool

ErrorsRetryable returns if the errors in the stack are retryable later

func Find 

func Find(slice []string, val string) (int, bool)

Find takes a slice and looks for an element in it. If found it will return it's key, otherwise it will return -1 and a bool of false. From https://golangcode.com/check-if-element-exists-in-slice/

func GetBestCache added in v6.10.0 

func GetBestCache(cacheListName string) ([]string, error)

func GetCacheHostnames added in v6.10.0 

func GetCacheHostnames(testFile string) (urls []string, err error)

func GetErrors 

func GetErrors() string

func HasPort 

func HasPort(host string) bool

HasPort test the host if it includes a port

func IsDir 

func IsDir(dirUrl *url.URL, token string, namespace Namespace) (bool, error)

func IsProxyEnabled added in v6.10.0 

func IsProxyEnabled() bool

Determines whether or not we can interact with the site HTTP proxy

func IsRetryable 

func IsRetryable(err error) bool

IsRetryable will return true if the error is retryable

func RegisterClient added in v6.10.0 

func RegisterClient(namespace Namespace) (*config.PrefixEntry, error)

func TokenIsAcceptable added in v6.10.0 

func TokenIsAcceptable(jwtSerialized string, osdfPath string, namespace Namespace, isWrite bool) bool

func TokenIsExpired added in v6.10.0 

func TokenIsExpired(jwtSerialized string) bool

func UploadFile 

func UploadFile(src string, dest *url.URL, token string, namespace Namespace) (int64, error)

UploadFile Uploads a file using HTTP

Types

type Cache 

type Cache struct {
	AuthEndpoint string `json:"auth_endpoint"`
	Endpoint     string `json:"endpoint"`
	Resource     string `json:"resource"`
}

Cache

func GetCachesFromNamespace added in v6.10.0 

func GetCachesFromNamespace(namespace Namespace) (caches []Cache, err error)

type ConnectionSetupError 

type ConnectionSetupError struct {
	URL string
	Err error
}

ConnectionSetupError is an error that is returned when a connection to the remote server fails

func (*ConnectionSetupError) Error 

func (e *ConnectionSetupError) Error() string

func (*ConnectionSetupError) Is 

func (e *ConnectionSetupError) Is(target error) bool

func (*ConnectionSetupError) Unwrap 

func (e *ConnectionSetupError) Unwrap() error

type CredentialGeneration added in v6.12.0 

type CredentialGeneration struct {
	Issuer        *string `json:"issuer"`
	MaxScopeDepth *int    `json:"max_scope_depth"`
	Strategy      *string `json:"strategy"`
	VaultServer   *string `json:"vault_server"`
}

Credential generation information

type FileDownloadError added in v6.10.0 

type FileDownloadError struct {
	Text string
	Err  error
}

func (*FileDownloadError) Error added in v6.10.0 

func (e *FileDownloadError) Error() string

func (*FileDownloadError) Unwrap added in v6.10.0 

func (e *FileDownloadError) Unwrap() error

type Namespace 

type Namespace struct {
	Caches         []Cache               `json:"caches"`
	Path           string                `json:"path"`
	CredentialGen  *CredentialGeneration `json:"credential_generation"`
	ReadHTTPS      bool                  `json:"readhttps"`
	UseTokenOnRead bool                  `json:"usetokenonread"`
	WriteBackHost  string                `json:"writebackhost"`
	DirListHost    string                `json:"dirlisthost"`
}

Namespace holds the structure of stash namespaces

func GetNamespaces 

func GetNamespaces() ([]Namespace, error)

GetNamespaces returns the list of namespaces

func MatchNamespace 

func MatchNamespace(path string) (Namespace, error)

MatchNamespace matches the namespace passed in to the namespaces in the list

func (*Namespace) GetCacheHosts 

func (ns *Namespace) GetCacheHosts() []string

func (*Namespace) GetCaches 

func (ns *Namespace) GetCaches() []Cache

GetCaches returns the list of caches for the namespace

func (*Namespace) MatchCaches 

func (ns *Namespace) MatchCaches(caches []string) []Cache

MatchCaches compares the caches passed in (presumably from an ordered list of caches) to the caches for the namespace, and returns the intersection of the two

type NamespaceFull 

type NamespaceFull struct {
	Caches     []Cache     `json:"caches"`
	Namespaces []Namespace `json:"namespaces"`
}

type OptionsStruct 

type OptionsStruct struct {
	ProgressBars bool
	Recursive    bool
	Token        string
	Version      string
}
var Options OptionsStruct

type ProgressReader 

type ProgressReader struct {
	// contains filtered or unexported fields
}

ProgressReader wraps the io.Reader to get progress Adapted from https://stackoverflow.com/questions/26050380/go-tracking-post-request-progress

func (*ProgressReader) Close 

func (pr *ProgressReader) Close() error

Close implments the close function of io.Closer

func (*ProgressReader) Read 

func (pr *ProgressReader) Read(p []byte) (n int, err error)

Read implements the common read function for io.Reader

type SlowTransferError 

type SlowTransferError struct {
	BytesTransferred int64
	BytesPerSecond   int64
	BytesTotal       int64
	Duration         time.Duration
}

SlowTransferError is an error that is returned when a transfer takes longer than the configured timeout

func (*SlowTransferError) Error 

func (e *SlowTransferError) Error() string

func (*SlowTransferError) Is 

func (e *SlowTransferError) Is(target error) bool

type StoppedTransferError added in v6.11.0 

type StoppedTransferError struct {
	Err string
}

func (*StoppedTransferError) Error added in v6.11.0 

func (e *StoppedTransferError) Error() string

type TimestampedError added in v6.10.0 

type TimestampedError struct {
	// contains filtered or unexported fields
}

type TransferDetails 

type TransferDetails struct {
	// Url is the url.URL of the cache and port
	Url url.URL

	// Proxy specifies if a proxy should be used
	Proxy bool
}

func NewTransferDetails 

func NewTransferDetails(cache Cache, https bool) []TransferDetails

NewTransferDetails creates the TransferDetails struct with the given cache

type TransferResults 

type TransferResults struct {
	Error      error
	Downloaded int64
}

Source Files

View all Source files

Directories

Path Synopsis
cmd

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.