cmd

Documentation

Overview

Copyright © Microsoft <wastore@microsoft.com>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Index

• Constants
• Variables
• func Execute(azsAppPathFolder, logPathFolder string, jobPlanFolder string, ...)
• func GetAccountRoot(resource common.ResourceString, location common.Location) (string, error)
• func GetContainerName(path string, location common.Location) (string, error)
• func GetCredTypeFromEnvVar() common.CredentialType
• func GetOAuthTokenManagerInstance() (*common.UserOAuthTokenManager, error)
• func GetResourceRoot(resource string, location common.Location) (resourceBase string, err error)
• func GetUserOAuthTokenManagerInstance() *common.UserOAuthTokenManager
• func HandleListJobsCommand(jobStatus common.JobStatus) error
• func HandlePauseCommand(jobIdString string)
• func HandleShowCommand(listRequest common.ListRequest) error
• func IsDestinationCaseInsensitive(fromTo common.FromTo) bool
• func ParseSizeString(s string, name string) (int64, error)
• func PrintExistingJobIds(listJobResponse common.ListJobsResponse) error
• func PrintJobProgressSummary(summary common.ListJobSummaryResponse)
• func PrintJobTransfers(listTransfersResponse common.ListJobTransfersResponse)
• func SplitResourceString(raw string, loc common.Location) (common.ResourceString, error)
• func UnfurlSymlinks(symlinkPath string) (result string, err error)
• func WalkWithSymlinks(fullPath string, walkFunc filepath.WalkFunc, followSymlinks bool) (err error)
• func WarnStdoutAndScanningLog(toLog string)
• func WrapFolder(fullpath string, stat os.FileInfo) (os.FileInfo, error)
• type BlobFSAccountTraverser
• type BucketToContainerNameResolver
• type GCPBucketNameToAzureResourcesResolver
  • func NewGCPBucketNameToAzureResourcesResolver(gcpBucketNames []string) *GCPBucketNameToAzureResourcesResolver
  • func (resolver *GCPBucketNameToAzureResourcesResolver) ResolveName(bucketName string) (string, error)
• type HTTPClient
  • func NewHttpClient(url string) *HTTPClient
• type ListReq
• type ListResponse
• type LocationLevel
  • func (LocationLevel) Container() LocationLevel
  • func (LocationLevel) Object() LocationLevel
  • func (LocationLevel) Service() LocationLevel
• type S3BucketNameToAzureResourcesResolver
  • func NewS3BucketNameToAzureResourcesResolver(s3BucketNames []string) *S3BucketNameToAzureResourcesResolver
  • func (s3Resolver *S3BucketNameToAzureResourcesResolver) ResolveName(bucketName string) (string, error)
• type Version
  • func NewVersion(raw string) (*Version, error)
  • func (v Version) NewerThan(v2 Version) bool
  • func (v Version) OlderThan(v2 Version) bool

Constants

const (
	NumOfFilesPerDispatchJobPart = 10000
)

const PreservePermissionsFlag = "preserve-permissions"

Variables

var FinalPartCreatedMessage = "Final job part has been created"

var IPv4Regex = regexp.MustCompile(`\d+\.\d+\.\d+\.\d+`) // simple regex

var NothingScheduledError = errors.New("no transfers were scheduled because no files matched the specified criteria")

var NothingToRemoveError = errors.New("nothing found to remove")

var Rpc = func(cmd common.RpcCmd, request interface{}, response interface{}) {
	err := inprocSend(cmd, request, response)
	common.PanicIfErr(err)
}

Global singleton for sending RPC requests from the frontend to the STE

Functions

func Execute

func Execute(azsAppPathFolder, logPathFolder string, jobPlanFolder string, maxFileAndSocketHandles int)

Execute adds all child commands to the root command and sets flags appropriately. This is called by main.main(). It only needs to happen once to the rootCmd.

func GetAccountRoot

func GetAccountRoot(resource common.ResourceString, location common.Location) (string, error)

func GetContainerName

func GetContainerName(path string, location common.Location) (string, error)

func GetCredTypeFromEnvVar

func GetCredTypeFromEnvVar() common.CredentialType

GetCredTypeFromEnvVar tries to get credential type from environment variable defined by envVarCredentialType.

func GetOAuthTokenManagerInstance

func GetOAuthTokenManagerInstance() (*common.UserOAuthTokenManager, error)

* GetInstanceOAuthTokenInfo returns OAuth token, obtained by auto-login, * for current instance of AzCopy.

func GetResourceRoot

func GetResourceRoot(resource string, location common.Location) (resourceBase string, err error)

GetResourceRoot should eliminate wildcards and error out in invalid scenarios. This is intended for the jobPartOrder.SourceRoot.

func GetUserOAuthTokenManagerInstance

func GetUserOAuthTokenManagerInstance() *common.UserOAuthTokenManager

GetUserOAuthTokenManagerInstance gets or creates OAuthTokenManager for current user. Note: Currently, only support to have TokenManager for one user mapping to one tenantID.

func HandleListJobsCommand

func HandleListJobsCommand(jobStatus common.JobStatus) error

HandleListJobsCommand sends the ListJobs request to transfer engine Print the Jobs in the history of Azcopy

func HandlePauseCommand

func HandlePauseCommand(jobIdString string)

handles the pause command dispatches the pause Job order to the storage engine

func HandleShowCommand

func HandleShowCommand(listRequest common.ListRequest) error

handles the list command dispatches the list order to the transfer engine

func IsDestinationCaseInsensitive

func IsDestinationCaseInsensitive(fromTo common.FromTo) bool

func ParseSizeString

func ParseSizeString(s string, name string) (int64, error)

func PrintExistingJobIds

func PrintExistingJobIds(listJobResponse common.ListJobsResponse) error

PrintExistingJobIds prints the response of listOrder command when listOrder command requested the list of existing jobs

func PrintJobProgressSummary

func PrintJobProgressSummary(summary common.ListJobSummaryResponse)

PrintJobProgressSummary prints the response of listOrder command when listOrder command requested the progress summary of an existing job

func PrintJobTransfers

func PrintJobTransfers(listTransfersResponse common.ListJobTransfersResponse)

PrintJobTransfers prints the response of listOrder command when list Order command requested the list of specific transfer of an existing job

func SplitResourceString

func SplitResourceString(raw string, loc common.Location) (common.ResourceString, error)

func UnfurlSymlinks

func UnfurlSymlinks(symlinkPath string) (result string, err error)

func WalkWithSymlinks

func WalkWithSymlinks(fullPath string, walkFunc filepath.WalkFunc, followSymlinks bool) (err error)

WalkWithSymlinks is a symlinks-aware, parallelized, version of filePath.Walk. Separate this from the traverser for two purposes: 1) Cleaner code 2) Easier to test individually than to test the entire traverser.

func WarnStdoutAndScanningLog

func WarnStdoutAndScanningLog(toLog string)

func WrapFolder

func WrapFolder(fullpath string, stat os.FileInfo) (os.FileInfo, error)

Types

type BlobFSAccountTraverser

type BlobFSAccountTraverser struct {
	// contains filtered or unexported fields
}

We don't allow S2S from BlobFS, but what this gives us is the ability for users to download entire accounts at once. This is just added to create that feature parity. Enumerates an entire blobFS account, looking into each matching filesystem as it goes

type BucketToContainerNameResolver

type BucketToContainerNameResolver interface {
	ResolveName(bucketName string) (string, error)
}

type GCPBucketNameToAzureResourcesResolver

type GCPBucketNameToAzureResourcesResolver struct {
	// contains filtered or unexported fields
}

func NewGCPBucketNameToAzureResourcesResolver

func NewGCPBucketNameToAzureResourcesResolver(gcpBucketNames []string) *GCPBucketNameToAzureResourcesResolver

func (*GCPBucketNameToAzureResourcesResolver) ResolveName

func (resolver *GCPBucketNameToAzureResourcesResolver) ResolveName(bucketName string) (string, error)

type HTTPClient

type HTTPClient struct {
	// contains filtered or unexported fields
}

todo : use url in case of string

func NewHttpClient

func NewHttpClient(url string) *HTTPClient

NewHttpClient returns the instance of struct containing an instance of http.client and url

type ListReq

type ListReq struct {
	JobID    common.JobID
	OfStatus string
}

type ListResponse

type ListResponse struct {
	ErrorMsg string
}

type LocationLevel

type LocationLevel uint8

----- LOCATION LEVEL HANDLING -----

var ELocationLevel LocationLevel = 0

func (LocationLevel) Container

func (LocationLevel) Container() LocationLevel

func (LocationLevel) Object

func (LocationLevel) Object() LocationLevel

func (LocationLevel) Service

func (LocationLevel) Service() LocationLevel

type S3BucketNameToAzureResourcesResolver

type S3BucketNameToAzureResourcesResolver struct {
	// contains filtered or unexported fields
}

S3BucketNameToAzureResourcesResolver resolves s3 bucket name to Azure Blob container/ADLS Gen2 filesystem/File share. For Azure, container/filesystem/share's naming follows: 1. Lower case letters, numbers and hyphen. 2. 3-63 length. 3. Name should not contain two consecutive hyphens. 4. Name should not start or end with hyphen. For S3, bucket's naming follows: 1. The bucket name can be between 3 and 63 characters long, and can contain only lower-case characters, numbers, periods, and dashes. 2. Each label in the bucket name must start with a lowercase letter or number. 3. The bucket name cannot contain underscores, end with a dash or period, have consecutive periods, or use dashes adjacent to periods. 4. The bucket name cannot be formatted as an IP address (198.51.100.24). Two common cases need be solved are: 1. bucket name with period. In this case, AzCopy try to replace period with hyphen. e.g. bucket.with.period -> bucket-with-period 2. bucket name with consecutive hyphens. In this case, AzCopy try to replace consecutive hyphen, with -[numberOfHyphens]-. e.g. bucket----hyphens -> bucket-4-hyphens The resolver checks if there are naming collision with other existing bucket names, and try to add suffix when there is any collision. e.g. There is buckets with name: bucket-name, bucket.name. Azcopy will resolve bucket.name -> bucket-name -> bucket-name-2 All the resolving should be logged and warned to user.

func NewS3BucketNameToAzureResourcesResolver

func NewS3BucketNameToAzureResourcesResolver(s3BucketNames []string) *S3BucketNameToAzureResourcesResolver

NewS3BucketNameToAzureResourcesResolver creates S3BucketNameToAzureResourcesResolver. Users can provide bucket names upfront and on-demand via ResolveName. Previously resolved names will be returned outright by ResolveName.

func (*S3BucketNameToAzureResourcesResolver) ResolveName

func (s3Resolver *S3BucketNameToAzureResourcesResolver) ResolveName(bucketName string) (string, error)

ResolveName returns resolved name for given bucket name.

type Version

type Version struct {
	// contains filtered or unexported fields
}

func NewVersion

func NewVersion(raw string) (*Version, error)

To keep the code simple, we assume we only use a simple subset of semantic versions. Namely, the version is either a normal stable version, or a pre-release version with '-preview' attached. Examples: 10.1.0, 11.2.0-preview

func (Version) NewerThan

func (v Version) NewerThan(v2 Version) bool

detect if version v is newer than v2

func (Version) OlderThan

func (v Version) OlderThan(v2 Version) bool

detect if version v is older than v2