cmd

package
v10.0.0-...-53ebdbe Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 29, 2023 License: MIT Imports: 42 Imported by: 0

Documentation

Overview

Copyright © Microsoft <wastore@microsoft.com>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Index

Constants

View Source 
const MAX_SYMLINKS_TO_FOLLOW = 40
View Source 
const (
	NumOfFilesPerDispatchJobPart = 10000
)
View Source 
const PreservePermissionsFlag = "preserve-permissions"

Variables

View Source 
var AzcopyAppPathFolder string
View Source 
var AzcopyLogVerbosity common.LogLevel
View Source 
var EnumerationParallelStatFiles = false
View Source 
var EnumerationParallelism = 1
View Source 
var FinalPartCreatedMessage = "Final job part has been created"
View Source 
var IPv4Regex = regexp.MustCompile(`\d+\.\d+\.\d+\.\d+`) // simple regex
View Source 
var NothingScheduledError = errors.New("no transfers were scheduled because no files matched the specified criteria")
View Source 
var NothingToRemoveError = errors.New("nothing found to remove")
View Source 
var Rpc = func(cmd common.RpcCmd, request interface{}, response interface{}) {
	err := inprocSend(cmd, request, response)
	common.PanicIfErr(err)
}

Global singleton for sending RPC requests from the frontend to the STE

Functions

func Execute 

func Execute(logPathFolder, jobPlanFolder string, maxFileAndSocketHandles int, jobID common.JobID)

func GetAccountRoot 

func GetAccountRoot(resource common.ResourceString, location common.Location) (string, error)

func GetContainerName 

func GetContainerName(path string, location common.Location) (string, error)

func GetCredTypeFromEnvVar 

func GetCredTypeFromEnvVar() common.CredentialType

GetCredTypeFromEnvVar tries to get credential type from environment variable defined by envVarCredentialType.

func GetCredentialInfoForLocation 

func GetCredentialInfoForLocation(ctx context.Context, location common.Location, resource, resourceSAS string, isSource bool, cpkOptions common.CpkOptions) (credInfo common.CredentialInfo, isPublic bool, err error)

func GetOAuthTokenManagerInstance 

func GetOAuthTokenManagerInstance() (*common.UserOAuthTokenManager, error)

* GetInstanceOAuthTokenInfo returns OAuth token, obtained by auto-login, * for current instance of AzCopy.

func GetResourceRoot 

func GetResourceRoot(resource string, location common.Location) (resourceBase string, err error)

GetResourceRoot should eliminate wildcards and error out in invalid scenarios. This is intended for the jobPartOrder.SourceRoot.

func GetUserOAuthTokenManagerInstance 

func GetUserOAuthTokenManagerInstance() *common.UserOAuthTokenManager

GetUserOAuthTokenManagerInstance gets or creates OAuthTokenManager for current user. Note: Currently, only support to have TokenManager for one user mapping to one tenantID.

func HandleListJobsCommand 

func HandleListJobsCommand(jobStatus common.JobStatus) error

HandleListJobsCommand sends the ListJobs request to transfer engine Print the Jobs in the history of Azcopy

func HandlePauseCommand 

func HandlePauseCommand(jobIdString string)

handles the pause command dispatches the pause Job order to the storage engine

func HandleShowCommand 

func HandleShowCommand(listRequest common.ListRequest) error

handles the list command dispatches the list order to the transfer engine

func InferArgumentLocation 

func InferArgumentLocation(arg string) common.Location

func InitPipeline 

func InitPipeline(ctx context.Context, location common.Location, credential common.CredentialInfo, logLevel pipeline.LogLevel) (p pipeline.Pipeline, err error)

func IsDestinationCaseInsensitive 

func IsDestinationCaseInsensitive(fromTo common.FromTo) bool

func ParseSizeString 

func ParseSizeString(s string, name string) (int64, error)

func PrintExistingJobIds 

func PrintExistingJobIds(listJobResponse common.ListJobsResponse) error

PrintExistingJobIds prints the response of listOrder command when listOrder command requested the list of existing jobs

func PrintJobProgressSummary 

func PrintJobProgressSummary(summary common.ListJobSummaryResponse)

PrintJobProgressSummary prints the response of listOrder command when listOrder command requested the progress summary of an existing job

func PrintJobTransfers 

func PrintJobTransfers(listTransfersResponse common.ListJobTransfersResponse)

PrintJobTransfers prints the response of listOrder command when list Order command requested the list of specific transfer of an existing job

func SplitResourceString 

func SplitResourceString(raw string, loc common.Location) (common.ResourceString, error)
func UnfurlSymlinks(symlinkPath string) (result string, err error)

func ValidateFromTo 

func ValidateFromTo(src, dst string, userSpecifiedFromTo string) (common.FromTo, error)
func WalkWithSymlinks(appCtx context.Context, fullPath string, walkFunc filepath.WalkFunc, followSymlinks bool, errorChannel chan ErrorFileInfo, getObjectIndexerMapSize func() int64,
	orderedTqueue parallel.OrderedTqueueInterface, isSource bool, isSync bool, maxObjectIndexerSizeInGB uint32, scannerLogger common.ILoggerResetable) (err error)

WalkWithSymlinks is a symlinks-aware, parallelized, version of filePath.Walk. Separate this from the traverser for two purposes: 1) Cleaner code 2) Easier to test individually than to test the entire traverser.

func WarnStdoutAndScanningLog 

func WarnStdoutAndScanningLog(toLog string)

func WrapFolder 

func WrapFolder(fullpath string, stat os.FileInfo) (os.FileInfo, error)

Types

type AccountTraverser 

type AccountTraverser interface {
	ResourceTraverser
	// contains filtered or unexported methods
}

type BlobFSAccountTraverser 

type BlobFSAccountTraverser struct {
	// contains filtered or unexported fields
}

We don't allow S2S from BlobFS, but what this gives us is the ability for users to download entire accounts at once. This is just added to create that feature parity. Enumerates an entire blobFS account, looking into each matching filesystem as it goes

func (*BlobFSAccountTraverser) IsDirectory 

func (t *BlobFSAccountTraverser) IsDirectory(isSource bool) bool

func (*BlobFSAccountTraverser) Traverse 

func (t *BlobFSAccountTraverser) Traverse(preprocessor objectMorpher, processor objectProcessor, filters []ObjectFilter) error

type BucketToContainerNameResolver 

type BucketToContainerNameResolver interface {
	ResolveName(bucketName string) (string, error)
}

type CookedCopyCmdArgs 

type CookedCopyCmdArgs struct {
	// from arguments
	Source      common.ResourceString
	Destination common.ResourceString

	FromTo common.FromTo

	// new include/exclude only apply to file names
	// implemented for remove (and sync) only
	// includePathPatterns are handled like a list-of-files. Do not panic. This is not a bug that it is not present here.
	IncludePatterns       []string
	ExcludePatterns       []string
	ExcludePathPatterns   []string
	IncludeFileAttributes []string
	ExcludeFileAttributes []string
	IncludeBefore         *time.Time
	IncludeAfter          *time.Time

	// list of version ids
	ListOfVersionIDs chan string
	// filters from flags
	ListOfFilesChannel chan string // Channels are nullable.
	Recursive          bool
	StripTopDir        bool
	FollowSymlinks     bool
	ForceWrite         common.OverwriteOption // says whether we should try to overwrite
	ForceIfReadOnly    bool                   // says whether we should _force_ any overwrites (triggered by forceWrite) to work on Azure Files objects that are set to read-only

	CheckLength bool

	// Whether the user wants to preserve the SMB ACLs assigned to their files when moving between resources that are SMB ACL aware.
	PreservePermissions common.PreservePermissionsOption
	// Whether the user wants to preserve the SMB properties ...
	PreserveSMBInfo bool
	// Whether the user wants to preserve the POSIX properties ...
	PreservePOSIXProperties bool

	// Whether to rename/share the root
	AsSubdir bool

	// To specify whether user wants to preserve the blob index tags during service to service transfer.
	S2sPreserveBlobTags bool

	// whether to include blobs that have metadata 'hdi_isfolder = true'
	IncludeDirectoryStubs bool

	CpkOptions common.CpkOptions

	// Optional flag that permanently deletes soft deleted blobs
	PermanentDeleteOption common.PermanentDeleteOption
	// contains filtered or unexported fields
}

represents the processed copy command input from the user

func (*CookedCopyCmdArgs) Cancel 

func (cca *CookedCopyCmdArgs) Cancel(lcm common.LifecycleMgr)

func (*CookedCopyCmdArgs) InitModularFilters 

func (cca *CookedCopyCmdArgs) InitModularFilters() []ObjectFilter

Initialize the modular filters outside of copy to increase readability.

func (*CookedCopyCmdArgs) MakeEscapedRelativePath 

func (cca *CookedCopyCmdArgs) MakeEscapedRelativePath(source bool, dstIsDir bool, AsSubdir bool, object StoredObject) (relativePath string)

func (*CookedCopyCmdArgs) ReportProgressOrExit 

func (cca *CookedCopyCmdArgs) ReportProgressOrExit(lcm common.LifecycleMgr) (totalKnownCount uint32)

type CopyEnumerator 

type CopyEnumerator struct {
	Traverser ResourceTraverser

	// general filters apply to the objects returned by the traverser
	Filters []ObjectFilter

	// receive objects from the traverser and dispatch them for transferring
	ObjectDispatcher objectProcessor

	// a finalizer that is always called if the enumeration finishes properly
	Finalize func() error
}

func NewCopyEnumerator 

func NewCopyEnumerator(traverser ResourceTraverser, filters []ObjectFilter, objectDispatcher objectProcessor, finalizer func() error) *CopyEnumerator

type ErrorFileInfo 

type ErrorFileInfo struct {
	FileName             string
	FilePath             string
	FileSize             int64
	FileLastModifiedTime time.Time
	IsDir                bool
	ErrorMsg             error
	IsSource             bool
}

ErrorFileInfo holds information about files and folders that failed enumeration.

type FilterSet 

type FilterSet []ObjectFilter

func (FilterSet) GetEnumerationPreFilter 

func (fs FilterSet) GetEnumerationPreFilter(recursive bool) string

GetEnumerationPreFilter returns a prefix that is common to all the include filters, or "" if no such prefix can be found. (The implementation may return "" even in cases where such a prefix does exist, but in at least the simplest cases, it should return a non-empty prefix.) The result can be used to optimize enumeration, since anything without this prefix will fail the FilterSet

type GCPBucketNameToAzureResourcesResolver 

type GCPBucketNameToAzureResourcesResolver struct {
	// contains filtered or unexported fields
}

func NewGCPBucketNameToAzureResourcesResolver 

func NewGCPBucketNameToAzureResourcesResolver(gcpBucketNames []string) *GCPBucketNameToAzureResourcesResolver

func (*GCPBucketNameToAzureResourcesResolver) ResolveName 

func (resolver *GCPBucketNameToAzureResourcesResolver) ResolveName(bucketName string) (string, error)

type HTTPClient 

type HTTPClient struct {
	// contains filtered or unexported fields
}

todo : use url in case of string

func NewHttpClient 

func NewHttpClient(url string) *HTTPClient

NewHttpClient returns the instance of struct containing an instance of http.client and url

type IncludeAfterDateFilter 

type IncludeAfterDateFilter struct {
	Threshold time.Time
}

includeAfterDateFilter includes files with Last Modified Times >= the specified threshold Used for copy, but doesn't make conceptual sense for sync

func (*IncludeAfterDateFilter) AppliesOnlyToFiles 

func (f *IncludeAfterDateFilter) AppliesOnlyToFiles() bool

func (*IncludeAfterDateFilter) DoesPass 

func (f *IncludeAfterDateFilter) DoesPass(storedObject StoredObject) bool

func (*IncludeAfterDateFilter) DoesSupportThisOS 

func (f *IncludeAfterDateFilter) DoesSupportThisOS() (msg string, supported bool)

func (IncludeAfterDateFilter) FormatAsUTC 

func (_ IncludeAfterDateFilter) FormatAsUTC(t time.Time) string

func (IncludeAfterDateFilter) ParseISO8601 

func (_ IncludeAfterDateFilter) ParseISO8601(s string, chooseEarliest bool) (time.Time, error)

type IncludeBeforeDateFilter 

type IncludeBeforeDateFilter struct {
	Threshold time.Time
}

IncludeBeforeDateFilter includes files with Last Modified Times <= the specified Threshold Used for copy, but doesn't make conceptual sense for sync

func (*IncludeBeforeDateFilter) AppliesOnlyToFiles 

func (f *IncludeBeforeDateFilter) AppliesOnlyToFiles() bool

func (*IncludeBeforeDateFilter) DoesPass 

func (f *IncludeBeforeDateFilter) DoesPass(storedObject StoredObject) bool

func (*IncludeBeforeDateFilter) DoesSupportThisOS 

func (f *IncludeBeforeDateFilter) DoesSupportThisOS() (msg string, supported bool)

func (IncludeBeforeDateFilter) FormatAsUTC 

func (_ IncludeBeforeDateFilter) FormatAsUTC(t time.Time) string

func (IncludeBeforeDateFilter) ParseISO8601 

func (_ IncludeBeforeDateFilter) ParseISO8601(s string, chooseEarliest bool) (time.Time, error)

type IncludeFilter 

type IncludeFilter struct {
	// contains filtered or unexported fields
}

design explanation: include filters are different from the exclude ones, which work together in the "AND" manner meaning and if an StoredObject is rejected by any of the exclude filters, then it is rejected by all of them as a result, the exclude filters can be in their own struct, and work correctly on the other hand, include filters work in the "OR" manner meaning that if an StoredObject is accepted by any of the include filters, then it is accepted by all of them consequently, all the include Patterns must be stored together

func (*IncludeFilter) AppliesOnlyToFiles 

func (f *IncludeFilter) AppliesOnlyToFiles() bool

func (*IncludeFilter) DoesPass 

func (f *IncludeFilter) DoesPass(storedObject StoredObject) bool

func (*IncludeFilter) DoesSupportThisOS 

func (f *IncludeFilter) DoesSupportThisOS() (msg string, supported bool)

type ListReq 

type ListReq struct {
	JobID    common.JobID
	OfStatus string
}

type ListResponse 

type ListResponse struct {
	ErrorMsg string
}

type LocationLevel 

type LocationLevel uint8

----- LOCATION LEVEL HANDLING ----- 

var ELocationLevel LocationLevel = 0

func DetermineLocationLevel 

func DetermineLocationLevel(location string, locationType common.Location, source bool) (LocationLevel, error)

Uses syntax to assume the "level" of a location. This is typically used to

func (LocationLevel) Container 

func (LocationLevel) Container() LocationLevel

func (LocationLevel) Object 

func (LocationLevel) Object() LocationLevel

func (LocationLevel) Service 

func (LocationLevel) Service() LocationLevel

type ObjectFilter 

type ObjectFilter interface {
	DoesSupportThisOS() (msg string, supported bool)
	DoesPass(storedObject StoredObject) bool
	AppliesOnlyToFiles() bool
}

given a StoredObject, verify if it satisfies the defined conditions if yes, return true

type RawSyncCmdArgs 

type RawSyncCmdArgs struct {
	Src       string
	Dst       string
	Recursive bool
	FromTo    string

	// options from flags
	BlockSizeMB           float64
	Include               string
	Exclude               string
	ExcludePath           string
	IncludeFileAttributes string
	ExcludeFileAttributes string
	LegacyInclude         string // for warning messages only
	LegacyExclude         string // for warning messages only
	IncludeRegex          string
	ExcludeRegex          string

	PreservePermissions     bool
	PreserveSMBPermissions  bool // deprecated and synonymous with preservePermissions
	PreserveOwner           bool
	PreserveSMBInfo         bool
	PreservePOSIXProperties bool
	FollowSymlinks          bool

	Md5ValidationOption string

	AzcopyCurrentJobID common.JobID
	// this flag indicates the user agreement with respect to deleting the extra files at the destination
	// which do not exists at source. With this flag turned on/off, users will not be asked for permission.
	// otherwise the user is prompted to make a decision
	DeleteDestination string

	// this flag is to disable comparator and overwrite files at destination irrespective
	MirrorMode bool

	ForceIfReadOnly bool

	// Optional flag to encrypt user data with user provided key.
	// Key is provide in the REST request itself
	// Provided key (EncryptionKey and EncryptionKeySHA256) and its hash will be fetched from environment variables
	// Set EncryptionAlgorithm = "AES256" by default.
	CpkInfo bool
	// Key is present in AzureKeyVault and Azure KeyVault is linked with storage account.
	// Provided key name will be fetched from Azure Key Vault and will be used to encrypt the data
	CpkScopeInfo string

	// Limit on size of ObjectIndexerMap in memory.
	// For more information, please refer to cookedSyncCmdArgs.
	MaxObjectIndexerMapSizeInGB string

	// Change file detection mode.
	// For more information, please refer to cookedSyncCmdArgs.
	CfdMode string

	// For more information, please refer to cookedSyncCmdArgs.
	MetaDataOnlySync bool

	// This is the time of last sync in ISO8601 format. For more information, please refer to cookedSyncCmdArgs.
	LastSyncTime string
	// contains filtered or unexported fields
}

func (*RawSyncCmdArgs) Cook 

func (raw *RawSyncCmdArgs) Cook() (cookedSyncCmdArgs, error)

validates and transform raw input into cooked input

type ResourceTraverser 

type ResourceTraverser interface {
	Traverse(preprocessor objectMorpher, processor objectProcessor, filters []ObjectFilter) error
	IsDirectory(isSource bool) bool
}

capable of traversing a structured resource like container or local directory pass each StoredObject to the given objectProcessor if it passes all the filters

func InitResourceTraverser 

func InitResourceTraverser(resource common.ResourceString, location common.Location, ctx *context.Context,
	credential *common.CredentialInfo, followSymlinks *bool, listOfFilesChannel chan string, recursive, getProperties,
	includeDirectoryStubs bool, permanentDeleteOption common.PermanentDeleteOption, incrementEnumerationCounter enumerationCounterFunc, listOfVersionIds chan string,
	s2sPreserveBlobTags bool, logLevel pipeline.LogLevel, cpkOptions common.CpkOptions, errorChannel chan ErrorFileInfo,
	indexerMap *folderIndexer, possiblyRenamedMap *possiblyRenamedMap, orderedTqueue parallel.OrderedTqueueInterface, isSource bool, isSync bool, maxObjectIndexerSizeInGB uint32, lastSyncTime time.Time, cfdMode common.CFDMode,
	metaDataOnlySync bool, scannerLogger common.ILoggerResetable) (ResourceTraverser, error)

source, location, recursive, and incrementEnumerationCounter are always required. ctx, pipeline are only required for remote resources. followSymlinks is only required for local resources (defaults to false) errorOnDirWOutRecursive is used by copy. If errorChannel is non-nil, all errors encountered during enumeration will be conveyed through this channel. To avoid slowdowns, use a buffered channel of enough capacity. orderedTqueue is communication channel b/w source and destination and required in case of sync, maxObjectIndexerSizeInGB for auto pacing. lastSyncTime and CFDModeFlags for change detection.

type S3BucketNameToAzureResourcesResolver 

type S3BucketNameToAzureResourcesResolver struct {
	// contains filtered or unexported fields
}

S3BucketNameToAzureResourcesResolver resolves s3 bucket name to Azure Blob container/ADLS Gen2 filesystem/File share. For Azure, container/filesystem/share's naming follows: 1. Lower case letters, numbers and hyphen. 2. 3-63 length. 3. Name should not contain two consecutive hyphens. 4. Name should not start or end with hyphen. For S3, bucket's naming follows: 1. The bucket name can be between 3 and 63 characters long, and can contain only lower-case characters, numbers, periods, and dashes. 2. Each label in the bucket name must start with a lowercase letter or number. 3. The bucket name cannot contain underscores, end with a dash or period, have consecutive periods, or use dashes adjacent to periods. 4. The bucket name cannot be formatted as an IP address (198.51.100.24). Two common cases need be solved are: 1. bucket name with period. In this case, AzCopy try to replace period with hyphen. e.g. bucket.with.period -> bucket-with-period 2. bucket name with consecutive hyphens. In this case, AzCopy try to replace consecutive hyphen, with -[numberOfHyphens]-. e.g. bucket----hyphens -> bucket-4-hyphens The resolver checks if there are naming collision with other existing bucket names, and try to add suffix when there is any collision. e.g. There is buckets with name: bucket-name, bucket.name. Azcopy will resolve bucket.name -> bucket-name -> bucket-name-2 All the resolving should be logged and warned to user.

func NewS3BucketNameToAzureResourcesResolver 

func NewS3BucketNameToAzureResourcesResolver(s3BucketNames []string) *S3BucketNameToAzureResourcesResolver

NewS3BucketNameToAzureResourcesResolver creates S3BucketNameToAzureResourcesResolver. Users can provide bucket names upfront and on-demand via ResolveName. Previously resolved names will be returned outright by ResolveName.

func (*S3BucketNameToAzureResourcesResolver) ResolveName 

func (s3Resolver *S3BucketNameToAzureResourcesResolver) ResolveName(bucketName string) (string, error)

ResolveName returns resolved name for given bucket name.

type StoredObject 

type StoredObject struct {

	// container source, only included by account traversers.
	ContainerName string
	// destination container name. Included in the processor after resolving container names.
	DstContainerName string

	// metadata, included in S2S transfers
	Metadata common.Metadata
	// contains filtered or unexported fields
}

represent a local or remote resource object (ex: local file, blob, etc.) we can add more properties if needed, as this is easily extensible ** DO NOT instantiate directly, always use newStoredObject ** (to make sure its fully populated and any preprocessor method runs) Note: If you make any changes made to this struct, make sure storedObjectSize() in syncIndexer.go is updated accordingly.

func (*StoredObject) ToNewCopyTransfer 

func (s *StoredObject) ToNewCopyTransfer(
	steWillAutoDecompress bool,
	Source string,
	Destination string,
	preserveBlobTier bool,
	folderPropertiesOption common.FolderPropertyOption) (transfer common.CopyTransfer, shouldSendToSte bool)

type Version 

type Version struct {
	// contains filtered or unexported fields
}

func NewVersion 

func NewVersion(raw string) (*Version, error)

To keep the code simple, we assume we only use a simple subset of semantic versions. Namely, the version is either a normal stable version, or a pre-release version with '-preview' attached. Examples: 10.1.0, 11.2.0-preview

func (Version) NewerThan 

func (v Version) NewerThan(v2 Version) bool

detect if version v is newer than v2

func (Version) OlderThan 

func (v Version) OlderThan(v2 Version) bool

detect if version v is older than v2

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.