Documentation ¶
Overview ¶
Copyright © Microsoft <wastore@microsoft.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Index ¶
- Constants
- Variables
- func Execute(azsAppPathFolder, logPathFolder string, jobPlanFolder string, ...)
- func GetAccountRoot(resource common.ResourceString, location common.Location) (string, error)
- func GetContainerName(path string, location common.Location) (string, error)
- func GetCredTypeFromEnvVar() common.CredentialType
- func GetOAuthTokenManagerInstance() (*common.UserOAuthTokenManager, error)
- func GetResourceRoot(resource string, location common.Location) (resourceBase string, err error)
- func GetUserOAuthTokenManagerInstance() *common.UserOAuthTokenManager
- func HandleListJobsCommand(jobStatus common.JobStatus) error
- func HandlePauseCommand(jobIdString string)
- func HandleShowCommand(listRequest common.ListRequest) error
- func IsDestinationCaseInsensitive(fromTo common.FromTo) bool
- func ParseSizeString(s string, name string) (int64, error)
- func PrintExistingJobIds(listJobResponse common.ListJobsResponse) error
- func PrintJobProgressSummary(summary common.ListJobSummaryResponse)
- func PrintJobTransfers(listTransfersResponse common.ListJobTransfersResponse)
- func SplitResourceString(raw string, loc common.Location) (common.ResourceString, error)
- func UnfurlSymlinks(symlinkPath string) (result string, err error)
- func WalkWithSymlinks(fullPath string, walkFunc filepath.WalkFunc, followSymlinks bool) (err error)
- func WarnStdoutAndScanningLog(toLog string)
- func WrapFolder(fullpath string, stat os.FileInfo) (os.FileInfo, error)
- type BlobFSAccountTraverser
- type BucketToContainerNameResolver
- type GCPBucketNameToAzureResourcesResolver
- type HTTPClient
- type ListReq
- type ListResponse
- type LocationLevel
- type S3BucketNameToAzureResourcesResolver
- type Version
Constants ¶
const (
NumOfFilesPerDispatchJobPart = 10000
)
const PreservePermissionsFlag = "preserve-permissions"
Variables ¶
var FinalPartCreatedMessage = "Final job part has been created"
var IPv4Regex = regexp.MustCompile(`\d+\.\d+\.\d+\.\d+`) // simple regex
var NothingScheduledError = errors.New("no transfers were scheduled because no files matched the specified criteria")
var NothingToRemoveError = errors.New("nothing found to remove")
var Rpc = func(cmd common.RpcCmd, request interface{}, response interface{}) { err := inprocSend(cmd, request, response) common.PanicIfErr(err) }
Global singleton for sending RPC requests from the frontend to the STE
Functions ¶
func Execute ¶
func Execute(azsAppPathFolder, logPathFolder string, jobPlanFolder string, maxFileAndSocketHandles int)
Execute adds all child commands to the root command and sets flags appropriately. This is called by main.main(). It only needs to happen once to the rootCmd.
func GetAccountRoot ¶
func GetContainerName ¶
func GetCredTypeFromEnvVar ¶
func GetCredTypeFromEnvVar() common.CredentialType
GetCredTypeFromEnvVar tries to get credential type from environment variable defined by envVarCredentialType.
func GetOAuthTokenManagerInstance ¶
func GetOAuthTokenManagerInstance() (*common.UserOAuthTokenManager, error)
* GetInstanceOAuthTokenInfo returns OAuth token, obtained by auto-login, * for current instance of AzCopy.
func GetResourceRoot ¶
GetResourceRoot should eliminate wildcards and error out in invalid scenarios. This is intended for the jobPartOrder.SourceRoot.
func GetUserOAuthTokenManagerInstance ¶
func GetUserOAuthTokenManagerInstance() *common.UserOAuthTokenManager
GetUserOAuthTokenManagerInstance gets or creates OAuthTokenManager for current user. Note: Currently, only support to have TokenManager for one user mapping to one tenantID.
func HandleListJobsCommand ¶
HandleListJobsCommand sends the ListJobs request to transfer engine Print the Jobs in the history of Azcopy
func HandlePauseCommand ¶
func HandlePauseCommand(jobIdString string)
handles the pause command dispatches the pause Job order to the storage engine
func HandleShowCommand ¶
func HandleShowCommand(listRequest common.ListRequest) error
handles the list command dispatches the list order to the transfer engine
func IsDestinationCaseInsensitive ¶ added in v10.12.0
func PrintExistingJobIds ¶
func PrintExistingJobIds(listJobResponse common.ListJobsResponse) error
PrintExistingJobIds prints the response of listOrder command when listOrder command requested the list of existing jobs
func PrintJobProgressSummary ¶
func PrintJobProgressSummary(summary common.ListJobSummaryResponse)
PrintJobProgressSummary prints the response of listOrder command when listOrder command requested the progress summary of an existing job
func PrintJobTransfers ¶
func PrintJobTransfers(listTransfersResponse common.ListJobTransfersResponse)
PrintJobTransfers prints the response of listOrder command when list Order command requested the list of specific transfer of an existing job
func SplitResourceString ¶
func UnfurlSymlinks ¶
func WalkWithSymlinks ¶
WalkWithSymlinks is a symlinks-aware, parallelized, version of filePath.Walk. Separate this from the traverser for two purposes: 1) Cleaner code 2) Easier to test individually than to test the entire traverser.
func WarnStdoutAndScanningLog ¶ added in v10.11.0
func WarnStdoutAndScanningLog(toLog string)
Types ¶
type BlobFSAccountTraverser ¶
type BlobFSAccountTraverser struct {
// contains filtered or unexported fields
}
We don't allow S2S from BlobFS, but what this gives us is the ability for users to download entire accounts at once. This is just added to create that feature parity. Enumerates an entire blobFS account, looking into each matching filesystem as it goes
type GCPBucketNameToAzureResourcesResolver ¶
type GCPBucketNameToAzureResourcesResolver struct {
// contains filtered or unexported fields
}
func NewGCPBucketNameToAzureResourcesResolver ¶
func NewGCPBucketNameToAzureResourcesResolver(gcpBucketNames []string) *GCPBucketNameToAzureResourcesResolver
func (*GCPBucketNameToAzureResourcesResolver) ResolveName ¶
func (resolver *GCPBucketNameToAzureResourcesResolver) ResolveName(bucketName string) (string, error)
type HTTPClient ¶
type HTTPClient struct {
// contains filtered or unexported fields
}
todo : use url in case of string
func NewHttpClient ¶
func NewHttpClient(url string) *HTTPClient
NewHttpClient returns the instance of struct containing an instance of http.client and url
type ListResponse ¶
type ListResponse struct {
ErrorMsg string
}
type LocationLevel ¶
type LocationLevel uint8
----- LOCATION LEVEL HANDLING -----
var ELocationLevel LocationLevel = 0
func (LocationLevel) Container ¶
func (LocationLevel) Container() LocationLevel
func (LocationLevel) Object ¶
func (LocationLevel) Object() LocationLevel
func (LocationLevel) Service ¶
func (LocationLevel) Service() LocationLevel
type S3BucketNameToAzureResourcesResolver ¶
type S3BucketNameToAzureResourcesResolver struct {
// contains filtered or unexported fields
}
S3BucketNameToAzureResourcesResolver resolves s3 bucket name to Azure Blob container/ADLS Gen2 filesystem/File share. For Azure, container/filesystem/share's naming follows: 1. Lower case letters, numbers and hyphen. 2. 3-63 length. 3. Name should not contain two consecutive hyphens. 4. Name should not start or end with hyphen. For S3, bucket's naming follows: 1. The bucket name can be between 3 and 63 characters long, and can contain only lower-case characters, numbers, periods, and dashes. 2. Each label in the bucket name must start with a lowercase letter or number. 3. The bucket name cannot contain underscores, end with a dash or period, have consecutive periods, or use dashes adjacent to periods. 4. The bucket name cannot be formatted as an IP address (198.51.100.24). Two common cases need be solved are: 1. bucket name with period. In this case, AzCopy try to replace period with hyphen. e.g. bucket.with.period -> bucket-with-period 2. bucket name with consecutive hyphens. In this case, AzCopy try to replace consecutive hyphen, with -[numberOfHyphens]-. e.g. bucket----hyphens -> bucket-4-hyphens The resolver checks if there are naming collision with other existing bucket names, and try to add suffix when there is any collision. e.g. There is buckets with name: bucket-name, bucket.name. Azcopy will resolve bucket.name -> bucket-name -> bucket-name-2 All the resolving should be logged and warned to user.
func NewS3BucketNameToAzureResourcesResolver ¶
func NewS3BucketNameToAzureResourcesResolver(s3BucketNames []string) *S3BucketNameToAzureResourcesResolver
NewS3BucketNameToAzureResourcesResolver creates S3BucketNameToAzureResourcesResolver. Users can provide bucket names upfront and on-demand via ResolveName. Previously resolved names will be returned outright by ResolveName.
func (*S3BucketNameToAzureResourcesResolver) ResolveName ¶
func (s3Resolver *S3BucketNameToAzureResourcesResolver) ResolveName(bucketName string) (string, error)
ResolveName returns resolved name for given bucket name.
type Version ¶
type Version struct {
// contains filtered or unexported fields
}
func NewVersion ¶
To keep the code simple, we assume we only use a simple subset of semantic versions. Namely, the version is either a normal stable version, or a pre-release version with '-preview' attached. Examples: 10.1.0, 11.2.0-preview
Source Files ¶
- benchmark.go
- cancel.go
- copy.go
- copyEnumeratorHelper.go
- copyEnumeratorInit.go
- copyUtil.go
- credentialUtil.go
- doc.go
- env.go
- gcpNameResolver.go
- helpMessages.go
- jobs.go
- jobsClean.go
- jobsList.go
- jobsRemove.go
- jobsResume.go
- jobsShow.go
- list.go
- load.go
- loadCLFS.go
- loadOutputTranslation.go
- login.go
- loginStatus.go
- logout.go
- make.go
- pathUtils.go
- pause.go
- remove.go
- removeEnumerator.go
- removeProcessor.go
- root.go
- rpc.go
- s3NameResolver.go
- sync.go
- syncComparator.go
- syncEnumerator.go
- syncIndexer.go
- syncProcessor.go
- validators.go
- versionChecker.go
- zc_attr_filter_notwin.go
- zc_enumerator.go
- zc_filter.go
- zc_newobjectadapters.go
- zc_pipeline_init.go
- zc_processor.go
- zc_traverser_benchmark.go
- zc_traverser_blob.go
- zc_traverser_blob_account.go
- zc_traverser_blob_versions.go
- zc_traverser_blobfs.go
- zc_traverser_blobfs_account.go
- zc_traverser_file.go
- zc_traverser_file_account.go
- zc_traverser_gcp.go
- zc_traverser_gcp_service.go
- zc_traverser_list.go
- zc_traverser_local.go
- zc_traverser_local_other.go
- zc_traverser_s3.go
- zc_traverser_s3_service.go