azdatalake

README

ADLS Gen2 Storage SDK for Go

Service Version: 2021-06-08

Azure Data Lake Storage Gen2 (ADLS Gen2) is Microsoft's hierarchical object storage solution for the cloud with converged capabilities with Azure Blob Storage. For example, Data Lake Storage Gen2 provides file system semantics, file-level security, and scale. Because these capabilities are built on Blob storage, you also get low-cost, tiered storage, with high availability/disaster recovery capabilities. ADLS Gen2 makes Azure Storage the foundation for building enterprise data lakes on Azure. Designed from the start to service multiple petabytes of information while sustaining hundreds of gigabits of throughput, ADLS Gen2 allows you to easily manage massive amounts of data.

Source code | API reference documentation | REST API documentation

Getting started

Install the package

Install the ADLS Gen2 Storage SDK for Go with go get:

go get github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake

If you're going to authenticate with Azure Active Directory (recommended), install the azidentity module.

go get github.com/Azure/azure-sdk-for-go/sdk/azidentity

Prerequisites

A supported Go version (the Azure SDK supports the two most recent Go releases).

You need an Azure subscription and a Storage Account to use this package.

To create a new Storage Account, you can use the Azure Portal, Azure PowerShell, or the Azure CLI. Here's an example using the Azure CLI:

az storage account create --name MyStorageAccount --resource-group MyResourceGroup --location westus --sku Standard_LRS

Authenticate the client

In order to interact with the ADLS Gen2 Storage service, you'll need to create an instance of the Client type. The azidentity module makes it easy to add Azure Active Directory support for authenticating Azure SDK clients with their corresponding Azure services.

// create a credential for authenticating with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle err

// create a service.Client for the specified storage account that uses the above credential
client, err := service.NewClient("https://MYSTORAGEACCOUNT.dfs.core.windows.net/", cred, nil)
// TODO: handle err
// you can also create filesystem, file and directory clients

Learn more about enabling Azure Active Directory for authentication with Azure Storage in our documentation and our samples.

Key concepts

ADLS Gen2 provides:

• Hadoop-compatible access
• Hierarchical directory structure
• Optimized cost and performance
• Finer grain security model
• Massive scalability

ADLS Gen2 storage is designed for:

• Serving images or documents directly to a browser.
• Storing files for distributed access.
• Streaming video and audio.
• Writing to log files.
• Storing data for backup and restore, disaster recovery, and archiving.
• Storing data for analysis by an on-premises or Azure-hosted service.

ADLS Gen2 storage offers three types of resources:

• The storage account
• One or more filesystems in a storage account
• One or more files or directories in a filesystem

Instances of the Client type provide methods for manipulating filesystems and paths within a storage account. The storage account is specified when the Client is constructed. The clients available are referenced below. Use the appropriate client constructor function for the authentication mechanism you wish to use.

Goroutine safety

We guarantee that all client instance methods are goroutine-safe and independent of each other (guideline). This ensures that the recommendation of reusing client instances is always safe, even across goroutines.

About metadata

ADLS Gen2 metadata name/value pairs are valid HTTP headers and should adhere to all restrictions governing HTTP headers. Metadata names must be valid HTTP header names, may contain only ASCII characters, and should be treated as case-insensitive. Base64-encode or URL-encode metadata values containing non-ASCII characters.

Additional concepts

Client options | Accessing the response | Handling failures | Logging

Examples

Creating and uploading a file (assuming filesystem exists)

const (
	path = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs/sample-file"
)

// authenticate with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle error

// create a client for the specified storage account
client, err := file.NewClient(path, cred, nil)
// TODO: handle error

_, err = client.Create(context.TODO(), nil)
// TODO: handle error

// open the file for reading
fh, err := os.OpenFile(sampleFile, os.O_RDONLY, 0)
// TODO: handle error
defer fh.Close()

// upload the file to the specified filesystem with the specified file name
_, err = client.UploadFile(context.TODO(), fh, nil)
// TODO: handle error

Downloading a file

const (
    path = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs/cloud.jpg"
)

// authenticate with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle error

// create a client for the specified storage account
client, err := file.NewClient(path, cred, nil)
// TODO: handle error

// create or open a local file where we can download the file
file, err := os.Create("cloud.jpg")
// TODO: handle error
defer file.Close()

// download the file
_, err = client.DownloadFile(context.TODO(), file, nil)
// TODO: handle error

Creating and deleting a filesystem

const (
	fs = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs"
)

// authenticate with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle error

// create a client for the specified storage account
client, err := filesystem.NewClient(fs, cred, nil)
// TODO: handle error

_, err = client.Create(context.TODO(), nil)
// TODO: handle error

_, err = client.Delete(context.TODO(), nil)
// TODO: handle error

Enumerating paths (assuming filesystem exists)

const (
	fs = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs"
)

// authenticate with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle error

// create a filesystem client for the specified storage account
client, err := filesystem.NewClient(fs, cred, nil)
// TODO: handle error

// path listings are returned across multiple pages
pager := client.NewListPathsPager(true, nil)

// continue fetching pages until no more remain
for pager.More() {
  // advance to the next page
	page, err := pager.NextPage(context.TODO())
	// TODO: handle error

	// print the path names for this page
	for _, path := range page.PathList.Paths {
		fmt.Println(*path.Name)
        fmt.Println(*path.IsDirectory)
	}
}

Troubleshooting

All Datalake service operations will return an *azcore.ResponseError on failure with a populated ErrorCode field. Many of these errors are recoverable. The datalakeerror package provides the possible Storage error codes along with various helper facilities for error handling.

Specialized clients

The ADLS Gen2 Storage SDK for Go provides specialized clients in various subpackages.

The file package contains APIs related to file path types.

The directory package contains APIs related to directory path types.

The lease package contains clients for managing leases on paths (paths represent both directory and file paths) and filesystems. Please see the reference docs for general information on leases.

The filesystem package contains APIs specific to filesystems. This includes APIs setting access policies or properties, and more.

The service package contains APIs specific to Datalake service. This includes APIs for manipulating filesystems, retrieving account information, and more.

The sas package contains utilities to aid in the creation and manipulation of Shared Access Signature tokens. See the package's documentation for more information.

You can find additional context and examples in our samples for each subpackage (named examples_test.go).

Contributing

See the Storage CONTRIBUTING.md for details on building, testing, and contributing to this library.

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit cla.microsoft.com.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Documentation

Index

• Constants
• type DurationType
  • func PossibleDurationTypeValues() []DurationType
• type HTTPRange
• type SharedKeyCredential
  • func NewSharedKeyCredential(accountName, accountKey string) (*SharedKeyCredential, error)
• type StateType
  • func PossibleStateTypeValues() []StateType
• type StatusType
  • func PossibleStatusTypeValues() []StatusType
• type URLParts
  • func ParseURL(u string) (URLParts, error)

Constants

const (
	// EventUpload is used for logging events related to upload operation.
	EventUpload = exported.EventUpload

	// EventError is used for logging errors.
	EventError = exported.EventError
)

Types

type DurationType

type DurationType = lease.DurationType

DurationType defines values for DurationType

const (
	DurationTypeInfinite DurationType = lease.DurationTypeInfinite
	DurationTypeFixed    DurationType = lease.DurationTypeFixed
)

func PossibleDurationTypeValues

func PossibleDurationTypeValues() []DurationType

PossibleDurationTypeValues returns the possible values for the DurationType const type.

type HTTPRange

type HTTPRange = exported.HTTPRange

HTTPRange defines a range of bytes within an HTTP resource, starting at offset and ending at offset+count. A zero-value HTTPRange indicates the entire resource. An HTTPRange which has an offset and zero value count indicates from the offset to the resource's end.

type SharedKeyCredential

type SharedKeyCredential = exported.SharedKeyCredential

SharedKeyCredential contains an account's name and its primary or secondary key.

func NewSharedKeyCredential

func NewSharedKeyCredential(accountName, accountKey string) (*SharedKeyCredential, error)

NewSharedKeyCredential creates an immutable SharedKeyCredential containing the storage account's name and either its primary or secondary key.

type StateType

type StateType = lease.StateType

StateType defines values for StateType

const (
	StateTypeAvailable StateType = lease.StateTypeAvailable
	StateTypeLeased    StateType = lease.StateTypeLeased
	StateTypeExpired   StateType = lease.StateTypeExpired
	StateTypeBreaking  StateType = lease.StateTypeBreaking
	StateTypeBroken    StateType = lease.StateTypeBroken
)

func PossibleStateTypeValues

func PossibleStateTypeValues() []StateType

PossibleStateTypeValues returns the possible values for the StateType const type.

type StatusType

type StatusType = lease.StatusType

StatusType defines values for StatusType

const (
	StatusTypeLocked   StatusType = lease.StatusTypeLocked
	StatusTypeUnlocked StatusType = lease.StatusTypeUnlocked
)

func PossibleStatusTypeValues

func PossibleStatusTypeValues() []StatusType

PossibleStatusTypeValues returns the possible values for the StatusType const type.

type URLParts

type URLParts = sas.URLParts

URLParts object represents the components that make up an Azure Storage Container/Blob URL. NOTE: Changing any SAS-related field requires computing a new SAS signature.

func ParseURL

func ParseURL(u string) (URLParts, error)

ParseURL parses a URL initializing URLParts' fields including any SAS-related & snapshot query parameters. Any other query parameters remain in the UnparsedParams field. This method overwrites all fields in the URLParts object.