azblob

package module
v1.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 22, 2025 License: MIT Imports: 14 Imported by: 394

README

Azure Blob Storage module for Go

PkgGoDev Build Status Code Coverage

Service Version: 2023-11-03

Azure Blob Storage is Microsoft's object storage solution for the cloud. Blob Storage is optimized for storing massive amounts of unstructured data - data that does not adhere to a particular data model or definition, such as text or binary data. For more information, see Introduction to Azure Blob Storage.

Use the Azure Blob Storage client module github.com/Azure/azure-sdk-for-go/sdk/storage/azblob to:

  • Authenticate clients with Azure Blob Storage
  • Manipulate containers and blobs in an Azure storage account

Key links:

Source code | API reference documentation | REST API documentation | Product documentation | Samples

Getting started

Prerequisites
az storage account create --name MyStorageAccount --resource-group MyResourceGroup --location westus --sku Standard_LRS
Install the package

Install the Azure Blob Storage client module for Go with go get:

go get github.com/Azure/azure-sdk-for-go/sdk/storage/azblob

If you plan to authenticate with Azure Active Directory (recommended), also install the azidentity module.

go get github.com/Azure/azure-sdk-for-go/sdk/azidentity
Authenticate the client

To interact with the Azure Blob Storage service, you'll need to create an instance of the azblob.Client type. The azidentity module makes it easy to add Azure Active Directory support for authenticating Azure SDK clients with their corresponding Azure services.

// create a credential for authenticating with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle err

// create an azblob.Client for the specified storage account that uses the above credential
client, err := azblob.NewClient("https://MYSTORAGEACCOUNT.blob.core.windows.net/", cred, nil)
// TODO: handle err

Learn more about enabling Azure Active Directory for authentication with Azure Storage:

Other options for authentication include connection strings, shared key, shared access signatures (SAS), and anonymous public access. Use the appropriate client constructor function for the authentication mechanism you wish to use. For examples, see:

Key concepts

Blob Storage is designed for:

  • Serving images or documents directly to a browser.
  • Storing files for distributed access.
  • Streaming video and audio.
  • Writing to log files.
  • Storing data for backup and restore, disaster recovery, and archiving.
  • Storing data for analysis by an on-premises or Azure-hosted service.

Blob Storage offers three types of resources:

  • The storage account
  • One or more containers in a storage account
  • One or more blobs in a container

Instances of the azblob.Client type provide methods for manipulating containers and blobs within a storage account. The storage account is specified when the azblob.Client is constructed.

Specialized clients

The Azure Blob Storage client module for Go also provides specialized clients in various subpackages. Use these clients when you need to interact with a specific kind of blob. Learn more about block blobs, append blobs, and page blobs.

The blob package contains APIs common to all blob types. This includes APIs for deleting and undeleting a blob, setting metadata, and more.

The lease package contains clients for managing leases on blobs and containers. See the REST API reference for general information on leases.

The container package contains APIs specific to containers. This includes APIs for setting access policies or properties, and more.

The service package contains APIs specific to the Blob service. This includes APIs for manipulating containers, retrieving account information, and more.

The sas package contains utilities to aid in the creation and manipulation of shared access signature (SAS) tokens. See the package's documentation for more information.

Goroutine safety

We guarantee that all client instance methods are goroutine-safe and independent of each other (see guideline). This ensures that the recommendation to reuse client instances is always safe, even across goroutines.

Blob metadata

Blob metadata name-value pairs are valid HTTP headers and should adhere to all restrictions governing HTTP headers. Metadata names must be valid HTTP header names, may contain only ASCII characters, and should be treated as case-insensitive. Base64-encode or URL-encode metadata values containing non-ASCII characters.

Additional concepts

Client options | Accessing the response | Handling failures | Logging

Examples

Upload a blob
const (
	account       = "https://MYSTORAGEACCOUNT.blob.core.windows.net/"
	containerName = "sample-container"
	blobName      = "sample-blob"
	sampleFile    = "path/to/sample/file"
)

// authenticate with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle error

// create a client for the specified storage account
client, err := azblob.NewClient(account, cred, nil)
// TODO: handle error

// open the file for reading
file, err := os.OpenFile(sampleFile, os.O_RDONLY, 0)
// TODO: handle error
defer file.Close()

// upload the file to the specified container with the specified blob name
_, err = client.UploadFile(context.TODO(), containerName, blobName, file, nil)
// TODO: handle error
Download a blob
// this example accesses a public blob via anonymous access, so no credentials are required
client, err := azblob.NewClientWithNoCredential("https://azurestoragesamples.blob.core.windows.net/", nil)
// TODO: handle error

// create or open a local file where we can download the blob
file, err := os.Create("cloud.jpg")
// TODO: handle error
defer file.Close()

// download the blob
_, err = client.DownloadFile(context.TODO(), "samples", "cloud.jpg", file, nil)
// TODO: handle error
Enumerate blobs
const (
	account       = "https://MYSTORAGEACCOUNT.blob.core.windows.net/"
	containerName = "sample-container"
)

// authenticate with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle error

// create a client for the specified storage account
client, err := azblob.NewClient(account, cred, nil)
// TODO: handle error

// blob listings are returned across multiple pages
pager := client.NewListBlobsFlatPager(containerName, nil)

// continue fetching pages until no more remain
for pager.More() {
  // advance to the next page
	page, err := pager.NextPage(context.TODO())
	// TODO: handle error

	// print the blob names for this page
	for _, blob := range page.Segment.BlobItems {
		fmt.Println(*blob.Name)
	}
}

Troubleshooting

All Blob service operations will return an *azcore.ResponseError on failure with a populated ErrorCode field. Many of these errors are recoverable. The bloberror package provides the possible Storage error codes along with helper facilities for error handling.

const (
	connectionString = "<connection_string>"
	containerName    = "sample-container"
)

// create a client with the provided connection string
client, err := azblob.NewClientFromConnectionString(connectionString, nil)
// TODO: handle error

// try to delete the container, avoiding any potential race conditions with an in-progress or completed deletion
_, err = client.DeleteContainer(context.TODO(), containerName, nil)

if bloberror.HasCode(err, bloberror.ContainerBeingDeleted, bloberror.ContainerNotFound) {
	// ignore any errors if the container is being deleted or already has been deleted
} else if err != nil {
	// TODO: some other error
}

Next steps

Get started with our Blob samples. They contain complete examples of the above snippets and more.

Contributing

See the Storage CONTRIBUTING.md for details on building, testing, and contributing to this library.

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit cla.microsoft.com.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Impressions

Documentation

Overview

Example

This example is a quick-starter and demonstrates how to get started using the Azure Blob Storage SDK for Go. 

Output:
Example (Blob_AccessConditions)

This example shows how to perform operations on blob conditionally. 

Output:
Example (Blob_Client_Download)

This example shows how to download a large stream with intelligent retries. Specifically, if the connection fails while reading, continuing to read from this stream initiates a new GetBlob call passing a range that starts from the last byte successfully read before the failure. 

Output:
Example (Client_CreateContainer) 
Output:
Example (Client_DeleteBlob) 
Output:
Example (Client_DeleteContainer) 
Output:
Example (Client_DownloadFile) 
Output:
Example (Client_DownloadStream) 
Output:
Example (Client_NewClient) 
Output:
Example (Client_NewClientFromConnectionString) 
Output:
Example (Client_NewClientWithSharedKeyCredential) 
Output:
Example (Client_NewListBlobsPager) 
Output:
Example (Client_NewListContainersPager) 
Output:
Example (Client_UploadFile) 
Output:
Example (Client_UploadStream) 
Output:
Example (Client_anonymous_NewClientWithNoCredential) 
Output:
Example (ProgressUploadDownload) 
Output:

Index

Examples

Constants

View Source 
const (
	// EventUpload is used for logging events related to upload operation.
	EventUpload = exported.EventUpload

	// EventSubmitBatch is used for logging events related to submit blob batch operation.
	EventSubmitBatch = exported.EventSubmitBatch
)

Variables

This section is empty.

Functions

This section is empty.

Types

type AccessConditions added in v0.5.0 

type AccessConditions = exported.BlobAccessConditions

AccessConditions identifies blob-specific access conditions which you optionally set.

type CPKInfo added in v1.0.0 

type CPKInfo = blob.CPKInfo

CPKInfo contains a group of parameters for client provided encryption key.

type CPKScopeInfo added in v1.0.0 

type CPKScopeInfo = container.CPKScopeInfo

CPKScopeInfo contains a group of parameters for the ContainerClient.Create method.

type Client added in v0.5.0 

type Client struct {
	// contains filtered or unexported fields
}

Client represents a URL to an Azure Storage blob; the blob may be a block blob, append blob, or page blob.

func NewClient added in v0.5.0 

func NewClient(serviceURL string, cred azcore.TokenCredential, options *ClientOptions) (*Client, error)

NewClient creates an instance of Client with the specified values.

  • serviceURL - the URL of the storage account e.g. https://<account>.blob.core.windows.net/
  • cred - an Azure AD credential, typically obtained via the azidentity module
  • options - client options; pass nil to accept the default values

func NewClientFromConnectionString added in v0.5.0 

func NewClientFromConnectionString(connectionString string, options *ClientOptions) (*Client, error)

NewClientFromConnectionString creates an instance of Client with the specified values.

  • connectionString - a connection string for the desired storage account
  • options - client options; pass nil to accept the default values

func NewClientWithNoCredential added in v0.5.0 

func NewClientWithNoCredential(serviceURL string, options *ClientOptions) (*Client, error)

NewClientWithNoCredential creates an instance of Client with the specified values. This is used to anonymously access a storage account or with a shared access signature (SAS) token.

  • serviceURL - the URL of the storage account e.g. https://<account>.blob.core.windows.net/?<sas token>
  • options - client options; pass nil to accept the default values

func NewClientWithSharedKeyCredential added in v0.5.0 

func NewClientWithSharedKeyCredential(serviceURL string, cred *SharedKeyCredential, options *ClientOptions) (*Client, error)

NewClientWithSharedKeyCredential creates an instance of Client with the specified values.

  • serviceURL - the URL of the storage account e.g. https://<account>.blob.core.windows.net/
  • cred - a SharedKeyCredential created with the matching storage account and access key
  • options - client options; pass nil to accept the default values

func (*Client) CreateContainer added in v0.5.0 

func (c *Client) CreateContainer(ctx context.Context, containerName string, o *CreateContainerOptions) (CreateContainerResponse, error)

CreateContainer is a lifecycle method to creates a new container under the specified account. If the container with the same name already exists, a ContainerAlreadyExists Error will be raised. This method returns a client with which to interact with the newly created container.

func (*Client) DeleteBlob added in v0.5.0 

func (c *Client) DeleteBlob(ctx context.Context, containerName string, blobName string, o *DeleteBlobOptions) (DeleteBlobResponse, error)

DeleteBlob marks the specified blob or snapshot for deletion. The blob is later deleted during garbage collection. Note that deleting a blob also deletes all its snapshots. For more information, see https://docs.microsoft.com/rest/api/storageservices/delete-blob.

func (*Client) DeleteContainer added in v0.5.0 

func (c *Client) DeleteContainer(ctx context.Context, containerName string, o *DeleteContainerOptions) (DeleteContainerResponse, error)

DeleteContainer is a lifecycle method that marks the specified container for deletion. The container and any blobs contained within it are later deleted during garbage collection. If the container is not found, a ResourceNotFoundError will be raised.

func (*Client) DownloadBuffer added in v0.5.0 

func (c *Client) DownloadBuffer(ctx context.Context, containerName string, blobName string, buffer []byte, o *DownloadBufferOptions) (int64, error)

DownloadBuffer downloads an Azure blob to a buffer with parallel.

func (*Client) DownloadFile added in v0.5.0 

func (c *Client) DownloadFile(ctx context.Context, containerName string, blobName string, file *os.File, o *DownloadFileOptions) (int64, error)

DownloadFile downloads an Azure blob to a local file. The file would be truncated if the size doesn't match.

func (*Client) DownloadStream added in v0.5.0 

func (c *Client) DownloadStream(ctx context.Context, containerName string, blobName string, o *DownloadStreamOptions) (DownloadStreamResponse, error)

DownloadStream reads a range of bytes from a blob. The response also includes the blob's properties and metadata. For more information, see https://docs.microsoft.com/rest/api/storageservices/get-blob.

func (*Client) NewListBlobsFlatPager added in v0.5.0 

func (c *Client) NewListBlobsFlatPager(containerName string, o *ListBlobsFlatOptions) *runtime.Pager[ListBlobsFlatResponse]

NewListBlobsFlatPager returns a pager for blobs starting from the specified Marker. Use an empty Marker to start enumeration from the beginning. Blob names are returned in lexicographic order. For more information, see https://docs.microsoft.com/rest/api/storageservices/list-blobs.

func (*Client) NewListContainersPager added in v0.5.0 

func (c *Client) NewListContainersPager(o *ListContainersOptions) *runtime.Pager[ListContainersResponse]

NewListContainersPager operation returns a pager of the containers under the specified account. Use an empty Marker to start enumeration from the beginning. Container names are returned in lexicographic order. For more information, see https://docs.microsoft.com/rest/api/storageservices/list-containers2.

func (*Client) ServiceClient added in v0.6.0 

func (c *Client) ServiceClient() *service.Client

ServiceClient returns the embedded service client for this client.

func (*Client) URL added in v0.5.0 

func (c *Client) URL() string

URL returns the URL endpoint used by the BlobClient object.

func (*Client) UploadBuffer added in v0.5.0 

func (c *Client) UploadBuffer(ctx context.Context, containerName string, blobName string, buffer []byte, o *UploadBufferOptions) (UploadBufferResponse, error)

UploadBuffer uploads a buffer in blocks to a block blob.

func (*Client) UploadFile added in v0.5.0 

func (c *Client) UploadFile(ctx context.Context, containerName string, blobName string, file *os.File, o *UploadFileOptions) (UploadFileResponse, error)

UploadFile uploads a file in blocks to a block blob.

func (*Client) UploadStream added in v0.5.0 

func (c *Client) UploadStream(ctx context.Context, containerName string, blobName string, body io.Reader, o *UploadStreamOptions) (UploadStreamResponse, error)

UploadStream copies the file held in io.Reader to the Blob at blockBlobClient. A Context deadline or cancellation will cause this to error.

type ClientOptions 

type ClientOptions base.ClientOptions

ClientOptions contains the optional parameters when creating a Client.

type CreateContainerOptions 

type CreateContainerOptions = service.CreateContainerOptions

CreateContainerOptions contains the optional parameters for the ContainerClient.Create method.

type CreateContainerResponse added in v0.5.0 

type CreateContainerResponse = service.CreateContainerResponse

CreateContainerResponse contains the response from method container.Client.Create.

type DeleteBlobOptions 

type DeleteBlobOptions = blob.DeleteOptions

DeleteBlobOptions contains the optional parameters for the Client.Delete method.

type DeleteBlobResponse added in v0.5.0 

type DeleteBlobResponse = blob.DeleteResponse

DeleteBlobResponse contains the response from method blob.Client.Delete.

type DeleteContainerOptions 

type DeleteContainerOptions = service.DeleteContainerOptions

DeleteContainerOptions contains the optional parameters for the container.Client.Delete method.

type DeleteContainerResponse added in v0.5.0 

type DeleteContainerResponse = service.DeleteContainerResponse

DeleteContainerResponse contains the response from method container.Client.Delete

type DeleteSnapshotsOptionType 

type DeleteSnapshotsOptionType = generated.DeleteSnapshotsOptionType

DeleteSnapshotsOptionType defines values for DeleteSnapshotsOptionType. 

const (
	DeleteSnapshotsOptionTypeInclude DeleteSnapshotsOptionType = generated.DeleteSnapshotsOptionTypeInclude
	DeleteSnapshotsOptionTypeOnly    DeleteSnapshotsOptionType = generated.DeleteSnapshotsOptionTypeOnly
)

func PossibleDeleteSnapshotsOptionTypeValues 

func PossibleDeleteSnapshotsOptionTypeValues() []DeleteSnapshotsOptionType

PossibleDeleteSnapshotsOptionTypeValues returns the possible values for the DeleteSnapshotsOptionType const type.

type DownloadBufferOptions added in v0.5.0 

type DownloadBufferOptions = blob.DownloadBufferOptions

DownloadBufferOptions identifies options used by the DownloadBuffer and DownloadFile functions.

type DownloadFileOptions added in v0.5.0 

type DownloadFileOptions = blob.DownloadFileOptions

DownloadFileOptions identifies options used by the DownloadBuffer and DownloadFile functions.

type DownloadStreamOptions added in v0.5.0 

type DownloadStreamOptions = blob.DownloadStreamOptions

DownloadStreamOptions contains the optional parameters for the Client.DownloadStream method.

type DownloadStreamResponse added in v0.5.0 

type DownloadStreamResponse = blob.DownloadStreamResponse

DownloadStreamResponse wraps AutoRest generated BlobDownloadResponse and helps to provide info for retry.

type HTTPRange added in v0.5.0 

type HTTPRange = exported.HTTPRange

HTTPRange defines a range of bytes within an HTTP resource, starting at offset and ending at offset+count. A zero-value HTTPRange indicates the entire resource. An HTTPRange which has an offset and zero value count indicates from the offset to the resource's end.

type ListBlobsFlatOptions added in v0.5.0 

type ListBlobsFlatOptions = container.ListBlobsFlatOptions

ListBlobsFlatOptions contains the optional parameters for the container.Client.ListBlobFlatSegment method.

type ListBlobsFlatResponse added in v0.5.0 

type ListBlobsFlatResponse = container.ListBlobsFlatResponse

ListBlobsFlatResponse contains the response from method container.Client.ListBlobFlatSegment.

type ListBlobsFlatSegmentResponse 

type ListBlobsFlatSegmentResponse = generated.ListBlobsFlatSegmentResponse

ListBlobsFlatSegmentResponse - An enumeration of blobs

type ListBlobsInclude added in v0.5.0 

type ListBlobsInclude = container.ListBlobsInclude

ListBlobsInclude indicates what additional information the service should return with each blob.

type ListContainersInclude added in v0.5.0 

type ListContainersInclude = service.ListContainersInclude

ListContainersInclude indicates what additional information the service should return with each container.

type ListContainersOptions 

type ListContainersOptions = service.ListContainersOptions

ListContainersOptions contains the optional parameters for the container.Client.ListContainers operation

type ListContainersResponse added in v0.5.0 

type ListContainersResponse = service.ListContainersResponse

ListContainersResponse contains the response from method service.Client.ListContainersSegment.

type ListContainersSegmentResponse 

type ListContainersSegmentResponse = generated.ListContainersSegmentResponse

ListContainersSegmentResponse - An enumeration of containers

type ObjectReplicationPolicy 

type ObjectReplicationPolicy = blob.ObjectReplicationPolicy

ObjectReplicationPolicy are deserialized attributes

type PublicAccessType 

type PublicAccessType = generated.PublicAccessType

PublicAccessType defines values for AccessType - private (default) or blob or container. 

const (
	PublicAccessTypeBlob      PublicAccessType = generated.PublicAccessTypeBlob
	PublicAccessTypeContainer PublicAccessType = generated.PublicAccessTypeContainer
)

func PossiblePublicAccessTypeValues 

func PossiblePublicAccessTypeValues() []PublicAccessType

PossiblePublicAccessTypeValues returns the possible values for the PublicAccessType const type.

type RetryReaderOptions 

type RetryReaderOptions = blob.RetryReaderOptions

RetryReaderOptions contains properties which can help to decide when to do retry.

type SharedKeyCredential 

type SharedKeyCredential = exported.SharedKeyCredential

SharedKeyCredential contains an account's name and its primary or secondary key.

func NewSharedKeyCredential 

func NewSharedKeyCredential(accountName, accountKey string) (*SharedKeyCredential, error)

NewSharedKeyCredential creates an immutable SharedKeyCredential containing the storage account's name and either its primary or secondary key.

type URLParts added in v0.5.0 

type URLParts = sas.URLParts

URLParts object represents the components that make up an Azure Storage Container/Blob URL. NOTE: Changing any SAS-related field requires computing a new SAS signature.

func ParseURL added in v0.5.0 

func ParseURL(u string) (URLParts, error)

ParseURL parses a URL initializing URLParts' fields including any SAS-related & snapshot query parameters. Any other query parameters remain in the UnparsedParams field. This method overwrites all fields in the URLParts object.

type UploadBufferOptions added in v0.5.0 

type UploadBufferOptions = blockblob.UploadBufferOptions

UploadBufferOptions provides set of configurations for UploadBuffer operation

type UploadBufferResponse added in v0.5.0 

type UploadBufferResponse = blockblob.UploadBufferResponse

UploadBufferResponse contains the response from method Client.UploadBuffer/Client.UploadFile.

type UploadFileOptions added in v0.5.0 

type UploadFileOptions = blockblob.UploadFileOptions

UploadFileOptions provides set of configurations for UploadFile operation

type UploadFileResponse added in v0.5.0 

type UploadFileResponse = blockblob.UploadFileResponse

UploadFileResponse contains the response from method Client.UploadBuffer/Client.UploadFile.

type UploadResponse added in v0.5.0 

type UploadResponse = blockblob.CommitBlockListResponse

UploadResponse contains the response from method blockblob.Client.CommitBlockList.

type UploadStreamOptions added in v0.4.0 

type UploadStreamOptions = blockblob.UploadStreamOptions

UploadStreamOptions provides set of configurations for UploadStream operation

type UploadStreamResponse added in v0.5.0 

type UploadStreamResponse = blockblob.CommitBlockListResponse

UploadStreamResponse contains the response from method Client.CommitBlockList.

Source Files

View all Source files

Directories

Path Synopsis
internal

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.