azureblob

Documentation

Overview

Package azureblob provides a blob implementation that uses Azure Storage’s BlockBlob. Use OpenBucket to construct a *blob.Bucket.

NOTE: SignedURLs for PUT created with this package are not fully portable; they will not work unless the PUT request includes a "x-ms-blob-type" header set to "BlockBlob". See https://stackoverflow.com/questions/37824136/put-on-sas-blob-url-without-specifying-x-ms-blob-type-header.

URLs

For blob.OpenBucket, azureblob registers for the scheme "azblob".

The default URL opener will use environment variables to generate credentials and a service URL; see https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob for a more complete descriptions of each approach.

• AZURE_STORAGE_ACCOUNT: The service account name. Required if used along with AZURE_STORAGE KEY, because it defines authentication mechanism to be azblob.NewSharedKeyCredential, which creates immutable shared key credentials. Otherwise, "storage_account" in the URL query string parameter can be used.
• AZURE_STORAGE_KEY: To use a shared key credential. The service account name and key are passed to NewSharedKeyCredential and then the resulting credential is passed to NewServiceClientWithSharedKey.
• AZURE_STORAGE_CONNECTION_STRING: To use a connection string, passed to NewServiceClientFromConnectionString.
• AZURE_STORAGE_SAS_TOKEN: To use a SAS token. The SAS token is added as a URL parameter to the service URL, and passed to NewServiceClientWithNoCredential.
• If none of the above are provided, azureblob defaults to azidentity.NewDefaultAzureCredential: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#NewDefaultAzureCredential. See the documentation there for the credential types it supports, including CLI creds, environment variables like AZURE_CLIENT_ID, AZURE_TENANT_ID, etc.

In addition, the environment variables AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_DOMAIN, AZURE_STORAGE_PROTOCOL, AZURE_STORAGE_IS_CDN, and AZURE_STORAGE_IS_LOCAL_EMULATOR can be used to configure how the default URLOpener generates the Azure Service URL via ServiceURLOptions. These can all be configured via URL parameters as well. See ServiceURLOptions and NewDefaultServiceURL for more details.

To customize the URL opener, or for more details on the URL format, see URLOpener.

See https://gocloud.dev/concepts/urls/ for background information.

Escaping

Go CDK supports all UTF-8 strings; to make this work with services lacking full UTF-8 support, strings must be escaped (during writes) and unescaped (during reads). The following escapes are performed for azureblob:

• Blob keys: ASCII characters 0-31, 92 ("\"), and 127 are escaped to "__0x<hex>__". Additionally, the "/" in "../" and a trailing "/" in a key (e.g., "foo/") are escaped in the same way.
• Metadata keys: Per https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata, Azure only allows C# identifiers as metadata keys. Therefore, characters other than "[a-z][A-z][0-9]_" are escaped using "__0x<hex>__". In addition, characters "[0-9]" are escaped when they start the string. URL encoding would not work since "%" is not valid.
• Metadata values: Escaped using URL encoding.

As

azureblob exposes the following types for As:

• Bucket: *azblob.ContainerClient
• Error: *azcore.ReponseError, *azblob.InternalError, *azblob.StorageError
• ListObject: azblob.BlobItemInternal for objects, azblob.BlobPrefix for "directories"
• ListOptions.BeforeList: *azblob.ContainerListBlobsHierarchyOption
• Reader: azblob.BlobDownloadResponse
• Reader.BeforeRead: *azblob.BlockDownloadOptions
• Attributes: azblob.BlobGetPropertiesResponse
• CopyOptions.BeforeCopy: *azblob.BlobStartCopyOptions
• WriterOptions.BeforeWrite: *azblob.UploadStreamOptions
• SignedURLOptions.BeforeSign: *azblob.BlobSASPermissions

Example (OpenBucketFromURL)

package main

import (
	"context"
	"log"

	"gocloud.dev/blob"
)

func main() {
	// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
	// PRAGMA: On gocloud.dev, add a blank import: _ "gocloud.dev/blob/azureblob"
	// PRAGMA: On gocloud.dev, hide lines until the next blank line.
	ctx := context.Background()

	// blob.OpenBucket creates a *blob.Bucket from a URL.
	// This URL will open the container "my-container" using default
	// credentials found in environment variables as documented in
	// the package.
	// Assuming AZURE_STORAGE_ACCOUNT is set to "myaccount",
	// and other options aren't set, the service URL will look like:
	// "https://myaccount.blob.core.windows.net/my-container".
	bucket, err := blob.OpenBucket(ctx, "azblob://my-container")
	if err != nil {
		log.Fatal(err)
	}
	defer bucket.Close()

	// Another example, against a local emulator.
	// Assuming AZURE_STORAGE_ACCOUNT is set to "myaccount",
	// the service URL will look like:
	// "http://localhost:10001/myaccount/my-container".
	localbucket, err := blob.OpenBucket(ctx, "azblob://my-container?protocol=http&domain=localhost:10001")
	if err != nil {
		log.Fatal(err)
	}
	defer localbucket.Close()
}

Index

• Constants
• Variables
• func NewDefaultServiceClient(svcURL ServiceURL) (*azblob.ServiceClient, error)
• func OpenBucket(ctx context.Context, svcClient *azblob.ServiceClient, containerName string, ...) (*blob.Bucket, error)
• type Options
• type ServiceURL
  • func NewServiceURL(opts *ServiceURLOptions) (ServiceURL, error)
• type ServiceURLOptions
  • func NewDefaultServiceURLOptions() *ServiceURLOptions
• type URLOpener
  • func (o *URLOpener) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error)

Examples

• Package (OpenBucketFromURL)
• OpenBucket

Constants

const Scheme = "azblob"

Scheme is the URL scheme gcsblob registers its URLOpener under on blob.DefaultMux.

Variables

var Set = wire.NewSet(
	NewDefaultServiceURLOptions,
	NewServiceURL,
	NewDefaultServiceClient,
)

Set holds Wire providers for this package.

Functions

func NewDefaultServiceClient

func NewDefaultServiceClient(svcURL ServiceURL) (*azblob.ServiceClient, error)

NewDefaultServiceClient returns an Azure Blob service client with credentials from the environment as described in the package docstring.

func OpenBucket

func OpenBucket(ctx context.Context, svcClient *azblob.ServiceClient, containerName string, opts *Options) (*blob.Bucket, error)

OpenBucket returns a *blob.Bucket backed by Azure Storage Account. See the package documentation for an example and https://godoc.org/github.com/Azure/azure-storage-blob-go/azblob for more details.

Example

package main

import (
	"context"
	"log"

	"gocloud.dev/blob/azureblob"
)

func main() {
	// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
	// PRAGMA: On gocloud.dev, hide lines until the next blank line.
	ctx := context.Background()

	const (
		// The storage container to access.
		containerName = "my-container"
	)

	// Construct the service URL.
	// There are many forms of service URLs, see ServiceURLOptions.
	opts := azureblob.NewDefaultServiceURLOptions()
	serviceURL, err := azureblob.NewServiceURL(opts)
	if err != nil {
		log.Fatal(err)
	}

	// There are many ways to authenticate to Azure.
	// This approach uses environment variables as described in azureblob package
	// documentation.
	// For example, to use shared key authentication, you would set
	// AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY.
	// To use a SAS token, you would set AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_SAS_TOKEN.
	// You can also construct a client using the azblob constructors directly, like
	// azblob.NewServiceClientWithSharedKey.
	client, err := azureblob.NewDefaultServiceClient(serviceURL)
	if err != nil {
		log.Fatal(err)
	}

	// Create a *blob.Bucket.
	b, err := azureblob.OpenBucket(ctx, client, containerName, nil)
	if err != nil {
		log.Fatal(err)
	}
	defer b.Close()

	// Now we can use b to read or write files to the container.
	data, err := b.ReadAll(ctx, "my-key")
	if err != nil {
		log.Fatal(err)
	}
	_ = data
}

Types

type Options

type Options struct{}

Options sets options for constructing a *blob.Bucket backed by Azure Blob.

type ServiceURL

type ServiceURL string

ServiceURL represents an Azure service URL.

func NewServiceURL

func NewServiceURL(opts *ServiceURLOptions) (ServiceURL, error)

NewServiceURL generates a URL for addressing an Azure Blob service account. It uses several parameters, each of which can be specified via ServiceURLOptions.

The generated URL is "<protocol>://<account name>.<domain>" with the following caveats:

• If opts.SASToken is provided, it is appended to the URL as a query parameter.
• If opts.IsCDN is true, the <account name> part is dropped.
• If opts.IsLocalEmulator is true, or the domain starts with "localhost" or "127.0.0.1", the account name and domain are flipped, e.g.: http://127.0.0.1:10000/myaccount

type ServiceURLOptions

type ServiceURLOptions struct {
	// AccountName is the account name the credentials are for.
	AccountName string

	// SASToken will be appended to the service URL.
	// See https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1#shared-access-signature-parameters.
	SASToken string

	// StorageDomain can be provided to specify an Azure Cloud Environment
	// domain to target for the blob storage account (i.e. public, government, china).
	// Defaults to "blob.core.windows.net". Possible values will look similar
	// to this but are different for each cloud (i.e. "blob.core.govcloudapi.net" for USGovernment).
	// Check the Azure developer guide for the cloud environment where your bucket resides.
	// See the docstring for NewServiceURL to see examples of how this is used
	// along with the other Options fields.
	StorageDomain string

	// Protocol can be provided to specify protocol to access Azure Blob Storage.
	// Protocols that can be specified are "http" for local emulator and "https" for general.
	// Defaults to "https".
	// See the docstring for NewServiceURL to see examples of how this is used
	// along with the other Options fields.
	Protocol string

	// IsCDN can be set to true when using a CDN URL pointing to a blob storage account:
	// https://docs.microsoft.com/en-us/azure/cdn/cdn-create-a-storage-account-with-cdn
	// See the docstring for NewServiceURL to see examples of how this is used
	// along with the other Options fields.
	IsCDN bool

	// IsLocalEmulator should be set to true when targeting Local Storage Emulator (Azurite).
	// See the docstring for NewServiceURL to see examples of how this is used
	// along with the other Options fields.
	IsLocalEmulator bool
}

ServiceURLOptions sets options for constructing a service URL for Azure Blob.

func NewDefaultServiceURLOptions

func NewDefaultServiceURLOptions() *ServiceURLOptions

NewDefaultServiceURLOptions generates a ServiceURLOptions based on environment variables.

type URLOpener

type URLOpener struct {
	// MakeClient must be set to a non-nil value.
	MakeClient func(svcURL ServiceURL) (*azblob.ServiceClient, error)

	// ServiceURLOptions specifies default options for generating the service URL.
	// Some options can be overridden in the URL as described above.
	ServiceURLOptions ServiceURLOptions

	// Options specifies the options to pass to OpenBucket.
	Options Options
}

URLOpener opens Azure URLs like "azblob://mybucket".

The URL host is used as the bucket name.

The following query options are supported:

• domain: Overrides Options.StorageDomain.
• protocol: Overrides Options.Protocol.
• cdn: Overrides Options.IsCDN.
• localemu: Overrides Options.IsLocalEmulator.

func (*URLOpener) OpenBucketURL

func (o *URLOpener) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error)

OpenBucketURL opens a blob.Bucket based on u.