azureblob

package
v0.33.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 2, 2023 License: Apache-2.0 Imports: 29 Imported by: 113

Documentation

Overview

Package azureblob provides a blob implementation that uses Azure Storage’s BlockBlob. Use OpenBucket to construct a *blob.Bucket.

NOTE: SignedURLs for PUT created with this package are not fully portable; they will not work unless the PUT request includes a "x-ms-blob-type" header set to "BlockBlob". See https://stackoverflow.com/questions/37824136/put-on-sas-blob-url-without-specifying-x-ms-blob-type-header.

URLs

For blob.OpenBucket, azureblob registers for the scheme "azblob".

The default URL opener will use environment variables to generate credentials and a service URL; see https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob for a more complete descriptions of each approach.

  • AZURE_STORAGE_ACCOUNT: The service account name. Required if used along with AZURE_STORAGE KEY, because it defines authentication mechanism to be azblob.NewSharedKeyCredential, which creates immutable shared key credentials. Otherwise, "storage_account" in the URL query string parameter can be used.
  • AZURE_STORAGE_KEY: To use a shared key credential. The service account name and key are passed to NewSharedKeyCredential and then the resulting credential is passed to NewClientWithSharedKeyCredential.
  • AZURE_STORAGE_CONNECTION_STRING: To use a connection string, passed to NewClientFromConnectionString.
  • AZURE_STORAGE_SAS_TOKEN: To use a SAS token. The SAS token is added as a URL parameter to the service URL, and passed to NewClientWithNoCredential.
  • If none of the above are provided, azureblob defaults to azidentity.NewDefaultAzureCredential: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#NewDefaultAzureCredential. See the documentation there for the credential types it supports, including CLI creds, environment variables like AZURE_CLIENT_ID, AZURE_TENANT_ID, etc.

In addition, the environment variables AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_DOMAIN, AZURE_STORAGE_PROTOCOL, AZURE_STORAGE_IS_CDN, and AZURE_STORAGE_IS_LOCAL_EMULATOR can be used to configure how the default URLOpener generates the Azure Service URL via ServiceURLOptions. These can all be configured via URL parameters as well. See ServiceURLOptions and NewDefaultServiceURL for more details.

To customize the URL opener, or for more details on the URL format, see URLOpener.

See https://gocloud.dev/concepts/urls/ for background information.

Escaping

Go CDK supports all UTF-8 strings; to make this work with services lacking full UTF-8 support, strings must be escaped (during writes) and unescaped (during reads). The following escapes are performed for azureblob:

  • Blob keys: ASCII characters 0-31, 34 ("\""), 35 ("#"), 37 ("%"), 63 ("?"), 92 ("\"), and 127 are escaped to "__0x<hex>__". Additionally, the "/" in "../" and a trailing "/" in a key (e.g., "foo/") are escaped in the same way.
  • Metadata keys: Per https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata, Azure only allows C# identifiers as metadata keys. Therefore, characters other than "[a-z][A-z][0-9]_" are escaped using "__0x<hex>__". In addition, characters "[0-9]" are escaped when they start the string. URL encoding would not work since "%" is not valid.
  • Metadata values: Escaped using URL encoding.

As

azureblob exposes the following types for As:

  • Bucket: *container.Client
  • Error: *azcore.ReponseError. You can use bloberror.HasCode directly though.
  • ListObject: container.BlobItem for objects, container.BlobPrefix for "directories"
  • ListOptions.BeforeList: *container.ListBlobsHierarchyOptions
  • Reader: azblobblob.DownloadStreamResponse
  • Reader.BeforeRead: *azblob.DownloadStreamOptions
  • Attributes: azblobblob.GetPropertiesResponse
  • CopyOptions.BeforeCopy: *azblobblob.StartCopyFromURLOptions
  • WriterOptions.BeforeWrite: *azblob.UploadStreamOptions
  • SignedURLOptions.BeforeSign: *sas.BlobPermissions
Example (OpenBucketFromURL) 
package main

import (
	"context"
	"log"

	"gocloud.dev/blob"
)

func main() {
	// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
	// PRAGMA: On gocloud.dev, add a blank import: _ "gocloud.dev/blob/azureblob"
	// PRAGMA: On gocloud.dev, hide lines until the next blank line.
	ctx := context.Background()

	// blob.OpenBucket creates a *blob.Bucket from a URL.
	// This URL will open the container "my-container" using default
	// credentials found in environment variables as documented in
	// the package.
	// Assuming AZURE_STORAGE_ACCOUNT is set to "myaccount",
	// and other options aren't set, the service URL will look like:
	// "https://myaccount.blob.core.windows.net/my-container".
	bucket, err := blob.OpenBucket(ctx, "azblob://my-container")
	if err != nil {
		log.Fatal(err)
	}
	defer bucket.Close()

	// Another example, against a local emulator.
	// Assuming AZURE_STORAGE_ACCOUNT is set to "myaccount",
	// the service URL will look like:
	// "http://localhost:10001/myaccount/my-container".
	localbucket, err := blob.OpenBucket(ctx, "azblob://my-container?protocol=http&domain=localhost:10001")
	if err != nil {
		log.Fatal(err)
	}
	defer localbucket.Close()
}

Output:

Index

Examples

Constants

View Source 
const Scheme = "azblob"

Scheme is the URL scheme gcsblob registers its URLOpener under on blob.DefaultMux.

Variables

View Source 
var Set = wire.NewSet(
	NewDefaultServiceURLOptions,
	NewServiceURL,
	NewDefaultClient,
)

Set holds Wire providers for this package.

Functions

func NewDefaultClient added in v0.28.0 

func NewDefaultClient(svcURL ServiceURL, containerName ContainerName) (*container.Client, error)

NewDefaultClient returns an Azure Blob container client with credentials from the environment as described in the package docstring.

func OpenBucket 

func OpenBucket(ctx context.Context, client *container.Client, opts *Options) (*blob.Bucket, error)

OpenBucket returns a *blob.Bucket backed by Azure Storage Account. See the package documentation for an example and https://godoc.org/github.com/Azure/azure-storage-blob-go/azblob for more details.

Example 
package main

import (
	"context"
	"log"

	"gocloud.dev/blob/azureblob"
)

func main() {
	// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
	// PRAGMA: On gocloud.dev, hide lines until the next blank line.
	ctx := context.Background()

	const (
		// The storage container to access.
		containerName = "my-container"
	)

	// Construct the service URL.
	// There are many forms of service URLs, see ServiceURLOptions.
	opts := azureblob.NewDefaultServiceURLOptions()
	serviceURL, err := azureblob.NewServiceURL(opts)
	if err != nil {
		log.Fatal(err)
	}

	// There are many ways to authenticate to Azure.
	// This approach uses environment variables as described in azureblob package
	// documentation.
	// For example, to use shared key authentication, you would set
	// AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY.
	// To use a SAS token, you would set AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_SAS_TOKEN.
	// You can also construct a client using the azblob constructors directly, like
	// azblob.NewServiceClientWithSharedKey.
	client, err := azureblob.NewDefaultClient(serviceURL, containerName)
	if err != nil {
		log.Fatal(err)
	}

	// Create a *blob.Bucket.
	b, err := azureblob.OpenBucket(ctx, client, nil)
	if err != nil {
		log.Fatal(err)
	}
	defer b.Close()

	// Now we can use b to read or write files to the container.
	data, err := b.ReadAll(ctx, "my-key")
	if err != nil {
		log.Fatal(err)
	}
	_ = data
}

Output:

Types

type ContainerName added in v0.28.0 

type ContainerName string

ContainerName represents an Azure blob container name.

type Options 

type Options struct{}

Options sets options for constructing a *blob.Bucket backed by Azure Blob.

type ServiceURL added in v0.27.0 

type ServiceURL string

ServiceURL represents an Azure service URL.

func NewServiceURL added in v0.27.0 

func NewServiceURL(opts *ServiceURLOptions) (ServiceURL, error)

NewServiceURL generates a URL for addressing an Azure Blob service account. It uses several parameters, each of which can be specified via ServiceURLOptions.

The generated URL is "<protocol>://<account name>.<domain>" with the following caveats:

  • If opts.SASToken is provided, it is appended to the URL as a query parameter.
  • If opts.IsCDN is true, the <account name> part is dropped.
  • If opts.IsLocalEmulator is true, or the domain starts with "localhost" or "127.0.0.1", the account name and domain are flipped, e.g.: http://127.0.0.1:10000/myaccount

type ServiceURLOptions added in v0.27.0 

type ServiceURLOptions struct {
	// AccountName is the account name the credentials are for.
	AccountName string

	// SASToken will be appended to the service URL.
	// See https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1#shared-access-signature-parameters.
	SASToken string

	// StorageDomain can be provided to specify an Azure Cloud Environment
	// domain to target for the blob storage account (i.e. public, government, china).
	// Defaults to "blob.core.windows.net". Possible values will look similar
	// to this but are different for each cloud (i.e. "blob.core.govcloudapi.net" for USGovernment).
	// Check the Azure developer guide for the cloud environment where your bucket resides.
	// See the docstring for NewServiceURL to see examples of how this is used
	// along with the other Options fields.
	StorageDomain string

	// Protocol can be provided to specify protocol to access Azure Blob Storage.
	// Protocols that can be specified are "http" for local emulator and "https" for general.
	// Defaults to "https".
	// See the docstring for NewServiceURL to see examples of how this is used
	// along with the other Options fields.
	Protocol string

	// IsCDN can be set to true when using a CDN URL pointing to a blob storage account:
	// https://docs.microsoft.com/en-us/azure/cdn/cdn-create-a-storage-account-with-cdn
	// See the docstring for NewServiceURL to see examples of how this is used
	// along with the other Options fields.
	IsCDN bool

	// IsLocalEmulator should be set to true when targeting Local Storage Emulator (Azurite).
	// See the docstring for NewServiceURL to see examples of how this is used
	// along with the other Options fields.
	IsLocalEmulator bool
}

ServiceURLOptions sets options for constructing a service URL for Azure Blob.

func NewDefaultServiceURLOptions added in v0.27.0 

func NewDefaultServiceURLOptions() *ServiceURLOptions

NewDefaultServiceURLOptions generates a ServiceURLOptions based on environment variables.

type URLOpener added in v0.10.0 

type URLOpener struct {
	// MakeClient must be set to a non-nil value.
	MakeClient func(svcURL ServiceURL, containerName ContainerName) (*container.Client, error)

	// ServiceURLOptions specifies default options for generating the service URL.
	// Some options can be overridden in the URL as described above.
	ServiceURLOptions ServiceURLOptions

	// Options specifies the options to pass to OpenBucket.
	Options Options
}

URLOpener opens Azure URLs like "azblob://mybucket".

The URL host is used as the bucket name.

The following query options are supported:

  • domain: Overrides Options.StorageDomain.
  • protocol: Overrides Options.Protocol.
  • cdn: Overrides Options.IsCDN.
  • localemu: Overrides Options.IsLocalEmulator.

func (*URLOpener) OpenBucketURL added in v0.10.0 

func (o *URLOpener) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error)

OpenBucketURL opens a blob.Bucket based on u.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.