azblob

package
v0.18.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 20, 2024 License: MIT Imports: 23 Imported by: 11

Documentation

Overview

Package azblob reads/writes files to Azure blob storage in Chunks.

Package azblob reads/writes files to Azure blob storage in Chunks.

Package azblob reads/writes files to Azure blob storage in Chunks.

Package azblob reads/writes files to Azure blob storage in Chunks.

Package azblob reads/writes files to Azure blob storage in Chunks.

Package azblob reads/writes files to Azure blob storage in Chunks.

Package azblob reads/writes files to Azure blob storage in Chunks.

Index

Constants

View Source 
const (
	// metadata keys
	ContentKey = "content_type"
	HashKey    = "hash"
	MimeKey    = "mime_type"
	SizeKey    = "size"
	TimeKey    = "time_accepted"
)
View Source 
const (
	DebugLevel = logger.DebugLevel
)

Variables

View Source 
var (
	ErrMustSupportSeek0 = errors.New("must be seekable to position 0")
)
View Source 
var (
	ErrUnspecifiedContainer = errors.New("storer: container is unspecified")
)

Functions

This section is empty.

Types

type BytesSeekableReaderCloser 

type BytesSeekableReaderCloser struct {
	*bytes.Reader
}

BytesSeekableReader closer provides reader that has a No-Op Close and a usuable Seek. Because we need Seek, we can't use ioutil.NopCloser

func NewBytesReaderCloser 

func NewBytesReaderCloser(b []byte) *BytesSeekableReaderCloser

func (*BytesSeekableReaderCloser) Close 

func (io *BytesSeekableReaderCloser) Close() error

type ContainerClient 

type ContainerClient = azStorageBlob.ContainerClient

so we dont have to import azure blob package anywhere else

type DevConfig 

type DevConfig struct {
	AccountName string
	Key         string
	URL         string
}

func NewDevConfigFromEnv 

func NewDevConfigFromEnv() DevConfig

NewDevConfigFromEnv reads azurite (azure emulator) config from the standard azure env vars and falls back to the docmented defaults if they are not set. If overriding any settings via env, be sure to also configure AZURITE_ACCOUNTS for the emulator See: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite

type ETagCondition 

type ETagCondition int
const (
	EtagNotUsed ETagCondition = iota
	ETagMatch
	ETagNoneMatch
	TagsWhere
)

type Error 

type Error struct {
	// contains filtered or unexported fields
}

func ErrorFromError 

func ErrorFromError(err error) *Error

func NewStatusError 

func NewStatusError(text string, statusCode int) *Error

func (*Error) Error 

func (e *Error) Error() string

func (*Error) IsConditionNotMet 

func (e *Error) IsConditionNotMet() bool

IsConditionNotMet returns true if the err is the storage code indicating that a If- header predicate (eg ETag) was not met

func (*Error) StatusCode 

func (e *Error) StatusCode() int

StatusCode returns status code for failing request or 500 if code is not available on the error

func (*Error) StorageErrorCode 

func (e *Error) StorageErrorCode() string

StorageErrorCode returns the underlying azure storage ErrorCode string eg "BlobNotFound"

func (*Error) Unwrap 

func (e *Error) Unwrap() error

type FilterResponse added in v0.14.1 

type FilterResponse struct {
	Marker ListMarker // nil if no more pages

	// Standard request status things
	StatusCode int // For If- header fails, err can be nil and code can be 304
	Status     string

	Items []*azStorageBlob.FilterBlobItem
}

type GetMetadata 

type GetMetadata int
const (
	NoMetadata GetMetadata = iota
	OnlyMetadata
	BothMetadataAndBlob
)

func (GetMetadata) String 

func (g GetMetadata) String() string

type HTTPError 

type HTTPError interface {
	Error() string
	StatusCode() int
	Unwrap() error
}

HTTPError error type with info about http.StatusCode

type IfSinceCondition 

type IfSinceCondition int
const (
	IfConditionNotUsed IfSinceCondition = iota
	IfConditionModifiedSince
	IfConditionUnmodifiedSince
)

type LeaseRenewer 

type LeaseRenewer func(ctx context.Context) error

type ListMarker added in v0.10.2 

type ListMarker *string

type ListerResponse added in v0.10.2 

type ListerResponse struct {
	Marker ListMarker // nil if no more pages
	Prefix string

	// Standard request status things
	StatusCode int // For If- header fails, err can be nil and code can be 304
	Status     string

	Items []*azStorageBlob.BlobItemInternal
}

type Logger added in v0.17.2 

type Logger = logger.Logger

type Option 

type Option func(*StorerOptions)

func WithEtagMatch 

func WithEtagMatch(etag string) Option

WithEtagMatch succeed if the blob etag matches the provied value Typically used to make optimistic concurrency updates safe.

func WithEtagNoneMatch 

func WithEtagNoneMatch(etag string) Option

WithEtagNoneMatch succeed if the blob etag does *not* match the supplied value

func WithGetMetadata 

func WithGetMetadata(value GetMetadata) Option

WithGetMetadata specifies to get metadata - Reader() only.

func WithGetTags 

func WithGetTags() Option

func WithLeaseID 

func WithLeaseID(leaseID string) Option

WithLeaseID - specifies LeaseId - Reader() and Write()

func WithListDelim added in v0.10.2 

func WithListDelim(delim string) Option

TODO: this is an sdk v1.2.1 feature

func WithListMarker added in v0.10.2 

func WithListMarker(marker ListMarker) Option

func WithListMaxResults added in v0.14.0 

func WithListMaxResults(maxResults int32) Option

WithListMaxResults adds an option to limit the page size, default at time of writing 5000

func WithListMetadata added in v0.10.2 

func WithListMetadata() Option

func WithListPrefix added in v0.10.2 

func WithListPrefix(prefix string) Option

func WithListTags added in v0.10.2 

func WithListTags() Option

func WithMetadata 

func WithMetadata(metadata map[string]string) Option

WithMetadata specifies metadata to add - Write() only

func WithModifiedSince 

func WithModifiedSince(since *time.Time) Option

func WithSizeLimit 

func WithSizeLimit(sizeLimit int64) Option

WithSizeLimit specifies the size limit of the blob. -1 for unlimited. 0+ for limited.

func WithTags 

func WithTags(tags map[string]string) Option

WithTags specifies tags to add - Reader() and Write(). For Write) the tags are written with the blob. For Reader() the tags are used to apply ownership permissions.

func WithUnmodifiedSince 

func WithUnmodifiedSince(since *time.Time) Option

func WithWhereTags 

func WithWhereTags(whereTags string) Option

WithWhereTags succeed if the where clause matches the blob tags)

type Reader added in v0.12.2 

type Reader interface {
	Reader(
		ctx context.Context,
		identity string,
		opts ...Option,
	) (*ReaderResponse, error)
	FilteredList(ctx context.Context, tagsFilter string, opts ...Option) (*FilterResponse, error)
	List(ctx context.Context, opts ...Option) (*ListerResponse, error)
}

Reader is the interface in order to carry out read operations on the azure blob storage

func NewReaderDefaultAuth added in v0.18.5 

func NewReaderDefaultAuth(url string, opts ...ReaderOption) (Reader, error)

NewReaderDefaultAuth is a azure blob reader client that obtains credentials from the environment - including aad pod identity / workload identity.

func NewReaderNoAuth added in v0.12.2 

func NewReaderNoAuth(url string, opts ...ReaderOption) (Reader, error)

NewReaderNoAuth is a azure blob reader client that has no credentials.

Paramaters: 

	accountName: used only for logging purposes and may be empty
	url: The root path for the blob store requests, must not be empty
	opts: optional arguments specific to creating a reader with no auth

   * WithContainer() - specifies the azblob container for point get in the container
   * WithAccountName() - specifies the azblob account name for logging

NOTE: due to having no credentials, this can only read from public blob storage. or proxied private blob storage.

NOTE: if no optional container is specified than the Reader() method on the interface will error, as we cannot create a container client reader.

example: 

url: https://app.datatrails.ai/verifiabledata

type ReaderOption added in v0.16.0 

type ReaderOption func(*ReaderOptions)

func WithAccountName added in v0.16.0 

func WithAccountName(accountName string) ReaderOption

WithAccountName sets the azblob account name for the reader

func WithContainer added in v0.16.0 

func WithContainer(container string) ReaderOption

WithContainer sets the azblob container

type ReaderOptions added in v0.16.0 

type ReaderOptions struct {
	// contains filtered or unexported fields
}

ReaderOptions - optional args for specifying optional behaviour for 

azblob readers

func ParseReaderOptions added in v0.16.0 

func ParseReaderOptions(options ...ReaderOption) ReaderOptions

ParseReaderOptions parses the given options into a ReaderOptions struct

type ReaderResponse 

type ReaderResponse struct {
	Reader            io.ReadCloser
	HashValue         string
	MimeType          string
	ContentLength     int64
	Size              int64 // MIME size
	Tags              map[string]string
	TimestampAccepted string
	ScannedStatus     string
	ScannedBadReason  string
	ScannedTimestamp  string

	BlobClient *azStorageBlob.BlobClient

	// The following are copied as appropriate from the azure sdk response.
	// See also WriterResponse
	ETag         *string
	LastModified *time.Time
	Metadata     map[string]string // x-ms-meta header
	StatusCode   int               // For If- header fails, err can be nil and code can be 304
	Status       string
	XMsErrorCode string // will be "ConditioNotMet" for If- header predicate fails, even when err is nil
}

func (*ReaderResponse) ConditionNotMet 

func (r *ReaderResponse) ConditionNotMet() bool

func (*ReaderResponse) DownloadToWriter 

func (r *ReaderResponse) DownloadToWriter(w io.Writer) error

func (*ReaderResponse) Ok 

func (r *ReaderResponse) Ok() bool

Ok returns true if the http status was 200 or 201 This method is provided for use in combination with specific headers like If-Match and ETags conditions. In thos circumstances we often get err=nil but no content.

type ServiceClient 

type ServiceClient = azStorageBlob.ServiceClient

type SharedKeyCredential 

type SharedKeyCredential = azStorageBlob.SharedKeyCredential

type Storer 

type Storer struct {
	AccountName   string
	ResourceGroup string
	Subscription  string
	Container     string
	// contains filtered or unexported fields
}

Storer implements usage of Reader/Writer backed by azblob

func New 

func New(
	accountName string,
	resourceGroup string,
	subscription string,
	container string,
) (*Storer, error)

New returns new az blob read/write object

func NewDev 

func NewDev(cfg DevConfig, container string) (*Storer, error)

NewDev returns a normal blob client but connected for the azurite local emulator It uses the well known account name and key by default. If overriding, be sure to also configure AZURITE_ACCOUNTS for the emulator See: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite

func (*Storer) AcquireLease 

func (azp *Storer) AcquireLease(
	ctx context.Context, objectname string, leaseTimeout int32,
) (string, error)

AcquireLease gets a lease on a blob

func (*Storer) AcquireLeaseRenewable 

func (azp *Storer) AcquireLeaseRenewable(
	ctx context.Context, objectname string, leaseTimeout int32,
) (string, LeaseRenewer, error)

func (*Storer) Count 

func (azp *Storer) Count(ctx context.Context, tagsFilter string, opts ...Option) (int64, error)

Count counts the number of blobs filtered by the given tags filter

func (*Storer) Delete added in v0.10.2 

func (azp *Storer) Delete(
	ctx context.Context,
	identity string,
) error

Delete the identified blob

func (*Storer) FilteredList 

func (azp *Storer) FilteredList(ctx context.Context, tagsFilter string, opts ...Option) (*FilterResponse, error)

FilteredList returns a list of blobs filtered on their tag values.

tagsFilter examples: 

	 All tenants with more than one massif
	     "firstindex">'0000000000000000'

	 All tenants whose logs have been updated since a particular idtimestamp
	     "lastid > '018e84dbbb6513a6'"

 note: in the case where you are making up the id timestamp from a time
 reading, set the least significant 24 bits to zero and use the hex encoding
 of the resulting value

	All blobs in a storage account
		"cat='tiger' AND penguin='emperorpenguin'"
	All blobs in a specific container
		"@container='zoo' AND cat='tiger' AND penguin='emperorpenguin'"

See also: https://learn.microsoft.com/en-us/rest/api/storageservices/find-blobs-by-tags-container?tabs=microsoft-entra-id

Returns all blobs with the specific tag filter.

func (*Storer) GetContainerClient 

func (s *Storer) GetContainerClient() *azStorageBlob.ContainerClient

GetContainerClient returns the underlying container client

func (*Storer) GetServiceClient 

func (s *Storer) GetServiceClient() *azStorageBlob.ServiceClient

GetServiceClient returns the underlying service client

func (*Storer) List added in v0.10.2 

func (azp *Storer) List(ctx context.Context, opts ...Option) (*ListerResponse, error)

func (*Storer) Put 

func (azp *Storer) Put(
	ctx context.Context,
	identity string,
	source io.ReadSeekCloser,
	opts ...Option,
) (*WriteResponse, error)

Put creates or replaces a blob metadata and tags are set in the same operation as the content update.

func (*Storer) Reader 

func (azp *Storer) Reader(
	ctx context.Context,
	identity string,
	opts ...Option,
) (*ReaderResponse, error)

Reader creates a reader.

func (*Storer) ReleaseLease 

func (azp *Storer) ReleaseLease(ctx context.Context, objectname string, leaseID string,
) error

ReleaseLease release a lease on a blob

func (*Storer) ReleaseLeaseDeferable 

func (azp *Storer) ReleaseLeaseDeferable(ctx context.Context, objectname string, leaseID string)

ReleaseLeaseDeferable this is intended to use with defer - doesn't return error so we don't need to check it

func (*Storer) Write 

func (azp *Storer) Write(
	ctx context.Context,
	identity string,
	source io.Reader,
	opts ...Option,
) (*WriteResponse, error)

Write writes to blob from io.Reader.

func (*Storer) WriteStream 

func (azp *Storer) WriteStream(
	ctx context.Context,
	identity string,
	source *http.Request,
	opts ...Option,
) (*WriteResponse, error)

Write writes to blob from http request.

type StorerOptions 

type StorerOptions struct {
	// contains filtered or unexported fields
}

StorerOptions - optional args for specifying optional behaviour

type WriteResponse 

type WriteResponse struct {
	HashValue         string
	MimeType          string
	Size              int64
	TimestampAccepted string

	// The following fields are copied from the sdk response. nil pointers mean
	// 'not set'.  Note that different azure blob sdk write mode apis (put,
	// create, stream) etc return different generated types. So we copy only the
	// fields which make sense into this unified type. And we only copy values
	// which can be made use of in the api presented by this package.
	ETag         *string
	LastModified *time.Time
	StatusCode   int // For If- header fails, err can be nil and code can be 304
	Status       string
	XMsErrorCode string // will be "ConditioNotMet" for If- header predicate fails, even when err is nil
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.