blob

package
v0.0.0-...-379680d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 15, 2019 License: MIT Imports: 17 Imported by: 10

Documentation

Overview

Package blob defines types to refer to and retrieve low-level Perkeep blobs.

Index

Constants

View Source 
const Pattern = `\b([a-z][a-z0-9]*)-([a-f0-9]+)\b`

Pattern is the regular expression which matches a blobref. It does not contain ^ or $.

Variables

View Source 
var (
	ErrNegativeSubFetch         = errors.New("invalid negative subfetch parameters")
	ErrOutOfRangeOffsetSubFetch = errors.New("subfetch offset greater than blob size")
)
View Source 
var ErrUnimplemented = errors.New("optional method not implemented")

ErrUnimplemented is returned by optional interfaces when their wrapped values don't implemented the optional interface.

Functions

func HashFuncs 

func HashFuncs() []string

HashFuncs returns the names of the supported hash functions.

func NewHash 

func NewHash() hash.Hash

NewHash returns a new hash.Hash of the currently recommended hash type. Currently this is SHA-224, but is subject to change over time.

func ReaderAt 

func ReaderAt(ctx context.Context, sf SubFetcher, br Ref) io.ReaderAt

ReaderAt returns an io.ReaderAt of br, fetching against sf. The context is stored in and used by the returned ReaderAt.

func TypeAlphabet 

func TypeAlphabet(typ string) string

TypeAlphabet returns the valid characters in the given blobref type. It returns the empty string if the typ is unknown.

func ValidRefString 

func ValidRefString(s string) bool

Types

type Blob 

type Blob struct {
	// contains filtered or unexported fields
}

Blob represents a blob. Use the methods Size, SizedRef and ReadAll to query and get data from Blob.

func FromFetcher 

func FromFetcher(ctx context.Context, fetcher Fetcher, br Ref) (*Blob, error)

FromFetcher fetches br from fetcher and slurps its contents to memory. It does not validate the blob's digest. Use the Blob.ValidContents method for that.

func FromReader 

func FromReader(ctx context.Context, br Ref, r io.Reader, size uint32) (*Blob, error)

FromReader slurps the given blob from r to memory. It does not validate the blob's digest. Use the Blob.ValidContents method for that.

func NewBlob 

func NewBlob(ref Ref, size uint32, readAll func(ctx context.Context) ([]byte, error)) *Blob

NewBlob constructs a Blob from its Ref, size and a function that returns the contents of the blob. Any error in the function newReader when constructing the io.ReadCloser should be returned upon the first call to Read or Close.

func (*Blob) IsUTF8 

func (b *Blob) IsUTF8(ctx context.Context) (bool, error)

IsUTF8 reports whether the blob is entirely UTF-8.

func (*Blob) ReadAll 

func (b *Blob) ReadAll(ctx context.Context) (*bytes.Reader, error)

ReadAll reads the blob completely to memory, using the provided context, and then returns a reader over it. The Reader will not have errors except EOF.

The provided is only ctx is used until ReadAll returns.

func (*Blob) Ref 

func (b *Blob) Ref() Ref

Ref returns the blob's reference.

func (*Blob) Size 

func (b *Blob) Size() uint32

Size returns the size of the blob (in bytes).

func (*Blob) SizedRef 

func (b *Blob) SizedRef() SizedRef

SizedRef returns the SizedRef corresponding to the blob.

func (*Blob) ValidContents 

func (b *Blob) ValidContents(ctx context.Context) error

ValidContents reports whether the hash of blob's content matches its reference.

type ByRef 

type ByRef []Ref

ByRef sorts blob references.

func (ByRef) Len 

func (s ByRef) Len() int

func (ByRef) Less 

func (s ByRef) Less(i, j int) bool

func (ByRef) Swap 

func (s ByRef) Swap(i, j int)

type ChanPeeker 

type ChanPeeker struct {
	Ch <-chan SizedRef
	// contains filtered or unexported fields
}

ChanPeeker wraps a channel receiving SizedRefs and adds a 1 element buffer on it, with Peek and Take methods.

func (*ChanPeeker) Closed 

func (cp *ChanPeeker) Closed() bool

Closed reports true if no more SizedRef values are available.

func (*ChanPeeker) ConsumeAll 

func (cp *ChanPeeker) ConsumeAll()

ConsumeAll drains the channel of all items.

func (*ChanPeeker) MustPeek 

func (cp *ChanPeeker) MustPeek() SizedRef

MustPeek returns the next SizedRef or panics if none is available.

func (*ChanPeeker) MustTake 

func (cp *ChanPeeker) MustTake() SizedRef

MustTake returns the next SizedRef, else panics if none is available.

func (*ChanPeeker) Peek 

func (cp *ChanPeeker) Peek() (sr SizedRef, ok bool)

Peek returns the next SizedRef and whether one was available.

func (*ChanPeeker) Take 

func (cp *ChanPeeker) Take() (sr SizedRef, ok bool)

Take returns the next SizedRef and whether one was available for the taking.

type DirFetcher 

type DirFetcher struct {
	// contains filtered or unexported fields
}

func NewSimpleDirectoryFetcher 

func NewSimpleDirectoryFetcher(dir string) *DirFetcher

func (*DirFetcher) Fetch 

func (df *DirFetcher) Fetch(ctx context.Context, r Ref) (file io.ReadCloser, size uint32, err error)

type Fetcher 

type Fetcher interface {
	// Fetch returns a blob. If the blob is not found then
	// os.ErrNotExist should be returned for the error (not a wrapped
	// error with a ErrNotExist inside)
	//
	// The contents are not guaranteed to match the digest of the
	// provided Ref (e.g. when streamed over HTTP). Paranoid
	// callers should verify them.
	//
	// The caller must close blob.
	//
	// The provided context is used until blob is closed and its
	// cancelation should but may not necessarily cause reads from
	// blob to fail with an error.
	Fetch(context.Context, Ref) (blob io.ReadCloser, size uint32, err error)
}

Fetcher is the minimal interface for retrieving a blob from storage. The full storage interface is blobserver.Storage.

func NewSerialFetcher 

func NewSerialFetcher(fetchers ...Fetcher) Fetcher

type Ref 

type Ref struct {
	// contains filtered or unexported fields
}

Ref is a reference to a Perkeep blob. It is used as a value type and supports equality (with ==) and the ability to use it as a map key.

func MustParse 

func MustParse(s string) Ref

MustParse parse s as a blobref and panics on failure.

func Parse 

func Parse(s string) (ref Ref, ok bool)

Parse parse s as a blobref and returns the ref and whether it was parsed successfully.

func ParseBytes 

func ParseBytes(s []byte) (ref Ref, ok bool)

ParseBytes is like Parse, but parses from a byte slice.

func ParseKnown 

func ParseKnown(s string) (ref Ref, ok bool)

ParseKnown is like Parse, but only parse blobrefs known to this server. It returns ok == false for well-formed but unsupported blobrefs.

func ParseOrZero 

func ParseOrZero(s string) Ref

ParseOrZero parses as a blobref. If s is invalid, a zero Ref is returned which can be tested with the Valid method.

func RefFromBytes 

func RefFromBytes(b []byte) Ref

RefFromBytes returns a blobref from the given string, for the currently recommended hash function.

func RefFromHash 

func RefFromHash(h hash.Hash) Ref

RefFromHash returns a blobref representing the given hash. It panics if the hash isn't of a known type.

func RefFromString 

func RefFromString(s string) Ref

RefFromString returns a blobref from the given string, for the currently recommended hash function.

func (Ref) Digest 

func (r Ref) Digest() string

Digest returns the lower hex digest of the blobref, without the e.g. "sha224-" prefix. It panics if r is zero.

func (Ref) DigestPrefix 

func (r Ref) DigestPrefix(digits int) string

func (Ref) DomID 

func (r Ref) DomID() string

func (Ref) EqualString 

func (r Ref) EqualString(s string) bool

EqualString reports whether r.String() is equal to s. It does not allocate.

func (Ref) HasPrefix 

func (r Ref) HasPrefix(s string) bool

HasPrefix reports whether s is a prefix of r.String(). It returns false if s does not contain at least the digest name prefix (e.g. "sha224-") and one byte of digest. It does not allocate.

func (Ref) Hash 

func (r Ref) Hash() hash.Hash

Hash returns a new hash.Hash of r's type. It panics if r is zero.

func (Ref) HashMatches 

func (r Ref) HashMatches(h hash.Hash) bool

func (Ref) HashName 

func (r Ref) HashName() string

HashName returns the lowercase hash function name of the reference. It panics if r is zero.

func (Ref) IsSupported 

func (r Ref) IsSupported() bool

func (Ref) Less 

func (r Ref) Less(o Ref) bool

Less reports whether r sorts before o. Invalid references blobs sort first.

func (Ref) MarshalBinary 

func (r Ref) MarshalBinary() (data []byte, err error)

MarshalBinary implements Go's encoding.BinaryMarshaler interface.

func (Ref) MarshalJSON 

func (r Ref) MarshalJSON() ([]byte, error)

func (Ref) String 

func (r Ref) String() string

func (Ref) StringMinusOne 

func (r Ref) StringMinusOne() string

StringMinusOne returns the first string that's before String.

func (Ref) Sum32 

func (r Ref) Sum32() uint32

func (Ref) Sum64 

func (r Ref) Sum64() uint64

func (*Ref) UnmarshalBinary 

func (r *Ref) UnmarshalBinary(data []byte) error

UnmarshalBinary implements Go's encoding.BinaryUnmarshaler interface.

func (*Ref) UnmarshalJSON 

func (r *Ref) UnmarshalJSON(d []byte) error

func (Ref) Valid 

func (r Ref) Valid() bool

type SizedByRef 

type SizedByRef []SizedRef

SizedByRef sorts SizedRefs by their blobref.

func (SizedByRef) Len 

func (s SizedByRef) Len() int

func (SizedByRef) Less 

func (s SizedByRef) Less(i, j int) bool

func (SizedByRef) Swap 

func (s SizedByRef) Swap(i, j int)

type SizedRef 

type SizedRef struct {
	Ref  Ref    `json:"blobRef"`
	Size uint32 `json:"size"`
}

SizedRef is like a Ref but includes a size. It should also be used as a value type and supports equality.

func (SizedRef) HashMatches 

func (sr SizedRef) HashMatches(h hash.Hash) bool

func (SizedRef) Less 

func (sr SizedRef) Less(o SizedRef) bool

Less reports whether sr sorts before o. Invalid references blobs sort first.

func (SizedRef) String 

func (sr SizedRef) String() string

func (SizedRef) Valid 

func (sr SizedRef) Valid() bool

type SubFetcher 

type SubFetcher interface {
	// SubFetch returns part of a blob.
	// The caller must close the returned io.ReadCloser.
	// The Reader may return fewer than 'length' bytes. Callers should
	// check. The returned error should be: ErrNegativeSubFetch if any of
	// offset or length is negative, or os.ErrNotExist if the blob
	// doesn't exist, or ErrOutOfRangeOffsetSubFetch if offset goes over
	// the size of the blob. If the error is ErrUnimplemented, the caller should
	// treat this Fetcher as if it doesn't implement SubFetcher.
	SubFetch(ctx context.Context, ref Ref, offset, length int64) (io.ReadCloser, error)
}

A SubFetcher is a Fetcher that can retrieve part of a blob.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.