cos

package
v1.3.12 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 7, 2022 License: MIT Imports: 45 Imported by: 9

Documentation

Overview

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2021-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Index

Constants

View Source 
const (
	ExtTar    = ".tar"
	ExtTgz    = ".tgz"
	ExtTarTgz = ".tar.gz"
	ExtZip    = ".zip"

	// msgpack doesn't have a "common extension", see for instance:
	// * https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types
	// however, there seems to be a de-facto agreement wrt Content-Type
	// * application/msgpack
	// * application/x-msgpack (<<< recommended)
	// * application/*+msgpack
	// AIS uses the following single constant for both the default file extension
	// and for the Content-Type (the latter with offset [1:])
	ExtMsgpack = ".msgpack"
)

supported archive types (file extensions)

View Source 
const (
	ChecksumNone   = "none"
	ChecksumXXHash = "xxhash"
	ChecksumMD5    = "md5"
	ChecksumCRC32C = "crc32c"
	ChecksumSHA256 = "sha256" // crypto.SHA512_256 (SHA-2)
	ChecksumSHA512 = "sha512" // crypto.SHA512 (SHA-2)
)

checksums

View Source 
const (
	KiB = 1024
	MiB = 1024 * KiB
	GiB = 1024 * MiB
	TiB = 1024 * GiB
)

IEC (binary) units

View Source 
const (
	SizeofI64 = int(unsafe.Sizeof(uint64(0)))
	SizeofI32 = int(unsafe.Sizeof(uint32(0)))
	SizeofI16 = int(unsafe.Sizeof(uint16(0)))

	MLCG32 = 1103515245 // xxhash seed

	PermRWR   os.FileMode = 0o640 // POSIX perms
	PermRWXRX os.FileMode = 0o750
)

assorted common constants

View Source 
const (
	HdrRange                 = "Range" // Ref: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35
	HdrRangeValPrefix        = "bytes="
	HdrContentRange          = "Content-Range"
	HdrContentRangeValPrefix = "bytes " // Ref: https://tools.ietf.org/html/rfc7233#section-4.2
	HdrAcceptRanges          = "Accept-Ranges"
	HdrContentType           = "Content-Type"
	HdrContentTypeOptions    = "X-Content-Type-Options"
	HdrContentLength         = "Content-Length"
	HdrUserAgent             = "User-Agent"
	HdrAccept                = "Accept"
	HdrLocation              = "Location"
	HdrServer                = "Server"
	HdrETag                  = "ETag" // Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Hdrs/ETag
)
View Source 
const (
	// https://cloud.google.com/storage/docs/xml-api/reference-headers
	GsCksumHeader   = "x-goog-hash"
	GsVersionHeader = "x-goog-generation"

	// https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html
	// https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html
	S3CksumHeader   = "ETag"
	S3VersionHeader = "x-amz-version-id"

	// s3 api request headers
	S3HdrObjSrc        = "x-amz-copy-source"
	S3HdrMptCnt        = "x-amz-mp-parts-count"
	S3HdrContentSHA256 = "x-amz-content-sha256"
	S3HdrBckRegion     = "x-amz-bucket-region"

	S3ChecksumCRC32  = "x-amz-checksum-crc32"
	S3ChecksumCRC32C = "x-amz-checksum-crc32c"
	S3ChecksumSHA1   = "x-amz-checksum-sha1"
	S3ChecksumSHA256 = "x-amz-checksum-sha256"
	S3LastModified   = "Last-Modified"

	S3MetadataChecksumType = "x-amz-meta-ais-cksum-type"
	S3MetadataChecksumVal  = "x-amz-meta-ais-cksum-val"

	// https://docs.microsoft.com/en-us/rest/api/storageservices/get-blob-properties#response-headers
	AzCksumHeader   = "Content-MD5"
	AzVersionHeader = "ETag"
)

provider-specific headers (=> custom props, and more)

View Source 
const (
	ContentJSON           = "application/json"
	ContentJSONCharsetUTF = "application/json; charset=utf-8"
	ContentMsgPack        = "application/msgpack"
	ContentXML            = "application/xml"
	ContentBinary         = "application/octet-stream"
)

Ref: https://www.iana.org/assignments/media-types/media-types.xhtml

View Source 
const (
	CtxReadWrapper contextID = "readWrapper" // context key for ReadWrapperFunc
	CtxSetSize     contextID = "setSize"     // context key for SetSizeFunc
	CtxOriginalURL contextID = "origURL"     // context key for OriginalURL for HTTP cloud
)
View Source 
const (
	QuantityPercent = "percent"
	QuantityBytes   = "bytes"
)
View Source 
const (
	LetterRunes = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
	LenRunes    = len(LetterRunes)
)
View Source 
const ContentLengthUnknown = -1
View Source 
const (
	// Number of sync maps
	MultiSyncMapCount = 0x40
)
View Source 
const SizeofLen = SizeofI32

Internally size of byte slice or string is packed as this integer type. Since binary packed structs are used for fairly small amount of data, it is possible to use int32 instead of int64 to keep the length of the data.

View Source 
const TarBlockSize = 512 // Size of each block in a tar stream

Variables

View Source 
var (
	ErrInvalidQuantityUsage = errors.New("invalid quantity, format should be '81%' or '1GB'")

	ErrInvalidQuantityPercent = errors.New("percent must be in the range (0, 100)")
	ErrInvalidQuantityBytes   = errors.New("value (bytes) must be non-negative")
)
View Source 
var ArchExtensions = []string{ExtTar, ExtTgz, ExtTarTgz, ExtZip, ExtMsgpack}
View Source 
var ErrBufferUnderrun = errors.New("buffer underrun")
View Source 
var JSON jsoniter.API

JSON is used to Marshal/Unmarshal API json messages and is initialized in init function.

View Source 
var NoneCksum = NewCksum(ChecksumNone, "")

Functions

func Abs 

func Abs(a int) int

func AbsI64 

func AbsI64(a int64) int64

func AnyHasPrefixInSlice 

func AnyHasPrefixInSlice(prefix string, arr []string) bool

func Assert 

func Assert(cond bool)

func AssertMsg 

func AssertMsg(cond bool, msg string)

NOTE: when using Sprintf and such, `if (!cond) { AssertMsg(false, msg) }` is the preferable usage.

func AssertNoErr 

func AssertNoErr(err error)

func Assertf 

func Assertf(cond bool, f string, a ...any)

NOTE: Not to be used in the datapath - consider instead one of the flavors below.

func B2S 

func B2S(b int64, digits int) string

func BHead 

func BHead(b []byte) string

func BoolExtVar 

func BoolExtVar(f *flag.FlagSet, p *BoolExt, name, usage string)

func CapitalizeString 

func CapitalizeString(s string) string

func CeilAlign 

func CeilAlign(val, align uint) uint

CeilAlign returns smallest number bigger or equal to val, which is divisible by align

func CeilAlignInt64 

func CeilAlignInt64(val, align int64) int64

func ClearfAtomic 

func ClearfAtomic(f *uint64, flags uint64) (ok bool)

func Close 

func Close(closer io.Closer)

func ConvertToString 

func ConvertToString(value any) (valstr string, err error)

func CopyStruct 

func CopyStruct(dst, src any)

shallow copy

func CreateDir 

func CreateDir(dir string) error

CreateDir creates directory if does not exist. Does not return error when directory already exists.

func CreateFile 

func CreateFile(fqn string) (*os.File, error)

CreateFile creates a new write-only (O_WRONLY) file with default cos.PermRWR permissions. NOTE: if the file pathname doesn't exist it'll be created. NOTE: if the file already exists it'll be also silently truncated.

func DamerauLevenstheinDistance 

func DamerauLevenstheinDistance(s, t string) int

based on Wikipedia https://en.wikipedia.org/wiki/Damerau–Levenshtein_distance

func DivCeil 

func DivCeil(a, b int64) int64

func DivRound 

func DivRound(a, b int64) int64

func DivRoundU64 

func DivRoundU64(a, b uint64) uint64

func DrainReader 

func DrainReader(r io.Reader)

DrainReader reads and discards all the data from a reader. No need for `io.CopyBuffer` as `io.Discard` has efficient `io.ReaderFrom` implementation.

func DurationExtVar 

func DurationExtVar(f *flag.FlagSet, p *DurationExt, name string, defValue time.Duration, usage string)

func Either 

func Either(lhs, rhs string) string

Either returns either lhs or rhs depending on which one is non-empty

func Errorf 

func Errorf(format string, a ...any)

func ExitLogf 

func ExitLogf(f string, a ...any)

ExitLogf is glog + Exitf.

func Exitf 

func Exitf(f string, a ...any)

Exitf writes formatted message to STDERR and exits with non-zero status code.

func ExpandPath 

func ExpandPath(path string) string

ExpandPath replaces common abbreviations in file path (eg. `~` with absolute path to the current user home directory) and cleans the path.

func FastLog2 

func FastLog2(c uint64) uint

FastLog2 returns floor(log2(c))

func FastLog2Ceil 

func FastLog2Ceil(c uint64) uint

func FloodWriter 

func FloodWriter(w io.Writer, n int64) error

FloodWriter writes `n` random bytes to provided writer.

func FlushClose 

func FlushClose(file *os.File) (err error)

func FormatMilli 

func FormatMilli(tm time.Duration) string

FormatMilli returns a duration formatted as milliseconds. For values bigger than millisecond, it returns an integer number "#ms". For values smaller than millisecond, the function returns fractional number "0.##ms"

func FormatTimestamp 

func FormatTimestamp(tm time.Time) string

func FormatUnixNano 

func FormatUnixNano(unixnano int64, format string) string

func FreeMemToOS 

func FreeMemToOS(d ...time.Duration)

FreeMemToOS calls GC and returns allocated memory to OS after that Use to clean up memory after a huge amount of memory becomes "free" to return it to OS immediately without waiting for GC does it automatically Params: 

d - a delay before starting memory cleanup

func GenDaemonID 

func GenDaemonID() string

func GenTestingDaemonID 

func GenTestingDaemonID(suffix string) string

func GenTie 

func GenTie() string

func GenUUID 

func GenUUID() (uuid string)

GenUUID generates unique and human-readable IDs.

func HomeConfigDir 

func HomeConfigDir(subdir string) (configDir string)

func HomeDir 

func HomeDir() (string, error)

func Infof 

func Infof(format string, a ...any)

func InitShortID 

func InitShortID(seed uint64)

func IsAlphaPlus 

func IsAlphaPlus(s string, withPeriod bool) bool

alpha-numeric++ including letters, numbers, dashes (-), and underscores (_) period (.) is allowed conditionally except for '..'

func IsAnySetfAtomic 

func IsAnySetfAtomic(f *uint64, flags uint64) (yes bool)

func IsAzureURL 

func IsAzureURL(u *url.URL) bool

func IsEOF 

func IsEOF(err error) bool

including "unexpecting EOF" to accommodate unsized streaming and early termination of the other side (prior to sending the first byte)

func IsErrBadCksum 

func IsErrBadCksum(err error) bool

func IsErrBrokenPipe 

func IsErrBrokenPipe(err error) (yes bool)

func IsErrConnectionNotAvail 

func IsErrConnectionNotAvail(err error) (yes bool)

likely out of socket descriptors

func IsErrConnectionRefused 

func IsErrConnectionRefused(err error) (yes bool)

retriable conn errs

func IsErrConnectionReset 

func IsErrConnectionReset(err error) (yes bool)

func IsErrOOS 

func IsErrOOS(err error) bool

func IsErrXattrNotFound 

func IsErrXattrNotFound(err error) bool

func IsGoogleAPIURL 

func IsGoogleAPIURL(u *url.URL) bool

func IsGoogleStorageURL 

func IsGoogleStorageURL(u *url.URL) bool

func IsGzipped 

func IsGzipped(filename string) bool

func IsHTTP 

func IsHTTP(url string) bool

func IsHTTPS 

func IsHTTPS(url string) bool

func IsIOError 

func IsIOError(err error) bool

Checks if the error is generated by any IO operation and if the error is severe enough to run the FSHC for mountpath testing

For mountpath definition, see fs/mountfs.go

func IsParseBool 

func IsParseBool(s string) bool

func IsRetriableConnErr 

func IsRetriableConnErr(err error) (yes bool)

func IsS3URL 

func IsS3URL(link string) bool

func IsSetfAtomic 

func IsSetfAtomic(f *uint64, flags uint64) (yes bool)

func IsTimeZero 

func IsTimeZero(t time.Time) bool

func IsUnreachable 

func IsUnreachable(err error, status int) bool

func IsValidUUID 

func IsValidUUID(uuid string) bool

func JoinPath 

func JoinPath(url, path string) string

JoinPath joins two path elements that may (or may not) be prefixed/suffixed with a slash.

func JoinWords 

func JoinWords(w string, words ...string) (path string)

JoinWords uses forward slash to join any number of words into a single path. Returned path is prefixed with a slash.

func Max 

func Max(xs ...int) int

Max returns max value from given ints.

func MaxDuration 

func MaxDuration(a, b time.Duration) time.Duration

MaxDuration returns min value of a and b time.Duration types

func MaxI32 

func MaxI32(a, b int32) int32

MaxI32 returns max value of a and b for int32 types

func MaxI64 

func MaxI64(a, b int64) int64

MaxI64 returns max value of a and b for int64 types

func MaxU64 

func MaxU64(a, b uint64) uint64

MaxU64 returns max value of a and b for uint64 types

func MaxUint 

func MaxUint(a, b uint) uint

func Mime 

func Mime(mime, filename string) (ext string, err error)

Map user-specified mime type OR the filename's extension to one of the supported ArchExtensions

func Min 

func Min(xs ...int) int

Min returns min value from given ints.

func MinDuration 

func MinDuration(a, b time.Duration) time.Duration

MinDuration returns min value of a and b time.Duration types

func MinF64 

func MinF64(a, b float64) float64

func MinI32 

func MinI32(a, b int32) int32

MinI32 returns min value of a and b for int32 types

func MinI64 

func MinI64(a, b int64) int64

MinI64 returns min value of a and b for int64 types

func MinTime 

func MinTime(a, b time.Time) time.Time

func MinU64 

func MinU64(a, b uint64) uint64

MinU64 returns min value of a and b for uint64 types

func MinUint 

func MinUint(a, b uint) uint

func MorphMarshal 

func MorphMarshal(data, v any) error

func MustMarshal 

func MustMarshal(v any) []byte

MustMarshal marshals v and panics if error occurs.

func MustMarshalToString 

func MustMarshalToString(v any) string

func MustMorphMarshal 

func MustMorphMarshal(data, v any)

func NewBadDataCksumError 

func NewBadDataCksumError(a, b *Cksum, context ...string) error

func NewBadMetaCksumError 

func NewBadMetaCksumError(a, b uint64, context ...string) error

func NewCRC32C 

func NewCRC32C() hash.Hash

func NopReader 

func NopReader(size int64) io.Reader

func NounEnding 

func NounEnding(count int) string

func NowRand 

func NowRand() *rand.Rand

func OpenTarForAppend 

func OpenTarForAppend(objName, workFQN string) (*os.File, error)

OpenTarForAppend opens a TAR and uses tar's reader Next() to skip to the position right _after_ the last file in the TAR (padding bytes including).

Background: 

TAR file is padded with one or more 512-byte blocks of zero bytes.
The blocks must be overwritten, otherwise newly added files won't be
accessible. Different TAR formats (such as `ustar`, `pax` and `GNU`)
write different number of zero blocks.

func PackedStrLen 

func PackedStrLen(s string) int

PackedStrLen returns the size occupied by a given string in the output

func ParseBool 

func ParseBool(s string) (value bool, err error)

ParseBool converts string to bool (case-insensitive): 

y, yes, on -> true
n, no, off, <empty value> -> false

strconv handles the following: 

1, true, t -> true
0, false, f -> false

func ParseEnvVariables 

func ParseEnvVariables(fpath string, delimiter ...string) map[string]string

ParseEnvVariables takes in a .env file and parses its contents

func ParseURL 

func ParseURL(s string) (u *url.URL, valid bool)

func Plural 

func Plural(num int) (s string)

(common use)

func ProbingFrequency 

func ProbingFrequency(dur time.Duration) time.Duration

wait duration => probing frequency

func RandStringStrong 

func RandStringStrong(n int) string

func RandStringWithSrc 

func RandStringWithSrc(src rand.Source, n int) string

func Ratio 

func Ratio(high, low, curr int64) float32

func RatioPct 

func RatioPct(high, low, curr int64) int64

func ReadLines 

func ReadLines(filename string, cb func(string) error) error

Read a file line by line and call a callback for each line until the file ends or a callback returns io.EOF

func ReadOneInt64 

func ReadOneInt64(filename string) (int64, error)

Read only the first line of a file and return it as int64 Do not use for big files: it reads all the content and then extracts the first line. Use for files that may contains a few lines with trailing EOL

func ReadOneLine 

func ReadOneLine(filename string) (string, error)

Read only the first line of a file. Do not use for big files: it reads all the content and then extracts the first line. Use for files that may contains a few lines with trailing EOL

func ReadOneUint64 

func ReadOneUint64(filename string) (uint64, error)

Read only the first line of a file and return it as uint64 Do not use for big files: it reads all the content and then extracts the first line. Use for files that may contains a few lines with trailing EOL

func RemoveFile 

func RemoveFile(path string) (err error)

RemoveFile removes path; returns nil upon success or if the path does not exist.

func Rename 

func Rename(src, dst string) (err error)

(creates destination directory if doesn't exist)

func ReparseQuery 

func ReparseQuery(r *http.Request)

WARNING: `ReparseQuery` might affect non-tensorflow clients using S3-compatible API with AIStore. To be used with caution.

func S2B 

func S2B(s string) (int64, error)

func S2Duration 

func S2Duration(s string) (time.Duration, error)

func S2UnixNano 

func S2UnixNano(s string) (int64, error)

func SHead 

func SHead(s string) string

func SetfAtomic 

func SetfAtomic(f *uint64, flags uint64) (ok bool)

func Stat 

func Stat(path string) error

syscall to check that path exists (see bench/lstat)

func StrSlicesEqual 

func StrSlicesEqual(lhs, rhs []string) bool

StrSlicesEqual compares content of two string slices. It is replacement for reflect.DeepEqual because the latter returns false if slices have the same values but in different order.

func StrToSentence 

func StrToSentence(str string) string

func StringInSlice 

func StringInSlice(s string, arr []string) bool

func StringSliceToIntSlice 

func StringSliceToIntSlice(strs []string) ([]int64, error)

func SupportedChecksums 

func SupportedChecksums() (types []string)

func UnixNano2S 

func UnixNano2S(unixnano int64) string

func UnsafeS 

func UnsafeS(b []byte) string

UnsafeS casts bytes to an immutable string. ***** CAUTION! the resulting string must never change *****

func UnsignedB2S 

func UnsignedB2S(b uint64, digits int) string

func UnwrapSyscallErr 

func UnwrapSyscallErr(err error) error

func ValidateCksumType 

func ValidateCksumType(ty string, emptyOK ...bool) (err error)

func ValidateDaemonID 

func ValidateDaemonID(id string) error

func ValidateEtlID 

func ValidateEtlID(id string) error

func Warningf 

func Warningf(format string, a ...any)

Types

type BitFlags 

type BitFlags uint64

func (BitFlags) Clear 

func (f BitFlags) Clear(flags BitFlags) BitFlags

func (BitFlags) IsAnySet 

func (f BitFlags) IsAnySet(flags BitFlags) bool

func (BitFlags) IsSet 

func (f BitFlags) IsSet(flags BitFlags) bool

func (BitFlags) Set 

func (f BitFlags) Set(flags BitFlags) BitFlags

type Bits 

type Bits uint8

func (*Bits) Clear 

func (b *Bits) Clear(flag Bits)

func (*Bits) Has 

func (b *Bits) Has(flag Bits) bool

func (*Bits) Set 

func (b *Bits) Set(flag Bits)

func (*Bits) Toggle 

func (b *Bits) Toggle(flag Bits)

type BoolExt 

type BoolExt struct {
	IsSet bool
	Val   bool
}

func (*BoolExt) Get 

func (b *BoolExt) Get() any

func (*BoolExt) IsBoolFlag 

func (*BoolExt) IsBoolFlag() bool

func (*BoolExt) Set 

func (b *BoolExt) Set(s string) (err error)

func (*BoolExt) String 

func (b *BoolExt) String() string

type ByteHandle 

type ByteHandle struct {
	*bytes.Reader
	// contains filtered or unexported fields
}

ByteHandle is a byte buffer(made from []byte) that implements ReadOpenCloser interface

func NewByteHandle 

func NewByteHandle(bt []byte) *ByteHandle

func (*ByteHandle) Close 

func (*ByteHandle) Close() error

func (*ByteHandle) Open 

func (b *ByteHandle) Open() (ReadOpenCloser, error)

type BytePack 

type BytePack struct {
	// contains filtered or unexported fields
}

func NewPacker 

func NewPacker(buf []byte, bufLen int) *BytePack

func (*BytePack) Bytes 

func (bw *BytePack) Bytes() []byte

func (*BytePack) WriteAny 

func (bw *BytePack) WriteAny(st Packer)

func (*BytePack) WriteBool 

func (bw *BytePack) WriteBool(b bool)

func (*BytePack) WriteByte 

func (bw *BytePack) WriteByte(b byte)

func (*BytePack) WriteBytes 

func (bw *BytePack) WriteBytes(b []byte)

func (*BytePack) WriteInt16 

func (bw *BytePack) WriteInt16(i int16)

func (*BytePack) WriteInt32 

func (bw *BytePack) WriteInt32(i int32)

func (*BytePack) WriteInt64 

func (bw *BytePack) WriteInt64(i int64)

func (*BytePack) WriteMapStrUint16 

func (bw *BytePack) WriteMapStrUint16(mp MapStrUint16)

func (*BytePack) WriteString 

func (bw *BytePack) WriteString(s string)

func (*BytePack) WriteUint16 

func (bw *BytePack) WriteUint16(i uint16)

func (*BytePack) WriteUint32 

func (bw *BytePack) WriteUint32(i uint32)

func (*BytePack) WriteUint64 

func (bw *BytePack) WriteUint64(i uint64)

type ByteUnpack 

type ByteUnpack struct {
	// contains filtered or unexported fields
}

func NewUnpacker 

func NewUnpacker(buf []byte) *ByteUnpack

func (*ByteUnpack) Bytes 

func (br *ByteUnpack) Bytes() []byte

func (*ByteUnpack) Len 

func (br *ByteUnpack) Len() int

func (*ByteUnpack) ReadAny 

func (br *ByteUnpack) ReadAny(st Unpacker) error

func (*ByteUnpack) ReadBool 

func (br *ByteUnpack) ReadBool() (bool, error)

func (*ByteUnpack) ReadByte 

func (br *ByteUnpack) ReadByte() (byte, error)

func (*ByteUnpack) ReadBytes 

func (br *ByteUnpack) ReadBytes() ([]byte, error)

func (*ByteUnpack) ReadInt16 

func (br *ByteUnpack) ReadInt16() (int16, error)

func (*ByteUnpack) ReadInt32 

func (br *ByteUnpack) ReadInt32() (int32, error)

func (*ByteUnpack) ReadInt64 

func (br *ByteUnpack) ReadInt64() (int64, error)

func (*ByteUnpack) ReadMapStrUint16 

func (br *ByteUnpack) ReadMapStrUint16() (MapStrUint16, error)

func (*ByteUnpack) ReadString 

func (br *ByteUnpack) ReadString() (string, error)

func (*ByteUnpack) ReadUint16 

func (br *ByteUnpack) ReadUint16() (uint16, error)

func (*ByteUnpack) ReadUint32 

func (br *ByteUnpack) ReadUint32() (uint32, error)

func (*ByteUnpack) ReadUint64 

func (br *ByteUnpack) ReadUint64() (uint64, error)

type CallbackROC 

type CallbackROC struct {
	// contains filtered or unexported fields
}

readers

func NewCallbackReadOpenCloser 

func NewCallbackReadOpenCloser(r ReadOpenCloser, readCb func(int, error), reportedBytes ...int) *CallbackROC

func (*CallbackROC) Close 

func (r *CallbackROC) Close() error

func (*CallbackROC) Open 

func (r *CallbackROC) Open() (ReadOpenCloser, error)

func (*CallbackROC) Read 

func (r *CallbackROC) Read(p []byte) (n int, err error)

type Cksum 

type Cksum struct {
	// contains filtered or unexported fields
}

func ChecksumBytes 

func ChecksumBytes(b []byte, cksumType string) (cksum *Cksum, err error)

ChecksumBytes computes checksum of given bytes using additional buffer.

func NewCksum 

func NewCksum(ty, value string) *Cksum

func (*Cksum) Clone 

func (ck *Cksum) Clone() *Cksum

func (*Cksum) Equal 

func (ck *Cksum) Equal(to *Cksum) bool

func (*Cksum) Get 

func (ck *Cksum) Get() (string, string)

func (*Cksum) IsEmpty 

func (ck *Cksum) IsEmpty() bool

func (*Cksum) MarshalJSON 

func (ck *Cksum) MarshalJSON() ([]byte, error)

func (*Cksum) String 

func (ck *Cksum) String() string

func (*Cksum) Ty 

func (ck *Cksum) Ty() string

func (*Cksum) Type 

func (ck *Cksum) Type() string

func (*Cksum) Val 

func (ck *Cksum) Val() string

func (*Cksum) Value 

func (ck *Cksum) Value() string

type CksumHash 

type CksumHash struct {
	Cksum
	H hash.Hash
	// contains filtered or unexported fields
}

func CopyAndChecksum 

func CopyAndChecksum(w io.Writer, r io.Reader, buf []byte, cksumType string) (n int64, cksum *CksumHash, err error)

CopyAndChecksum reads io.Reader and writes io.Writer; returns bytes written, checksum, and error

func CopyFile 

func CopyFile(src, dst string, buf []byte, cksumType string) (written int64, cksum *CksumHash, err error)

and computes checksum if requested

func NewCksumHash 

func NewCksumHash(ty string) (ck *CksumHash)

func SaveReader 

func SaveReader(fqn string, reader io.Reader, buf []byte, cksumType string,
	size int64, dirMustExist string) (cksum *CksumHash, err error)

Saves the reader directly to a local file, xxhash-checksums if requested

func SaveReaderSafe 

func SaveReaderSafe(tmpfqn, fqn string, reader io.Reader, buf []byte, cksumType string,
	size int64, dirMustExist string) (cksum *CksumHash, err error)

same as above, plus rename

func (*CksumHash) Equal 

func (ck *CksumHash) Equal(to *Cksum) bool

func (*CksumHash) Finalize 

func (ck *CksumHash) Finalize()

func (*CksumHash) Init 

func (ck *CksumHash) Init(ty string)

func (*CksumHash) Sum 

func (ck *CksumHash) Sum() []byte

type CksumHashSize 

type CksumHashSize struct {
	CksumHash
	Size int64
}

func (*CksumHashSize) Write 

func (ck *CksumHashSize) Write(b []byte) (n int, err error)

type Duration 

type Duration time.Duration

func (Duration) D 

func (d Duration) D() time.Duration

func (Duration) MarshalJSON 

func (d Duration) MarshalJSON() ([]byte, error)

func (Duration) String 

func (d Duration) String() (s string)

func (*Duration) UnmarshalJSON 

func (d *Duration) UnmarshalJSON(b []byte) (err error)

type DurationExt 

type DurationExt struct {
	IsSet bool
	Val   time.Duration
}

func (*DurationExt) Get 

func (d *DurationExt) Get() any

func (*DurationExt) IsBoolFlag 

func (*DurationExt) IsBoolFlag() bool

func (*DurationExt) Set 

func (d *DurationExt) Set(s string) (err error)

func (*DurationExt) String 

func (d *DurationExt) String() string

type DynSemaphore 

type DynSemaphore struct {
	// contains filtered or unexported fields
}

DynSemaphore implements sempahore which can change its size during usage.

func NewDynSemaphore 

func NewDynSemaphore(n int) *DynSemaphore

func (*DynSemaphore) Acquire 

func (s *DynSemaphore) Acquire(cnts ...int)

func (*DynSemaphore) Release 

func (s *DynSemaphore) Release(cnts ...int)

func (*DynSemaphore) SetSize 

func (s *DynSemaphore) SetSize(n int)

func (*DynSemaphore) Size 

func (s *DynSemaphore) Size() int

type ErrBadCksum 

type ErrBadCksum struct {
	// contains filtered or unexported fields
}

func (*ErrBadCksum) Error 

func (e *ErrBadCksum) Error() string

type ErrSignal 

type ErrSignal struct {
	// contains filtered or unexported fields
}

func NewSignalError 

func NewSignalError(s syscall.Signal) *ErrSignal

func (*ErrSignal) Error 

func (e *ErrSignal) Error() string

func (*ErrSignal) ExitCode 

func (e *ErrSignal) ExitCode() int

https://tldp.org/LDP/abs/html/exitcodes.html

type ErrTemplate 

type ErrTemplate struct {
	// contains filtered or unexported fields
}

func (*ErrTemplate) Error 

func (e *ErrTemplate) Error() string

type ErrUnknownMime 

type ErrUnknownMime struct {
	// contains filtered or unexported fields
}

func NewUnknownMimeError 

func NewUnknownMimeError(d string) *ErrUnknownMime

func (*ErrUnknownMime) Error 

func (e *ErrUnknownMime) Error() string

type ErrValue 

type ErrValue struct {
	atomic.Value
	// contains filtered or unexported fields
}

func (*ErrValue) Err 

func (ea *ErrValue) Err() (err error)

func (*ErrValue) IsNil 

func (ea *ErrValue) IsNil() bool

func (*ErrValue) Load 

func (*ErrValue) Load() any

NOTE: hide atomic.Value.Load() - must use Err() below

func (*ErrValue) Store 

func (ea *ErrValue) Store(err error)

type FileHandle 

type FileHandle struct {
	*os.File
	// contains filtered or unexported fields
}

handles (and more readers)

func NewFileHandle 

func NewFileHandle(fqn string) (*FileHandle, error)

func (*FileHandle) Open 

func (f *FileHandle) Open() (ReadOpenCloser, error)

type FileSectionHandle 

type FileSectionHandle struct {
	// contains filtered or unexported fields
}

FileSectionHandle opens a file and reads a section of it with optional padding. It implements the ReadOpenCloser interface.

func NewFileSectionHandle 

func NewFileSectionHandle(fqn string, offset, size int64) (*FileSectionHandle, error)

NewFileSectionHandle opens file which is expected at `fqn` and defines a SectionHandle on it to only read a specified section.

func (*FileSectionHandle) Close 

func (f *FileSectionHandle) Close() error

func (*FileSectionHandle) Open 

func (f *FileSectionHandle) Open() (ReadOpenCloser, error)

func (*FileSectionHandle) Read 

func (f *FileSectionHandle) Read(buf []byte) (int, error)

type FsID 

type FsID [2]int32

FsID is unified and cross-platform syscall.Fsid type which implements JSON marshaling.

func (FsID) MarshalJSON 

func (d FsID) MarshalJSON() ([]byte, error)

func (FsID) String 

func (d FsID) String() string

func (*FsID) UnmarshalJSON 

func (d *FsID) UnmarshalJSON(b []byte) error

type JSONRawMsgs 

type JSONRawMsgs map[string]jsoniter.RawMessage

type LimitedWaitGroup 

type LimitedWaitGroup struct {
	// contains filtered or unexported fields
}

LimitedWaitGroup is helper struct which combines standard wait group and semaphore to limit the number of goroutines created.

func (*LimitedWaitGroup) Add 

func (lwg *LimitedWaitGroup) Add(n int)

func (*LimitedWaitGroup) Done 

func (lwg *LimitedWaitGroup) Done()

func (*LimitedWaitGroup) Wait 

func (lwg *LimitedWaitGroup) Wait()

type MapStrUint16 

type MapStrUint16 map[string]uint16

type MemCPUInfo 

type MemCPUInfo struct {
	MemUsed    uint64  `json:"mem_used"`
	MemAvail   uint64  `json:"mem_avail"`
	PctMemUsed float64 `json:"pct_mem_used"`
	PctCPUUsed float64 `json:"pct_cpu_used"`
}

type MultiSyncMap 

type MultiSyncMap struct {
	M [MultiSyncMapCount]sync.Map
}

func (*MultiSyncMap) Get 

func (msm *MultiSyncMap) Get(idx int) *sync.Map

func (*MultiSyncMap) GetByHash 

func (msm *MultiSyncMap) GetByHash(hash uint32) *sync.Map

type NamedVal64 

type NamedVal64 struct {
	Name       string
	NameSuffix string // forces immediate send when non-empty (see NOTE below)
	Value      int64
}

type Packer 

type Packer interface {
	Pack(packer *BytePack)
	PackedSize() int
}

type ParsedQuantity 

type ParsedQuantity struct {
	Type  string
	Value uint64
}

func ParseQuantity 

func ParseQuantity(quantity string) (ParsedQuantity, error)

func (ParsedQuantity) String 

func (pq ParsedQuantity) String() string

type ParsedTemplate 

type ParsedTemplate struct {
	Prefix string
	Ranges []TemplateRange
	// contains filtered or unexported fields
}

func NewParsedTemplate 

func NewParsedTemplate(template string) (ParsedTemplate, error)

func ParseAtTemplate 

func ParseAtTemplate(template string) (pt ParsedTemplate, err error)

e.g. multi-range template: "prefix-@00001-gap-@100-suffix" 

single range:         "prefix@00100suffix"

func ParseBashTemplate 

func ParseBashTemplate(template string) (pt ParsedTemplate, err error)

e.g. single-range template: "prefix{0001..0010}suffix" 

multi-range:           "prefix-{00001..00010..2}-gap-{001..100..2}-suffix"

(both prefix and suffix are optional, here and elsewhere)

func ParseFmtTemplate 

func ParseFmtTemplate(template string) (pt ParsedTemplate, err error)

template: "prefix-%06d-suffix" (both prefix and suffix are optional, here and elsewhere)

func (*ParsedTemplate) Count 

func (pt *ParsedTemplate) Count() int64

func (*ParsedTemplate) InitIter 

func (pt *ParsedTemplate) InitIter()

func (*ParsedTemplate) Next 

func (pt *ParsedTemplate) Next() (string, bool)

func (*ParsedTemplate) ToSlice 

func (pt *ParsedTemplate) ToSlice(maxLen ...int) []string

maxLen specifies maximum objects to be returned

type ReadCloseSizer 

type ReadCloseSizer interface {
	io.ReadCloser
	Size() int64
}

ReadCloseSizer is the interface that adds Size method to io.ReadCloser.

func NewDeferRCS 

func NewDeferRCS(r ReadCloseSizer, cb func()) ReadCloseSizer

func NewSizedRC 

func NewSizedRC(r io.ReadCloser, size int64) ReadCloseSizer

type ReadOpenCloseSizer 

type ReadOpenCloseSizer interface {
	ReadOpenCloser
	Size() int64
}

ReadOpenCloseSizer is the interface that adds Size method to ReadOpenCloser.

type ReadOpenCloser 

type ReadOpenCloser interface {
	io.ReadCloser
	Open() (ReadOpenCloser, error)
}

readers

func NopOpener 

func NopOpener(r io.ReadCloser) ReadOpenCloser

type ReadReaderAt 

type ReadReaderAt interface {
	io.Reader
	io.ReaderAt
}

readers

type ReadSizer 

type ReadSizer interface {
	io.Reader
	Size() int64
}

ReadSizer is the interface that adds Size method to io.Reader.

func NewSizedReader 

func NewSizedReader(r io.Reader, size int64) ReadSizer

type ReadWrapperFunc 

type ReadWrapperFunc func(r io.ReadCloser) io.ReadCloser

type ReaderArgs 

type ReaderArgs struct {
	R       io.Reader
	ReadCb  func(int, error)
	DeferCb func()
	Size    int64
}

readers

type ReaderWithArgs 

type ReaderWithArgs struct {
	// contains filtered or unexported fields
}

readers

func NewReaderWithArgs 

func NewReaderWithArgs(args ReaderArgs) *ReaderWithArgs

func (*ReaderWithArgs) Close 

func (r *ReaderWithArgs) Close() (err error)

func (*ReaderWithArgs) Open 

func (*ReaderWithArgs) Open() (ReadOpenCloser, error)

func (*ReaderWithArgs) Read 

func (r *ReaderWithArgs) Read(p []byte) (n int, err error)

func (*ReaderWithArgs) Size 

func (r *ReaderWithArgs) Size() int64

type Runner 

type Runner interface {
	Name() string
	Run() error
	Stop(error)
}

type SectionHandle 

type SectionHandle struct {
	// contains filtered or unexported fields
}

SectionHandle is a section of reader with optional padding that implements ReadOpenCloser interface.

func NewSectionHandle 

func NewSectionHandle(r io.ReaderAt, offset, size, padding int64) *SectionHandle

func (*SectionHandle) Close 

func (*SectionHandle) Close() error

func (*SectionHandle) Open 

func (f *SectionHandle) Open() (ReadOpenCloser, error)

func (*SectionHandle) Read 

func (f *SectionHandle) Read(buf []byte) (n int, err error)

Reads a reader section. When the slice finishes but the buffer is not filled yet, act as if it reads a few more bytes from somewhere.

type Semaphore 

type Semaphore struct {
	// contains filtered or unexported fields
}

Semaphore implements sempahore which is just a nice wrapper on `chan struct{}`.

func NewSemaphore 

func NewSemaphore(n int) *Semaphore

func (*Semaphore) Acquire 

func (s *Semaphore) Acquire()

func (*Semaphore) Release 

func (s *Semaphore) Release()

func (*Semaphore) TryAcquire 

func (s *Semaphore) TryAcquire() <-chan struct{}

type SetSizeFunc 

type SetSizeFunc func(size int64)

type Size 

type Size int64

func (Size) MarshalJSON 

func (siz Size) MarshalJSON() ([]byte, error)

func (Size) String 

func (siz Size) String() string

func (*Size) UnmarshalJSON 

func (siz *Size) UnmarshalJSON(b []byte) (err error)

type StatsTracker 

type StatsTracker interface {
	Add(name string, val int64)
	Get(name string) int64
	AddMany(namedVal64 ...NamedVal64)
}

type StopCh 

type StopCh struct {
	// contains filtered or unexported fields
}

StopCh is specialized channel for stopping things.

func NewStopCh 

func NewStopCh() *StopCh

func (*StopCh) Close 

func (sch *StopCh) Close()

func (*StopCh) Init 

func (sch *StopCh) Init()

func (*StopCh) Listen 

func (sch *StopCh) Listen() <-chan struct{}

type StrKVs 

type StrKVs map[string]string

func NewStrKVs 

func NewStrKVs(pairs ...string) (kvs StrKVs)

func (StrKVs) Compare 

func (kvs StrKVs) Compare(other StrKVs) bool

func (StrKVs) Contains 

func (kvs StrKVs) Contains(key string) (ok bool)

func (StrKVs) ContainsAnyMatch 

func (kvs StrKVs) ContainsAnyMatch(in []string) string

func (StrKVs) KeyFor 

func (kvs StrKVs) KeyFor(value string) (key string)

func (StrKVs) Keys 

func (kvs StrKVs) Keys() []string

type StrSet 

type StrSet map[string]struct{}

func NewStrSet 

func NewStrSet(keys ...string) (ss StrSet)

func (StrSet) Add 

func (ss StrSet) Add(keys ...string)

func (StrSet) All 

func (ss StrSet) All(xs ...string) bool

func (StrSet) Clone 

func (ss StrSet) Clone() StrSet

func (StrSet) Contains 

func (ss StrSet) Contains(key string) (yes bool)

func (StrSet) Delete 

func (ss StrSet) Delete(key string)

func (StrSet) Intersection 

func (ss StrSet) Intersection(other StrSet) StrSet

func (StrSet) String 

func (ss StrSet) String() string

func (StrSet) ToSlice 

func (ss StrSet) ToSlice() []string

type TemplateRange 

type TemplateRange struct {
	Gap        string // characters after the range (to the next range or end of the string)
	Start      int64
	End        int64
	Step       int64
	DigitCount int
}

type TimeoutGroup 

type TimeoutGroup struct {
	// contains filtered or unexported fields
}

TimeoutGroup is similar to sync.WaitGroup with the difference on Wait where we only allow timing out.

WARNING: It should not be used in critical code as it may have worse performance than sync.WaitGroup - use only if its needed.

WARNING: It is not safe to wait on completion in multiple threads!

WARNING: It is not recommended to reuse the TimeoutGroup - it was not designed for that and bugs can be expected, especially when previous group was not called with successful (without timeout) WaitTimeout.

func NewTimeoutGroup 

func NewTimeoutGroup() *TimeoutGroup

func (*TimeoutGroup) Add 

func (twg *TimeoutGroup) Add(n int)

func (*TimeoutGroup) Done 

func (twg *TimeoutGroup) Done()

Done decrements number of jobs left to do. Panics if the number jobs left is less than 0.

func (*TimeoutGroup) Wait 

func (twg *TimeoutGroup) Wait()

Wait waits until the Added pending count goes to zero. NOTE: must be invoked after _all_ Adds.

func (*TimeoutGroup) WaitTimeout 

func (twg *TimeoutGroup) WaitTimeout(timeout time.Duration) bool

Wait waits until the Added pending count goes to zero _or_ timeout. NOTE: must be invoked after _all_ Adds.

func (*TimeoutGroup) WaitTimeoutWithStop 

func (twg *TimeoutGroup) WaitTimeoutWithStop(timeout time.Duration, stop <-chan struct{}) (timed, stopped bool)

Wait waits until the Added pending count goes to zero _or_ timeout _or_ stop. NOTE: must be invoked after _all_ Adds.

type Unpacker 

type Unpacker interface {
	Unpack(unpacker *ByteUnpack) error
}

Every object that is going to use binary representation instead of JSON must implement two following methods

type WG 

type WG interface {
	Add(int)
	Done()
	Wait()
}

WG is an interface for wait group

func NewLimitedWaitGroup 

func NewLimitedWaitGroup(limit, have int) WG

usage: no more than `limit` goroutines in parallel

type WriteSizer 

type WriteSizer interface {
	io.Writer
	Size() int64
}

writers

type WriterAt 

type WriterAt interface {
	io.Writer
	io.WriterAt
}

writers

type WriterMulti 

type WriterMulti struct {
	// contains filtered or unexported fields
}

writers

func NewWriterMulti 

func NewWriterMulti(w ...io.Writer) *WriterMulti

func (*WriterMulti) Write 

func (mw *WriterMulti) Write(b []byte) (n int, err error)

type WriterOnly 

type WriterOnly struct{ io.Writer }

WriterOnly is helper struct to hide `io.ReaderFrom` implementation which can use some heuristics to improve performance but can result in not using `buffer` provided in `io.CopyBuffer`. See: https://golang.org/doc/go1.15#os.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.