cos

package
v1.3.24 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 27, 2024 License: MIT Imports: 44 Imported by: 9

Documentation

Overview

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2021-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2021-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2023-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2022-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Index

Constants

View Source 
const (
	ChecksumNone   = "none"
	ChecksumXXHash = "xxhash"
	ChecksumMD5    = "md5"
	ChecksumCRC32C = "crc32c"
	ChecksumSHA256 = "sha256" // crypto.SHA512_256 (SHA-2)
	ChecksumSHA512 = "sha512" // crypto.SHA512 (SHA-2)
)

checksums

View Source 
const (
	CtxReadWrapper contextID = "readWrapper" // context key for ReadWrapperFunc
	CtxSetSize     contextID = "setSize"     // context key for SetSizeFunc
	CtxOriginalURL contextID = "origURL"     // context key for OriginalURL for HTTP cloud
)
View Source 
const (
	ContentJSON           = "application/json"
	ContentJSONCharsetUTF = "application/json; charset=utf-8"
	ContentMsgPack        = "application/msgpack"
	ContentXML            = "application/xml"
	ContentBinary         = "application/octet-stream"

	// not present in IANA registry
	// mozilla.org has it though, and also https://en.wikipedia.org/wiki/List_of_archive_formats
	ContentTar = "application/x-tar"

	// not currently used
	ContentZip = "application/zip"
)

standard MIME types - https://www.iana.org/assignments/media-types/media-types.xhtml - https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types

View Source 
const (
	// range to read:
	HdrRange          = "Range" // Ref: https://www.rfc-editor.org/rfc/rfc7233#section-2.1
	HdrRangeValPrefix = "bytes="
	// range read response:
	HdrContentRange          = "Content-Range"
	HdrContentRangeValPrefix = "bytes " // Ref: https://tools.ietf.org/html/rfc7233#section-4.2
	HdrAcceptRanges          = "Accept-Ranges"

	// content length & type
	HdrContentType        = "Content-Type"
	HdrContentTypeOptions = "X-Content-Type-Options"
	HdrContentLength      = "Content-Length"

	// misc. gen
	HdrUserAgent = "User-Agent"
	HdrAccept    = "Accept"
	HdrLocation  = "Location"
	HdrServer    = "Server"
	HdrETag      = "ETag" // Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag

	HdrHSTS = "Strict-Transport-Security"
)

Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers

View Source 
const (
	// https://cloud.google.com/storage/docs/xml-api/reference-headers
	GsCksumHeader   = "x-goog-hash"
	GsVersionHeader = "x-goog-generation"
)
View Source 
const (
	// https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html
	// https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html
	S3CksumHeader   = HdrETag
	S3VersionHeader = "x-amz-version-id"

	// s3 api request headers
	S3HdrObjSrc = "x-amz-copy-source"
	S3HdrMptCnt = "x-amz-mp-parts-count"

	// https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
	S3UnsignedPayload  = "UNSIGNED-PAYLOAD"
	S3HdrContentSHA256 = "x-amz-content-sha256"

	S3HdrBckRegion = "x-amz-bucket-region"

	S3ChecksumCRC32  = "x-amz-checksum-crc32"
	S3ChecksumCRC32C = "x-amz-checksum-crc32c"
	S3ChecksumSHA1   = "x-amz-checksum-sha1"
	S3ChecksumSHA256 = "x-amz-checksum-sha256"

	S3MetadataChecksumType = "x-amz-meta-ais-cksum-type"
	S3MetadataChecksumVal  = "x-amz-meta-ais-cksum-val"

	S3LastModified = "Last-Modified"
)
View Source 
const (
	// https://docs.microsoft.com/en-us/rest/api/storageservices/get-blob-properties#response-headers
	AzCksumHeader   = "Content-MD5"
	AzVersionHeader = HdrETag
)
View Source 
const (
	PermRWR   os.FileMode = 0o640
	PermRWRR  os.FileMode = 0o644 // (archived)
	PermRWXRX os.FileMode = 0o750
)

POSIX permissions

View Source 
const (
	SmoduleTransport = 1 << iota
	SmoduleAIS
	SmoduleMemsys
	SmoduleCore
	SmoduleFS
	SmoduleReb
	SmoduleEC
	SmoduleStats
	SmoduleIOS
	SmoduleXs
	SmoduleBackend
	SmoduleSpace
	SmoduleMirror
	SmoduleDsort
	SmoduleDload
	SmoduleETL
	SmoduleS3
)
View Source 
const (
	VoteInProgress       = NodeStateFlags(1 << iota) // warning
	ClusterStarted                                   // info: (primary: cluster-started | all other nodes: joined-cluster)
	NodeStarted                                      // info: (started; possibly, not joined yet)
	Rebalancing                                      // warning
	RebalanceInterrupted                             // warning
	Resilvering                                      // warning
	ResilverInterrupted                              // warning
	Restarted                                        // warning
	OOS                                              // red alert (see IsRed below)
	OOM                                              // red alert
	MaintenanceMode                                  // warning
	LowCapacity                                      // (used > high); warning: OOS possible soon..
	LowMemory                                        // ditto OOM
	DiskFault                                        // red
	NoMountpaths                                     // red: (reserved, not used)
	NumGoroutines                                    // red
	CertWillSoonExpire                               // warning X.509
	CertificateExpired                               // red --/--
	CertificateInvalid                               // red --/--
	KeepAliveErrors                                  // warning (new keep-alive errors during the last 5m)
)
View Source 
const (
	QuantityPercent = "percent"
	QuantityBytes   = "bytes"
)
View Source 
const (
	LetterRunes = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
	LenRunes    = len(LetterRunes)
)
View Source 
const (
	UnitsIEC = "iec" // default
	UnitsSI  = "si"  // NOTE: currently, SI system is CLI-only (compare with cmn/cos/size.go)
	UnitsRaw = "raw"
)
View Source 
const (
	KiB = 1024
	MiB = 1024 * KiB
	GiB = 1024 * MiB
	TiB = 1024 * GiB
)

IEC (binary) units

View Source 
const (
	KB = 1000
	MB = 1000 * KB
	GB = 1000 * MB
	TB = 1000 * GB
)

IS (metric) units

View Source 
const (
	StreamsOutObjCount = "stream.out.n"
	StreamsOutObjSize  = "stream.out.size"
	StreamsInObjCount  = "stream.in.n"
	StreamsInObjSize   = "stream.in.size"
)

intra-cluster transmit & receive (cumulative counters)

View Source 
const (
	// Number of sync maps
	MultiSyncMapCount = 0x40 // m.b. a power of two
	MultiSyncMapMask  = MultiSyncMapCount - 1
)
View Source 
const (
	WildcardMatchAll = "*"
	EmptyMatchAll    = ""
)
View Source 
const (
	StampMicro = "15:04:05.000000" // time.StampMicro without a date
	StampSec   = "15:04:05"        // time.Stamp without a date

	// S3 ListObjectsV2
	// https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html
	ISO8601 = "2006-01-02T15:04:05.000Z"

	// S3 HeadObject
	// https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html#API_HeadObject_Examples
	RFC1123GMT = "Mon, 17 Dec 2012 02:14:10 GMT"
)

in addition to standard layouts at /usr/local/go/src/time/format.go

View Source 
const (
	SizeofI64 = int(unsafe.Sizeof(uint64(0)))
	SizeofI32 = int(unsafe.Sizeof(uint32(0)))
	SizeofI16 = int(unsafe.Sizeof(uint16(0)))
)

assorted common constants

View Source 
const (
	OnlyNice = "must be less than 32 characters and " + mayOnlyContain // NOTE tooLongID
	OnlyPlus = mayOnlyContain + ", and dots (.)"
)
View Source 
const ContentLengthUnknown = -1
View Source 
const (
	LenShortID = 9 // UUID length, as per https://github.com/teris-io/shortid#id-length

)
View Source 
const MLCG32 = 1103515245 // xxhash seed
View Source 
const NodeAlerts = "state.flags"
View Source 
const PathSeparator = string(filepath.Separator)
View Source 
const SizeofLen = SizeofI32

Internally size of byte slice or string is packed as this integer type. Since binary packed structs are used for fairly small amount of data, it is possible to use int32 instead of int64 to keep the length of the data.

Variables

View Source 
var (
	ErrQuantityUsage   = errors.New("invalid quantity, format should be '81%' or '1GB'")
	ErrQuantityPercent = errors.New("percent must be in the range (0, 100)")
	ErrQuantityBytes   = errors.New("value (bytes) must be non-negative")
)
View Source 
var (
	ErrEmptyTemplate = errors.New("empty range template")
)
View Source 
var ErrWorkChanFull = errors.New("work channel full")
View Source 
var JSON jsoniter.API

JSON is used to Marshal/Unmarshal API json messages and is initialized in init function.

View Source 
var NoneCksum = NewCksum(ChecksumNone, "")
View Source 
var Smodules = [...]string{
	"transport", "ais", "memsys", "cluster", "fs", "reb", "ec", "stats",
	"ios", "xs", "backend", "space", "mirror", "dsort", "downloader", "etl",
	"s3",
}

NOTE: keep in-sync with the above

Functions

func AnyHasPrefixInSlice 

func AnyHasPrefixInSlice(prefix string, arr []string) bool

func Assert 

func Assert(cond bool)

func AssertMsg 

func AssertMsg(cond bool, msg string)

NOTE: when using Sprintf and such, `if (!cond) { AssertMsg(false, msg) }` is the preferable usage.

func AssertNoErr 

func AssertNoErr(err error)

func Assertf 

func Assertf(cond bool, f string, a ...any)

NOTE: Not to be used in the datapath - consider instead one of the flavors below.

func BHead 

func BHead(b []byte, ls ...int) string

func CeilAlign 

func CeilAlign(val, align uint) uint

returns smallest number divisible by `align` that is greater or equal `val`

func CeilAlignInt64 

func CeilAlignInt64(val, align int64) int64

func CheckAlphaPlus added in v1.3.24 

func CheckAlphaPlus(s, tag string) error

alpha-numeric++ including letters, numbers, dashes (-), and underscores (_) period (.) is allowed except for '..' (OnlyPlus const)

func ClearfAtomic 

func ClearfAtomic(f *uint64, flags uint64) (ok bool)

func Close 

func Close(closer io.Closer)

func ConvertToString 

func ConvertToString(value any) (valstr string, err error)

func CopyBuffer added in v1.3.22 

func CopyBuffer(dst io.Writer, src io.Reader, buf []byte) (written int64, err error)

a slightly modified excerpt from https://github.com/golang/go/blob/master/src/io/io.go#L407 - regular streaming copy with `io.WriteTo` and `io.ReaderFrom` not checked and not used - buffer _must_ be provided - see also: WriterOnly comment (above)

func CopyStruct 

func CopyStruct(dst, src any)

shallow copy

func CreateDir 

func CreateDir(dir string) error

CreateDir creates directory if does not exist. If the directory already exists returns nil.

func CreateFile 

func CreateFile(fqn string) (*os.File, error)

CreateFile creates a new write-only (O_WRONLY) file with default cos.PermRWR permissions. NOTE: if the file pathname doesn't exist it'll be created. NOTE: if the file already exists it'll be also silently truncated.

func CryptoRandS added in v1.3.16 

func CryptoRandS(n int) string

func DivCeil 

func DivCeil(a, b int64) int64

func DivRound 

func DivRound(a, b int64) int64

func DivRoundU64 

func DivRoundU64(a, b uint64) uint64

func DrainReader 

func DrainReader(r io.Reader)

DrainReader reads and discards all the data from a reader. No need for `io.CopyBuffer` as `io.Discard` has efficient `io.ReaderFrom` implementation.

func Err2ClientURLErr added in v1.3.16 

func Err2ClientURLErr(err error) (uerr *url.Error)

func Errorf 

func Errorf(format string, a ...any)

func Errorln added in v1.3.24 

func Errorln(a ...any)

func ExitAssertLog added in v1.3.22 

func ExitAssertLog(cond bool, a ...any)

+condition

func ExitLog added in v1.3.18 

func ExitLog(a ...any)

func ExitLogf 

func ExitLogf(f string, a ...any)

+log

func Exitf 

func Exitf(f string, a ...any)

func ExpandPath 

func ExpandPath(path string) string

ExpandPath replaces common abbreviations in file path (eg. `~` with absolute path to the current user home directory) and cleans the path.

func Ext added in v1.3.18 

func Ext(path string) (ext string)

e.g.: "/aaa/bbb/ccc.tar.gz" => ".tar.gz", while "/aaa/bb.b/ccctargz" => ""

func FloodWriter 

func FloodWriter(w io.Writer, n int64) error

FloodWriter writes `n` random bytes to provided writer.

func FlushClose 

func FlushClose(file *os.File) (err error)

func FormatBigNum added in v1.3.20 

func FormatBigNum(n int) (s string)

func FormatMilli 

func FormatMilli(tm time.Duration) string

FormatMilli returns a duration formatted as milliseconds. For values bigger than millisecond, it returns an integer number "#ms". For values smaller than millisecond, the function returns fractional number "0.##ms"

func FormatNanoTime added in v1.3.16 

func FormatNanoTime(unixnano int64, format string) string

`unixnano` here is expected to be "nanoseconds since January 1, 1970 UTC"

func FormatNowStamp added in v1.3.16 

func FormatNowStamp() string

func FormatTime added in v1.3.16 

func FormatTime(t time.Time, format string) string

func FreeMemToOS 

func FreeMemToOS(force bool) bool

FreeMemToOS executes in a separate goroutine with at least so-many minutes (above) between the runs. It calls GC and returns allocated memory to the operating system. Notes:

  • `forceFreeMemTime` is expected to be significantly greater than the time spent in the goroutine
  • still, an unlikely case of overlapping runs is treated via `runningOOM`
  • for somewhat related OOS handling - esp. constants - see ais/tgtspace.go

func GenBEID added in v1.3.19 

func GenBEID(val uint64, l int) string

"best-effort ID" - to independently and locally generate globally unique ID called by xreg.GenBEID

func GenDaemonID 

func GenDaemonID() string

func GenTestingDaemonID 

func GenTestingDaemonID(suffix string) string

(when config.TestingEnv)

func GenTie 

func GenTie() string

3-letter tie breaker (fast)

func GenUUID 

func GenUUID() (uuid string)

compare with xreg.GenBEID

func GetEnvOrDefault added in v1.3.24 

func GetEnvOrDefault(envVar, defaultValue string) string

getEnvOrDefault returns the value of the environment variable if it exists, otherwise it returns the provided default value.

func HashK8sProxyID added in v1.3.23 

func HashK8sProxyID(nodeName string) (pid string)

func HomeConfigDir 

func HomeConfigDir(subdir string) (configDir string)

func HomeDir 

func HomeDir() (string, error)

func Infoln added in v1.3.24 

func Infoln(a ...any)

func InitShortID 

func InitShortID(seed uint64)

func IsAlphaNice added in v1.3.16 

func IsAlphaNice(s string) bool

letters and numbers w/ '-' and '_' permitted with limitations (see OnlyNice const)

func IsAnySetfAtomic 

func IsAnySetfAtomic(f *uint64, flags uint64) (yes bool)

func IsAzureURL 

func IsAzureURL(u *url.URL) bool

func IsEOF 

func IsEOF(err error) bool

including "unexpecting EOF" to accommodate unsized streaming and early termination of the other side (prior to sending the first byte)

func IsErrBadCksum 

func IsErrBadCksum(err error) bool

func IsErrBrokenPipe 

func IsErrBrokenPipe(err error) (yes bool)

func IsErrClientURLTimeout added in v1.3.16 

func IsErrClientURLTimeout(err error) bool

func IsErrConnectionNotAvail 

func IsErrConnectionNotAvail(err error) (yes bool)

likely out of socket descriptors

func IsErrConnectionRefused 

func IsErrConnectionRefused(err error) (yes bool)

retriable conn errs

func IsErrConnectionReset 

func IsErrConnectionReset(err error) (yes bool)

func IsErrDNSLookup added in v1.3.24 

func IsErrDNSLookup(err error) bool

func IsErrNotFound added in v1.3.18 

func IsErrNotFound(err error) bool

func IsErrOOS 

func IsErrOOS(err error) bool

func IsErrSyscallTimeout added in v1.3.16 

func IsErrSyscallTimeout(err error) bool

func IsErrXattrNotFound 

func IsErrXattrNotFound(err error) bool

func IsGoogleAPIURL 

func IsGoogleAPIURL(u *url.URL) bool

func IsGoogleStorageURL 

func IsGoogleStorageURL(u *url.URL) bool

func IsHT added in v1.3.24 

func IsHT(url string) bool

func IsHTTPS 

func IsHTTPS(url string) bool

func IsIOError 

func IsIOError(err error) bool

Checks if the error is generated by any IO operation and if the error is severe enough to run the FSHC for mountpath testing

func IsLastB added in v1.3.22 

func IsLastB(s string, b byte) bool

func IsNotExist added in v1.3.22 

func IsNotExist(err error, ecode int) bool

NOTE: compare with cmn.IsErrObjNought() that also includes lmeta-not-found et al.

func IsParseBool 

func IsParseBool(s string) bool

func IsParseEnvBoolOrDefault added in v1.3.24 

func IsParseEnvBoolOrDefault(envVar string, defaultValue bool) (bool, error)

IsParseBoolOrDefault parses a boolean from the environment variable string or returns the default value if parsing fails.

func IsRetriableConnErr 

func IsRetriableConnErr(err error) (yes bool)

func IsS3URL 

func IsS3URL(link string) bool

func IsSetfAtomic 

func IsSetfAtomic(f *uint64, flags uint64) (yes bool)

func IsTimeZero 

func IsTimeZero(t time.Time) bool

func IsUnreachable 

func IsUnreachable(err error, status int) bool

func IsValidAtime added in v1.3.18 

func IsValidAtime(atime int64) bool

access time validity; prefetch special (negative) case - sets atime=-now 946771140000000000 = time.Parse(time.RFC3339Nano, "2000-01-01T23:59:00Z").UnixNano()

func IsValidUUID 

func IsValidUUID(uuid string) bool

func JoinPath 

func JoinPath(url, path string) string

JoinPath joins two path elements that may (or may not) be prefixed/suffixed with a slash.

func JoinWords 

func JoinWords(w string, words ...string) (path string)

JoinWords uses forward slash to join any number of words into a single path. Returned path is prefixed with a slash.

func Left added in v1.3.24 

func Left(left, right string) string

left if non-empty; otherwise right

func MatchAll added in v1.3.23 

func MatchAll(template string) bool

func MorphMarshal 

func MorphMarshal(data, v any) error

func MustMarshal 

func MustMarshal(v any) []byte

MustMarshal marshals v and panics if error occurs.

func MustMarshalToString 

func MustMarshalToString(v any) string

func MustMorphMarshal 

func MustMorphMarshal(data, v any)

func NewCRC32C 

func NewCRC32C() hash.Hash

func NewErrDataCksum added in v1.3.18 

func NewErrDataCksum(a, b *Cksum, context ...string) error

func NewErrMetaCksum added in v1.3.18 

func NewErrMetaCksum(a, b uint64, context ...string) error

func NewNowSource added in v1.3.23 

func NewNowSource() rand.Source

func NewRandSource added in v1.3.23 

func NewRandSource(seed uint64) rand.Source

func NopReader 

func NopReader(size int64) io.Reader

func NowRand 

func NowRand() *rand.Rand

func PackedBytesLen added in v1.3.24 

func PackedBytesLen(b []byte) int

func PackedStrLen 

func PackedStrLen(s string) int

PackedStrLen returns the size occupied by a given string in the output

func ParseBool 

func ParseBool(s string) (bool, error)

ParseBool converts string to bool (case-insensitive): 

y, yes, on -> true
n, no, off, <empty value> -> false

strconv handles the following: 

1, true, t -> true
0, false, f -> false

func ParseSize added in v1.3.16 

func ParseSize(size, units string) (int64, error)

when `units` arg is empty conversion is defined by the suffix

func ParseURL 

func ParseURL(s string) (u *url.URL, valid bool)

func Plural 

func Plural(num int) (s string)

(common use)

func ProbingFrequency 

func ProbingFrequency(dur time.Duration) time.Duration

wait duration => probing frequency

func RandStringWithSrc 

func RandStringWithSrc(src rand.Source, n int) string

func Ratio 

func Ratio(high, low, curr int64) float32

func RatioPct 

func RatioPct(high, low, curr int64) int64

func ReadAll added in v1.3.24 

func ReadAll(r io.Reader) ([]byte, error)

func ReadAllN added in v1.3.24 

func ReadAllN(r io.Reader, size int64) (b []byte, err error)

instead of os.ReadAll

func ReadLines 

func ReadLines(filename string, cb func(string) error) error

Read a file line by line and call a callback for each line until the file ends or a callback returns io.EOF

func ReadOneInt64 

func ReadOneInt64(filename string) (int64, error)

Read only the first line of a file and return it as int64 Do not use for big files: it reads all the content and then extracts the first line. Use for files that may contains a few lines with trailing EOL

func ReadOneLine 

func ReadOneLine(filename string) (string, error)

Read only the first line of a file. Do not use for big files: it reads all the content and then extracts the first line. Use for files that may contains a few lines with trailing EOL

func ReadOneUint64 

func ReadOneUint64(filename string) (uint64, error)

Read only the first line of a file and return it as uint64 Do not use for big files: it reads all the content and then extracts the first line. Use for files that may contains a few lines with trailing EOL

func RemoveFile 

func RemoveFile(path string) (err error)

RemoveFile removes path; returns nil upon success or if the path does not exist.

func Rename 

func Rename(src, dst string) (err error)

(creates destination directory if doesn't exist)

func ReparseQuery 

func ReparseQuery(r *http.Request)

WARNING: `ReparseQuery` might affect non-tensorflow clients using S3-compatible API with AIStore. To be used with caution. 

func Right(left, right string) string

right if non-empty; otherwise left

func S2Duration 

func S2Duration(s string) (time.Duration, error)

func S2UnixNano 

func S2UnixNano(s string) (int64, error)

func SHead 

func SHead(s string) string

func SetfAtomic 

func SetfAtomic(f *uint64, flags uint64) (ok bool)

"set" as in: "add"

func Stat 

func Stat(path string) error

syscall to check that path exists (see bench/lstat)

func StrSlicesEqual 

func StrSlicesEqual(lhs, rhs []string) bool

StrSlicesEqual compares content of two string slices. It is replacement for reflect.DeepEqual because the latter returns false if slices have the same values but in different order.

func StringInSlice 

func StringInSlice(s string, arr []string) bool

func SupportedChecksums 

func SupportedChecksums() (types []string)

func ToSizeIEC added in v1.3.16 

func ToSizeIEC(b int64, digits int) string

(compare w/ CLI `ToSizeIS`)

func TrimLastB added in v1.3.23 

func TrimLastB(s string, b byte) string

func UnixNano2S 

func UnixNano2S(unixnano int64) string

func UnsafeB added in v1.3.16 

func UnsafeB(s string) []byte

cast string to immutable bytes

func UnsafeBptr added in v1.3.24 

func UnsafeBptr(s *string) *[]byte

*string to *[]bytes

func UnsafeS 

func UnsafeS(b []byte) string

cast bytes to an immutable string

func UnsafeSptr added in v1.3.24 

func UnsafeSptr(b []byte) *string

bytes to *string

func UnwrapSyscallErr 

func UnwrapSyscallErr(err error) error

func ValidateCksumType 

func ValidateCksumType(ty string, emptyOK ...bool) (err error)

func ValidateDaemonID 

func ValidateDaemonID(id string) error

func WdsKey added in v1.3.23 

func WdsKey(path string) (key string)

WebDataset convention - not to confuse with filepath.Base (!) * see https://github.com/webdataset/webdataset#the-webdataset-format

Types

type BitFlags 

type BitFlags uint64

func (BitFlags) Clear 

func (f BitFlags) Clear(flags BitFlags) BitFlags

func (BitFlags) IsAnySet 

func (f BitFlags) IsAnySet(flags BitFlags) bool

func (BitFlags) IsSet 

func (f BitFlags) IsSet(flags BitFlags) bool

func (BitFlags) Set 

func (f BitFlags) Set(flags BitFlags) BitFlags

type Bits 

type Bits uint8

func (*Bits) Clear 

func (b *Bits) Clear(flag Bits)

func (*Bits) Has 

func (b *Bits) Has(flag Bits) bool

func (*Bits) Set 

func (b *Bits) Set(flag Bits)

func (*Bits) Toggle 

func (b *Bits) Toggle(flag Bits)

type Buffer added in v1.3.23 

type Buffer struct {
	// contains filtered or unexported fields
}

writers

func NewBuffer added in v1.3.23 

func NewBuffer(b []byte) *Buffer

func (*Buffer) WriteTo2 added in v1.3.23 

func (w *Buffer) WriteTo2(dst io.Writer) (err error)

type ByteHandle 

type ByteHandle struct {
	*bytes.Reader
	// contains filtered or unexported fields
}

ByteHandle is a byte buffer(made from []byte) that implements ReadOpenCloser interface

func NewByteHandle 

func NewByteHandle(bt []byte) *ByteHandle

func (*ByteHandle) Close 

func (*ByteHandle) Close() error

func (*ByteHandle) Open 

func (b *ByteHandle) Open() (ReadOpenCloser, error)

type BytePack 

type BytePack struct {
	// contains filtered or unexported fields
}

func NewPacker 

func NewPacker(buf []byte, bufLen int) *BytePack

func (*BytePack) Bytes 

func (bw *BytePack) Bytes() []byte

func (*BytePack) WriteAny 

func (bw *BytePack) WriteAny(st Packer)

func (*BytePack) WriteBool 

func (bw *BytePack) WriteBool(b bool)

func (*BytePack) WriteByte 

func (bw *BytePack) WriteByte(b byte)

func (*BytePack) WriteBytes 

func (bw *BytePack) WriteBytes(b []byte)

func (*BytePack) WriteInt16 

func (bw *BytePack) WriteInt16(i int16)

func (*BytePack) WriteInt32 

func (bw *BytePack) WriteInt32(i int32)

func (*BytePack) WriteInt64 

func (bw *BytePack) WriteInt64(i int64)

func (*BytePack) WriteMapStrUint16 

func (bw *BytePack) WriteMapStrUint16(mp MapStrUint16)

func (*BytePack) WriteString 

func (bw *BytePack) WriteString(s string)

func (*BytePack) WriteUint16 

func (bw *BytePack) WriteUint16(i uint16)

func (*BytePack) WriteUint32 

func (bw *BytePack) WriteUint32(i uint32)

func (*BytePack) WriteUint64 

func (bw *BytePack) WriteUint64(i uint64)

type ByteUnpack 

type ByteUnpack struct {
	// contains filtered or unexported fields
}

func NewUnpacker 

func NewUnpacker(buf []byte) *ByteUnpack

func (*ByteUnpack) Bytes 

func (br *ByteUnpack) Bytes() []byte

func (*ByteUnpack) Len 

func (br *ByteUnpack) Len() int

func (*ByteUnpack) ReadAny 

func (br *ByteUnpack) ReadAny(st Unpacker) error

func (*ByteUnpack) ReadBool 

func (br *ByteUnpack) ReadBool() (bool, error)

func (*ByteUnpack) ReadByte 

func (br *ByteUnpack) ReadByte() (byte, error)

func (*ByteUnpack) ReadBytes 

func (br *ByteUnpack) ReadBytes() ([]byte, error)

func (*ByteUnpack) ReadInt16 

func (br *ByteUnpack) ReadInt16() (int16, error)

func (*ByteUnpack) ReadInt32 

func (br *ByteUnpack) ReadInt32() (int32, error)

func (*ByteUnpack) ReadInt64 

func (br *ByteUnpack) ReadInt64() (int64, error)

func (*ByteUnpack) ReadMapStrUint16 

func (br *ByteUnpack) ReadMapStrUint16() (MapStrUint16, error)

func (*ByteUnpack) ReadString 

func (br *ByteUnpack) ReadString() (string, error)

func (*ByteUnpack) ReadUint16 

func (br *ByteUnpack) ReadUint16() (uint16, error)

func (*ByteUnpack) ReadUint32 

func (br *ByteUnpack) ReadUint32() (uint32, error)

func (*ByteUnpack) ReadUint64 

func (br *ByteUnpack) ReadUint64() (uint64, error)

type CallbackROC 

type CallbackROC struct {
	// contains filtered or unexported fields
}

readers: implementations

func NewCallbackReadOpenCloser 

func NewCallbackReadOpenCloser(r ReadOpenCloser, readCb func(int, error), reportedBytes ...int) *CallbackROC

func (*CallbackROC) Close 

func (r *CallbackROC) Close() error

func (*CallbackROC) Open 

func (r *CallbackROC) Open() (ReadOpenCloser, error)

func (*CallbackROC) Read 

func (r *CallbackROC) Read(p []byte) (n int, err error)

type Cksum 

type Cksum struct {
	// contains filtered or unexported fields
}

func ChecksumBytes 

func ChecksumBytes(b []byte, cksumType string) (cksum *Cksum, err error)

ChecksumBytes computes checksum of given bytes using additional buffer.

func NewCksum 

func NewCksum(ty, value string) *Cksum

func (*Cksum) Clone 

func (ck *Cksum) Clone() *Cksum

func (*Cksum) Equal 

func (ck *Cksum) Equal(to *Cksum) bool

func (*Cksum) Get 

func (ck *Cksum) Get() (string, string)

func (*Cksum) IsEmpty 

func (ck *Cksum) IsEmpty() bool

func (*Cksum) MarshalJSON 

func (ck *Cksum) MarshalJSON() ([]byte, error)

func (*Cksum) String 

func (ck *Cksum) String() string

func (*Cksum) Ty 

func (ck *Cksum) Ty() string

func (*Cksum) Type 

func (ck *Cksum) Type() string

func (*Cksum) Val 

func (ck *Cksum) Val() string

func (*Cksum) Value 

func (ck *Cksum) Value() string

type CksumHash 

type CksumHash struct {
	Cksum
	H hash.Hash
	// contains filtered or unexported fields
}

func CopyAndChecksum 

func CopyAndChecksum(w io.Writer, r io.Reader, buf []byte, cksumType string) (n int64, cksum *CksumHash, err error)

CopyAndChecksum reads from `r` and writes to `w`; returns num bytes copied and checksum, or error

func CopyFile 

func CopyFile(src, dst string, buf []byte, cksumType string) (written int64, cksum *CksumHash, err error)

and computes checksum if requested

func NewCksumHash 

func NewCksumHash(ty string) (ck *CksumHash)

func SaveReader 

func SaveReader(fqn string, reader io.Reader, buf []byte, cksumType string, size int64) (*CksumHash, error)

Saves the `reader` directly to `fqn`, checksums if requested

func (*CksumHash) Equal 

func (ck *CksumHash) Equal(to *Cksum) bool

func (*CksumHash) Finalize 

func (ck *CksumHash) Finalize()

func (*CksumHash) Init 

func (ck *CksumHash) Init(ty string)

func (*CksumHash) Sum 

func (ck *CksumHash) Sum() []byte

type CksumHashSize 

type CksumHashSize struct {
	CksumHash
	Size int64
}

func (*CksumHashSize) Write 

func (ck *CksumHashSize) Write(b []byte) (n int, err error)

type Duration 

type Duration time.Duration

func (Duration) D 

func (d Duration) D() time.Duration

func (Duration) MarshalJSON 

func (d Duration) MarshalJSON() ([]byte, error)

func (Duration) String 

func (d Duration) String() (s string)

func (*Duration) UnmarshalJSON 

func (d *Duration) UnmarshalJSON(b []byte) (err error)

type DynSemaphore 

type DynSemaphore struct {
	// contains filtered or unexported fields
}

DynSemaphore implements sempahore which can change its size during usage.

func NewDynSemaphore 

func NewDynSemaphore(n int) *DynSemaphore

func (*DynSemaphore) Acquire 

func (s *DynSemaphore) Acquire(cnts ...int)

func (*DynSemaphore) Release 

func (s *DynSemaphore) Release(cnts ...int)

func (*DynSemaphore) SetSize 

func (s *DynSemaphore) SetSize(n int)

func (*DynSemaphore) Size 

func (s *DynSemaphore) Size() int

type ErrBadCksum 

type ErrBadCksum struct {
	// contains filtered or unexported fields
}

in-cluster checksum validation (compare with cmn.ErrInvalidCksum)

func (*ErrBadCksum) Error 

func (e *ErrBadCksum) Error() string

type ErrNotFound added in v1.3.18 

type ErrNotFound struct {
	// contains filtered or unexported fields
}

func NewErrNotFound added in v1.3.18 

func NewErrNotFound(where fmt.Stringer, what string) *ErrNotFound

func (*ErrNotFound) Error added in v1.3.18 

func (e *ErrNotFound) Error() string

type ErrSignal 

type ErrSignal struct {
	// contains filtered or unexported fields
}

func NewSignalError 

func NewSignalError(s syscall.Signal) *ErrSignal

func (*ErrSignal) Error 

func (e *ErrSignal) Error() string

func (*ErrSignal) ExitCode 

func (e *ErrSignal) ExitCode() int

https://tldp.org/LDP/abs/html/exitcodes.html

type Errs added in v1.3.18 

type Errs struct {
	// contains filtered or unexported fields
}

func (*Errs) Add added in v1.3.18 

func (e *Errs) Add(err error)

func (*Errs) Cnt added in v1.3.18 

func (e *Errs) Cnt() int

func (*Errs) Error added in v1.3.18 

func (e *Errs) Error() (s string)

Errs is an error

func (*Errs) JoinErr added in v1.3.18 

func (e *Errs) JoinErr() (cnt int, err error)

type FS added in v1.3.16 

type FS struct {
	Fs     string
	FsType string
	FsID   FsID
}

func (*FS) Equal added in v1.3.16 

func (fs *FS) Equal(otherFs FS) bool

func (*FS) String added in v1.3.16 

func (fs *FS) String() string

type FileHandle 

type FileHandle struct {
	*os.File
	// contains filtered or unexported fields
}

handles (and even more readers)

func NewFileHandle 

func NewFileHandle(fqn string) (*FileHandle, error)

func (*FileHandle) Open 

func (f *FileHandle) Open() (ReadOpenCloser, error)

type FileSectionHandle 

type FileSectionHandle struct {
	// contains filtered or unexported fields
}

FileSectionHandle opens a file and reads a section of it with optional padding. It implements the ReadOpenCloser interface.

func NewFileSectionHandle 

func NewFileSectionHandle(fqn string, offset, size int64) (*FileSectionHandle, error)

NewFileSectionHandle opens file which is expected at `fqn` and defines a SectionHandle on it to only read a specified section.

func (*FileSectionHandle) Close 

func (f *FileSectionHandle) Close() error

func (*FileSectionHandle) Open 

func (f *FileSectionHandle) Open() (ReadOpenCloser, error)

func (*FileSectionHandle) Read 

func (f *FileSectionHandle) Read(buf []byte) (int, error)

type FsID 

type FsID [2]int32

FsID is unified and cross-platform syscall.Fsid type which implements JSON marshaling.

func AddUniqueFsID added in v1.3.23 

func AddUniqueFsID(ids []FsID, nid FsID) ([]FsID, bool)

func (FsID) MarshalJSON 

func (d FsID) MarshalJSON() ([]byte, error)

func (FsID) String 

func (d FsID) String() string

func (*FsID) UnmarshalJSON 

func (d *FsID) UnmarshalJSON(b []byte) error

type JSONRawMsgs 

type JSONRawMsgs map[string]jsoniter.RawMessage

type LimitedWaitGroup 

type LimitedWaitGroup struct {
	// contains filtered or unexported fields
}

LimitedWaitGroup is helper struct which combines standard wait group and semaphore to limit the number of goroutines created.

func (*LimitedWaitGroup) Add 

func (lwg *LimitedWaitGroup) Add(n int)

func (*LimitedWaitGroup) Done 

func (lwg *LimitedWaitGroup) Done()

func (*LimitedWaitGroup) Wait 

func (lwg *LimitedWaitGroup) Wait()

type LogLevel added in v1.3.18 

type LogLevel string

func (LogLevel) Parse added in v1.3.18 

func (l LogLevel) Parse() (level, modules int)

func (*LogLevel) Set added in v1.3.18 

func (l *LogLevel) Set(level int, sm []string)

func (LogLevel) String added in v1.3.18 

func (l LogLevel) String() (s string)

func (LogLevel) Validate added in v1.3.18 

func (l LogLevel) Validate() (err error)

type LomReader added in v1.3.24 

type LomReader interface {
	io.ReadCloser
	io.ReaderAt
}

readers: interfaces

type LomWriter added in v1.3.24 

type LomWriter interface {
	io.WriteCloser
	Sync() error
}

readers: interfaces

type MapStrUint16 

type MapStrUint16 map[string]uint16

type MultiSyncMap 

type MultiSyncMap struct {
	M [MultiSyncMapCount]sync.Map
}

func (*MultiSyncMap) Get 

func (msm *MultiSyncMap) Get(idx int) *sync.Map

func (*MultiSyncMap) GetByHash 

func (msm *MultiSyncMap) GetByHash(hash uint32) *sync.Map

type NamedVal64 

type NamedVal64 struct {
	Name       string
	NameSuffix string // forces immediate send when non-empty
	Value      int64
}

type NodeStateFlags added in v1.3.23 

type NodeStateFlags BitFlags

func (NodeStateFlags) Clear added in v1.3.23 

func (f NodeStateFlags) Clear(flags NodeStateFlags) NodeStateFlags

func (NodeStateFlags) IsOK added in v1.3.24 

func (f NodeStateFlags) IsOK() bool

func (NodeStateFlags) IsRed added in v1.3.24 

func (f NodeStateFlags) IsRed() bool

func (NodeStateFlags) IsSet added in v1.3.23 

func (f NodeStateFlags) IsSet(flag NodeStateFlags) bool

func (NodeStateFlags) IsWarn added in v1.3.24 

func (f NodeStateFlags) IsWarn() bool

func (NodeStateFlags) Set added in v1.3.23 

func (f NodeStateFlags) Set(flags NodeStateFlags) NodeStateFlags

func (NodeStateFlags) String added in v1.3.24 

func (f NodeStateFlags) String() string

type NodeStateInfo added in v1.3.23 

type NodeStateInfo struct {
	Smap struct {
		Primary struct {
			PubURL  string `json:"pub_url"`
			CtrlURL string `json:"control_url"`
			ID      string `json:"id"`
		}
		Version int64  `json:"version,string"`
		UUID    string `json:"uuid"`
	} `json:"smap"`
	BMD struct {
		UUID    string `json:"uuid"`
		Version int64  `json:"version,string"`
	} `json:"bmd"`
	RMD struct {
		Version int64 `json:"version,string"`
	} `json:"rmd"`
	Config struct {
		Version int64 `json:"version,string"`
	} `json:"config"`
	EtlMD struct {
		Version int64 `json:"version,string"`
	} `json:"etlmd"`
	Flags NodeStateFlags `json:"flags"`
}

func (*NodeStateInfo) SmapEqual added in v1.3.23 

func (nsti *NodeStateInfo) SmapEqual(other *NodeStateInfo) (ok bool)

func (*NodeStateInfo) String added in v1.3.23 

func (nsti *NodeStateInfo) String() string

type NopLocker added in v1.3.18 

type NopLocker struct{}

func (NopLocker) Lock added in v1.3.18 

func (NopLocker) Lock()

func (NopLocker) Unlock added in v1.3.18 

func (NopLocker) Unlock()

type OAH added in v1.3.18 

type OAH interface {
	Lsize(special ...bool) int64
	Version(special ...bool) string
	VersionPtr() *string
	Checksum() *Cksum
	AtimeUnix() int64
	GetCustomMD() StrKVs
	GetCustomKey(key string) (val string, exists bool)
	SetCustomKey(k, v string)
	String() string
}

type Packer 

type Packer interface {
	Pack(packer *BytePack)
	PackedSize() int
}

type ParsedQuantity 

type ParsedQuantity struct {
	Type  string
	Value uint64
}

func ParseQuantity 

func ParseQuantity(quantity string) (ParsedQuantity, error)

func (ParsedQuantity) String 

func (pq ParsedQuantity) String() string

type ParsedTemplate 

type ParsedTemplate struct {
	Prefix string
	Ranges []TemplateRange
	// contains filtered or unexported fields
}

func NewParsedTemplate 

func NewParsedTemplate(template string) (parsed ParsedTemplate, err error)

func ParseAtTemplate 

func ParseAtTemplate(template string) (pt ParsedTemplate, err error)

e.g.: - multi range: "prefix-@00001-gap-@100-suffix" - single range: "prefix@00100suffix"

func ParseBashTemplate 

func ParseBashTemplate(template string) (pt ParsedTemplate, err error)

examples - single-range: "prefix{0001..0010}suffix" - multi-range: "prefix-{00001..00010..2}-gap-{001..100..2}-suffix" (both prefix and suffix are optional, here and elsewhere)

func ParseFmtTemplate 

func ParseFmtTemplate(template string) (pt ParsedTemplate, err error)

template: "prefix-%06d-suffix" (both prefix and suffix are optional, here and elsewhere)

func (*ParsedTemplate) Clone added in v1.3.22 

func (pt *ParsedTemplate) Clone() *ParsedTemplate

func (*ParsedTemplate) Count 

func (pt *ParsedTemplate) Count() int64

func (*ParsedTemplate) InitIter 

func (pt *ParsedTemplate) InitIter()

func (*ParsedTemplate) Next 

func (pt *ParsedTemplate) Next() (string, bool)

func (*ParsedTemplate) ToSlice 

func (pt *ParsedTemplate) ToSlice(maxLen ...int) []string

maxLen specifies maximum objects to be returned

type ReadCloseSizer 

type ReadCloseSizer interface {
	io.ReadCloser
	Size() int64
}

readers: interfaces

func NewDeferRCS 

func NewDeferRCS(r ReadCloseSizer, cb func()) ReadCloseSizer

type ReadOpenCloseSizer 

type ReadOpenCloseSizer interface {
	ReadOpenCloser
	Size() int64
}

readers: interfaces

type ReadOpenCloser 

type ReadOpenCloser interface {
	io.ReadCloser
	Open() (ReadOpenCloser, error)
}

readers: interfaces

func NopOpener 

func NopOpener(r io.ReadCloser) ReadOpenCloser

type ReadReaderAt 

type ReadReaderAt interface {
	io.Reader
	io.ReaderAt
}

readers: interfaces

type ReadSizer 

type ReadSizer interface {
	io.Reader
	Size() int64
}

readers: interfaces

func NewSizedReader 

func NewSizedReader(r io.Reader, size int64) ReadSizer

type ReadWrapperFunc 

type ReadWrapperFunc func(r io.ReadCloser) io.ReadCloser

type ReaderArgs 

type ReaderArgs struct {
	R       io.Reader
	ReadCb  func(int, error)
	DeferCb func()
	Size    int64
}

readers: implementations

type ReaderWithArgs 

type ReaderWithArgs struct {
	// contains filtered or unexported fields
}

readers: implementations

func NewReaderWithArgs 

func NewReaderWithArgs(args ReaderArgs) *ReaderWithArgs

func (*ReaderWithArgs) Close 

func (r *ReaderWithArgs) Close() (err error)

func (*ReaderWithArgs) Open 

func (*ReaderWithArgs) Open() (ReadOpenCloser, error)

func (*ReaderWithArgs) Read 

func (r *ReaderWithArgs) Read(p []byte) (n int, err error)

func (*ReaderWithArgs) Size 

func (r *ReaderWithArgs) Size() int64

type Runner 

type Runner interface {
	Name() string
	Run() error
	Stop(error)
}

type SectionHandle 

type SectionHandle struct {
	// contains filtered or unexported fields
}

SectionHandle is a section of reader with optional padding that implements ReadOpenCloser interface.

func NewSectionHandle 

func NewSectionHandle(r io.ReaderAt, offset, size, padding int64) *SectionHandle

func (*SectionHandle) Close 

func (*SectionHandle) Close() error

func (*SectionHandle) Open 

func (f *SectionHandle) Open() (ReadOpenCloser, error)

func (*SectionHandle) Read 

func (f *SectionHandle) Read(buf []byte) (n int, err error)

Reads a reader section. When the slice finishes but the buffer is not filled yet, act as if it reads a few more bytes from somewhere.

type Semaphore 

type Semaphore struct {
	// contains filtered or unexported fields
}

Semaphore is a textbook _sempahore_ implemented as a wrapper on `chan struct{}`.

func NewSemaphore 

func NewSemaphore(n int) *Semaphore

func (*Semaphore) Acquire 

func (s *Semaphore) Acquire()

func (*Semaphore) Release 

func (s *Semaphore) Release()

func (*Semaphore) TryAcquire 

func (s *Semaphore) TryAcquire() <-chan struct{}

type SetSizeFunc 

type SetSizeFunc func(size int64)

type SimpleOAH added in v1.3.18 

type SimpleOAH struct {
	Size  int64
	Atime int64
}

convenience/shortcut

func (SimpleOAH) AtimeUnix added in v1.3.18 

func (s SimpleOAH) AtimeUnix() int64

func (SimpleOAH) Checksum added in v1.3.18 

func (SimpleOAH) Checksum() *Cksum

func (SimpleOAH) GetCustomKey added in v1.3.18 

func (SimpleOAH) GetCustomKey(string) (string, bool)

func (SimpleOAH) GetCustomMD added in v1.3.18 

func (SimpleOAH) GetCustomMD() StrKVs

func (SimpleOAH) Lsize added in v1.3.24 

func (s SimpleOAH) Lsize(...bool) int64

func (SimpleOAH) SetCustomKey added in v1.3.18 

func (SimpleOAH) SetCustomKey(_, _ string)

func (SimpleOAH) String added in v1.3.18 

func (SimpleOAH) String() string

func (SimpleOAH) Version added in v1.3.18 

func (SimpleOAH) Version(...bool) string

func (SimpleOAH) VersionPtr added in v1.3.24 

func (SimpleOAH) VersionPtr() *string

type SizeIEC added in v1.3.16 

type SizeIEC int64

func (SizeIEC) MarshalJSON added in v1.3.16 

func (siz SizeIEC) MarshalJSON() ([]byte, error)

func (SizeIEC) String added in v1.3.16 

func (siz SizeIEC) String() string

func (*SizeIEC) UnmarshalJSON added in v1.3.16 

func (siz *SizeIEC) UnmarshalJSON(b []byte) (err error)

type StatsUpdater added in v1.3.16 

type StatsUpdater interface {
	Inc(name string)
	Add(name string, val int64)
	SetFlag(name string, set NodeStateFlags)
	ClrFlag(name string, clr NodeStateFlags)
	SetClrFlag(name string, set, clr NodeStateFlags)
	Get(name string) int64
	AddMany(namedVal64 ...NamedVal64)
}

type StopCh 

type StopCh struct {
	// contains filtered or unexported fields
}

StopCh is a channel for stopping running things.

func NewStopCh 

func NewStopCh() *StopCh

func (*StopCh) Close 

func (sch *StopCh) Close()

func (*StopCh) Init 

func (sch *StopCh) Init()

func (*StopCh) Listen 

func (sch *StopCh) Listen() <-chan struct{}

type StrKVs 

type StrKVs map[string]string

func NewStrKVs 

func NewStrKVs(l int) StrKVs

func (StrKVs) Compare 

func (kvs StrKVs) Compare(other StrKVs) bool

func (StrKVs) Contains 

func (kvs StrKVs) Contains(key string) (ok bool)

func (StrKVs) ContainsAnyMatch 

func (kvs StrKVs) ContainsAnyMatch(in []string) string

func (StrKVs) Delete added in v1.3.23 

func (kvs StrKVs) Delete(key string)

func (StrKVs) KeyFor 

func (kvs StrKVs) KeyFor(value string) (key string)

func (StrKVs) Keys 

func (kvs StrKVs) Keys() []string

type StrSet 

type StrSet map[string]struct{}

func NewStrSet 

func NewStrSet(keys ...string) (ss StrSet)

func (StrSet) Add 

func (ss StrSet) Add(keys ...string)

func (StrSet) All 

func (ss StrSet) All(xs ...string) bool

func (StrSet) Clone 

func (ss StrSet) Clone() StrSet

func (StrSet) Contains 

func (ss StrSet) Contains(key string) (yes bool)

func (StrSet) Delete 

func (ss StrSet) Delete(key string)

func (StrSet) Intersection 

func (ss StrSet) Intersection(other StrSet) StrSet

func (StrSet) Set added in v1.3.16 

func (ss StrSet) Set(key string)

func (StrSet) String 

func (ss StrSet) String() string

func (StrSet) ToSlice 

func (ss StrSet) ToSlice() []string

type TemplateRange 

type TemplateRange struct {
	Gap        string // characters after the range (to the next range or end of the string)
	Start      int64
	End        int64
	Step       int64
	DigitCount int
}

type TimeoutGroup 

type TimeoutGroup struct {
	// contains filtered or unexported fields
}

TimeoutGroup is similar to sync.WaitGroup with the difference on Wait where we only allow timing out.

WARNING: It should not be used in critical code as it may have worse performance than sync.WaitGroup - use only if its needed.

WARNING: It is not safe to wait on completion in multiple threads!

WARNING: It is not recommended to reuse the TimeoutGroup - it was not designed for that and bugs can be expected, especially when previous group was not called with successful (without timeout) WaitTimeout.

func NewTimeoutGroup 

func NewTimeoutGroup() *TimeoutGroup

func (*TimeoutGroup) Add 

func (twg *TimeoutGroup) Add(n int)

func (*TimeoutGroup) Done 

func (twg *TimeoutGroup) Done()

Done decrements number of jobs left to do. Panics if the number jobs left is less than 0.

func (*TimeoutGroup) Wait 

func (twg *TimeoutGroup) Wait()

Wait waits until the Added pending count goes to zero. NOTE: must be invoked after _all_ Adds.

func (*TimeoutGroup) WaitTimeout 

func (twg *TimeoutGroup) WaitTimeout(timeout time.Duration) bool

Wait waits until the Added pending count goes to zero _or_ timeout. NOTE: must be invoked after _all_ Adds.

func (*TimeoutGroup) WaitTimeoutWithStop 

func (twg *TimeoutGroup) WaitTimeoutWithStop(timeout time.Duration, stop <-chan struct{}) (timed, stopped bool)

Wait waits until the Added pending count goes to zero _or_ timeout _or_ stop. NOTE: must be invoked after _all_ Adds.

type Unpacker 

type Unpacker interface {
	Unpack(unpacker *ByteUnpack) error
}

Every object that is going to use binary representation instead of JSON must implement two following methods

type WG 

type WG interface {
	Add(int)
	Done()
	Wait()
}

WG is an interface for wait group

func NewLimitedWaitGroup 

func NewLimitedWaitGroup(limit, wanted int) WG

usage: no more than `limit` (e.g., sys.NumCPU()) goroutines in parallel

type WriteSizer 

type WriteSizer interface {
	io.Writer
	Size() int64
}

writers

type WriterAt 

type WriterAt interface {
	io.Writer
	io.WriterAt
}

writers

type WriterMulti 

type WriterMulti struct {
	// contains filtered or unexported fields
}

writers

func NewWriterMulti 

func NewWriterMulti(w ...io.Writer) *WriterMulti

func (*WriterMulti) Write 

func (mw *WriterMulti) Write(b []byte) (n int, err error)

type WriterOnly 

type WriterOnly struct{ io.Writer }

WriterOnly is a helper struct to hide `io.ReaderFrom` interface implementation As far as http.ResponseWriter (and its underlying tcp conn.), the following are tradeoffs: [-] sendfile (when sending), or [-] copy_file_range (when writing local files) [+] use (reusable) buffer, reduce code path, reduce locking

type WriterTo2 added in v1.3.23 

type WriterTo2 interface {
	WriteTo2(dst io.Writer) error
}

common between `Buffer` (below) and `memsys.SGL`

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.