cos

package
v1.3.24 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 27, 2024 License: MIT Imports: 44 Imported by: 9

Documentation

Overview

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2021-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2021-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2023-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2022-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Index

Constants

View Source
const (
	ChecksumNone   = "none"
	ChecksumXXHash = "xxhash"
	ChecksumMD5    = "md5"
	ChecksumCRC32C = "crc32c"
	ChecksumSHA256 = "sha256" // crypto.SHA512_256 (SHA-2)
	ChecksumSHA512 = "sha512" // crypto.SHA512 (SHA-2)
)

checksums

View Source
const (
	CtxReadWrapper contextID = "readWrapper" // context key for ReadWrapperFunc
	CtxSetSize     contextID = "setSize"     // context key for SetSizeFunc
	CtxOriginalURL contextID = "origURL"     // context key for OriginalURL for HTTP cloud
)
View Source
const (
	ContentJSON           = "application/json"
	ContentJSONCharsetUTF = "application/json; charset=utf-8"
	ContentMsgPack        = "application/msgpack"
	ContentXML            = "application/xml"
	ContentBinary         = "application/octet-stream"

	// not present in IANA registry
	// mozilla.org has it though, and also https://en.wikipedia.org/wiki/List_of_archive_formats
	ContentTar = "application/x-tar"

	// not currently used
	ContentZip = "application/zip"
)

standard MIME types - https://www.iana.org/assignments/media-types/media-types.xhtml - https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types

View Source
const (
	// range to read:
	HdrRange          = "Range" // Ref: https://www.rfc-editor.org/rfc/rfc7233#section-2.1
	HdrRangeValPrefix = "bytes="
	// range read response:
	HdrContentRange          = "Content-Range"
	HdrContentRangeValPrefix = "bytes " // Ref: https://tools.ietf.org/html/rfc7233#section-4.2
	HdrAcceptRanges          = "Accept-Ranges"

	// content length & type
	HdrContentType        = "Content-Type"
	HdrContentTypeOptions = "X-Content-Type-Options"
	HdrContentLength      = "Content-Length"

	// misc. gen
	HdrUserAgent = "User-Agent"
	HdrAccept    = "Accept"
	HdrLocation  = "Location"
	HdrServer    = "Server"
	HdrETag      = "ETag" // Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag

	HdrHSTS = "Strict-Transport-Security"
)

Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers

View Source
const (
	// https://cloud.google.com/storage/docs/xml-api/reference-headers
	GsCksumHeader   = "x-goog-hash"
	GsVersionHeader = "x-goog-generation"
)
View Source
const (
	// https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html
	// https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html
	S3CksumHeader   = HdrETag
	S3VersionHeader = "x-amz-version-id"

	// s3 api request headers
	S3HdrObjSrc = "x-amz-copy-source"
	S3HdrMptCnt = "x-amz-mp-parts-count"

	// https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
	S3UnsignedPayload  = "UNSIGNED-PAYLOAD"
	S3HdrContentSHA256 = "x-amz-content-sha256"

	S3HdrBckRegion = "x-amz-bucket-region"

	S3ChecksumCRC32  = "x-amz-checksum-crc32"
	S3ChecksumCRC32C = "x-amz-checksum-crc32c"
	S3ChecksumSHA1   = "x-amz-checksum-sha1"
	S3ChecksumSHA256 = "x-amz-checksum-sha256"

	S3MetadataChecksumType = "x-amz-meta-ais-cksum-type"
	S3MetadataChecksumVal  = "x-amz-meta-ais-cksum-val"

	S3LastModified = "Last-Modified"
)
View Source
const (
	PermRWR   os.FileMode = 0o640
	PermRWRR  os.FileMode = 0o644 // (archived)
	PermRWXRX os.FileMode = 0o750
)

POSIX permissions

View Source
const (
	SmoduleTransport = 1 << iota
	SmoduleAIS
	SmoduleMemsys
	SmoduleCore
	SmoduleFS
	SmoduleReb
	SmoduleEC
	SmoduleStats
	SmoduleIOS
	SmoduleXs
	SmoduleBackend
	SmoduleSpace
	SmoduleMirror
	SmoduleDsort
	SmoduleDload
	SmoduleETL
	SmoduleS3
)
View Source
const (
	VoteInProgress       = NodeStateFlags(1 << iota) // warning
	ClusterStarted                                   // info: (primary: cluster-started | all other nodes: joined-cluster)
	NodeStarted                                      // info: (started; possibly, not joined yet)
	Rebalancing                                      // warning
	RebalanceInterrupted                             // warning
	Resilvering                                      // warning
	ResilverInterrupted                              // warning
	Restarted                                        // warning
	OOS                                              // red alert (see IsRed below)
	OOM                                              // red alert
	MaintenanceMode                                  // warning
	LowCapacity                                      // (used > high); warning: OOS possible soon..
	LowMemory                                        // ditto OOM
	DiskFault                                        // red
	NoMountpaths                                     // red: (reserved, not used)
	NumGoroutines                                    // red
	CertWillSoonExpire                               // warning X.509
	CertificateExpired                               // red --/--
	CertificateInvalid                               // red --/--
	KeepAliveErrors                                  // warning (new keep-alive errors during the last 5m)
)
View Source
const (
	QuantityPercent = "percent"
	QuantityBytes   = "bytes"
)
View Source
const (
	LetterRunes = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
	LenRunes    = len(LetterRunes)
)
View Source
const (
	UnitsIEC = "iec" // default
	UnitsSI  = "si"  // NOTE: currently, SI system is CLI-only (compare with cmn/cos/size.go)
	UnitsRaw = "raw"
)
View Source
const (
	KiB = 1024
	MiB = 1024 * KiB
	GiB = 1024 * MiB
	TiB = 1024 * GiB
)

IEC (binary) units

View Source
const (
	KB = 1000
	MB = 1000 * KB
	GB = 1000 * MB
	TB = 1000 * GB
)

IS (metric) units

View Source
const (
	StreamsOutObjCount = "stream.out.n"
	StreamsOutObjSize  = "stream.out.size"
	StreamsInObjCount  = "stream.in.n"
	StreamsInObjSize   = "stream.in.size"
)

intra-cluster transmit & receive (cumulative counters)

View Source
const (
	// Number of sync maps
	MultiSyncMapCount = 0x40 // m.b. a power of two
	MultiSyncMapMask  = MultiSyncMapCount - 1
)
View Source
const (
	WildcardMatchAll = "*"
	EmptyMatchAll    = ""
)
View Source
const (
	StampMicro = "15:04:05.000000" // time.StampMicro without a date
	StampSec   = "15:04:05"        // time.Stamp without a date

	// S3 ListObjectsV2
	// https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html
	ISO8601 = "2006-01-02T15:04:05.000Z"

	// S3 HeadObject
	// https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html#API_HeadObject_Examples
	RFC1123GMT = "Mon, 17 Dec 2012 02:14:10 GMT"
)

in addition to standard layouts at /usr/local/go/src/time/format.go

View Source
const (
	SizeofI64 = int(unsafe.Sizeof(uint64(0)))
	SizeofI32 = int(unsafe.Sizeof(uint32(0)))
	SizeofI16 = int(unsafe.Sizeof(uint16(0)))
)

assorted common constants

View Source
const (
	OnlyNice = "must be less than 32 characters and " + mayOnlyContain // NOTE tooLongID
	OnlyPlus = mayOnlyContain + ", and dots (.)"
)
View Source
const ContentLengthUnknown = -1
View Source
const (
	LenShortID = 9 // UUID length, as per https://github.com/teris-io/shortid#id-length

)
View Source
const MLCG32 = 1103515245 // xxhash seed
View Source
const NodeAlerts = "state.flags"
View Source
const PathSeparator = string(filepath.Separator)
View Source
const SizeofLen = SizeofI32

Internally size of byte slice or string is packed as this integer type. Since binary packed structs are used for fairly small amount of data, it is possible to use int32 instead of int64 to keep the length of the data.

Variables

View Source
var (
	ErrQuantityUsage   = errors.New("invalid quantity, format should be '81%' or '1GB'")
	ErrQuantityPercent = errors.New("percent must be in the range (0, 100)")
	ErrQuantityBytes   = errors.New("value (bytes) must be non-negative")
)
View Source
var (
	ErrEmptyTemplate = errors.New("empty range template")
)
View Source
var ErrWorkChanFull = errors.New("work channel full")

JSON is used to Marshal/Unmarshal API json messages and is initialized in init function.

View Source
var NoneCksum = NewCksum(ChecksumNone, "")
View Source
var Smodules = [...]string{
	"transport", "ais", "memsys", "cluster", "fs", "reb", "ec", "stats",
	"ios", "xs", "backend", "space", "mirror", "dsort", "downloader", "etl",
	"s3",
}

NOTE: keep in-sync with the above

Functions

func AnyHasPrefixInSlice

func AnyHasPrefixInSlice(prefix string, arr []string) bool

func Assert

func Assert(cond bool)

func AssertMsg

func AssertMsg(cond bool, msg string)

NOTE: when using Sprintf and such, `if (!cond) { AssertMsg(false, msg) }` is the preferable usage.

func AssertNoErr

func AssertNoErr(err error)

func Assertf

func Assertf(cond bool, f string, a ...any)

NOTE: Not to be used in the datapath - consider instead one of the flavors below.

func BHead

func BHead(b []byte, ls ...int) string

func CeilAlign

func CeilAlign(val, align uint) uint

returns smallest number divisible by `align` that is greater or equal `val`

func CeilAlignInt64

func CeilAlignInt64(val, align int64) int64

func CheckAlphaPlus added in v1.3.24

func CheckAlphaPlus(s, tag string) error

alpha-numeric++ including letters, numbers, dashes (-), and underscores (_) period (.) is allowed except for '..' (OnlyPlus const)

func ClearfAtomic

func ClearfAtomic(f *uint64, flags uint64) (ok bool)

func Close

func Close(closer io.Closer)

func ConvertToString

func ConvertToString(value any) (valstr string, err error)

func CopyBuffer added in v1.3.22

func CopyBuffer(dst io.Writer, src io.Reader, buf []byte) (written int64, err error)

a slightly modified excerpt from https://github.com/golang/go/blob/master/src/io/io.go#L407 - regular streaming copy with `io.WriteTo` and `io.ReaderFrom` not checked and not used - buffer _must_ be provided - see also: WriterOnly comment (above)

func CopyStruct

func CopyStruct(dst, src any)

shallow copy

func CreateDir

func CreateDir(dir string) error

CreateDir creates directory if does not exist. If the directory already exists returns nil.

func CreateFile

func CreateFile(fqn string) (*os.File, error)

CreateFile creates a new write-only (O_WRONLY) file with default cos.PermRWR permissions. NOTE: if the file pathname doesn't exist it'll be created. NOTE: if the file already exists it'll be also silently truncated.

func CryptoRandS added in v1.3.16

func CryptoRandS(n int) string

func DivCeil

func DivCeil(a, b int64) int64

func DivRound

func DivRound(a, b int64) int64

func DivRoundU64

func DivRoundU64(a, b uint64) uint64

func DrainReader

func DrainReader(r io.Reader)

DrainReader reads and discards all the data from a reader. No need for `io.CopyBuffer` as `io.Discard` has efficient `io.ReaderFrom` implementation.

func Err2ClientURLErr added in v1.3.16

func Err2ClientURLErr(err error) (uerr *url.Error)

func Errorf

func Errorf(format string, a ...any)

func Errorln added in v1.3.24

func Errorln(a ...any)

func ExitAssertLog added in v1.3.22

func ExitAssertLog(cond bool, a ...any)

+condition

func ExitLog added in v1.3.18

func ExitLog(a ...any)

func ExitLogf

func ExitLogf(f string, a ...any)

+log

func Exitf

func Exitf(f string, a ...any)

func ExpandPath

func ExpandPath(path string) string

ExpandPath replaces common abbreviations in file path (eg. `~` with absolute path to the current user home directory) and cleans the path.

func Ext added in v1.3.18

func Ext(path string) (ext string)

e.g.: "/aaa/bbb/ccc.tar.gz" => ".tar.gz", while "/aaa/bb.b/ccctargz" => ""

func FloodWriter

func FloodWriter(w io.Writer, n int64) error

FloodWriter writes `n` random bytes to provided writer.

func FlushClose

func FlushClose(file *os.File) (err error)

func FormatBigNum added in v1.3.20

func FormatBigNum(n int) (s string)

func FormatMilli

func FormatMilli(tm time.Duration) string

FormatMilli returns a duration formatted as milliseconds. For values bigger than millisecond, it returns an integer number "#ms". For values smaller than millisecond, the function returns fractional number "0.##ms"

func FormatNanoTime added in v1.3.16

func FormatNanoTime(unixnano int64, format string) string

`unixnano` here is expected to be "nanoseconds since January 1, 1970 UTC"

func FormatNowStamp added in v1.3.16

func FormatNowStamp() string

func FormatTime added in v1.3.16

func FormatTime(t time.Time, format string) string

func FreeMemToOS

func FreeMemToOS(force bool) bool

FreeMemToOS executes in a separate goroutine with at least so-many minutes (above) between the runs. It calls GC and returns allocated memory to the operating system. Notes:

  • `forceFreeMemTime` is expected to be significantly greater than the time spent in the goroutine
  • still, an unlikely case of overlapping runs is treated via `runningOOM`
  • for somewhat related OOS handling - esp. constants - see ais/tgtspace.go

func GenBEID added in v1.3.19

func GenBEID(val uint64, l int) string

"best-effort ID" - to independently and locally generate globally unique ID called by xreg.GenBEID

func GenDaemonID

func GenDaemonID() string

func GenTestingDaemonID

func GenTestingDaemonID(suffix string) string

(when config.TestingEnv)

func GenTie

func GenTie() string

3-letter tie breaker (fast)

func GenUUID

func GenUUID() (uuid string)

compare with xreg.GenBEID

func GetEnvOrDefault added in v1.3.24

func GetEnvOrDefault(envVar, defaultValue string) string

getEnvOrDefault returns the value of the environment variable if it exists, otherwise it returns the provided default value.

func HashK8sProxyID added in v1.3.23

func HashK8sProxyID(nodeName string) (pid string)

func HomeConfigDir

func HomeConfigDir(subdir string) (configDir string)

func HomeDir

func HomeDir() (string, error)

func Infoln added in v1.3.24

func Infoln(a ...any)

func InitShortID

func InitShortID(seed uint64)

func IsAlphaNice added in v1.3.16

func IsAlphaNice(s string) bool

letters and numbers w/ '-' and '_' permitted with limitations (see OnlyNice const)

func IsAnySetfAtomic

func IsAnySetfAtomic(f *uint64, flags uint64) (yes bool)

func IsAzureURL

func IsAzureURL(u *url.URL) bool

func IsEOF

func IsEOF(err error) bool

including "unexpecting EOF" to accommodate unsized streaming and early termination of the other side (prior to sending the first byte)

func IsErrBadCksum

func IsErrBadCksum(err error) bool

func IsErrBrokenPipe

func IsErrBrokenPipe(err error) (yes bool)

func IsErrClientURLTimeout added in v1.3.16

func IsErrClientURLTimeout(err error) bool

func IsErrConnectionNotAvail

func IsErrConnectionNotAvail(err error) (yes bool)

likely out of socket descriptors

func IsErrConnectionRefused

func IsErrConnectionRefused(err error) (yes bool)

retriable conn errs

func IsErrConnectionReset

func IsErrConnectionReset(err error) (yes bool)

func IsErrDNSLookup added in v1.3.24

func IsErrDNSLookup(err error) bool

func IsErrNotFound added in v1.3.18

func IsErrNotFound(err error) bool

func IsErrOOS

func IsErrOOS(err error) bool

func IsErrSyscallTimeout added in v1.3.16

func IsErrSyscallTimeout(err error) bool

func IsErrXattrNotFound

func IsErrXattrNotFound(err error) bool

func IsGoogleAPIURL

func IsGoogleAPIURL(u *url.URL) bool

func IsGoogleStorageURL

func IsGoogleStorageURL(u *url.URL) bool

func IsHT added in v1.3.24

func IsHT(url string) bool

func IsHTTPS

func IsHTTPS(url string) bool

func IsIOError

func IsIOError(err error) bool

Checks if the error is generated by any IO operation and if the error is severe enough to run the FSHC for mountpath testing

func IsLastB added in v1.3.22

func IsLastB(s string, b byte) bool

func IsNotExist added in v1.3.22

func IsNotExist(err error, ecode int) bool

NOTE: compare with cmn.IsErrObjNought() that also includes lmeta-not-found et al.

func IsParseBool

func IsParseBool(s string) bool

func IsParseEnvBoolOrDefault added in v1.3.24

func IsParseEnvBoolOrDefault(envVar string, defaultValue bool) (bool, error)

IsParseBoolOrDefault parses a boolean from the environment variable string or returns the default value if parsing fails.

func IsRetriableConnErr

func IsRetriableConnErr(err error) (yes bool)

func IsS3URL

func IsS3URL(link string) bool

func IsSetfAtomic

func IsSetfAtomic(f *uint64, flags uint64) (yes bool)

func IsTimeZero

func IsTimeZero(t time.Time) bool

func IsUnreachable

func IsUnreachable(err error, status int) bool

func IsValidAtime added in v1.3.18

func IsValidAtime(atime int64) bool

access time validity; prefetch special (negative) case - sets atime=-now 946771140000000000 = time.Parse(time.RFC3339Nano, "2000-01-01T23:59:00Z").UnixNano()

func IsValidUUID

func IsValidUUID(uuid string) bool

func JoinPath

func JoinPath(url, path string) string

JoinPath joins two path elements that may (or may not) be prefixed/suffixed with a slash.

func JoinWords

func JoinWords(w string, words ...string) (path string)

JoinWords uses forward slash to join any number of words into a single path. Returned path is prefixed with a slash.

func Left added in v1.3.24

func Left(left, right string) string

left if non-empty; otherwise right

func MatchAll added in v1.3.23

func MatchAll(template string) bool

func MorphMarshal

func MorphMarshal(data, v any) error

func MustMarshal

func MustMarshal(v any) []byte

MustMarshal marshals v and panics if error occurs.

func MustMarshalToString

func MustMarshalToString(v any) string

func MustMorphMarshal

func MustMorphMarshal(data, v any)

func NewCRC32C

func NewCRC32C() hash.Hash

func NewErrDataCksum added in v1.3.18

func NewErrDataCksum(a, b *Cksum, context ...string) error

func NewErrMetaCksum added in v1.3.18

func NewErrMetaCksum(a, b uint64, context ...string) error

func NewNowSource added in v1.3.23

func NewNowSource() rand.Source

func NewRandSource added in v1.3.23

func NewRandSource(seed uint64) rand.Source

func NopReader

func NopReader(size int64) io.Reader

func NowRand

func NowRand() *rand.Rand

func PackedBytesLen added in v1.3.24

func PackedBytesLen(b []byte) int

func PackedStrLen

func PackedStrLen(s string) int

PackedStrLen returns the size occupied by a given string in the output

func ParseBool

func ParseBool(s string) (bool, error)

ParseBool converts string to bool (case-insensitive):

y, yes, on -> true
n, no, off, <empty value> -> false

strconv handles the following:

1, true, t -> true
0, false, f -> false

func ParseSize added in v1.3.16

func ParseSize(size, units string) (int64, error)

when `units` arg is empty conversion is defined by the suffix

func ParseURL

func ParseURL(s string) (u *url.URL, valid bool)

func Plural

func Plural(num int) (s string)

(common use)

func ProbingFrequency

func ProbingFrequency(dur time.Duration) time.Duration

wait duration => probing frequency

func RandStringWithSrc

func RandStringWithSrc(src rand.Source, n int) string

func Ratio

func Ratio(high, low, curr int64) float32

func RatioPct

func RatioPct(high, low, curr int64) int64

func ReadAll added in v1.3.24

func ReadAll(r io.Reader) ([]byte, error)

func ReadAllN added in v1.3.24

func ReadAllN(r io.Reader, size int64) (b []byte, err error)

instead of os.ReadAll

func ReadLines

func ReadLines(filename string, cb func(string) error) error

Read a file line by line and call a callback for each line until the file ends or a callback returns io.EOF

func ReadOneInt64

func ReadOneInt64(filename string) (int64, error)

Read only the first line of a file and return it as int64 Do not use for big files: it reads all the content and then extracts the first line. Use for files that may contains a few lines with trailing EOL

func ReadOneLine

func ReadOneLine(filename string) (string, error)

Read only the first line of a file. Do not use for big files: it reads all the content and then extracts the first line. Use for files that may contains a few lines with trailing EOL

func ReadOneUint64

func ReadOneUint64(filename string) (uint64, error)

Read only the first line of a file and return it as uint64 Do not use for big files: it reads all the content and then extracts the first line. Use for files that may contains a few lines with trailing EOL

func RemoveFile

func RemoveFile(path string) (err error)

RemoveFile removes path; returns nil upon success or if the path does not exist.

func Rename

func Rename(src, dst string) (err error)

(creates destination directory if doesn't exist)

func ReparseQuery

func ReparseQuery(r *http.Request)

WARNING: `ReparseQuery` might affect non-tensorflow clients using S3-compatible API with AIStore. To be used with caution.

func Right(left, right string) string

right if non-empty; otherwise left

func S2Duration

func S2Duration(s string) (time.Duration, error)

func S2UnixNano

func S2UnixNano(s string) (int64, error)

func SHead

func SHead(s string) string

func SetfAtomic

func SetfAtomic(f *uint64, flags uint64) (ok bool)

"set" as in: "add"

func Stat

func Stat(path string) error

syscall to check that path exists (see bench/lstat)

func StrSlicesEqual

func StrSlicesEqual(lhs, rhs []string) bool

StrSlicesEqual compares content of two string slices. It is replacement for reflect.DeepEqual because the latter returns false if slices have the same values but in different order.

func StringInSlice

func StringInSlice(s string, arr []string) bool

func SupportedChecksums

func SupportedChecksums() (types []string)

func ToSizeIEC added in v1.3.16

func ToSizeIEC(b int64, digits int) string

(compare w/ CLI `ToSizeIS`)

func TrimLastB added in v1.3.23

func TrimLastB(s string, b byte) string

func UnixNano2S

func UnixNano2S(unixnano int64) string

func UnsafeB added in v1.3.16

func UnsafeB(s string) []byte

cast string to immutable bytes

func UnsafeBptr added in v1.3.24

func UnsafeBptr(s *string) *[]byte

*string to *[]bytes

func UnsafeS

func UnsafeS(b []byte) string

cast bytes to an immutable string

func UnsafeSptr added in v1.3.24

func UnsafeSptr(b []byte) *string

bytes to *string

func UnwrapSyscallErr

func UnwrapSyscallErr(err error) error

func ValidateCksumType

func ValidateCksumType(ty string, emptyOK ...bool) (err error)

func ValidateDaemonID

func ValidateDaemonID(id string) error

func WdsKey added in v1.3.23

func WdsKey(path string) (key string)

WebDataset convention - not to confuse with filepath.Base (!) * see https://github.com/webdataset/webdataset#the-webdataset-format

Types

type BitFlags

type BitFlags uint64

func (BitFlags) Clear

func (f BitFlags) Clear(flags BitFlags) BitFlags

func (BitFlags) IsAnySet

func (f BitFlags) IsAnySet(flags BitFlags) bool

func (BitFlags) IsSet

func (f BitFlags) IsSet(flags BitFlags) bool

func (BitFlags) Set

func (f BitFlags) Set(flags BitFlags) BitFlags

type Bits

type Bits uint8

func (*Bits) Clear

func (b *Bits) Clear(flag Bits)

func (*Bits) Has

func (b *Bits) Has(flag Bits) bool

func (*Bits) Set

func (b *Bits) Set(flag Bits)

func (*Bits) Toggle

func (b *Bits) Toggle(flag Bits)

type Buffer added in v1.3.23

type Buffer struct {
	// contains filtered or unexported fields
}

writers

func NewBuffer added in v1.3.23

func NewBuffer(b []byte) *Buffer

func (*Buffer) WriteTo2 added in v1.3.23

func (w *Buffer) WriteTo2(dst io.Writer) (err error)

type ByteHandle

type ByteHandle struct {
	*bytes.Reader
	// contains filtered or unexported fields
}

ByteHandle is a byte buffer(made from []byte) that implements ReadOpenCloser interface

func NewByteHandle

func NewByteHandle(bt []byte) *ByteHandle

func (*ByteHandle) Close

func (*ByteHandle) Close() error

func (*ByteHandle) Open

func (b *ByteHandle) Open() (ReadOpenCloser, error)

type BytePack

type BytePack struct {
	// contains filtered or unexported fields
}

func NewPacker

func NewPacker(buf []byte, bufLen int) *BytePack

func (*BytePack) Bytes

func (bw *BytePack) Bytes() []byte

func (*BytePack) WriteAny

func (bw *BytePack) WriteAny(st Packer)

func (*BytePack) WriteBool

func (bw *BytePack) WriteBool(b bool)

func (*BytePack) WriteByte

func (bw *BytePack) WriteByte(b byte)

func (*BytePack) WriteBytes

func (bw *BytePack) WriteBytes(b []byte)

func (*BytePack) WriteInt16

func (bw *BytePack) WriteInt16(i int16)

func (*BytePack) WriteInt32

func (bw *BytePack) WriteInt32(i int32)

func (*BytePack) WriteInt64

func (bw *BytePack) WriteInt64(i int64)

func (*BytePack) WriteMapStrUint16

func (bw *BytePack) WriteMapStrUint16(mp MapStrUint16)

func (*BytePack) WriteString

func (bw *BytePack) WriteString(s string)

func (*BytePack) WriteUint16

func (bw *BytePack) WriteUint16(i uint16)

func (*BytePack) WriteUint32

func (bw *BytePack) WriteUint32(i uint32)

func (*BytePack) WriteUint64

func (bw *BytePack) WriteUint64(i uint64)

type ByteUnpack

type ByteUnpack struct {
	// contains filtered or unexported fields
}

func NewUnpacker

func NewUnpacker(buf []byte) *ByteUnpack

func (*ByteUnpack) Bytes

func (br *ByteUnpack) Bytes() []byte

func (*ByteUnpack) Len

func (br *ByteUnpack) Len() int

func (*ByteUnpack) ReadAny

func (br *ByteUnpack) ReadAny(st Unpacker) error

func (*ByteUnpack) ReadBool

func (br *ByteUnpack) ReadBool() (bool, error)

func (*ByteUnpack) ReadByte

func (br *ByteUnpack) ReadByte() (byte, error)

func (*ByteUnpack) ReadBytes

func (br *ByteUnpack) ReadBytes() ([]byte, error)

func (*ByteUnpack) ReadInt16

func (br *ByteUnpack) ReadInt16() (int16, error)

func (*ByteUnpack) ReadInt32

func (br *ByteUnpack) ReadInt32() (int32, error)

func (*ByteUnpack) ReadInt64

func (br *ByteUnpack) ReadInt64() (int64, error)

func (*ByteUnpack) ReadMapStrUint16

func (br *ByteUnpack) ReadMapStrUint16() (MapStrUint16, error)

func (*ByteUnpack) ReadString

func (br *ByteUnpack) ReadString() (string, error)

func (*ByteUnpack) ReadUint16

func (br *ByteUnpack) ReadUint16() (uint16, error)

func (*ByteUnpack) ReadUint32

func (br *ByteUnpack) ReadUint32() (uint32, error)

func (*ByteUnpack) ReadUint64

func (br *ByteUnpack) ReadUint64() (uint64, error)

type CallbackROC

type CallbackROC struct {
	// contains filtered or unexported fields
}

readers: implementations

func NewCallbackReadOpenCloser

func NewCallbackReadOpenCloser(r ReadOpenCloser, readCb func(int, error), reportedBytes ...int) *CallbackROC

func (*CallbackROC) Close

func (r *CallbackROC) Close() error

func (*CallbackROC) Open

func (r *CallbackROC) Open() (ReadOpenCloser, error)

func (*CallbackROC) Read

func (r *CallbackROC) Read(p []byte) (n int, err error)

type Cksum

type Cksum struct {
	// contains filtered or unexported fields
}

func ChecksumBytes

func ChecksumBytes(b []byte, cksumType string) (cksum *Cksum, err error)

ChecksumBytes computes checksum of given bytes using additional buffer.

func NewCksum

func NewCksum(ty, value string) *Cksum

func (*Cksum) Clone

func (ck *Cksum) Clone() *Cksum

func (*Cksum) Equal

func (ck *Cksum) Equal(to *Cksum) bool

func (*Cksum) Get

func (ck *Cksum) Get() (string, string)

func (*Cksum) IsEmpty

func (ck *Cksum) IsEmpty() bool

func (*Cksum) MarshalJSON

func (ck *Cksum) MarshalJSON() ([]byte, error)

func (*Cksum) String

func (ck *Cksum) String() string

func (*Cksum) Ty

func (ck *Cksum) Ty() string

func (*Cksum) Type

func (ck *Cksum) Type() string

func (*Cksum) Val

func (ck *Cksum) Val() string

func (*Cksum) Value

func (ck *Cksum) Value() string

type CksumHash

type CksumHash struct {
	Cksum
	H hash.Hash
	// contains filtered or unexported fields
}

func CopyAndChecksum

func CopyAndChecksum(w io.Writer, r io.Reader, buf []byte, cksumType string) (n int64, cksum *CksumHash, err error)

CopyAndChecksum reads from `r` and writes to `w`; returns num bytes copied and checksum, or error

func CopyFile

func CopyFile(src, dst string, buf []byte, cksumType string) (written int64, cksum *CksumHash, err error)

and computes checksum if requested

func NewCksumHash

func NewCksumHash(ty string) (ck *CksumHash)

func SaveReader

func SaveReader(fqn string, reader io.Reader, buf []byte, cksumType string, size int64) (*CksumHash, error)

Saves the `reader` directly to `fqn`, checksums if requested

func (*CksumHash) Equal

func (ck *CksumHash) Equal(to *Cksum) bool

func (*CksumHash) Finalize

func (ck *CksumHash) Finalize()

func (*CksumHash) Init

func (ck *CksumHash) Init(ty string)

func (*CksumHash) Sum

func (ck *CksumHash) Sum() []byte

type CksumHashSize

type CksumHashSize struct {
	CksumHash
	Size int64
}

func (*CksumHashSize) Write

func (ck *CksumHashSize) Write(b []byte) (n int, err error)

type Duration

type Duration time.Duration

func (Duration) D

func (d Duration) D() time.Duration

func (Duration) MarshalJSON

func (d Duration) MarshalJSON() ([]byte, error)

func (Duration) String

func (d Duration) String() (s string)

func (*Duration) UnmarshalJSON

func (d *Duration) UnmarshalJSON(b []byte) (err error)

type DynSemaphore

type DynSemaphore struct {
	// contains filtered or unexported fields
}

DynSemaphore implements sempahore which can change its size during usage.

func NewDynSemaphore

func NewDynSemaphore(n int) *DynSemaphore

func (*DynSemaphore) Acquire

func (s *DynSemaphore) Acquire(cnts ...int)

func (*DynSemaphore) Release

func (s *DynSemaphore) Release(cnts ...int)

func (*DynSemaphore) SetSize

func (s *DynSemaphore) SetSize(n int)

func (*DynSemaphore) Size

func (s *DynSemaphore) Size() int

type ErrBadCksum

type ErrBadCksum struct {
	// contains filtered or unexported fields
}

in-cluster checksum validation (compare with cmn.ErrInvalidCksum)

func (*ErrBadCksum) Error

func (e *ErrBadCksum) Error() string

type ErrNotFound added in v1.3.18

type ErrNotFound struct {
	// contains filtered or unexported fields
}

func NewErrNotFound added in v1.3.18

func NewErrNotFound(where fmt.Stringer, what string) *ErrNotFound

func (*ErrNotFound) Error added in v1.3.18

func (e *ErrNotFound) Error() string

type ErrSignal

type ErrSignal struct {
	// contains filtered or unexported fields
}

func NewSignalError

func NewSignalError(s syscall.Signal) *ErrSignal

func (*ErrSignal) Error

func (e *ErrSignal) Error() string

func (*ErrSignal) ExitCode

func (e *ErrSignal) ExitCode() int

https://tldp.org/LDP/abs/html/exitcodes.html

type Errs added in v1.3.18

type Errs struct {
	// contains filtered or unexported fields
}

func (*Errs) Add added in v1.3.18

func (e *Errs) Add(err error)

func (*Errs) Cnt added in v1.3.18

func (e *Errs) Cnt() int

func (*Errs) Error added in v1.3.18

func (e *Errs) Error() (s string)

Errs is an error

func (*Errs) JoinErr added in v1.3.18

func (e *Errs) JoinErr() (cnt int, err error)

type FS added in v1.3.16

type FS struct {
	Fs     string
	FsType string
	FsID   FsID
}

func (*FS) Equal added in v1.3.16

func (fs *FS) Equal(otherFs FS) bool

func (*FS) String added in v1.3.16

func (fs *FS) String() string

type FileHandle

type FileHandle struct {
	*os.File
	// contains filtered or unexported fields
}

handles (and even more readers)

func NewFileHandle

func NewFileHandle(fqn string) (*FileHandle, error)

func (*FileHandle) Open

func (f *FileHandle) Open() (ReadOpenCloser, error)

type FileSectionHandle

type FileSectionHandle struct {
	// contains filtered or unexported fields
}

FileSectionHandle opens a file and reads a section of it with optional padding. It implements the ReadOpenCloser interface.

func NewFileSectionHandle

func NewFileSectionHandle(fqn string, offset, size int64) (*FileSectionHandle, error)

NewFileSectionHandle opens file which is expected at `fqn` and defines a SectionHandle on it to only read a specified section.

func (*FileSectionHandle) Close

func (f *FileSectionHandle) Close() error

func (*FileSectionHandle) Open

func (f *FileSectionHandle) Open() (ReadOpenCloser, error)

func (*FileSectionHandle) Read

func (f *FileSectionHandle) Read(buf []byte) (int, error)

type FsID

type FsID [2]int32

FsID is unified and cross-platform syscall.Fsid type which implements JSON marshaling.

func AddUniqueFsID added in v1.3.23

func AddUniqueFsID(ids []FsID, nid FsID) ([]FsID, bool)

func (FsID) MarshalJSON

func (d FsID) MarshalJSON() ([]byte, error)

func (FsID) String

func (d FsID) String() string

func (*FsID) UnmarshalJSON

func (d *FsID) UnmarshalJSON(b []byte) error

type JSONRawMsgs

type JSONRawMsgs map[string]jsoniter.RawMessage

type LimitedWaitGroup

type LimitedWaitGroup struct {
	// contains filtered or unexported fields
}

LimitedWaitGroup is helper struct which combines standard wait group and semaphore to limit the number of goroutines created.

func (*LimitedWaitGroup) Add

func (lwg *LimitedWaitGroup) Add(n int)

func (*LimitedWaitGroup) Done

func (lwg *LimitedWaitGroup) Done()

func (*LimitedWaitGroup) Wait

func (lwg *LimitedWaitGroup) Wait()

type LogLevel added in v1.3.18

type LogLevel string

func (LogLevel) Parse added in v1.3.18

func (l LogLevel) Parse() (level, modules int)

func (*LogLevel) Set added in v1.3.18

func (l *LogLevel) Set(level int, sm []string)

func (LogLevel) String added in v1.3.18

func (l LogLevel) String() (s string)

func (LogLevel) Validate added in v1.3.18

func (l LogLevel) Validate() (err error)

type LomReader added in v1.3.24

type LomReader interface {
	io.ReadCloser
	io.ReaderAt
}

readers: interfaces

type LomWriter added in v1.3.24

type LomWriter interface {
	io.WriteCloser
	Sync() error
}

readers: interfaces

type MapStrUint16

type MapStrUint16 map[string]uint16

type MultiSyncMap

type MultiSyncMap struct {
	M [MultiSyncMapCount]sync.Map
}

func (*MultiSyncMap) Get

func (msm *MultiSyncMap) Get(idx int) *sync.Map

func (*MultiSyncMap) GetByHash

func (msm *MultiSyncMap) GetByHash(hash uint32) *sync.Map

type NamedVal64

type NamedVal64 struct {
	Name       string
	NameSuffix string // forces immediate send when non-empty
	Value      int64
}

type NodeStateFlags added in v1.3.23

type NodeStateFlags BitFlags

func (NodeStateFlags) Clear added in v1.3.23

func (NodeStateFlags) IsOK added in v1.3.24

func (f NodeStateFlags) IsOK() bool

func (NodeStateFlags) IsRed added in v1.3.24

func (f NodeStateFlags) IsRed() bool

func (NodeStateFlags) IsSet added in v1.3.23

func (f NodeStateFlags) IsSet(flag NodeStateFlags) bool

func (NodeStateFlags) IsWarn added in v1.3.24

func (f NodeStateFlags) IsWarn() bool

func (NodeStateFlags) Set added in v1.3.23

func (NodeStateFlags) String added in v1.3.24

func (f NodeStateFlags) String() string

type NodeStateInfo added in v1.3.23

type NodeStateInfo struct {
	Smap struct {
		Primary struct {
			PubURL  string `json:"pub_url"`
			CtrlURL string `json:"control_url"`
			ID      string `json:"id"`
		}
		Version int64  `json:"version,string"`
		UUID    string `json:"uuid"`
	} `json:"smap"`
	BMD struct {
		UUID    string `json:"uuid"`
		Version int64  `json:"version,string"`
	} `json:"bmd"`
	RMD struct {
		Version int64 `json:"version,string"`
	} `json:"rmd"`
	Config struct {
		Version int64 `json:"version,string"`
	} `json:"config"`
	EtlMD struct {
		Version int64 `json:"version,string"`
	} `json:"etlmd"`
	Flags NodeStateFlags `json:"flags"`
}

func (*NodeStateInfo) SmapEqual added in v1.3.23

func (nsti *NodeStateInfo) SmapEqual(other *NodeStateInfo) (ok bool)

func (*NodeStateInfo) String added in v1.3.23

func (nsti *NodeStateInfo) String() string

type NopLocker added in v1.3.18

type NopLocker struct{}

func (NopLocker) Lock added in v1.3.18

func (NopLocker) Lock()

func (NopLocker) Unlock added in v1.3.18

func (NopLocker) Unlock()

type OAH added in v1.3.18

type OAH interface {
	Lsize(special ...bool) int64
	Version(special ...bool) string
	VersionPtr() *string
	Checksum() *Cksum
	AtimeUnix() int64
	GetCustomMD() StrKVs
	GetCustomKey(key string) (val string, exists bool)
	SetCustomKey(k, v string)
	String() string
}

type Packer

type Packer interface {
	Pack(packer *BytePack)
	PackedSize() int
}

type ParsedQuantity

type ParsedQuantity struct {
	Type  string
	Value uint64
}

func ParseQuantity

func ParseQuantity(quantity string) (ParsedQuantity, error)

func (ParsedQuantity) String

func (pq ParsedQuantity) String() string

type ParsedTemplate

type ParsedTemplate struct {
	Prefix string
	Ranges []TemplateRange
	// contains filtered or unexported fields
}

func NewParsedTemplate

func NewParsedTemplate(template string) (parsed ParsedTemplate, err error)

func ParseAtTemplate

func ParseAtTemplate(template string) (pt ParsedTemplate, err error)

e.g.: - multi range: "prefix-@00001-gap-@100-suffix" - single range: "prefix@00100suffix"

func ParseBashTemplate

func ParseBashTemplate(template string) (pt ParsedTemplate, err error)

examples - single-range: "prefix{0001..0010}suffix" - multi-range: "prefix-{00001..00010..2}-gap-{001..100..2}-suffix" (both prefix and suffix are optional, here and elsewhere)

func ParseFmtTemplate

func ParseFmtTemplate(template string) (pt ParsedTemplate, err error)

template: "prefix-%06d-suffix" (both prefix and suffix are optional, here and elsewhere)

func (*ParsedTemplate) Clone added in v1.3.22

func (pt *ParsedTemplate) Clone() *ParsedTemplate

func (*ParsedTemplate) Count

func (pt *ParsedTemplate) Count() int64

func (*ParsedTemplate) InitIter

func (pt *ParsedTemplate) InitIter()

func (*ParsedTemplate) Next

func (pt *ParsedTemplate) Next() (string, bool)

func (*ParsedTemplate) ToSlice

func (pt *ParsedTemplate) ToSlice(maxLen ...int) []string

maxLen specifies maximum objects to be returned

type ReadCloseSizer

type ReadCloseSizer interface {
	io.ReadCloser
	Size() int64
}

readers: interfaces

func NewDeferRCS

func NewDeferRCS(r ReadCloseSizer, cb func()) ReadCloseSizer

type ReadOpenCloseSizer

type ReadOpenCloseSizer interface {
	ReadOpenCloser
	Size() int64
}

readers: interfaces

type ReadOpenCloser

type ReadOpenCloser interface {
	io.ReadCloser
	Open() (ReadOpenCloser, error)
}

readers: interfaces

func NopOpener

func NopOpener(r io.ReadCloser) ReadOpenCloser

type ReadReaderAt

type ReadReaderAt interface {
	io.Reader
	io.ReaderAt
}

readers: interfaces

type ReadSizer

type ReadSizer interface {
	io.Reader
	Size() int64
}

readers: interfaces

func NewSizedReader

func NewSizedReader(r io.Reader, size int64) ReadSizer

type ReadWrapperFunc

type ReadWrapperFunc func(r io.ReadCloser) io.ReadCloser

type ReaderArgs

type ReaderArgs struct {
	R       io.Reader
	ReadCb  func(int, error)
	DeferCb func()
	Size    int64
}

readers: implementations

type ReaderWithArgs

type ReaderWithArgs struct {
	// contains filtered or unexported fields
}

readers: implementations

func NewReaderWithArgs

func NewReaderWithArgs(args ReaderArgs) *ReaderWithArgs

func (*ReaderWithArgs) Close

func (r *ReaderWithArgs) Close() (err error)

func (*ReaderWithArgs) Open

func (*ReaderWithArgs) Open() (ReadOpenCloser, error)

func (*ReaderWithArgs) Read

func (r *ReaderWithArgs) Read(p []byte) (n int, err error)

func (*ReaderWithArgs) Size

func (r *ReaderWithArgs) Size() int64

type Runner

type Runner interface {
	Name() string
	Run() error
	Stop(error)
}

type SectionHandle

type SectionHandle struct {
	// contains filtered or unexported fields
}

SectionHandle is a section of reader with optional padding that implements ReadOpenCloser interface.

func NewSectionHandle

func NewSectionHandle(r io.ReaderAt, offset, size, padding int64) *SectionHandle

func (*SectionHandle) Close

func (*SectionHandle) Close() error

func (*SectionHandle) Open

func (f *SectionHandle) Open() (ReadOpenCloser, error)

func (*SectionHandle) Read

func (f *SectionHandle) Read(buf []byte) (n int, err error)

Reads a reader section. When the slice finishes but the buffer is not filled yet, act as if it reads a few more bytes from somewhere.

type Semaphore

type Semaphore struct {
	// contains filtered or unexported fields
}

Semaphore is a textbook _sempahore_ implemented as a wrapper on `chan struct{}`.

func NewSemaphore

func NewSemaphore(n int) *Semaphore

func (*Semaphore) Acquire

func (s *Semaphore) Acquire()

func (*Semaphore) Release

func (s *Semaphore) Release()

func (*Semaphore) TryAcquire

func (s *Semaphore) TryAcquire() <-chan struct{}

type SetSizeFunc

type SetSizeFunc func(size int64)

type SimpleOAH added in v1.3.18

type SimpleOAH struct {
	Size  int64
	Atime int64
}

convenience/shortcut

func (SimpleOAH) AtimeUnix added in v1.3.18

func (s SimpleOAH) AtimeUnix() int64

func (SimpleOAH) Checksum added in v1.3.18

func (SimpleOAH) Checksum() *Cksum

func (SimpleOAH) GetCustomKey added in v1.3.18

func (SimpleOAH) GetCustomKey(string) (string, bool)

func (SimpleOAH) GetCustomMD added in v1.3.18

func (SimpleOAH) GetCustomMD() StrKVs

func (SimpleOAH) Lsize added in v1.3.24

func (s SimpleOAH) Lsize(...bool) int64

func (SimpleOAH) SetCustomKey added in v1.3.18

func (SimpleOAH) SetCustomKey(_, _ string)

func (SimpleOAH) String added in v1.3.18

func (SimpleOAH) String() string

func (SimpleOAH) Version added in v1.3.18

func (SimpleOAH) Version(...bool) string

func (SimpleOAH) VersionPtr added in v1.3.24

func (SimpleOAH) VersionPtr() *string

type SizeIEC added in v1.3.16

type SizeIEC int64

func (SizeIEC) MarshalJSON added in v1.3.16

func (siz SizeIEC) MarshalJSON() ([]byte, error)

func (SizeIEC) String added in v1.3.16

func (siz SizeIEC) String() string

func (*SizeIEC) UnmarshalJSON added in v1.3.16

func (siz *SizeIEC) UnmarshalJSON(b []byte) (err error)

type StatsUpdater added in v1.3.16

type StatsUpdater interface {
	Inc(name string)
	Add(name string, val int64)
	SetFlag(name string, set NodeStateFlags)
	ClrFlag(name string, clr NodeStateFlags)
	SetClrFlag(name string, set, clr NodeStateFlags)
	Get(name string) int64
	AddMany(namedVal64 ...NamedVal64)
}

type StopCh

type StopCh struct {
	// contains filtered or unexported fields
}

StopCh is a channel for stopping running things.

func NewStopCh

func NewStopCh() *StopCh

func (*StopCh) Close

func (sch *StopCh) Close()

func (*StopCh) Init

func (sch *StopCh) Init()

func (*StopCh) Listen

func (sch *StopCh) Listen() <-chan struct{}

type StrKVs

type StrKVs map[string]string

func NewStrKVs

func NewStrKVs(l int) StrKVs

func (StrKVs) Compare

func (kvs StrKVs) Compare(other StrKVs) bool

func (StrKVs) Contains

func (kvs StrKVs) Contains(key string) (ok bool)

func (StrKVs) ContainsAnyMatch

func (kvs StrKVs) ContainsAnyMatch(in []string) string

func (StrKVs) Delete added in v1.3.23

func (kvs StrKVs) Delete(key string)

func (StrKVs) KeyFor

func (kvs StrKVs) KeyFor(value string) (key string)

func (StrKVs) Keys

func (kvs StrKVs) Keys() []string

type StrSet

type StrSet map[string]struct{}

func NewStrSet

func NewStrSet(keys ...string) (ss StrSet)

func (StrSet) Add

func (ss StrSet) Add(keys ...string)

func (StrSet) All

func (ss StrSet) All(xs ...string) bool

func (StrSet) Clone

func (ss StrSet) Clone() StrSet

func (StrSet) Contains

func (ss StrSet) Contains(key string) (yes bool)

func (StrSet) Delete

func (ss StrSet) Delete(key string)

func (StrSet) Intersection

func (ss StrSet) Intersection(other StrSet) StrSet

func (StrSet) Set added in v1.3.16

func (ss StrSet) Set(key string)

func (StrSet) String

func (ss StrSet) String() string

func (StrSet) ToSlice

func (ss StrSet) ToSlice() []string

type TemplateRange

type TemplateRange struct {
	Gap        string // characters after the range (to the next range or end of the string)
	Start      int64
	End        int64
	Step       int64
	DigitCount int
}

type TimeoutGroup

type TimeoutGroup struct {
	// contains filtered or unexported fields
}

TimeoutGroup is similar to sync.WaitGroup with the difference on Wait where we only allow timing out.

WARNING: It should not be used in critical code as it may have worse performance than sync.WaitGroup - use only if its needed.

WARNING: It is not safe to wait on completion in multiple threads!

WARNING: It is not recommended to reuse the TimeoutGroup - it was not designed for that and bugs can be expected, especially when previous group was not called with successful (without timeout) WaitTimeout.

func NewTimeoutGroup

func NewTimeoutGroup() *TimeoutGroup

func (*TimeoutGroup) Add

func (twg *TimeoutGroup) Add(n int)

func (*TimeoutGroup) Done

func (twg *TimeoutGroup) Done()

Done decrements number of jobs left to do. Panics if the number jobs left is less than 0.

func (*TimeoutGroup) Wait

func (twg *TimeoutGroup) Wait()

Wait waits until the Added pending count goes to zero. NOTE: must be invoked after _all_ Adds.

func (*TimeoutGroup) WaitTimeout

func (twg *TimeoutGroup) WaitTimeout(timeout time.Duration) bool

Wait waits until the Added pending count goes to zero _or_ timeout. NOTE: must be invoked after _all_ Adds.

func (*TimeoutGroup) WaitTimeoutWithStop

func (twg *TimeoutGroup) WaitTimeoutWithStop(timeout time.Duration, stop <-chan struct{}) (timed, stopped bool)

Wait waits until the Added pending count goes to zero _or_ timeout _or_ stop. NOTE: must be invoked after _all_ Adds.

type Unpacker

type Unpacker interface {
	Unpack(unpacker *ByteUnpack) error
}

Every object that is going to use binary representation instead of JSON must implement two following methods

type WG

type WG interface {
	Add(int)
	Done()
	Wait()
}

WG is an interface for wait group

func NewLimitedWaitGroup

func NewLimitedWaitGroup(limit, wanted int) WG

usage: no more than `limit` (e.g., sys.NumCPU()) goroutines in parallel

type WriteSizer

type WriteSizer interface {
	io.Writer
	Size() int64
}

writers

type WriterAt

type WriterAt interface {
	io.Writer
	io.WriterAt
}

writers

type WriterMulti

type WriterMulti struct {
	// contains filtered or unexported fields
}

writers

func NewWriterMulti

func NewWriterMulti(w ...io.Writer) *WriterMulti

func (*WriterMulti) Write

func (mw *WriterMulti) Write(b []byte) (n int, err error)

type WriterOnly

type WriterOnly struct{ io.Writer }

WriterOnly is a helper struct to hide `io.ReaderFrom` interface implementation As far as http.ResponseWriter (and its underlying tcp conn.), the following are tradeoffs: [-] sendfile (when sending), or [-] copy_file_range (when writing local files) [+] use (reusable) buffer, reduce code path, reduce locking

type WriterTo2 added in v1.3.23

type WriterTo2 interface {
	WriteTo2(dst io.Writer) error
}

common between `Buffer` (below) and `memsys.SGL`

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL