Documentation ¶
Overview ¶
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
msgp -file cmn/objlist.go -tests=false -marshal=false -unexported Code generated by the command above where msgp is tinylib/msgp; see docs/msgp.md. DO NOT EDIT.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2023-2024, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2022-2024, NVIDIA CORPORATION. All rights reserved.
Index ¶
- Constants
- Variables
- func CustomMD2S(md cos.StrKVs) string
- func DelBckFromQuery(query url.Values) url.Values
- func DirHasOrIsPrefix(dirPath, prefix string) bool
- func EnvToTLS(sargs *TLSArgs)
- func FreeBuffer(buf *bytes.Buffer)
- func FreeHra(a *HreqArgs)
- func FreeHterr(a *ErrHTTP)
- func Host2IP(host string) (net.IP, error)
- func InitErrs(a string, b func(error))
- func IsErrAborted(err error) bool
- func IsErrBckNotFound(err error) bool
- func IsErrBucketAlreadyExists(err error) bool
- func IsErrBucketLevel(err error) bool
- func IsErrBucketNought(err error) bool
- func IsErrCapExceeded(err error) bool
- func IsErrLmetaCorrupted(err error) bool
- func IsErrLmetaNotFound(err error) bool
- func IsErrMountpathNotFound(err error) bool
- func IsErrObjLevel(err error) bool
- func IsErrObjNought(err error) bool
- func IsErrRangeNotSatisfiable(err error) bool
- func IsErrRemoteBckNotFound(err error) bool
- func IsErrSoft(err error) bool
- func IsErrStreamTerminated(err error) bool
- func IsErrXactNotFound(err error) bool
- func IsErrXactUsePrev(err error) bool
- func IsFileAlreadyClosed(err error) bool
- func IsNestedMpath(a string, la int, b string) (err error)
- func IsS3MultipartEtag(etag string) bool
- func IsStatusBadGateway(err error) (yes bool)
- func IsStatusGone(err error) (yes bool)
- func IsStatusNotFound(err error) (yes bool)
- func IsStatusServiceUnavailable(err error) (yes bool)
- func IterFields(v any, updf updateFunc, opts ...IterOpts) error
- func KeepaliveRetryDuration(c *Config) time.Duration
- func LoadConfig(globalConfPath, localConfPath, daeRole string, config *Config) error
- func MakeRangeHdr(start, length int64) string
- func MaxParallelism() int
- func NetworkCallWithRetry(args *RetryArgs) (err error)
- func NetworkIsKnown(net string) bool
- func NewBuffer() (buf *bytes.Buffer)
- func NewClient(cargs TransportArgs) *http.Client
- func NewClientTLS(cargs TransportArgs, sargs TLSArgs) *http.Client
- func NewDefaultClients(timeout time.Duration) (clientH, clientTLS *http.Client)
- func NewIntraClientTLS(cargs TransportArgs, config *Config) *http.Client
- func NewTLS(sargs TLSArgs) (tlsConf *tls.Config, _ error)
- func NewTransport(cargs TransportArgs) *http.Transport
- func NormalizeProvider(provider string) (p string, err error)
- func ObjHasPrefix(objName, prefix string) bool
- func OrigURLBck2Name(origURLBck string) (bckName string)
- func ParseHost2IP(host string) (net.IP, error)
- func ParsePort(p string) (int, error)
- func ParseReadHeaderTimeout() (_ time.Duration, isSet bool)
- func ParseURL(path string, itemsPresent []string, itemsAfter int, splitAfter bool) ([]string, error)
- func ParseURLScheme(url string) (scheme, address string)
- func PrependProtocol(url string, protocol ...string) string
- func ReadBytes(r *http.Request) (b []byte, err error)
- func ReadJSON(w http.ResponseWriter, r *http.Request, out any) (err error)
- func S2CustomMD(custom, version string) (md cos.StrKVs)
- func SaveOverrideConfig(configDir string, toUpdate *ConfigToSet) error
- func SortLso(entries LsoEntries)
- func ToHeader(oah cos.OAH, hdr http.Header, size int64, cksums ...*cos.Cksum)
- func TokenGreaterEQ(token, objName string) bool
- func TypeCodeHTTPErr(s string) (tcode string)
- func UnquoteCEV(val string) string
- func UpdateFieldValue(s any, name string, value any) error
- func ValidateMpath(mpath string) (string, error)
- func ValidatePort(port int) (int, error)
- func ValidatePrefix(s string) error
- func ValidateRemAlias(alias string) (err error)
- func WriteErr(w http.ResponseWriter, r *http.Request, err error, opts ...int)
- func WriteErr405(w http.ResponseWriter, r *http.Request, methods ...string)
- func WriteErrJSON(w http.ResponseWriter, r *http.Request, out any, err error) error
- func WriteErrMsg(w http.ResponseWriter, r *http.Request, msg string, opts ...int)
- type AllBsummResults
- type ArchiveBckMsg
- type AuthConf
- type AuthConfToSet
- type BackendBckToSet
- type BackendConf
- func (c *BackendConf) EqualClouds(o *BackendConf) bool
- func (c *BackendConf) EqualRemAIS(o *BackendConf, sname string) bool
- func (c *BackendConf) Get(provider string) (conf any)
- func (c *BackendConf) MarshalJSON() (data []byte, err error)
- func (c *BackendConf) Set(provider string, newConf any)
- func (c *BackendConf) UnmarshalJSON(data []byte) error
- func (c *BackendConf) Validate() (err error)
- type BackendConfAIS
- type Bck
- func (b *Bck) AddToQuery(query url.Values) url.Values
- func (b *Bck) AddUnameToQuery(query url.Values, uparam string) url.Values
- func (b *Bck) Backend() *Bck
- func (b *Bck) Cname(objname string) (s string)
- func (b *Bck) Copy(src *Bck)
- func (bck *Bck) DefaultProps(c *ClusterConfig) *Bprops
- func (b Bck) Equal(other *Bck) bool
- func (b *Bck) HasProvider() bool
- func (b *Bck) HasVersioningMD() bool
- func (b *Bck) IsAIS() bool
- func (b *Bck) IsCloud() bool
- func (b *Bck) IsEmpty() bool
- func (b *Bck) IsHTTP() bool
- func (b *Bck) IsQuery() bool
- func (b *Bck) IsRemote() bool
- func (b *Bck) IsRemoteAIS() bool
- func (b *Bck) LenUnameGlob(objName string) int
- func (b *Bck) Less(other *Bck) bool
- func (b *Bck) MakeUname(objName string) string
- func (b *Bck) NewQuery() (q url.Values)
- func (b *Bck) RemoteBck() *Bck
- func (b Bck) String() (s string)
- func (b *Bck) Validate() (err error)
- func (b *Bck) ValidateName() (err error)
- type Bcks
- type Bprops
- type BpropsToSet
- type BsummResult
- type CksumConf
- type CksumConfToSet
- type ClientConf
- type ClientConfToSet
- type ClusterConfig
- type Config
- type ConfigToSet
- type DiskConf
- type DiskConfToSet
- type DownloaderConf
- type DownloaderConfToSet
- type DsortConf
- type DsortConfToSet
- type ECConf
- type ECConfToSet
- type ETLErrCtx
- type ErrAborted
- type ErrBckNotFound
- type ErrBucketAccessDenied
- type ErrBucketAlreadyExists
- type ErrBusy
- type ErrCapExceeded
- type ErrETL
- type ErrFailedTo
- type ErrHTTP
- type ErrInitBackend
- type ErrInvalidBackendProvider
- type ErrInvalidCksum
- type ErrInvalidFSPathsConf
- type ErrInvalidMountpath
- type ErrInvalidObjName
- type ErrLimitedCoexistence
- type ErrLmetaCorrupted
- type ErrLmetaNotFound
- type ErrMissingBackend
- type ErrMountpathNotFound
- type ErrNoNodes
- type ErrNotImpl
- type ErrNotRemoteBck
- type ErrObjDefunct
- type ErrObjectAccessDenied
- type ErrRangeNotSatisfiable
- type ErrRemoteBckNotFound
- type ErrRemoteBucketOffline
- type ErrSoft
- type ErrStreamTerminated
- type ErrUnsupp
- type ErrXactNotFound
- type ErrXactTgtInMaint
- type ErrXactUsePrev
- type ExtraProps
- type ExtraPropsAWS
- type ExtraPropsAWSToSet
- type ExtraPropsHDFS
- type ExtraPropsHDFSToSet
- type ExtraPropsHTTP
- type ExtraPropsHTTPToSet
- type ExtraToSet
- type FSHCConf
- type FSHCConfToSet
- type FSPConf
- type FSPConfV322
- type HTTPBckObj
- type HTTPConf
- type HTTPConfToSet
- type HreqArgs
- type IterField
- type IterOpts
- type KeepaliveConf
- type KeepaliveConfToSet
- type KeepaliveTrackerConf
- type KeepaliveTrackerConfToSet
- type L4Conf
- type LRUConf
- type LRUConfToSet
- type LocalConfig
- type LocalNetConfig
- type LogConf
- type LogConfToSet
- type LsoEnt
- func (be *LsoEnt) CopyWithProps(propsSet cos.StrSet) (ne *LsoEnt)
- func (z *LsoEnt) DecodeMsg(dc *msgp.Reader) (err error)
- func (z *LsoEnt) EncodeMsg(en *msgp.Writer) (err error)
- func (be *LsoEnt) IsDir() bool
- func (be *LsoEnt) IsInsideArch() bool
- func (be *LsoEnt) IsListedArch() bool
- func (be *LsoEnt) IsPresent() bool
- func (be *LsoEnt) IsStatusOK() bool
- func (be *LsoEnt) IsVerChanged() bool
- func (be *LsoEnt) IsVerRemoved() bool
- func (z *LsoEnt) Msgsize() (s int)
- func (be *LsoEnt) SetPresent()
- func (be *LsoEnt) SetVerChanged()
- func (be *LsoEnt) SetVerRemoved()
- func (be *LsoEnt) Status() uint16
- func (be *LsoEnt) String() string
- type LsoEntries
- type LsoRes
- type MemsysConf
- type MemsysConfToSet
- type MirrorConf
- type MirrorConfToSet
- type NetConf
- type NetConfToSet
- type Ns
- type OWT
- type ObjAttrs
- func (oa *ObjAttrs) AtimeUnix() int64
- func (oa *ObjAttrs) Checksum() *cos.Cksum
- func (oa *ObjAttrs) CopyFrom(oah cos.OAH, skipCksum bool)
- func (oa *ObjAttrs) DelCustomKeys(keys ...string)
- func (oa *ObjAttrs) Equal(rem cos.OAH) (eq bool)
- func (oa *ObjAttrs) FromHeader(hdr http.Header) (cksum *cos.Cksum)
- func (oa *ObjAttrs) FromLsoEntry(e *LsoEnt)
- func (oa *ObjAttrs) GetCustomKey(key string) (val string, exists bool)
- func (oa *ObjAttrs) GetCustomMD() cos.StrKVs
- func (oa *ObjAttrs) SetCksum(ty, val string)
- func (oa *ObjAttrs) SetCustomKey(k, v string)
- func (oa *ObjAttrs) SetCustomMD(md cos.StrKVs)
- func (oa *ObjAttrs) SetSize(size int64)
- func (oa *ObjAttrs) SizeBytes(_ ...bool) int64
- func (oa *ObjAttrs) String() string
- func (oa *ObjAttrs) Version(_ ...bool) string
- type ObjectProps
- type ParseURIOpts
- type PeriodConf
- type PeriodConfToSet
- type PropsValidator
- type ProxyConf
- type ProxyConfToSet
- type QueryBcks
- func (qbck *QueryBcks) AddToQuery(query url.Values)
- func (qbck QueryBcks) Contains(other *Bck) bool
- func (qbck QueryBcks) Equal(bck *Bck) bool
- func (qbck *QueryBcks) IsAIS() bool
- func (qbck *QueryBcks) IsBucket() bool
- func (qbck *QueryBcks) IsCloud() bool
- func (qbck *QueryBcks) IsEmpty() bool
- func (qbck *QueryBcks) IsHTTP() bool
- func (qbck *QueryBcks) IsRemoteAIS() bool
- func (qbck *QueryBcks) NewQuery() url.Values
- func (qbck QueryBcks) String() string
- func (qbck *QueryBcks) Validate() (err error)
- type RebalanceConf
- type RebalanceConfToSet
- type ResilverConf
- type ResilverConfToSet
- type RetryArgs
- type SpaceConf
- type SpaceConfToSet
- type TCBConf
- type TCBConfToSet
- type TCObjsMsg
- type TLSArgs
- type TestFSPConf
- type TimeoutConf
- type TimeoutConfToSet
- type TransportArgs
- type TransportConf
- type TransportConfToSet
- type Validator
- type VersionConf
- type VersionConfToSet
- type WritePolicyConf
- type WritePolicyConfToSet
Constants ¶
const ( PropBucketAccessAttrs = "access" // Bucket access attributes. PropBucketVerEnabled = "versioning.enabled" // Enable/disable object versioning in a bucket. PropBucketCreated = "created" // Bucket creation time. PropBackendBck = "backend_bck" PropBackendBckName = PropBackendBck + ".name" PropBackendBckProvider = PropBackendBck + ".provider" )
const ( IgnoreReaction = "ignore" WarnReaction = "warn" AbortReaction = "abort" )
dsort
const ( FmtErrIntegrity = "[%s%d, for troubleshooting see %s/blob/main/docs/troubleshooting.md]" FmtErrUnmarshal = "%s: failed to unmarshal %s (%s), err: %w" FmtErrMorphUnmarshal = "%s: failed to unmarshal %s (%T), err: %w" FmtErrUnknown = "%s: unknown %s %q" FmtErrBackwardCompat = "%v (backward compatibility is supported only one version back, e.g. 3.9 => 3.10)" BadSmapPrefix = "[bad cluster map]" StartupMayTimeout = "cluster startup is taking unusually long time..." // related ErrStartupTimeout )
const ( RetryLogVerbose = iota RetryLogQuiet RetryLogOff )
const ( NetPublic = "PUBLIC" NetIntraControl = "INTRA-CONTROL" NetIntraData = "INTRA-DATA" )
const ( DefaultMaxIdleConns = 64 DefaultMaxIdleConnsPerHost = 16 DefaultIdleConnTimeout = 8 * time.Second DefaultWriteBufferSize = 64 * cos.KiB DefaultReadBufferSize = 64 * cos.KiB DefaultSendRecvBufferSize = 128 * cos.KiB )
http.DefaultTransport has the following defaults: - MaxIdleConns: 100, - MaxIdleConnsPerHost : 2 (via DefaultMaxIdleConnsPerHost) - IdleConnTimeout: 90 * time.Second, - WriteBufferSize: 4KB - ReadBufferSize: 4KB Following are the constants we use by default:
const ( // source of the cold-GET and download; the values include all // 3rd party backend providers SourceObjMD = "source" // downloader' source is "web" WebObjMD = "web" VersionObjMD = "version" // "generation" for GCP, "version" for AWS but only if the bucket is versioned, etc. CRC32CObjMD = cos.ChecksumCRC32C MD5ObjMD = cos.ChecksumMD5 ETag = cos.HdrETag OrigURLObjMD = "orig_url" // additional backend LastModified = "LastModified" )
LOM custom metadata stored under `lomCustomMD`.
const ( VersionAIStore = "3.23" VersionCLI = "1.12" VersionLoader = "1.11" VersionAuthN = "1.0" )
const ( MetaverSmap = 2 // Smap (cluster map) formatting version a.k.a. meta-version (see core/meta/jsp.go) MetaverBMD = 2 // BMD (bucket metadata) --/-- MetaverRMD = 1 // Rebalance MD (jsp) MetaverVMD = 2 // Volume MD (jsp) MetaverEtlMD = 1 // ETL MD (jsp) MetaverLOM = 1 // LOM MetaverConfig = 4 // Global Configuration (jsp) MetaverAuthNConfig = 1 // Authn config (jsp) // ditto MetaverAuthTokens = 1 // Authn tokens (jsp) // ditto MetaverMetasync = 1 // metasync over network formatting version (jsp) MetaverJSP = jsp.Metaver // `jsp` own encoding version )
const AwsMultipartDelim = "-"
from https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html "The ETag may or may not be an MD5 digest of the object data. Whether or not it is depends on how the object was created and how it is encrypted..."
const GitHubHome = "https://github.com/NVIDIA/aistore"
const HostnameListSepa = ","
const IterFieldNameSepa = "."
const MsgpLsoBufSize = 32 * cos.KiB
const ( // NsGlobalUname is hardcoded here to avoid allocating it via Uname() // (the most common use case) NsGlobalUname = "@#" )
const (
ObjSizeToAlwaysReplicate = -1 // (see `ObjSizeLimit` comment above)
)
Variables ¶
var ( // NsGlobal represents *this* cluster's global namespace that is used by default when // no specific namespace was defined or provided by the user. NsGlobal = Ns{} // NsAnyRemote represents any remote cluster. As such, NsGlobalRemote applies // exclusively to AIS (provider) given that other Backend providers are remote by definition. NsAnyRemote = Ns{UUID: string(apc.NsUUIDPrefix)} )
var ( ErrSkip = errors.New("skip") ErrStartupTimeout = errors.New("startup timeout") // related StartupMayTimeout ErrQuiesceTimeout = errors.New("timed out waiting for quiescence") ErrNotEnoughTargets = errors.New("not enough target nodes") ErrNoMountpaths = errors.New("no mountpaths") // aborts ErrXactRenewAbort = errors.New("renewal abort") ErrXactUserAbort = errors.New("user abort") // via apc.ActXactStop ErrXactICNotifAbort = errors.New("IC(notifications) abort") // ditto )
var BackendHelpers = struct { Amazon backendFuncs Azure backendFuncs Google backendFuncs HTTP backendFuncs }{ Amazon: backendFuncs{ EncodeVersion: func(v any) (string, bool) { switch x := v.(type) { case *string: if awsIsVersionSet(x) { return *x, true } return "", false case string: if awsIsVersionSet(&x) { return x, true } return x, false default: debug.FailTypeCast(v) return "", false } }, EncodeCksum: func(v any) (string, bool) { switch x := v.(type) { case *string: if IsS3MultipartEtag(*x) { return *x, true } return UnquoteCEV(*x), true case string: return x, true default: debug.FailTypeCast(v) return "", false } }, }, Google: backendFuncs{ EncodeVersion: func(v any) (string, bool) { switch x := v.(type) { case string: return x, x != "" case int64: return strconv.FormatInt(x, 10), true default: debug.FailTypeCast(v) return "", false } }, EncodeCksum: func(v any) (string, bool) { switch x := v.(type) { case string: decoded, err := base64.StdEncoding.DecodeString(x) if err != nil { return "", false } return hex.EncodeToString(decoded), true case []byte: return hex.EncodeToString(x), true case uint32: b := []byte{byte(x >> 24), byte(x >> 16), byte(x >> 8), byte(x)} return base64.StdEncoding.EncodeToString(b), true default: debug.FailTypeCast(v) return "", false } }, }, HTTP: backendFuncs{ EncodeVersion: func(v any) (string, bool) { switch x := v.(type) { case string: x = strings.TrimPrefix(x, "W/") x = UnquoteCEV(x) return x, x != "" default: debug.FailTypeCast(v) return "", false } }, }, }
var ConfigRestartRequired = []string{"auth", "memsys", "net"}
assorted named fields that require (cluster | node) restart for changes to make an effect
var GCO *gco
var KnownNetworks = []string{NetPublic, NetIntraControl, NetIntraData}
var Rom readMostly
var SupportedReactions = []string{IgnoreReaction, WarnReaction, AbortReaction}
Functions ¶
func CustomMD2S ¶
func DirHasOrIsPrefix ¶ added in v1.3.16
Directory has to either: - include (or match) prefix, or - be contained in prefix - motivation: don't SkipDir a/b when looking for a/b/c An alternative name for this function could be smth. like SameBranch()
func FreeBuffer ¶ added in v1.3.18
func IsErrAborted ¶
func IsErrBckNotFound ¶
func IsErrBucketLevel ¶
func IsErrCapExceeded ¶ added in v1.3.19
func IsErrLmetaCorrupted ¶
func IsErrLmetaNotFound ¶
func IsErrMountpathNotFound ¶
func IsErrObjLevel ¶
func IsErrRangeNotSatisfiable ¶ added in v1.3.22
func IsErrRemoteBckNotFound ¶
func IsErrStreamTerminated ¶ added in v1.3.16
func IsErrXactNotFound ¶ added in v1.3.16
func IsErrXactUsePrev ¶ added in v1.3.18
func IsFileAlreadyClosed ¶ added in v1.3.16
func IsS3MultipartEtag ¶ added in v1.3.22
func IsStatusBadGateway ¶
func IsStatusGone ¶
func IsStatusNotFound ¶
func IterFields ¶
IterFields walks the struct and calls `updf` callback at every leaf field that it encounters. The (nested) names are created by joining the json tag with dot. Iteration supports reading another, custom tag `list` with values:
- `tagOmitempty` - omit empty fields (only for read run)
- `tagOmit` - omit field
- `tagReadonly` - field cannot be updated (returns error on `SetValue`)
Examples of usages for tags can be found in `BucketProps` or `Config` structs.
Passing additional options with `IterOpts` can for example call callback also at the non-leaf structures.
func KeepaliveRetryDuration ¶
func LoadConfig ¶
is called at startup
func MakeRangeHdr ¶ added in v1.3.16
Ref: https://www.rfc-editor.org/rfc/rfc7233#section-2.1 (compare w/ htrange.contentRange)
func MaxParallelism ¶ added in v1.3.22
func MaxParallelism() int
func NetworkCallWithRetry ¶
func NetworkIsKnown ¶
func NewClient ¶
func NewClient(cargs TransportArgs) *http.Client
NOTE: `NewTransport` (below) fills-in certain defaults
func NewClientTLS ¶ added in v1.3.21
func NewClientTLS(cargs TransportArgs, sargs TLSArgs) *http.Client
https client (ditto)
func NewDefaultClients ¶ added in v1.3.22
func NewIntraClientTLS ¶ added in v1.3.21
func NewIntraClientTLS(cargs TransportArgs, config *Config) *http.Client
func NewTransport ¶
func NewTransport(cargs TransportArgs) *http.Transport
{TransportArgs + defaults} => http.Transport for a variety of ais clients NOTE: TLS below, and separately
func NormalizeProvider ¶
func ObjHasPrefix ¶ added in v1.3.16
func OrigURLBck2Name ¶
func ParseReadHeaderTimeout ¶ added in v1.3.22
func ParseURL ¶ added in v1.3.18
func ParseURL(path string, itemsPresent []string, itemsAfter int, splitAfter bool) ([]string, error)
ParseURL splits URL path at "/" and matches resulting items against the specified, if any. - splitAfter == true: strings.Split() the entire path; - splitAfter == false: strings.SplitN(len(itemsPresent)+itemsAfter) Returns all items that follow the specified `items`.
func ParseURLScheme ¶
Splits url into [(scheme)://](address). It's not possible to use url.Parse as (from url.Parse() docs) 'Trying to parse a hostname and path without a scheme is invalid'
func PrependProtocol ¶
PrependProtocol prepends protocol in URL in case it is missing. By default it adds `http://` to the URL.
func S2CustomMD ¶ added in v1.3.22
func SaveOverrideConfig ¶
func SaveOverrideConfig(configDir string, toUpdate *ConfigToSet) error
func SortLso ¶
func SortLso(entries LsoEntries)
func ToHeader ¶
may set headers: - standard cos.HdrContentLength ("Content-Length") & cos.HdrETag ("ETag") - atime, version, etc. - all the rest "ais-" prefixed
func TokenGreaterEQ ¶
Returns true if the continuation token >= object's name (in other words, the object is already listed and must be skipped). Note that string `>=` is lexicographic.
func TypeCodeHTTPErr ¶ added in v1.3.19
func UnquoteCEV ¶ added in v1.3.22
unquote checksum, ETag, and version e.g., https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag
func UpdateFieldValue ¶
UpdateFieldValue updates the field in the struct with given value. Returns error if the field was not found or could not be updated.
func ValidateMpath ¶
common mountpath validation (NOTE: calls filepath.Clean() every time)
func ValidatePort ¶
func ValidateRemAlias ¶
func WriteErr405 ¶
func WriteErr405(w http.ResponseWriter, r *http.Request, methods ...string)
405 Method Not Allowed, see: * https://www.rfc-editor.org/rfc/rfc7231#section-6.5.5
func WriteErrJSON ¶
func WriteErrMsg ¶
Create ErrHTTP (based on `msg` and `opts`) and write it into HTTP response.
Types ¶
type AllBsummResults ¶
type AllBsummResults []*BsummResult
func (AllBsummResults) Aggregate ¶
func (s AllBsummResults) Aggregate(from *BsummResult) AllBsummResults
func (AllBsummResults) Finalize ¶
func (s AllBsummResults) Finalize(dsize map[string]uint64, testingEnv bool)
func (AllBsummResults) Len ¶
func (s AllBsummResults) Len() int
func (AllBsummResults) Less ¶
func (s AllBsummResults) Less(i, j int) bool
func (AllBsummResults) Swap ¶
func (s AllBsummResults) Swap(i, j int)
type ArchiveBckMsg ¶ added in v1.3.18
type ArchiveBckMsg struct { ToBck Bck `json:"tobck"` apc.ArchiveMsg }
ArchiveBckMsg contains parameters to archive mutiple objects from the specified (source) bucket. Destination bucket may the same as the source or a different one. -------------------- NOTE on terminology: --------------------- "archive" is any (.tar, .tgz/.tar.gz, .zip, .tar.lz4) formatted object often also called "shard"
See also: apc.PutApndArchArgs
func (*ArchiveBckMsg) Cname ¶ added in v1.3.18
func (msg *ArchiveBckMsg) Cname() string
type AuthConfToSet ¶ added in v1.3.21
type AuthConfToSet struct { Secret *string `json:"secret,omitempty"` Enabled *bool `json:"enabled,omitempty"` }
global configuration
type BackendBckToSet ¶ added in v1.3.21
type BackendConf ¶
type BackendConf struct { // provider implementation-dependent Conf map[string]any `json:"conf,omitempty"` // 3rd party Cloud(s) -- set during validation Providers map[string]Ns `json:"-"` }
global configuration
func (*BackendConf) EqualClouds ¶
func (c *BackendConf) EqualClouds(o *BackendConf) bool
func (*BackendConf) EqualRemAIS ¶
func (c *BackendConf) EqualRemAIS(o *BackendConf, sname string) bool
func (*BackendConf) Get ¶ added in v1.3.16
func (c *BackendConf) Get(provider string) (conf any)
func (*BackendConf) MarshalJSON ¶
func (c *BackendConf) MarshalJSON() (data []byte, err error)
func (*BackendConf) Set ¶ added in v1.3.16
func (c *BackendConf) Set(provider string, newConf any)
func (*BackendConf) UnmarshalJSON ¶
func (c *BackendConf) UnmarshalJSON(data []byte) error
func (*BackendConf) Validate ¶
func (c *BackendConf) Validate() (err error)
type BackendConfAIS ¶
global configuration
func (BackendConfAIS) String ¶
func (c BackendConfAIS) String() (s string)
type Bck ¶
type Bck struct { Props *Bprops `json:"-"` Name string `json:"name" yaml:"name"` Provider string `json:"provider" yaml:"provider"` // NOTE: see api/apc/provider.go for supported enum Ns Ns `json:"namespace" yaml:"namespace" list:"omitempty"` }
func ParseBckObjectURI ¶
func ParseBckObjectURI(uri string, opts ParseURIOpts) (bck Bck, objName string, err error)
func ParseUname ¶
unique name => Bck (use MakeUname above to perform the reverse translation)
func (*Bck) AddUnameToQuery ¶
func (*Bck) DefaultProps ¶
func (bck *Bck) DefaultProps(c *ClusterConfig) *Bprops
By default, created buckets inherit their properties from the cluster (global) configuration. Global configuration, in turn, is protected versioned, checksummed, and replicated across the entire cluster.
* Bucket properties can be changed at any time via `api.SetBprops`. * In addition, `api.CreateBucket` allows to specify (non-default) properties at bucket creation time. * Inherited defaults include checksum, LRU, etc. configurations - see below. * By default, LRU is disabled for AIS (`ais://`) buckets.
See also:
- github.com/NVIDIA/aistore/blob/main/docs/bucket.md#default-bucket-properties
- BpropsToSet (above)
- ais.defaultBckProps()
func (*Bck) HasProvider ¶
func (*Bck) HasVersioningMD ¶ added in v1.3.22
A subset of remote backends that maintain assorted items of versioning information - the items including ETag, checksum, etc. - that, in turn, can be used to populate `ObjAttrs` * see related: `ObjAttrs.Equal`
func (*Bck) IsRemoteAIS ¶
func (*Bck) LenUnameGlob ¶ added in v1.3.21
func (*Bck) ValidateName ¶
type Bprops ¶ added in v1.3.21
type Bprops struct { BackendBck Bck `json:"backend_bck,omitempty"` // makes remote bucket out of a given ais bucket Extra ExtraProps `json:"extra,omitempty" list:"omitempty"` WritePolicy WritePolicyConf `json:"write_policy"` Provider string `json:"provider" list:"readonly"` // backend provider Renamed string `list:"omit"` // non-empty if the bucket has been renamed Cksum CksumConf `json:"checksum"` // the bucket's checksum EC ECConf `json:"ec"` // erasure coding LRU LRUConf `json:"lru"` // LRU (watermarks and enabled/disabled) Mirror MirrorConf `json:"mirror"` // mirroring Access apc.AccessAttrs `json:"access,string"` // access permissions Features feat.Flags `json:"features,string"` // assorted features from feat.Bucket BID uint64 `json:"bid,string" list:"omit"` // unique ID Created int64 `json:"created,string" list:"readonly"` // creation timestamp Versioning VersionConf `json:"versioning"` // versioning (see "inherit") }
func (*Bprops) Apply ¶ added in v1.3.21
func (bp *Bprops) Apply(propsToSet *BpropsToSet)
func (*Bprops) SetProvider ¶ added in v1.3.21
type BpropsToSet ¶ added in v1.3.21
type BpropsToSet struct { BackendBck *BackendBckToSet `json:"backend_bck,omitempty"` Versioning *VersionConfToSet `json:"versioning,omitempty"` Cksum *CksumConfToSet `json:"checksum,omitempty"` LRU *LRUConfToSet `json:"lru,omitempty"` Mirror *MirrorConfToSet `json:"mirror,omitempty"` EC *ECConfToSet `json:"ec,omitempty"` Access *apc.AccessAttrs `json:"access,string,omitempty"` Features *feat.Flags `json:"features,string,omitempty"` WritePolicy *WritePolicyConfToSet `json:"write_policy,omitempty"` Extra *ExtraToSet `json:"extra,omitempty"` Force bool `json:"force,omitempty" copy:"skip" list:"omit"` }
Once validated, BpropsToSet are copied to Bprops. The struct may have extra fields that do not exist in Bprops. Add tag 'copy:"skip"' to ignore those fields when copying values.
func NewBpropsToSet ¶ added in v1.3.21
func NewBpropsToSet(nvs cos.StrKVs) (props *BpropsToSet, err error)
type BsummResult ¶
type BsummResult struct { Bck apc.BsummResult }
type CksumConf ¶
type CksumConf struct { // (note that `ChecksumNone` ("none") disables checksumming) Type string `json:"type"` // validate the checksum of the object that we cold-GET // or download from remote location (e.g., cloud bucket) ValidateColdGet bool `json:"validate_cold_get"` // validate object's version (if exists and provided) and its checksum - // if either value fail to match, the object is removed from ais. // // NOTE: object versioning is backend-specific and is may _not_ be supported by a given // (supported) backends - see docs for details. ValidateWarmGet bool `json:"validate_warm_get"` // determines whether to validate checksums of objects // migrated or replicated within the cluster ValidateObjMove bool `json:"validate_obj_move"` // EnableReadRange: Return read range checksum otherwise return entire object checksum. EnableReadRange bool `json:"enable_read_range"` }
global configuration
func (*CksumConf) ValidateAsProps ¶
type CksumConfToSet ¶ added in v1.3.21
type CksumConfToSet struct { Type *string `json:"type,omitempty"` ValidateColdGet *bool `json:"validate_cold_get,omitempty"` ValidateWarmGet *bool `json:"validate_warm_get,omitempty"` ValidateObjMove *bool `json:"validate_obj_move,omitempty"` EnableReadRange *bool `json:"enable_read_range,omitempty"` }
global configuration
type ClientConf ¶
type ClientConf struct { Timeout cos.Duration `json:"client_timeout"` TimeoutLong cos.Duration `json:"client_long_timeout"` ListObjTimeout cos.Duration `json:"list_timeout"` }
global configuration
func (*ClientConf) Validate ¶
func (c *ClientConf) Validate() error
type ClientConfToSet ¶ added in v1.3.21
type ClientConfToSet struct { Timeout *cos.Duration `json:"client_timeout,omitempty"` // readonly as far as intra-cluster TimeoutLong *cos.Duration `json:"client_long_timeout,omitempty"` ListObjTimeout *cos.Duration `json:"list_timeout,omitempty"` }
global configuration
type ClusterConfig ¶
type ClusterConfig struct { Ext any `json:"ext,omitempty"` // within meta-version extensions Backend BackendConf `json:"backend" allow:"cluster"` Mirror MirrorConf `json:"mirror" allow:"cluster"` EC ECConf `json:"ec" allow:"cluster"` Log LogConf `json:"log"` Periodic PeriodConf `json:"periodic"` Timeout TimeoutConf `json:"timeout"` Client ClientConf `json:"client"` Proxy ProxyConf `json:"proxy" allow:"cluster"` Space SpaceConf `json:"space"` LRU LRUConf `json:"lru"` Disk DiskConf `json:"disk"` Rebalance RebalanceConf `json:"rebalance" allow:"cluster"` Resilver ResilverConf `json:"resilver"` Cksum CksumConf `json:"checksum"` Versioning VersionConf `json:"versioning" allow:"cluster"` Net NetConf `json:"net"` FSHC FSHCConf `json:"fshc"` Auth AuthConf `json:"auth"` Keepalive KeepaliveConf `json:"keepalivetracker"` Downloader DownloaderConf `json:"downloader"` Dsort DsortConf `json:"distributed_sort"` Transport TransportConf `json:"transport"` Memsys MemsysConf `json:"memsys"` // Transform (offline) or Copy src Bucket => dst bucket TCB TCBConf `json:"tcb"` // metadata write policy: (immediate | delayed | never) WritePolicy WritePolicyConf `json:"write_policy"` // standalone enumerated features that can be configured // to flip assorted global defaults (see cmn/feat/feat.go) Features feat.Flags `json:"features,string" allow:"cluster"` // read-only LastUpdated string `json:"lastupdate_time"` // timestamp UUID string `json:"uuid"` // UUID Version int64 `json:"config_version,string"` // version }
global configuration
func (*ClusterConfig) Apply ¶
func (c *ClusterConfig) Apply(updateConf *ConfigToSet, asType string) error
func (*ClusterConfig) JspOpts ¶
func (*ClusterConfig) JspOpts() jsp.Options
func (*ClusterConfig) String ¶
func (c *ClusterConfig) String() string
type Config ¶
type Config struct { ClusterConfig `json:",inline"` LocalConfig `json:",inline"` // contains filtered or unexported fields }
Config is a single control structure (persistent and versioned) that contains both cluster (global) and node (local) configuration Naming convention for setting/getting values: (parent section json tag . child json tag) See also: `IterFields`, `IterFieldNameSepa`
func (*Config) TestingEnv ¶
TestingEnv returns true if config is set to a development environment where a single local filesystem is partitioned between all (locally running) targets and is used for both local and Cloud buckets See also: `rom.testingEnv`
func (*Config) UpdateClusterConfig ¶
func (c *Config) UpdateClusterConfig(updateConf *ConfigToSet, asType string) (err error)
type ConfigToSet ¶ added in v1.3.21
type ConfigToSet struct { // ClusterConfig Backend *BackendConf `json:"backend,omitempty"` Mirror *MirrorConfToSet `json:"mirror,omitempty"` EC *ECConfToSet `json:"ec,omitempty"` Log *LogConfToSet `json:"log,omitempty"` Periodic *PeriodConfToSet `json:"periodic,omitempty"` Timeout *TimeoutConfToSet `json:"timeout,omitempty"` Client *ClientConfToSet `json:"client,omitempty"` Space *SpaceConfToSet `json:"space,omitempty"` LRU *LRUConfToSet `json:"lru,omitempty"` Disk *DiskConfToSet `json:"disk,omitempty"` Rebalance *RebalanceConfToSet `json:"rebalance,omitempty"` Resilver *ResilverConfToSet `json:"resilver,omitempty"` Cksum *CksumConfToSet `json:"checksum,omitempty"` Versioning *VersionConfToSet `json:"versioning,omitempty"` Net *NetConfToSet `json:"net,omitempty"` FSHC *FSHCConfToSet `json:"fshc,omitempty"` Auth *AuthConfToSet `json:"auth,omitempty"` Keepalive *KeepaliveConfToSet `json:"keepalivetracker,omitempty"` Downloader *DownloaderConfToSet `json:"downloader,omitempty"` Dsort *DsortConfToSet `json:"distributed_sort,omitempty"` Transport *TransportConfToSet `json:"transport,omitempty"` Memsys *MemsysConfToSet `json:"memsys,omitempty"` TCB *TCBConfToSet `json:"tcb,omitempty"` WritePolicy *WritePolicyConfToSet `json:"write_policy,omitempty"` Proxy *ProxyConfToSet `json:"proxy,omitempty"` Features *feat.Flags `json:"features,string,omitempty"` // LocalConfig FSP *FSPConf `json:"fspaths,omitempty"` }
global configuration
func (*ConfigToSet) FillFromKVS ¶ added in v1.3.21
func (ctu *ConfigToSet) FillFromKVS(kvs []string) (err error)
FillFromKVS populates `ConfigToSet` from key value pairs of the form `key=value`
func (*ConfigToSet) FillFromQuery ¶ added in v1.3.21
func (ctu *ConfigToSet) FillFromQuery(query url.Values) error
FillFromQuery populates ConfigToSet from URL query values
func (*ConfigToSet) JspOpts ¶ added in v1.3.21
func (*ConfigToSet) JspOpts() jsp.Options
func (*ConfigToSet) Merge ¶ added in v1.3.21
func (ctu *ConfigToSet) Merge(update *ConfigToSet)
type DiskConf ¶
type DiskConf struct { DiskUtilLowWM int64 `json:"disk_util_low_wm"` // no throttling below DiskUtilHighWM int64 `json:"disk_util_high_wm"` // throttle longer when above DiskUtilMaxWM int64 `json:"disk_util_max_wm"` IostatTimeLong cos.Duration `json:"iostat_time_long"` IostatTimeShort cos.Duration `json:"iostat_time_short"` }
global configuration
type DiskConfToSet ¶ added in v1.3.21
type DiskConfToSet struct { DiskUtilLowWM *int64 `json:"disk_util_low_wm,omitempty"` DiskUtilHighWM *int64 `json:"disk_util_high_wm,omitempty"` DiskUtilMaxWM *int64 `json:"disk_util_max_wm,omitempty"` IostatTimeLong *cos.Duration `json:"iostat_time_long,omitempty"` IostatTimeShort *cos.Duration `json:"iostat_time_short,omitempty"` }
global configuration
type DownloaderConf ¶
global configuration
func (*DownloaderConf) Validate ¶
func (c *DownloaderConf) Validate() error
type DownloaderConfToSet ¶ added in v1.3.21
global configuration
type DsortConf ¶ added in v1.3.21
type DsortConf struct { DuplicatedRecords string `json:"duplicated_records"` MissingShards string `json:"missing_shards"` // cmn.SupportedReactions enum EKMMalformedLine string `json:"ekm_malformed_line"` EKMMissingKey string `json:"ekm_missing_key"` DefaultMaxMemUsage string `json:"default_max_mem_usage"` CallTimeout cos.Duration `json:"call_timeout"` DsorterMemThreshold string `json:"dsorter_mem_threshold"` Compression string `json:"compression"` // {CompressAlways,...} in api/apc/compression.go SbundleMult int `json:"bundle_multiplier"` // stream-bundle multiplier: num to destination }
global configuration
func (*DsortConf) ValidateWithOpts ¶ added in v1.3.21
type DsortConfToSet ¶ added in v1.3.21
type DsortConfToSet struct { DuplicatedRecords *string `json:"duplicated_records,omitempty"` MissingShards *string `json:"missing_shards,omitempty"` EKMMalformedLine *string `json:"ekm_malformed_line,omitempty"` EKMMissingKey *string `json:"ekm_missing_key,omitempty"` DefaultMaxMemUsage *string `json:"default_max_mem_usage,omitempty"` CallTimeout *cos.Duration `json:"call_timeout,omitempty"` DsorterMemThreshold *string `json:"dsorter_mem_threshold,omitempty"` Compression *string `json:"compression,omitempty"` SbundleMult *int `json:"bundle_multiplier,omitempty"` }
global configuration
type ECConf ¶
type ECConf struct { Compression string `json:"compression"` // enum { CompressAlways, ... } in api/apc/compression.go // ObjSizeLimit is object size threshold _separating_ intra-cluster mirroring from // erasure coding. // // The value 0 (zero) indicates that objects of any size // are to be sliced, to produce (D) data slices and (P) erasure coded parity slices. // On the other hand, the value -1 specifies that absolutely all objects of any size // must be replicated as is. In effect, the (-1) option provides data protection via // intra-cluster (P+1)-way replication (a.k.a. mirroring). // // In all cases, a given (D, P) configuration provides node-level redundancy, // whereby P nodes can be lost without incurring loss of data. ObjSizeLimit int64 `json:"objsize_limit"` // Number of data (D) slices; the value 1 will have an effect of producing // (P) additional full-size replicas. DataSlices int `json:"data_slices"` // Depending on the object size and `ObjSizeLimit`, the value of `ParitySlices` (or P) indicates: // - a number of additional parity slices (generated or _computed_ from the (D) data slices), // or: // - a number of full object replicas (copies). // In all cases, the same rule applies: all slices and/or all full copies are stored on different // storage nodes (a.k.a. targets). ParitySlices int `json:"parity_slices"` SbundleMult int `json:"bundle_multiplier"` // stream-bundle multiplier: num streams to destination Enabled bool `json:"enabled"` // EC is enabled DiskOnly bool `json:"disk_only"` // if true, EC does not use SGL - data goes directly to drives }
global configuration
func (*ECConf) RequiredRestoreTargets ¶
func (*ECConf) ValidateAsProps ¶
type ECConfToSet ¶ added in v1.3.21
type ECConfToSet struct { ObjSizeLimit *int64 `json:"objsize_limit,omitempty"` Compression *string `json:"compression,omitempty"` SbundleMult *int `json:"bundle_multiplier,omitempty"` DataSlices *int `json:"data_slices,omitempty"` ParitySlices *int `json:"parity_slices,omitempty"` Enabled *bool `json:"enabled,omitempty"` DiskOnly *bool `json:"disk_only,omitempty"` }
global configuration
type ErrAborted ¶
type ErrAborted struct {
// contains filtered or unexported fields
}
assorted aistore errors
func AsErrAborted ¶
func AsErrAborted(err error) (errAborted *ErrAborted)
func NewErrAborted ¶
func NewErrAborted(what, ctx string, err error) *ErrAborted
func (*ErrAborted) Error ¶
func (e *ErrAborted) Error() (s string)
func (*ErrAborted) Unwrap ¶
func (e *ErrAborted) Unwrap() (err error)
type ErrBckNotFound ¶
type ErrBckNotFound struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrBckNotFound ¶
func NewErrBckNotFound(bck *Bck) *ErrBckNotFound
func (*ErrBckNotFound) Error ¶
func (e *ErrBckNotFound) Error() string
type ErrBucketAccessDenied ¶
type ErrBucketAccessDenied struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewBucketAccessDenied ¶
func NewBucketAccessDenied(bucket, oper string, aattrs apc.AccessAttrs) *ErrBucketAccessDenied
func (*ErrBucketAccessDenied) Error ¶
func (e *ErrBucketAccessDenied) Error() string
type ErrBucketAlreadyExists ¶
type ErrBucketAlreadyExists struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrBckAlreadyExists ¶
func NewErrBckAlreadyExists(bck *Bck) *ErrBucketAlreadyExists
func (*ErrBucketAlreadyExists) Error ¶
func (e *ErrBucketAlreadyExists) Error() string
type ErrBusy ¶ added in v1.3.21
type ErrBusy struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrBusy ¶ added in v1.3.21
type ErrCapExceeded ¶ added in v1.3.19
type ErrCapExceeded struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrCapExceeded ¶ added in v1.3.19
func NewErrCapExceeded(totalBytesUsed, totalBytes uint64, highWM, cleanupWM int64, usedPct int32, oos bool) *ErrCapExceeded
func (*ErrCapExceeded) Error ¶ added in v1.3.19
func (e *ErrCapExceeded) Error() string
type ErrFailedTo ¶
type ErrFailedTo struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrFailedTo ¶
func (*ErrFailedTo) Error ¶
func (e *ErrFailedTo) Error() string
func (*ErrFailedTo) Unwrap ¶
func (e *ErrFailedTo) Unwrap() (err error)
type ErrHTTP ¶
type ErrHTTP struct { TypeCode string `json:"tcode,omitempty"` Message string `json:"message"` Method string `json:"method"` URLPath string `json:"url_path"` RemoteAddr string `json:"remote_addr"` Caller string `json:"caller"` Node string `json:"node"` Status int `json:"status"` // contains filtered or unexported fields }
API error structure is returned to aistore client and carries one of the specific errors enumerated below
func Err2HTTPErr ¶
func InitErrHTTP ¶
uses `allocHterr` to allocate - caller must free via `FreeHterr`
func Str2HTTPErr ¶ added in v1.3.18
type ErrInitBackend ¶
type ErrInitBackend struct {
Provider string
}
assorted aistore errors
func (*ErrInitBackend) Error ¶
func (e *ErrInitBackend) Error() string
type ErrInvalidBackendProvider ¶
type ErrInvalidBackendProvider struct {
// contains filtered or unexported fields
}
assorted aistore errors
func (*ErrInvalidBackendProvider) Error ¶
func (e *ErrInvalidBackendProvider) Error() string
func (*ErrInvalidBackendProvider) Is ¶
func (*ErrInvalidBackendProvider) Is(target error) bool
type ErrInvalidCksum ¶
type ErrInvalidCksum struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrInvalidCksum ¶
func NewErrInvalidCksum(eHash, aHash string) *ErrInvalidCksum
func (*ErrInvalidCksum) Error ¶
func (e *ErrInvalidCksum) Error() string
func (*ErrInvalidCksum) Expected ¶
func (e *ErrInvalidCksum) Expected() string
type ErrInvalidFSPathsConf ¶
type ErrInvalidFSPathsConf struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrInvalidFSPathsConf ¶
func NewErrInvalidFSPathsConf(err error) *ErrInvalidFSPathsConf
func (*ErrInvalidFSPathsConf) Error ¶
func (e *ErrInvalidFSPathsConf) Error() string
func (*ErrInvalidFSPathsConf) Unwrap ¶
func (e *ErrInvalidFSPathsConf) Unwrap() (err error)
type ErrInvalidMountpath ¶
type ErrInvalidMountpath struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrInvalidaMountpath ¶
func NewErrInvalidaMountpath(mpath, cause string) *ErrInvalidMountpath
func (*ErrInvalidMountpath) Error ¶
func (e *ErrInvalidMountpath) Error() string
type ErrInvalidObjName ¶ added in v1.3.22
type ErrInvalidObjName struct {
// contains filtered or unexported fields
}
assorted aistore errors
func ValidateObjName ¶ added in v1.3.22
func ValidateObjName(name string) (err *ErrInvalidObjName)
func (*ErrInvalidObjName) Error ¶ added in v1.3.22
func (e *ErrInvalidObjName) Error() string
type ErrLimitedCoexistence ¶
type ErrLimitedCoexistence struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrLimitedCoexistence ¶
func NewErrLimitedCoexistence(node, xaction, action, detail string) *ErrLimitedCoexistence
func (*ErrLimitedCoexistence) Error ¶
func (e *ErrLimitedCoexistence) Error() string
type ErrLmetaCorrupted ¶
type ErrLmetaCorrupted struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrLmetaCorrupted ¶
func NewErrLmetaCorrupted(err error) *ErrLmetaCorrupted
func (*ErrLmetaCorrupted) Error ¶
func (e *ErrLmetaCorrupted) Error() string
func (*ErrLmetaCorrupted) Unwrap ¶
func (e *ErrLmetaCorrupted) Unwrap() (err error)
type ErrLmetaNotFound ¶
type ErrLmetaNotFound struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrLmetaNotFound ¶
func NewErrLmetaNotFound(err error) *ErrLmetaNotFound
func (*ErrLmetaNotFound) Error ¶
func (e *ErrLmetaNotFound) Error() string
func (*ErrLmetaNotFound) Unwrap ¶
func (e *ErrLmetaNotFound) Unwrap() (err error)
type ErrMissingBackend ¶
assorted aistore errors
func (*ErrMissingBackend) Error ¶
func (e *ErrMissingBackend) Error() string
type ErrMountpathNotFound ¶
type ErrMountpathNotFound struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrMountpathNotFound ¶
func NewErrMountpathNotFound(mpath, fqn string, disabled bool) *ErrMountpathNotFound
func (*ErrMountpathNotFound) Error ¶
func (e *ErrMountpathNotFound) Error() string
type ErrNoNodes ¶
type ErrNoNodes struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrNoNodes ¶
func NewErrNoNodes(role string, mmcount int) *ErrNoNodes
func (*ErrNoNodes) Error ¶
func (e *ErrNoNodes) Error() (s string)
type ErrNotImpl ¶
type ErrNotImpl struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrNotImpl ¶
func NewErrNotImpl(action, what string) *ErrNotImpl
func (*ErrNotImpl) Error ¶
func (e *ErrNotImpl) Error() string
type ErrNotRemoteBck ¶ added in v1.3.22
type ErrNotRemoteBck struct {
// contains filtered or unexported fields
}
assorted aistore errors
func ValidateRemoteBck ¶ added in v1.3.22
func ValidateRemoteBck(act string, bck *Bck) (err *ErrNotRemoteBck)
func (*ErrNotRemoteBck) Error ¶ added in v1.3.22
func (e *ErrNotRemoteBck) Error() string
type ErrObjDefunct ¶
type ErrObjDefunct struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrObjDefunct ¶
func NewErrObjDefunct(name string, d1, d2 uint64) *ErrObjDefunct
func (*ErrObjDefunct) Error ¶
func (e *ErrObjDefunct) Error() string
type ErrObjectAccessDenied ¶
type ErrObjectAccessDenied struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewObjectAccessDenied ¶
func NewObjectAccessDenied(object, oper string, aattrs apc.AccessAttrs) *ErrObjectAccessDenied
func (*ErrObjectAccessDenied) Error ¶
func (e *ErrObjectAccessDenied) Error() string
type ErrRangeNotSatisfiable ¶ added in v1.3.22
type ErrRangeNotSatisfiable struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrRangeNotSatisfiable ¶ added in v1.3.22
func NewErrRangeNotSatisfiable(err error, ranges []string, size int64) *ErrRangeNotSatisfiable
func (*ErrRangeNotSatisfiable) Error ¶ added in v1.3.22
func (e *ErrRangeNotSatisfiable) Error() string
type ErrRemoteBckNotFound ¶
type ErrRemoteBckNotFound struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrRemoteBckNotFound ¶
func NewErrRemoteBckNotFound(bck *Bck) *ErrRemoteBckNotFound
func (*ErrRemoteBckNotFound) Error ¶
func (e *ErrRemoteBckNotFound) Error() string
type ErrRemoteBucketOffline ¶
type ErrRemoteBucketOffline struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrRemoteBckOffline ¶
func NewErrRemoteBckOffline(bck *Bck) *ErrRemoteBucketOffline
func (*ErrRemoteBucketOffline) Error ¶
func (e *ErrRemoteBucketOffline) Error() string
type ErrSoft ¶
type ErrSoft struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrSoft ¶
type ErrStreamTerminated ¶ added in v1.3.16
type ErrStreamTerminated struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrStreamTerminated ¶ added in v1.3.16
func NewErrStreamTerminated(stream string, err error, reason, detail string) *ErrStreamTerminated
func (*ErrStreamTerminated) Error ¶ added in v1.3.16
func (e *ErrStreamTerminated) Error() string
func (*ErrStreamTerminated) Unwrap ¶ added in v1.3.16
func (e *ErrStreamTerminated) Unwrap() (err error)
type ErrUnsupp ¶
type ErrUnsupp struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrUnsupp ¶
type ErrXactNotFound ¶ added in v1.3.16
type ErrXactNotFound struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrXactNotFoundError ¶
func NewErrXactNotFoundError(cause string) *ErrXactNotFound
func (*ErrXactNotFound) Error ¶ added in v1.3.16
func (e *ErrXactNotFound) Error() string
type ErrXactTgtInMaint ¶ added in v1.3.18
type ErrXactTgtInMaint struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrXactTgtInMaint ¶ added in v1.3.18
func NewErrXactTgtInMaint(xaction, tname string) *ErrXactTgtInMaint
func (*ErrXactTgtInMaint) Error ¶ added in v1.3.18
func (e *ErrXactTgtInMaint) Error() string
type ErrXactUsePrev ¶ added in v1.3.18
type ErrXactUsePrev struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrXactUsePrev ¶ added in v1.3.18
func NewErrXactUsePrev(xaction string) *ErrXactUsePrev
func (*ErrXactUsePrev) Error ¶ added in v1.3.18
func (e *ErrXactUsePrev) Error() string
type ExtraProps ¶
type ExtraProps struct { AWS ExtraPropsAWS `json:"aws,omitempty" list:"omitempty"` HTTP ExtraPropsHTTP `json:"http,omitempty" list:"omitempty"` HDFS ExtraPropsHDFS `json:"hdfs,omitempty" list:"omitempty"` // NOTE: obsolete; rm with meta-version }
func (*ExtraProps) ValidateAsProps ¶
func (c *ExtraProps) ValidateAsProps(arg ...any) error
type ExtraPropsAWS ¶
type ExtraPropsAWS struct { CloudRegion string `json:"cloud_region,omitempty"` // from https://github.com/aws/aws-sdk-go/blob/main/aws/config.go: // - "An optional endpoint URL (hostname only or fully qualified URI) // that overrides the default generated endpoint." Endpoint string `json:"endpoint,omitempty"` // from https://github.com/aws/aws-sdk-go/blob/main/aws/session/session.go: // - "Overrides the config profile the Session should be created from. If not // set the value of the environment variable will be loaded (AWS_PROFILE, // or AWS_DEFAULT_PROFILE if the Shared Config is enabled)." Profile string `json:"profile,omitempty"` // Amazon S3: 1000 // - https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-pagination.html#cli-usage-pagination-serverside // vs OpenStack Swift: 10,000 // - https://docs.openstack.org/swift/latest/api/pagination.html MaxPageSize int64 `json:"max_pagesize,omitempty"` }
type ExtraPropsAWSToSet ¶ added in v1.3.21
type ExtraPropsHDFS ¶
type ExtraPropsHDFS struct { // Reference directory. RefDirectory string `json:"ref_directory,omitempty"` }
type ExtraPropsHDFSToSet ¶ added in v1.3.21
type ExtraPropsHDFSToSet struct {
RefDirectory *string `json:"ref_directory"`
}
type ExtraPropsHTTP ¶
type ExtraPropsHTTP struct { // Original URL prior to hashing. OrigURLBck string `json:"original_url,omitempty" list:"readonly"` }
type ExtraPropsHTTPToSet ¶ added in v1.3.21
type ExtraPropsHTTPToSet struct {
OrigURLBck *string `json:"original_url"`
}
type ExtraToSet ¶ added in v1.3.21
type ExtraToSet struct { AWS *ExtraPropsAWSToSet `json:"aws"` HTTP *ExtraPropsHTTPToSet `json:"http"` HDFS *ExtraPropsHDFSToSet `json:"hdfs"` // ditto }
type FSHCConf ¶
type FSHCConf struct { TestFileCount int `json:"test_files"` // number of files to read/write ErrorLimit int `json:"error_limit"` // exceeding err limit causes disabling mountpath Enabled bool `json:"enabled"` }
global configuration
type FSHCConfToSet ¶ added in v1.3.21
type FSHCConfToSet struct { TestFileCount *int `json:"test_files,omitempty"` ErrorLimit *int `json:"error_limit,omitempty"` Enabled *bool `json:"enabled,omitempty"` }
global configuration
type FSPConf ¶
ais node: fspaths (a.k.a. mountpaths) and their respective (optional) labels
func (*FSPConf) MarshalJSON ¶
func (*FSPConf) UnmarshalJSON ¶
type FSPConfV322 ¶ added in v1.3.23
[backward compatibility]: v3.22 and prior
func (*FSPConfV322) MarshalJSON ¶ added in v1.3.23
func (c *FSPConfV322) MarshalJSON() ([]byte, error)
[backward compatibility]: used to generate fspath config for older ais versions
type HTTPBckObj ¶
type HTTPBckObj struct { Bck Bck ObjName string OrigURLBck string // HTTP URL of the bucket (object name excluded) }
Represents the AIS bucket, object and URL associated with a HTTP resource
func NewHTTPObj ¶
func NewHTTPObj(u *url.URL) *HTTPBckObj
func NewHTTPObjPath ¶
func NewHTTPObjPath(rawURL string) (*HTTPBckObj, error)
type HTTPConf ¶
type HTTPConf struct { Proto string `json:"-"` // http or https (set depending on `UseHTTPS`) Certificate string `json:"server_crt"` // HTTPS: X509 certificate CertKey string `json:"server_key"` // HTTPS: X509 key ServerNameTLS string `json:"domain_tls"` // #6410 ClientCA string `json:"client_ca_tls"` // #6410 ClientAuthTLS int `json:"client_auth_tls"` // #6410 tls.ClientAuthType enum WriteBufferSize int `json:"write_buffer_size"` // http.Transport.WriteBufferSize; zero defaults to 4KB ReadBufferSize int `json:"read_buffer_size"` // http.Transport.ReadBufferSize; ditto UseHTTPS bool `json:"use_https"` // use HTTPS SkipVerifyCrt bool `json:"skip_verify"` // skip X509 cert verification (used with self-signed certs) Chunked bool `json:"chunked_transfer"` // (https://tools.ietf.org/html/rfc7230#page-36; not used since 02/23) }
global configuration
type HTTPConfToSet ¶ added in v1.3.21
type HTTPConfToSet struct { Certificate *string `json:"server_crt,omitempty"` CertKey *string `json:"server_key,omitempty"` ServerNameTLS *string `json:"domain_tls,omitempty"` ClientCA *string `json:"client_ca_tls,omitempty"` WriteBufferSize *int `json:"write_buffer_size,omitempty" list:"readonly"` ReadBufferSize *int `json:"read_buffer_size,omitempty" list:"readonly"` ClientAuthTLS *int `json:"client_auth_tls,omitempty"` UseHTTPS *bool `json:"use_https,omitempty"` SkipVerifyCrt *bool `json:"skip_verify,omitempty"` Chunked *bool `json:"chunked_transfer,omitempty"` }
global configuration
type HreqArgs ¶
type HreqArgs struct { BodyR io.Reader Header http.Header // request headers Query url.Values // query, e.g. ?a=x&b=y&c=z RawQuery string // raw query Method string Base string // base URL, e.g. http://xyz.abc Path string // path URL, e.g. /x/y/z Body []byte }
usage 1: initialize and fill out HTTP request. usage 2: intra-cluster control-plane (except streams) usage 3: PUT and APPEND API BodyR optimizes-out allocations - if non-nil and implements `io.Closer`, will always be closed by `client.Do`
func (*HreqArgs) ReqWithCancel ¶
ReqWithCancel creates request with ability to cancel it.
func (*HreqArgs) ReqWithTimeout ¶
type IterField ¶
type IterField interface { Value() any // returns the value String() string // string representation of the value SetValue(v any, force ...bool) error // `force` ignores `tagReadonly` (to be used with caution!) }
Represents a single named field
type IterOpts ¶
type IterOpts struct { // Skip fields based on allowed tag Allowed string // Visits all the fields, not only the leaves. VisitAll bool // Read-only walk is true by default (compare with `UpdateFieldValue`) // Note that `tagOmitempty` is limited to read-only - has no effect when `OnlyRead == false`. OnlyRead bool }
type KeepaliveConf ¶
type KeepaliveConf struct { Proxy KeepaliveTrackerConf `json:"proxy"` // how proxy tracks target keepalives Target KeepaliveTrackerConf `json:"target"` // how target tracks primary proxies keepalives RetryFactor uint8 `json:"retry_factor"` }
global configuration
func (*KeepaliveConf) Validate ¶
func (c *KeepaliveConf) Validate() (err error)
type KeepaliveConfToSet ¶ added in v1.3.21
type KeepaliveConfToSet struct { Proxy *KeepaliveTrackerConfToSet `json:"proxy,omitempty"` Target *KeepaliveTrackerConfToSet `json:"target,omitempty"` RetryFactor *uint8 `json:"retry_factor,omitempty"` }
global configuration
type KeepaliveTrackerConf ¶
type KeepaliveTrackerConf struct { Name string `json:"name"` // "heartbeat" (other enumerated values TBD) Interval cos.Duration `json:"interval"` // keepalive interval Factor uint8 `json:"factor"` // only average }
keepalive tracker
type KeepaliveTrackerConfToSet ¶ added in v1.3.21
type KeepaliveTrackerConfToSet struct { Interval *cos.Duration `json:"interval,omitempty"` Name *string `json:"name,omitempty" list:"readonly"` Factor *uint8 `json:"factor,omitempty"` }
global configuration
type L4Conf ¶
type L4Conf struct { Proto string `json:"proto"` // tcp, udp SndRcvBufSize int `json:"sndrcv_buf_size"` // SO_RCVBUF and SO_SNDBUF }
global configuration
type LRUConf ¶
type LRUConf struct { // DontEvictTimeStr denotes the period of time during which eviction of an object // is forbidden [atime, atime + DontEvictTime] DontEvictTime cos.Duration `json:"dont_evict_time"` // CapacityUpdTimeStr denotes the frequency at which AIStore updates local capacity utilization CapacityUpdTime cos.Duration `json:"capacity_upd_time"` // Enabled: LRU will only run when set to true Enabled bool `json:"enabled"` }
global configuration
type LRUConfToSet ¶ added in v1.3.21
type LRUConfToSet struct { DontEvictTime *cos.Duration `json:"dont_evict_time,omitempty"` CapacityUpdTime *cos.Duration `json:"capacity_upd_time,omitempty"` Enabled *bool `json:"enabled,omitempty"` }
global configuration
type LocalConfig ¶
type LocalConfig struct { ConfigDir string `json:"confdir"` LogDir string `json:"log_dir"` HostNet LocalNetConfig `json:"host_net"` FSP FSPConf `json:"fspaths"` TestFSP TestFSPConf `json:"test_fspaths"` }
local config
func (*LocalConfig) AddPath ¶
func (c *LocalConfig) AddPath(mpath string)
func (*LocalConfig) DelPath ¶
func (c *LocalConfig) DelPath(mpath string)
func (*LocalConfig) JspOpts ¶
func (*LocalConfig) JspOpts() jsp.Options
func (*LocalConfig) TestingEnv ¶
func (c *LocalConfig) TestingEnv() bool
type LocalNetConfig ¶
type LocalNetConfig struct { Hostname string `json:"hostname"` HostnameIntraControl string `json:"hostname_intra_control"` HostnameIntraData string `json:"hostname_intra_data"` Port int `json:"port,string"` // listening port PortIntraControl int `json:"port_intra_control,string"` // --/-- for intra-cluster control PortIntraData int `json:"port_intra_data,string"` // --/-- for intra-cluster data // omit UseIntraControl bool `json:"-"` UseIntraData bool `json:"-"` }
ais node: (local) network config
func (*LocalNetConfig) Validate ¶
func (c *LocalNetConfig) Validate(contextConfig *Config) (err error)
type LogConf ¶
type LogConf struct { Level cos.LogLevel `json:"level"` // log level (aka verbosity) MaxSize cos.SizeIEC `json:"max_size"` // exceeding this size triggers log rotation MaxTotal cos.SizeIEC `json:"max_total"` // (sum individual log sizes); exceeding this number triggers cleanup FlushTime cos.Duration `json:"flush_time"` // log flush interval StatsTime cos.Duration `json:"stats_time"` // (not used) ToStderr bool `json:"to_stderr"` // Log only to stderr instead of files. }
global configuration
type LogConfToSet ¶ added in v1.3.21
type LogConfToSet struct { Level *cos.LogLevel `json:"level,omitempty"` ToStderr *bool `json:"to_stderr,omitempty"` MaxSize *cos.SizeIEC `json:"max_size,omitempty"` MaxTotal *cos.SizeIEC `json:"max_total,omitempty"` FlushTime *cos.Duration `json:"flush_time,omitempty"` StatsTime *cos.Duration `json:"stats_time,omitempty"` }
global configuration
type LsoEnt ¶ added in v1.3.23
type LsoEnt struct { Name string `json:"name" msg:"n"` // object name Checksum string `json:"checksum,omitempty" msg:"cs,omitempty"` // checksum Atime string `json:"atime,omitempty" msg:"a,omitempty"` // last access time; formatted as ListObjsMsg.TimeFormat Version string `json:"version,omitempty" msg:"v,omitempty"` // e.g., GCP int64 generation, AWS version (string), etc. Location string `json:"location,omitempty" msg:"t,omitempty"` // [tnode:mountpath] Custom string `json:"custom-md,omitempty" msg:"m,omitempty"` // custom metadata: ETag, MD5, CRC, user-defined ... Size int64 `json:"size,string,omitempty" msg:"s,omitempty"` // size in bytes Copies int16 `json:"copies,omitempty" msg:"c,omitempty"` // ## copies (NOTE: for non-replicated object copies == 1) Flags uint16 `json:"flags,omitempty" msg:"f,omitempty"` // enum { EntryIsCached, EntryIsDir, EntryInArch, ...} }
a single entry in LsoRes.Entries (below); contains list-objects results for the corresponding (listed) object or archived file; `Flags` is a bit field where `EntryStatusBits` bits [0-4] are reserved for object status (all statuses are mutually exclusive)
func DedupLso ¶ added in v1.3.16
func DedupLso(entries LsoEntries, maxSize int) []*LsoEnt
func HandleNoRecurs ¶ added in v1.3.23
no recursion (LsNoRecursion) helper function: - check the level of nesting - possibly, return virtual directory (to be included in LsoRes) and/or filepath.SkipDir
func (*LsoEnt) CopyWithProps ¶ added in v1.3.23
func (*LsoEnt) IsInsideArch ¶ added in v1.3.23
func (*LsoEnt) IsListedArch ¶ added in v1.3.23
func (*LsoEnt) IsPresent ¶ added in v1.3.23
The terms "cached" and "present" are interchangeable: "object is cached" and "is present" is actually the same thing
func (*LsoEnt) IsStatusOK ¶ added in v1.3.23
func (*LsoEnt) IsVerChanged ¶ added in v1.3.23
func (*LsoEnt) IsVerRemoved ¶ added in v1.3.23
func (*LsoEnt) Msgsize ¶ added in v1.3.23
Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
func (*LsoEnt) SetPresent ¶ added in v1.3.23
func (be *LsoEnt) SetPresent()
func (*LsoEnt) SetVerChanged ¶ added in v1.3.23
func (be *LsoEnt) SetVerChanged()
see also: "latest-ver", QparamLatestVer, et al.
func (*LsoEnt) SetVerRemoved ¶ added in v1.3.23
func (be *LsoEnt) SetVerRemoved()
type LsoEntries ¶
type LsoEntries []*LsoEnt
func (*LsoEntries) DecodeMsg ¶ added in v1.3.22
func (z *LsoEntries) DecodeMsg(dc *msgp.Reader) (err error)
DecodeMsg implements msgp.Decodable
func (LsoEntries) EncodeMsg ¶ added in v1.3.22
func (z LsoEntries) EncodeMsg(en *msgp.Writer) (err error)
EncodeMsg implements msgp.Encodable
func (LsoEntries) Msgsize ¶ added in v1.3.22
func (z LsoEntries) Msgsize() (s int)
Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
type LsoRes ¶ added in v1.3.23
type LsoRes struct { UUID string `json:"uuid"` ContinuationToken string `json:"continuation_token"` Entries LsoEntries `json:"entries"` Flags uint32 `json:"flags"` }
`api.ListObjects` and `Backend.ListObjects` results
func MergeLso ¶
MergeLso merges list-objects results received from targets. For the same object name (ie., the same object) the corresponding properties are merged. If maxSize is greater than 0, the resulting list is sorted and truncated.
type MemsysConf ¶
type MemsysConf struct { MinFree cos.SizeIEC `json:"min_free"` DefaultBufSize cos.SizeIEC `json:"default_buf"` SizeToGC cos.SizeIEC `json:"to_gc"` HousekeepTime cos.Duration `json:"hk_time"` MinPctTotal int `json:"min_pct_total"` MinPctFree int `json:"min_pct_free"` }
global configuration
func (*MemsysConf) Validate ¶
func (c *MemsysConf) Validate() (err error)
type MemsysConfToSet ¶ added in v1.3.21
type MemsysConfToSet struct { MinFree *cos.SizeIEC `json:"min_free,omitempty"` DefaultBufSize *cos.SizeIEC `json:"default_buf,omitempty"` SizeToGC *cos.SizeIEC `json:"to_gc,omitempty"` HousekeepTime *cos.Duration `json:"hk_time,omitempty"` MinPctTotal *int `json:"min_pct_total,omitempty"` MinPctFree *int `json:"min_pct_free,omitempty"` }
global configuration
type MirrorConf ¶
type MirrorConf struct { Copies int64 `json:"copies"` // num copies Burst int `json:"burst_buffer"` // xaction channel (buffer) size Enabled bool `json:"enabled"` // enabled (to generate copies) }
global configuration
func (*MirrorConf) String ¶
func (c *MirrorConf) String() string
func (*MirrorConf) Validate ¶
func (c *MirrorConf) Validate() error
func (*MirrorConf) ValidateAsProps ¶
func (c *MirrorConf) ValidateAsProps(...any) error
type MirrorConfToSet ¶ added in v1.3.21
type MirrorConfToSet struct { Copies *int64 `json:"copies,omitempty"` Burst *int `json:"burst_buffer,omitempty"` Enabled *bool `json:"enabled,omitempty"` }
global configuration
type NetConfToSet ¶ added in v1.3.21
type NetConfToSet struct {
HTTP *HTTPConfToSet `json:"http,omitempty"`
}
global configuration
type Ns ¶
type Ns struct { // UUID of other remote AIS cluster (for now only used for AIS). Note // that we can have different namespaces which refer to same UUID (cluster). // This means that in a sense UUID is a parent of the actual namespace. UUID string `json:"uuid" yaml:"uuid"` // Name uniquely identifies a namespace under the same UUID (which may // be empty) and is used in building FQN for the objects. Name string `json:"name" yaml:"name"` }
Ns (or Namespace) adds additional layer for scoping the data under the same provider. It allows to have same dataset and bucket names under different namespaces what allows for easy data manipulation without affecting data in different namespaces.
func ParseNsUname ¶
Parses [@uuid][#namespace]. It does a little bit more than just parsing a string from `Uname` so that logic can be reused in different places.
func (Ns) IsAnyRemote ¶
type OWT ¶
type OWT int
Object Write Transaction (OWT) controls certain aspects of creating new objects in the cluster. In particular, OwtGet* group below simultaneously specifies cold-GET variations (that all involve reading from a remote backend) and the associated locking (that will always reflect a tradeoff between consistency and parallelism) NOTE: enum entries' order is important
const ( // PUT and PUT-like transactions OwtPut OWT = iota // PUT OwtPromote // promote target-accessible files and directories OwtArchive // multi-obj arch OwtTransform // ETL OwtCopy // copy and move objects within cluster OwtRebalance // NOTE: must be the last in PUT* group // // GET and friends // OwtGetTryLock // if !try-lock(exclusive) { return error }; read from remote; ... OwtGetLock // lock(exclusive); read from remote; ... OwtGet // GET (with upgrading read-lock in the local-write path) OwtGetPrefetchLock // (used for maximum parallelism when prefetching) // // None of the above // OwtNone )
type ObjAttrs ¶
type ObjAttrs struct { Cksum *cos.Cksum `json:"checksum,omitempty"` // object checksum (cloned) CustomMD cos.StrKVs `json:"custom-md,omitempty"` // custom metadata: ETag, MD5, CRC, user-defined ... Ver string `json:"version,omitempty"` // object version Atime int64 `json:"atime,omitempty"` // access time (nanoseconds since UNIX epoch) Size int64 `json:"size,omitempty"` // object size (bytes) }
see also apc.HdrObjAtime et al. @ api/apc/const.go (and note that naming must be consistent)
func (*ObjAttrs) DelCustomKeys ¶
func (*ObjAttrs) Equal ¶
local <=> remote equality in the context of cold-GET and download. This function decides whether we need to go ahead and re-read the object from its remote location.
Other than a "binary" size and version checks, rest logic goes as follows: objects are considered equal if they have a) the same version and at least one matching checksum, or b) the same remote "source" and at least one matching checksum, or c) two matching checksums. (See also note below.)
Note that mismatch in any given checksum type immediately renders inequality and return from the function.
func (*ObjAttrs) FromHeader ¶
NOTE: returning checksum separately for subsequent validation
func (*ObjAttrs) FromLsoEntry ¶ added in v1.3.22
func (*ObjAttrs) GetCustomKey ¶
func (*ObjAttrs) GetCustomMD ¶
func (*ObjAttrs) SetCustomKey ¶
func (*ObjAttrs) SetCustomMD ¶
type ObjectProps ¶
type ObjectProps struct { Bck Bck `json:"bucket"` ObjAttrs Name string `json:"name"` Location string `json:"location"` // see also `GetPropsLocation` Mirror struct { Paths []string `json:"paths,omitempty"` Copies int `json:"copies,omitempty"` } `json:"mirror"` EC struct { Generation int64 `json:"generation"` DataSlices int `json:"data"` ParitySlices int `json:"parity"` IsECCopy bool `json:"replicated"` } `json:"ec"` Present bool `json:"present"` }
object properties NOTE: embeds system `ObjAttrs` that in turn includes custom user-defined NOTE: compare with `apc.LsoMsg`
type ParseURIOpts ¶
type PeriodConf ¶
type PeriodConf struct { StatsTime cos.Duration `json:"stats_time"` // collect and publish stats; other house-keeping RetrySyncTime cos.Duration `json:"retry_sync_time"` // metasync retry NotifTime cos.Duration `json:"notif_time"` // (IC notifications) }
NOTE: StatsTime is a one important timer
func (*PeriodConf) Validate ¶
func (c *PeriodConf) Validate() error
type PeriodConfToSet ¶ added in v1.3.21
type PeriodConfToSet struct { StatsTime *cos.Duration `json:"stats_time,omitempty"` RetrySyncTime *cos.Duration `json:"retry_sync_time,omitempty"` NotifTime *cos.Duration `json:"notif_time,omitempty"` }
global configuration
type PropsValidator ¶
type ProxyConf ¶
type ProxyConf struct { PrimaryURL string `json:"primary_url"` OriginalURL string `json:"original_url"` DiscoveryURL string `json:"discovery_url"` NonElectable bool `json:"non_electable"` }
global configuration
type ProxyConfToSet ¶ added in v1.3.21
type ProxyConfToSet struct { PrimaryURL *string `json:"primary_url,omitempty"` OriginalURL *string `json:"original_url,omitempty"` DiscoveryURL *string `json:"discovery_url,omitempty"` NonElectable *bool `json:"non_electable,omitempty"` }
global configuration
type QueryBcks ¶
type QueryBcks Bck
func (*QueryBcks) AddToQuery ¶
func (*QueryBcks) IsBucket ¶
QueryBcks is a Bck that _can_ have an empty Name. (TODO: extend to support prefix and regex.)
func (*QueryBcks) IsRemoteAIS ¶
type RebalanceConf ¶
type RebalanceConf struct { Compression string `json:"compression"` // enum { CompressAlways, ... } in api/apc/compression.go DestRetryTime cos.Duration `json:"dest_retry_time"` // max wait for ACKs & neighbors to complete SbundleMult int `json:"bundle_multiplier"` // stream-bundle multiplier: num streams to destination Enabled bool `json:"enabled"` // true=auto-rebalance | manual rebalancing }
global configuration
func (*RebalanceConf) String ¶
func (c *RebalanceConf) String() string
func (*RebalanceConf) Validate ¶
func (c *RebalanceConf) Validate() error
type RebalanceConfToSet ¶ added in v1.3.21
type RebalanceConfToSet struct { DestRetryTime *cos.Duration `json:"dest_retry_time,omitempty"` Compression *string `json:"compression,omitempty"` SbundleMult *int `json:"bundle_multiplier"` Enabled *bool `json:"enabled,omitempty"` }
global configuration
type ResilverConf ¶
type ResilverConf struct {
Enabled bool `json:"enabled"` // true=auto-resilver | manual resilvering
}
global configuration
func (*ResilverConf) String ¶
func (c *ResilverConf) String() string
func (*ResilverConf) Validate ¶
func (*ResilverConf) Validate() error
type ResilverConfToSet ¶ added in v1.3.21
type ResilverConfToSet struct {
Enabled *bool `json:"enabled,omitempty"`
}
global configuration
type RetryArgs ¶
type RetryArgs struct { Call func() (int, error) IsFatal func(error) bool Action string Caller string SoftErr uint // How many retires on ConnectionRefused or ConnectionReset error. HardErr uint // How many retries on any other error. Sleep time.Duration Verbosity int // Determine the verbosity level. BackOff bool // If requests should be retried less and less often. IsClient bool // true: client (e.g. dev tools, etc.) }
type SpaceConf ¶
type SpaceConf struct { // Storage Cleanup watermark: used capacity (%) that triggers cleanup // (deleted objects and buckets, extra copies, etc.) CleanupWM int64 `json:"cleanupwm"` // LowWM: used capacity low-watermark (% of total local storage capacity) LowWM int64 `json:"lowwm"` // HighWM: used capacity high-watermark (% of total local storage capacity) // - LRU starts evicting objects when the currently used capacity (used-cap) gets above HighWM // - and keeps evicting objects until the used-cap gets below LowWM // - while self-throttling itself in accordance with target utilization HighWM int64 `json:"highwm"` // Out-of-Space: if exceeded, the target starts failing new PUTs and keeps // failing them until its local used-cap gets back below HighWM (see above) OOS int64 `json:"out_of_space"` }
global configuration
func (*SpaceConf) ValidateAsProps ¶
type SpaceConfToSet ¶ added in v1.3.21
type SpaceConfToSet struct { CleanupWM *int64 `json:"cleanupwm,omitempty"` LowWM *int64 `json:"lowwm,omitempty"` HighWM *int64 `json:"highwm,omitempty"` OOS *int64 `json:"out_of_space,omitempty"` }
global configuration
type TCBConf ¶
type TCBConf struct { Compression string `json:"compression"` // enum { CompressAlways, ... } in api/apc/compression.go SbundleMult int `json:"bundle_multiplier"` // stream-bundle multiplier: num streams to destination }
global configuration
type TCBConfToSet ¶ added in v1.3.21
type TCBConfToSet struct { Compression *string `json:"compression,omitempty"` SbundleMult *int `json:"bundle_multiplier,omitempty"` }
global configuration
type TestFSPConf ¶
type TestFSPConf struct { Root string `json:"root"` Count int `json:"count"` Instance int `json:"instance"` }
local config
func (*TestFSPConf) Validate ¶
func (c *TestFSPConf) Validate(contextConfig *Config) (err error)
validate root and (NOTE: testing only) generate and fill-in counted FSP.Paths
func (*TestFSPConf) ValidateMpath ¶
func (c *TestFSPConf) ValidateMpath(p string) (err error)
type TimeoutConf ¶
type TimeoutConf struct { CplaneOperation cos.Duration `json:"cplane_operation"` // read-mostly via global cmn.Rom.CplaneOperation MaxKeepalive cos.Duration `json:"max_keepalive"` // ditto, cmn.Rom.MaxKeepalive - see below MaxHostBusy cos.Duration `json:"max_host_busy"` Startup cos.Duration `json:"startup_time"` JoinAtStartup cos.Duration `json:"join_startup_time"` // (join cluster at startup) timeout SendFile cos.Duration `json:"send_file_time"` }
maximum intra-cluster latencies (in the increasing order)
func (*TimeoutConf) Validate ¶
func (c *TimeoutConf) Validate() error
type TimeoutConfToSet ¶ added in v1.3.21
type TimeoutConfToSet struct { CplaneOperation *cos.Duration `json:"cplane_operation,omitempty"` MaxKeepalive *cos.Duration `json:"max_keepalive,omitempty"` MaxHostBusy *cos.Duration `json:"max_host_busy,omitempty"` Startup *cos.Duration `json:"startup_time,omitempty"` JoinAtStartup *cos.Duration `json:"join_startup_time,omitempty"` SendFile *cos.Duration `json:"send_file_time,omitempty"` }
global configuration
type TransportArgs ¶
type TransportArgs struct { DialTimeout time.Duration Timeout time.Duration IdleConnTimeout time.Duration IdleConnsPerHost int MaxIdleConns int SndRcvBufSize int WriteBufferSize int ReadBufferSize int UseHTTPProxyEnv bool }
assorted http(s) client options
func (*TransportArgs) ConnControl ¶
func (args *TransportArgs) ConnControl(_ syscall.RawConn) (cntl func(fd uintptr))
type TransportConf ¶
type TransportConf struct { MaxHeaderSize int `json:"max_header"` // max transport header buffer (default=4K) Burst int `json:"burst_buffer"` // num sends with no back pressure; see also AIS_STREAM_BURST_NUM // two no-new-transmissions durations: // * IdleTeardown: sender terminates the connection (to reestablish it upon the very first/next PDU) // * QuiesceTime: safe to terminate or transition to the next (in re: rebalance) stage IdleTeardown cos.Duration `json:"idle_teardown"` QuiesceTime cos.Duration `json:"quiescent"` // lz4 // max uncompressed block size, one of [64K, 256K(*), 1M, 4M] // fastcompression.blogspot.com/2013/04/lz4-streaming-format-final.html LZ4BlockMaxSize cos.SizeIEC `json:"lz4_block"` LZ4FrameChecksum bool `json:"lz4_frame_checksum"` }
global configuration
func (*TransportConf) Validate ¶
func (c *TransportConf) Validate() (err error)
NOTE: uncompressed block sizes - the enum currently supported by the github.com/pierrec/lz4
type TransportConfToSet ¶ added in v1.3.21
type TransportConfToSet struct { MaxHeaderSize *int `json:"max_header,omitempty" list:"readonly"` Burst *int `json:"burst_buffer,omitempty" list:"readonly"` IdleTeardown *cos.Duration `json:"idle_teardown,omitempty"` QuiesceTime *cos.Duration `json:"quiescent,omitempty"` LZ4BlockMaxSize *cos.SizeIEC `json:"lz4_block,omitempty"` LZ4FrameChecksum *bool `json:"lz4_frame_checksum,omitempty"` }
global configuration
type VersionConf ¶
type VersionConf struct { // Determines if versioning is enabled Enabled bool `json:"enabled"` // Validate remote version and, possibly, update in-cluster ("cached") copy. // Scenarios include (but are not limited to): // - warm GET // - prefetch bucket (*) // - prefetch multiple objects (see api/multiobj.go) // - copy bucket // Applies to Cloud and remote AIS buckets - generally, buckets that have remote backends // that in turn provide some form of object versioning. // (*) Xactions (ie., jobs) that read-access multiple objects (e.g., prefetch, copy-bucket) // may support operation-scope option to synchronize remote content (to aistore) - the option // not requiring changing bucket configuration. // See also: // - apc.QparamLatestVer, apc.PrefetchMsg, apc.CopyBckMsg ValidateWarmGet bool `json:"validate_warm_get"` // A stronger variant of the above that in addition entails: // - deleting in-cluster object if its remote ("cached") counterpart does not exist // See also: apc.QparamSync, apc.CopyBckMsg Sync bool `json:"synchronize"` }
global configuration
func (*VersionConf) String ¶
func (c *VersionConf) String() string
func (*VersionConf) Validate ¶
func (c *VersionConf) Validate() error
type VersionConfToSet ¶ added in v1.3.21
type VersionConfToSet struct { Enabled *bool `json:"enabled,omitempty"` ValidateWarmGet *bool `json:"validate_warm_get,omitempty"` Sync *bool `json:"synchronize,omitempty"` }
global configuration
type WritePolicyConf ¶
type WritePolicyConf struct { Data apc.WritePolicy `json:"data"` MD apc.WritePolicy `json:"md"` }
global configuration
func (*WritePolicyConf) Validate ¶
func (c *WritePolicyConf) Validate() (err error)
func (*WritePolicyConf) ValidateAsProps ¶
func (c *WritePolicyConf) ValidateAsProps(...any) error
type WritePolicyConfToSet ¶ added in v1.3.21
type WritePolicyConfToSet struct { Data *apc.WritePolicy `json:"data,omitempty" list:"readonly"` // NOTE: NIY MD *apc.WritePolicy `json:"md,omitempty"` }
global configuration
Source Files ¶
Directories ¶
Path | Synopsis |
---|---|
Package archive: write, read, copy, append, list primitives across all supported formats
|
Package archive: write, read, copy, append, list primitives across all supported formats |
Package atomic provides simple wrappers around numerics to enforce atomic access.
|
Package atomic provides simple wrappers around numerics to enforce atomic access. |
Package cos provides common low-level types and utilities for all aistore projects.
|
Package cos provides common low-level types and utilities for all aistore projects. |
Package provides debug utilities
|
Package provides debug utilities |
Package feat: global runtime-configurable feature flags
|
Package feat: global runtime-configurable feature flags |
Package fname contains filename constants and common system directories
|
Package fname contains filename constants and common system directories |
Package jsp (JSON persistence) provides utilities to store and load arbitrary JSON-encoded structures with optional checksumming and compression.
|
Package jsp (JSON persistence) provides utilities to store and load arbitrary JSON-encoded structures with optional checksumming and compression. |
Package k8s: initialization, client, and misc.
|
Package k8s: initialization, client, and misc. |
Package kvdb provides a local key/value database server for AIS.
|
Package kvdb provides a local key/value database server for AIS. |
Package mono provides low-level monotonic time
|
Package mono provides low-level monotonic time |
Package nlog - aistore logger, provides buffering, timestamping, writing, and flushing/syncing/rotating
|
Package nlog - aistore logger, provides buffering, timestamping, writing, and flushing/syncing/rotating |
Package prob implements fully features dynamic probabilistic filter.
|
Package prob implements fully features dynamic probabilistic filter. |
Package xoshiro256 implements the xoshiro256** RNG no-copyright
|
Package xoshiro256 implements the xoshiro256** RNG no-copyright |