api

package
v1.3.19 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 6, 2023 License: MIT Imports: 29 Imported by: 0

Documentation

Overview

Package api provides AIStore API over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides AIStore API over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides AIStore API over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides AIStore API over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides AIStore API over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides AIStore API over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides AIStore API over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides AIStore API over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides AIStore API over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides AIStore API over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides AIStore API over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides AIStore API over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides AIStore API over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AbortDSort 

func AbortDSort(bp BaseParams, managerUUID string) error

func AbortDownload 

func AbortDownload(bp BaseParams, id string) error

func AbortXaction 

func AbortXaction(bp BaseParams, args xact.ArgsMsg) (err error)

Abort ("stop") xactions

func AccessAttrs 

func AccessAttrs(v apc.AccessAttrs) *apc.AccessAttrs

AccessAttrs returns a pointer to the AccessAttr value passed in.

func AppendObject 

func AppendObject(args AppendArgs) (string, error)

AppendObject adds a reader (`args.Reader` - e.g., an open file) to an object. The API can be called multiple times - each call returns a handle that may be used for subsequent append requests. Once all the "appending" is done, the caller must call `api.FlushObject` to finalize the object. NOTE: object becomes visible and accessible only _after_ the call to `api.FlushObject`.

func ArchiveMultiObj 

func ArchiveMultiObj(bp BaseParams, bckFrom cmn.Bck, msg cmn.ArchiveBckMsg) (string, error)

Archive multiple objects from the specified source bucket. The option to append multiple objects to an existing archive is also supported. The source and the destination buckets are defined as `bckFrom` and `bckTo`, respectively (not necessarily distinct) For supported archiving formats, see `archive.FileExtensions`. See also: api.PutApndArch

func AttachMountpath 

func AttachMountpath(bp BaseParams, node *meta.Snode, mountpath string, force bool) error

TODO: rewrite tests that come here with `force`

func AttachRemoteAIS 

func AttachRemoteAIS(bp BaseParams, alias, u string) error

func Bool 

func Bool(v bool) *bool

Bool returns a pointer to the bool value passed in.

func CopyBucket 

func CopyBucket(bp BaseParams, bckFrom, bckTo cmn.Bck, msg *apc.CopyBckMsg, fltPresence ...int) (xid string, err error)

CopyBucket copies existing `bckFrom` bucket to the destination `bckTo` thus, effectively, creating a copy of the `bckFrom`.

  • AIS will create `bckTo` on the fly but only if the destination bucket does not exist and _is_ provided by AIStore; 3rd party backend destination must exist - otherwise the copy operation won't be successful.
  • There are no limitations on copying buckets across Backend providers: you can copy AIS bucket to (or from) AWS bucket, and the latter to Google or Azure bucket, etc.
  • Copying multiple buckets to the same destination bucket is also permitted.

`fltPresence` applies exclusively to remote `bckFrom` and is ignored if the source is ais:// The value is enum { apc.FltExists, apc.FltPresent, ... } - for complete enum, see api/apc/query.go Namely: * apc.FltExists - copy all objects, including those that are not (present) in AIS * apc.FltPresent - copy the current `bckFrom` content in the cluster (default) * apc.FltExistsOutside - copy only those remote objects that are not (present) in AIS

msg.Prefix, if specified, applies always and regardless.

Returns xaction ID if successful, an error otherwise. See also closely related api.ETLBucket

func CopyMultiObj 

func CopyMultiObj(bp BaseParams, bckFrom cmn.Bck, msg cmn.TCObjsMsg, fltPresence ...int) (xid string, err error)

func CreateBucket 

func CreateBucket(bp BaseParams, bck cmn.Bck, props *cmn.BucketPropsToUpdate) error

CreateBucket sends request to create an AIS bucket with the given name and, optionally, specific non-default properties (via cmn.BucketPropsToUpdate).

See also:

  • github.com/artashesbalabekyan/aistore/blob/master/docs/bucket.md#default-bucket-properties
  • cmn.BucketPropsToUpdate (cmn/api.go)

Bucket properties can be also changed at any time via SetBucketProps (above).

func DecommissionCluster 

func DecommissionCluster(bp BaseParams, rmUserData bool) error

DecommissionCluster permanently decommissions entire cluster

func DecommissionNode 

func DecommissionNode(bp BaseParams, actValue *apc.ActValRmNode) (xid string, err error)

func DeleteList 

func DeleteList(bp BaseParams, bck cmn.Bck, filesList []string) (string, error)

DeleteList sends request to remove a list of objects from a bucket.

func DeleteObject 

func DeleteObject(bp BaseParams, bck cmn.Bck, object string) error

DeleteObject deletes an object specified by bucket/object.

func DeleteRange 

func DeleteRange(bp BaseParams, bck cmn.Bck, rng string) (string, error)

DeleteRange sends request to remove a range of objects from a bucket.

func DestroyBucket 

func DestroyBucket(bp BaseParams, bck cmn.Bck) error

DestroyBucket sends request to remove an AIS bucket with the given name.

func DetachMountpath 

func DetachMountpath(bp BaseParams, node *meta.Snode, mountpath string, dontResilver bool) error

func DetachRemoteAIS 

func DetachRemoteAIS(bp BaseParams, alias string) error

func DisableMountpath 

func DisableMountpath(bp BaseParams, node *meta.Snode, mountpath string, dontResilver bool) error

func DoWithRetry 

func DoWithRetry(client *http.Client, cb NewRequestCB, reqArgs *cmn.HreqArgs) (resp *http.Response, err error)

DoWithRetry executes `http-client.Do` and retries *retriable connection errors*, such as "broken pipe" and "connection refused". This function always closes the `reqArgs.BodR`, even in case of error. Usage: PUT and simlar requests that transfer payload from the user side. NOTE: always closes request body reader (reqArgs.BodyR) - explicitly or via Do() TODO: refactor

func DownloadBackend 

func DownloadBackend(bp BaseParams, descr string, bck cmn.Bck, prefix, suffix string, ivals ...time.Duration) (string, error)

func DownloadGetList 

func DownloadGetList(bp BaseParams, regex string, onlyActive bool) (dlList dload.JobInfos, err error)

func DownloadMulti 

func DownloadMulti(bp BaseParams, description string, bck cmn.Bck, msg any, intervals ...time.Duration) (string, error)

func DownloadRange 

func DownloadRange(bp BaseParams, description string, bck cmn.Bck, template string, intervals ...time.Duration) (string, error)

func DownloadSingle 

func DownloadSingle(bp BaseParams, description string,
	bck cmn.Bck, objName, link string, intervals ...time.Duration) (string, error)

func DownloadStatus 

func DownloadStatus(bp BaseParams, id string, onlyActive bool) (dlStatus *dload.StatusResp, err error)

func DownloadWithParam 

func DownloadWithParam(bp BaseParams, dlt dload.Type, body any) (id string, err error)

func Duration 

func Duration(v time.Duration) *time.Duration

Duration returns a pointer to the time duration value passed in.

func ECEncodeBucket 

func ECEncodeBucket(bp BaseParams, bck cmn.Bck, data, parity int) (xid string, err error)

Erasure-code entire `bck` bucket at a given `data`:`parity` redundancy. The operation requires at least (`data + `parity` + 1) storage targets in the cluster. Returns xaction ID if successful, an error otherwise.

func ETLBucket 

func ETLBucket(bp BaseParams, bckFrom, bckTo cmn.Bck, msg *apc.TCBMsg, fltPresence ...int) (xid string, err error)

Transform src bucket => dst bucket, i.e.: - visit all (matching) source objects; for each object: - read it, transform using the specified (ID-ed) ETL, and write the result to dst bucket

`fltPresence` applies exclusively to remote `bckFrom` (is ignored otherwise) and is one of: { apc.FltExists, apc.FltPresent, ... } - for complete enum, see api/apc/query.go Namely: * apc.FltExists - copy all objects, including those that are not (present) in AIS * apc.FltPresent - copy the current `bckFrom` content in the cluster (default) * apc.FltExistsOutside - copy only those remote objects that are not (present) in AIS

msg.Prefix, if specified, applies always and regardless.

Returns xaction ID if successful, an error otherwise. See also: api.CopyBucket

func ETLDelete 

func ETLDelete(bp BaseParams, etlName string) (err error)

func ETLGetInitMsg 

func ETLGetInitMsg(params BaseParams, etlName string) (etl.InitMsg, error)

func ETLHealth 

func ETLHealth(params BaseParams, etlName string) (healths etl.HealthByTarget, err error)

func ETLInit 

func ETLInit(bp BaseParams, msg etl.InitMsg) (xid string, err error)

Initiate custom ETL workload by executing one of the documented `etl.InitMsg` message types. The API call results in deploying multiple ETL containers (K8s pods): one container per storage target. Returns xaction ID if successful, an error otherwise.

func ETLList 

func ETLList(bp BaseParams) (list []etl.Info, err error)

func ETLLogs 

func ETLLogs(bp BaseParams, etlName string, targetID ...string) (logs etl.LogsByTarget, err error)

func ETLMetrics 

func ETLMetrics(params BaseParams, etlName string) (healths etl.CPUMemByTarget, err error)

func ETLMultiObj 

func ETLMultiObj(bp BaseParams, bckFrom cmn.Bck, msg cmn.TCObjsMsg, fltPresence ...int) (xid string, err error)

func ETLObject 

func ETLObject(bp BaseParams, etlName string, bck cmn.Bck, objName string, w io.Writer) (err error)

TODO: add ETL-specific query param and change the examples/docs (!4455)

func ETLStart 

func ETLStart(bp BaseParams, etlName string) (err error)

func ETLStop 

func ETLStop(bp BaseParams, etlName string) (err error)

func EnableMountpath 

func EnableMountpath(bp BaseParams, node *meta.Snode, mountpath string) error

func EvictList 

func EvictList(bp BaseParams, bck cmn.Bck, fileslist []string) (string, error)

EvictList sends request to evict a list of objects from a remote bucket.

func EvictObject 

func EvictObject(bp BaseParams, bck cmn.Bck, object string) error

EvictObject evicts an object specified by bucket/object.

func EvictRange 

func EvictRange(bp BaseParams, bck cmn.Bck, rng string) (string, error)

EvictRange sends request to evict a range of objects from a remote bucket.

func EvictRemoteBucket 

func EvictRemoteBucket(bp BaseParams, bck cmn.Bck, keepMD bool) error

EvictRemoteBucket sends request to evict an entire remote bucket from the AIStore - keepMD: evict objects but keep bucket metadata

func FlushObject 

func FlushObject(args FlushArgs) error

FlushObject must be called after all the appends (via `api.AppendObject`). To "flush", it uses the handle returned by `api.AppendObject`. This call will create a fully operational and accessible object.

func FreeRp 

func FreeRp(reqParams *ReqParams)

func GetAllRunningXactions 

func GetAllRunningXactions(bp BaseParams, kindOrName string) (out []string, err error)

returns unique ':'-separated kind/ID pairs (strings) e.g.: put-copies[D-ViE6HEL_j] list[H96Y7bhR2s] copy-bck[matRQMRes] put-copies[pOibtHExY] TODO: return idle xactions separately

func GetAllXactionStatus 

func GetAllXactionStatus(bp BaseParams, args xact.ArgsMsg, force bool) (matching nl.StatusVec, err error)

same as above, except that it returns _all_ matching xactions

func GetBMD 

func GetBMD(bp BaseParams) (*meta.BMD, error)

GetBMD returns bucket metadata

func GetBucketInfo 

func GetBucketInfo(bp BaseParams, bck cmn.Bck, fltPresence int) (p *cmn.BucketProps, info *cmn.BsummResult, err error)

Bucket information - a runtime addendum to `BucketProps`.

`fltPresence` - as per QparamFltPresence enum (see api/apc/query.go)

Unlike `cmn.BucketProps` properties (which are user configurable), bucket runtime info: - includes usage, capacity, other statistics - is obtained via GetBucketInfo() API - delivered via apc.HdrBucketInfo header (compare with GetBucketSummary)

NOTE: the API utilizes HEAD method (compare with HeadBucket) and always executes the (cached-only, fast) version of the bucket summary

func GetBucketSummary 

func GetBucketSummary(bp BaseParams, qbck cmn.QueryBcks, msg *apc.BsummCtrlMsg) (cmn.AllBsummResults, error)

GetBucketSummary returns bucket summaries (capacity ulitization percentages, sizes, and numbers of objects) for the specified bucket or buckets, as per `cmn.QueryBcks` query. E.g., an empty bucket query corresponds to all buckets present in the cluster's metadata.

func GetClusterConfig 

func GetClusterConfig(bp BaseParams) (*cmn.ClusterConfig, error)

GetClusterConfig returns cluster-wide configuration (compare with `api.GetDaemonConfig`)

func GetClusterMap 

func GetClusterMap(bp BaseParams) (smap *meta.Smap, err error)

GetClusterMap retrieves AIStore cluster map.

func GetClusterStats 

func GetClusterStats(bp BaseParams) (res stats.Cluster, err error)

How to compute throughputs:

- AIS supports several enumerated metric "kinds", including `KindThroughput` (for complete enumeration, see stats/api.go) - By convention, metrics that have `KindThroughput` kind are named with ".bps" ("bytes per second") suffix. - ".bps" metrics reported by api.GetClusterStats and api.GetDaemonStats are, in fact, cumulative byte numbers. - It is the client's responsibility to compute the actual throughputs as only the client knows _when_ exactly the same ".bps" metric was queried the previous time.

- See also: `api.GetDaemonStats`, stats/api.go

func GetClusterSysInfo 

func GetClusterSysInfo(bp BaseParams) (info apc.ClusterSysInfo, err error)

GetClusterSysInfo retrieves AIStore system info.

func GetDaemonConfig 

func GetDaemonConfig(bp BaseParams, node *meta.Snode) (config *cmn.Config, err error)

GetDaemonConfig returns the configuration of a specific daemon in a cluster. (compare with `api.GetClusterConfig`)

func GetDaemonLog 

func GetDaemonLog(bp BaseParams, node *meta.Snode, args GetLogInput) (int64, error)

Returns log of a specific node in a cluster.

func GetDaemonStats 

func GetDaemonStats(bp BaseParams, node *meta.Snode) (ds *stats.Node, err error)

How to compute throughputs:

- AIS supports several enumerated metric "kinds", including `KindThroughput` (for complete enumeration, see stats/api.go) - By convention, metrics that have `KindThroughput` kind are named with ".bps" ("bytes per second") suffix. - ".bps" metrics reported by the API are, in fact, cumulative byte numbers. - It is the client's responsibility to compute the actual throughputs as only the client knows _when_ exactly the same ".bps" metric was queried the previous time.

See also: - api.GetClusterStats - api.GetStatsAndStatus (below) - stats/api.go

func GetDiskStats 

func GetDiskStats(bp BaseParams, tid string) (res ios.AllDiskStats, err error)

func GetMetricNames 

func GetMetricNames(bp BaseParams, node *meta.Snode) (kvs cos.StrKVs, err error)

names _and_ kinds, i.e. (name, kind) pairs

func GetMountpaths 

func GetMountpaths(bp BaseParams, node *meta.Snode) (mpl *apc.MountpathList, err error)

GetMountpaths given the direct public URL of the target, returns the target's mountpaths or error.

func GetNodeClusterMap 

func GetNodeClusterMap(bp BaseParams, sid string) (smap *meta.Smap, err error)

GetNodeClusterMap retrieves AIStore cluster map from a specific node.

func GetObjectReader 

func GetObjectReader(bp BaseParams, bck cmn.Bck, object string, args *GetArgs) (r io.ReadCloser, err error)

GetObjectReader returns reader of the requested object. It does not read body bytes, nor validates a checksum. Caller is responsible for closing the reader.

func GetObjectS3 

func GetObjectS3(bp BaseParams, bck cmn.Bck, objectName string, args ...GetArgs) (int64, error)

s3/<bucket-name>/<object-name>

func GetOneXactionStatus 

func GetOneXactionStatus(bp BaseParams, args xact.ArgsMsg) (status *nl.Status, err error)

GetOneXactionStatus queries one of the IC (proxy) members for status of the `args`-identified xaction. NOTE: - is used internally by the WaitForXactionIC() helper function (to wait on xaction) - returns a single matching xaction or none; - when the `args` filter "covers" multiple xactions the returned status corresponds to any matching xaction that's currently running, or - if nothing's running - the one that's finished most recently, if exists

func GetProxyReadiness 

func GetProxyReadiness(bp BaseParams) error

to be used by external watchdogs (Kubernetes, etc.) (compare with api.Health below)

func GetRemoteAIS 

func GetRemoteAIS(bp BaseParams) (remais cluster.Remotes, err error)

func GetStatsAndStatus 

func GetStatsAndStatus(bp BaseParams, node *meta.Snode) (daeStatus *stats.NodeStatus, err error)

Returns both node's stats and extended status

func GetWhatRawQuery 

func GetWhatRawQuery(getWhat, getProps string) string

func HTTPStatus 

func HTTPStatus(err error) int

HTTPStatus returns HTTP status or (-1) for non-HTTP error.

func HeadBucket 

func HeadBucket(bp BaseParams, bck cmn.Bck, dontAddRemote bool) (p *cmn.BucketProps, err error)

HEAD(bucket): apc.HdrBucketProps => cmn.BucketProps{} and apc.HdrBucketInfo => BucketInfo{}

Converts the string type fields returned from the HEAD request to their corresponding counterparts in the cmn.BucketProps struct.

By default, AIStore adds remote buckets to the cluster metadata on the fly. Remote bucket that was never accessed before just "shows up" when user performs HEAD, PUT, GET, SET-PROPS, and a variety of other operations. This is done only once (and after confirming the bucket's existence and accessibility) and doesn't require any action from the user. Use `dontAddRemote` to override the default behavior: as the name implies, setting `dontAddRemote = true` prevents AIS from adding remote bucket to the cluster's metadata.

func HeadObject 

func HeadObject(bp BaseParams, bck cmn.Bck, object string, fltPresence int) (*cmn.ObjectProps, error)

HeadObject returns object properties; can be conventionally used to establish in-cluster presence. `fltPresence` - as per QparamFltPresence enum (for values and comments, see api/apc/query.go)

func Health 

func Health(bp BaseParams, readyToRebalance ...bool) error

func HealthUptime 

func HealthUptime(bp BaseParams, readyToRebalance ...bool) (string, string, error)

func Int 

func Int(v int) *int

Int returns a pointer to the int value passed in.

func Int64 

func Int64(v int64) *int64

Int64 returns a pointer to the int64 value passed in.

func JoinCluster 

func JoinCluster(bp BaseParams, nodeInfo *meta.Snode) (rebID, sid string, err error)

JoinCluster add a node to a cluster.

func ListBuckets 

func ListBuckets(bp BaseParams, qbck cmn.QueryBcks, fltPresence int) (cmn.Bcks, error)

ListBuckets returns buckets for provided query, where

`fltPresence` is one of { apc.FltExists, apc.FltPresent, ... } - see api/apc/query.go

(ListBuckets must not be confused with `ListObjects()` and friends below).

func ListDSort 

func ListDSort(bp BaseParams, regex string, onlyActive bool) (jobInfos []*dsort.JobInfo, err error)

func ListObjects 

func ListObjects(bp BaseParams, bck cmn.Bck, lsmsg *apc.LsoMsg, numObj uint) (*cmn.LsoResult, error)

ListObjects returns a list of objects in a bucket - a slice of structures in the `cmn.LsoResult` that look like `cmn.LsoEntry`.

The `numObjects` argument is the maximum number of objects to be returned (where 0 (zero) means returning all objects in the bucket).

This API supports numerous options and flags. In particular, `apc.LsoMsg` structure supports "opening" objects formatted as one of the supported archival types and include contents of archived directories in generated result sets.

In addition, `lsmsg` (`apc.LsoMsg`) provides options (flags) to optimize the request's latency, to list anonymous public-access Cloud buckets, and more. Further details at `api/apc/lsmsg.go` source.

AIS supports listing buckets that have millions of objects. For large and very large buckets, it is strongly recommended to use the `ListObjectsPage` API - effectively, an iterator returning _next_ listed page along with associated _continuation token_.

See also: - `ListObjectsPage` - usage examples in CLI docs under docs/cli.

func ListObjectsInvalidateCache 

func ListObjectsInvalidateCache(bp BaseParams, bck cmn.Bck) error

TODO: obsolete this function after introducing mechanism to detect remote bucket changes.

func ListObjectsPage 

func ListObjectsPage(bp BaseParams, bck cmn.Bck, lsmsg *apc.LsoMsg) (*cmn.LsoResult, error)

ListObjectsPage returns the first page of bucket objects. On success the function updates `lsmsg.ContinuationToken` which client then can reuse to fetch the next page. See also: CLI and CLI usage examples See also: `apc.LsoMsg` See also: `api.ListObjectsInvalidateCache` See also: `api.ListObjects`

func ListObjectsWithOpts 

func ListObjectsWithOpts(bp BaseParams, bck cmn.Bck, lsmsg *apc.LsoMsg, numObj uint, progress *ProgressContext) (*cmn.LsoResult, error)

additional argument may include "progress-bar" context

func MakeNCopies 

func MakeNCopies(bp BaseParams, bck cmn.Bck, copies int) (xid string, err error)

MakeNCopies starts an extended action (xaction) to bring a given bucket to a certain redundancy level (num copies). Returns xaction ID if successful, an error otherwise.

func MetricsDSort 

func MetricsDSort(bp BaseParams, managerUUID string) (metrics map[string]*dsort.Metrics, err error)

func PrefetchList 

func PrefetchList(bp BaseParams, bck cmn.Bck, fileslist []string) (string, error)

PrefetchList sends request to prefetch a list of objects from a remote bucket.

func PrefetchRange 

func PrefetchRange(bp BaseParams, bck cmn.Bck, rng string) (string, error)

PrefetchRange sends request to prefetch a range of objects from a remote bucket.

func Promote 

func Promote(args *PromoteArgs) (xid string, err error)

promote files and directories to ais objects

func PutApndArch 

func PutApndArch(args PutApndArchArgs) (err error)

Archive the content of a reader (`args.Reader` - e.g., an open file). Destination, depending on the options, can be an existing (.tar, .tgz or .tar.gz, .zip, .tar.lz4) formatted object (aka "shard") or a new one (or, a new version). --- For the updated list of supported archival formats -- aka MIME types -- see cmn/cos/archive.go. -- See also: - api.ArchiveMultiObj(msg.AppendIfExists = true) - api.AppendObject

func QueryBuckets 

func QueryBuckets(bp BaseParams, qbck cmn.QueryBcks, fltPresence int) (bool, error)

QueryBuckets is a little convenience helper. It returns true if the selection contains at least one bucket that satisfies the (qbck) criteria.

`fltPresence` - as per QparamFltPresence enum (see api/apc/query.go)

func QueryXactionSnaps 

func QueryXactionSnaps(bp BaseParams, args xact.ArgsMsg) (xs xact.MultiSnap, err error)

QueryXactionSnaps gets all xaction snaps based on the specified selection. NOTE: args.Kind can be either xaction kind or name - here and elsewhere

func RemoveDSort 

func RemoveDSort(bp BaseParams, managerUUID string) error

func RemoveDownload 

func RemoveDownload(bp BaseParams, id string) error

func RemoveNodeUnsafe 

func RemoveNodeUnsafe(bp BaseParams, sid string) error

Remove node node from the cluster immediately. - NOTE: potential data loss, advanced usage only! - NOTE: the node remains running (compare w/ shutdown) and can be re-joined at a later time (see api.JoinCluster).

func RenameBucket 

func RenameBucket(bp BaseParams, bckFrom, bckTo cmn.Bck) (xid string, err error)

RenameBucket renames bckFrom as bckTo. Returns xaction ID if successful, an error otherwise.

func RenameObject 

func RenameObject(bp BaseParams, bck cmn.Bck, oldName, newName string) error

RenameObject renames object name from `oldName` to `newName`. Works only across single, specified bucket.

func ResetBucketProps 

func ResetBucketProps(bp BaseParams, bck cmn.Bck) (string, error)

ResetBucketProps resets the properties of a bucket to the global configuration.

func ResetClusterConfig 

func ResetClusterConfig(bp BaseParams) error

ResetClusterConfig resets the configuration of all nodes to the cluster configuration

func ResetClusterStats 

func ResetClusterStats(bp BaseParams, errorsOnly bool) (err error)

func ResetDaemonConfig 

func ResetDaemonConfig(bp BaseParams, nodeID string) error

ResetDaemonConfig resets the configuration for a specific node to the cluster configuration. TODO: revisit access control

func ResetDaemonStats 

func ResetDaemonStats(bp BaseParams, node *meta.Snode, errorsOnly bool) (err error)

see also: ResetClusterStats

func SetAuxHeaders 

func SetAuxHeaders(r *http.Request, bp *BaseParams)

func SetBucketProps 

func SetBucketProps(bp BaseParams, bck cmn.Bck, props *cmn.BucketPropsToUpdate) (string, error)

SetBucketProps sets the properties of a bucket. Validation of the properties passed in is performed by AIStore Proxy.

func SetClusterConfig 

func SetClusterConfig(bp BaseParams, nvs cos.StrKVs, transient bool) error

SetClusterConfig given key-value pairs of cluster configuration parameters, sets the cluster-wide configuration accordingly. Setting cluster-wide configuration requires sending the request to a proxy.

func SetClusterConfigUsingMsg 

func SetClusterConfigUsingMsg(bp BaseParams, configToUpdate *cmn.ConfigToUpdate, transient bool) error

SetClusterConfigUsingMsg sets the cluster-wide configuration using the `cmn.ConfigToUpdate` parameter provided.

func SetDaemonConfig 

func SetDaemonConfig(bp BaseParams, nodeID string, nvs cos.StrKVs, transient ...bool) error

SetDaemonConfig, given key value pairs, sets the configuration accordingly for a specific node.

func SetObjectCustomProps 

func SetObjectCustomProps(bp BaseParams, bck cmn.Bck, object string, custom cos.StrKVs, setNew bool) error

Given cos.StrKVs (map[string]string) keys and values, sets object's custom properties. By default, adds new or updates existing custom keys. Use `setNewCustomMDFlag` to _replace_ all existing keys with the specified (new) ones. See also: HeadObject() and apc.HdrObjCustomMD

func SetPrimaryProxy 

func SetPrimaryProxy(bp BaseParams, newPrimaryID string, force bool) error

SetPrimaryProxy given a daemonID sets that corresponding proxy as the primary proxy of the cluster.

func ShutdownCluster 

func ShutdownCluster(bp BaseParams) error

ShutdownCluster shuts down the whole cluster

func ShutdownNode 

func ShutdownNode(bp BaseParams, actValue *apc.ActValRmNode) (id string, err error)

ShutdownNode shuts down a specific node

func StartDSort 

func StartDSort(bp BaseParams, rs dsort.RequestSpec) (id string, err error)

func StartMaintenance 

func StartMaintenance(bp BaseParams, actValue *apc.ActValRmNode) (xid string, err error)

func StartXaction 

func StartXaction(bp BaseParams, args xact.ArgsMsg) (xid string, err error)

Start xaction

func StopMaintenance 

func StopMaintenance(bp BaseParams, actValue *apc.ActValRmNode) (xid string, err error)

func String 

func String(v string) *string

String returns a pointer to the string value passed in.

func WaitForXactionIC 

func WaitForXactionIC(bp BaseParams, args xact.ArgsMsg) (status *nl.Status, err error)

WaitForXactionIC waits for a given xaction to complete. Use it only for global xactions (those that execute on all targets and report their status to IC, e.g. rebalance).

func WaitForXactionIdle 

func WaitForXactionIdle(bp BaseParams, args xact.ArgsMsg) error

WaitForXactionIdle waits for a given on-demand xaction to be idle.

func WaitForXactionNode 

func WaitForXactionNode(bp BaseParams, args xact.ArgsMsg, fn func(xact.MultiSnap) (bool, bool)) error

WaitForXactionNode waits for a given xaction to complete. Use for xactions that do _not_ report their status to IC members, namely: - xact.IdlesBeforeFinishing() - x-resilver (as it usually runs on a single node)

func WritePolicy 

func WritePolicy(v apc.WritePolicy) *apc.WritePolicy

Types

type AppendArgs 

type AppendArgs struct {
	Reader     cos.ReadOpenCloser
	BaseParams BaseParams
	Bck        cmn.Bck
	Object     string
	Handle     string
	Size       int64
}

(see also: api.PutApndArchArgs)

type BaseParams 

type BaseParams struct {
	Client *http.Client
	URL    string
	Method string
	Token  string
	UA     string
}

type FlushArgs 

type FlushArgs struct {
	Cksum      *cos.Cksum
	BaseParams BaseParams
	Bck        cmn.Bck
	Object     string
	Handle     string
}

PUT, APPEND, PROMOTE (object)

type GetArgs 

type GetArgs struct {
	// If not specified (or same: if `nil`), Writer defaults to `io.Discard`
	// (in other words, with no writer the object that is being read will be discarded)
	Writer io.Writer

	// Currently, the Query field can optionally carry 2 (two) distinct values:
	// 1. `apc.QparamETLName`: named ETL to transform the object (i.e., perform "inline transformation")
	// 2. `apc.QparamOrigURL`: GET from a vanilla http(s) location (`ht://` bucket with the corresponding `OrigURLBck`)
	Query url.Values

	// The field is exclusively used to facilitate Range Read.
	// E.g. usage:
	// * Header.Set(cos.HdrRange, fmt.Sprintf("bytes=%d-%d", fromOffset, toOffset))
	// For range formatting, see the spec:
	// * https://www.rfc-editor.org/rfc/rfc7233#section-2.1
	Header http.Header
}

GET (object)

type GetLogInput 

type GetLogInput struct {
	Writer   io.Writer
	Severity string // one of: {cmn.LogInfo, ...}
	Offset   int64
}

type NewRequestCB 

type NewRequestCB func(args *cmn.HreqArgs) (*http.Request, error)

PUT, APPEND, PROMOTE (object)

type ObjAttrs 

type ObjAttrs struct {
	// contains filtered or unexported fields
}

`ObjAttrs` represents object attributes and can be further used to retrieve the object's size, checksum, version, and other metadata.

Note that while `GetObject()` and related GET APIs return `ObjAttrs`, `HeadObject()` API returns `cmn.ObjectProps` - a superset.

func GetObject 

func GetObject(bp BaseParams, bck cmn.Bck, object string, args *GetArgs) (oah ObjAttrs, err error)

Writes the response body if GetArgs.Writer is specified; otherwise, uses `io.Discard` to read all and discard

`io.Copy` is used internally to copy response bytes from the request to the writer.

Returns `ObjAttrs` that can be further used to get the size and other object metadata.

func GetObjectWithValidation 

func GetObjectWithValidation(bp BaseParams, bck cmn.Bck, object string, args *GetArgs) (oah ObjAttrs, err error)

Same as above with checksum validation.

Returns `cmn.ErrInvalidCksum` when the expected and actual checksum values are different.

func PutObject 

func PutObject(args PutArgs) (oah ObjAttrs, err error)

PutObject creates an object from the body of the reader (`args.Reader`) and puts it in the specified bucket.

Assumes that `args.Reader` is already opened and ready for usage. Returns `ObjAttrs` that can be further used to get the size and other object metadata.

func (*ObjAttrs) Attrs 

func (oah *ObjAttrs) Attrs() (out cmn.ObjAttrs)

func (*ObjAttrs) RespHeader 

func (oah *ObjAttrs) RespHeader() http.Header

e.g. usage: range read response

func (*ObjAttrs) Size 

func (oah *ObjAttrs) Size() int64

most often used (convenience) method

type ProgressCallback 

type ProgressCallback = func(pi *ProgressContext)

"progress bar" control structures and context

type ProgressContext 

type ProgressContext struct {
	// contains filtered or unexported fields
}

"progress bar" control structures and context

func NewProgressContext 

func NewProgressContext(cb ProgressCallback, after time.Duration) *ProgressContext

func (*ProgressContext) Elapsed 

func (ctx *ProgressContext) Elapsed() time.Duration

func (*ProgressContext) Info 

func (ctx *ProgressContext) Info() ProgressInfo

func (*ProgressContext) IsFinished 

func (ctx *ProgressContext) IsFinished() bool

type ProgressInfo 

type ProgressInfo struct {
	Percent float64
	Count   int
	Total   int
}

negative values indicate that progress information is unavailable

type PromoteArgs 

type PromoteArgs struct {
	BaseParams BaseParams
	Bck        cmn.Bck
	cluster.PromoteArgs
}

PUT, APPEND, PROMOTE (object)

type PutApndArchArgs 

type PutApndArchArgs struct {
	ArchPath string // filename _in_ archive
	Mime     string // user-specified mime type (NOTE: takes precedence if defined)
	Flags    int64  // apc.ArchAppend and apc.ArchAppendIfExist (the former requires destination shard to exist)
	PutArgs
}

Archive files and directories (see related: cmn.ArchiveBckMsg)

type PutArgs 

type PutArgs struct {
	Reader cos.ReadOpenCloser

	// optional; if provided:
	// - if object exists: load the object's metadata, compare checksums - skip writing if equal
	// - otherwise, compare the two checksums upon writing (aka, "end-to-end protection")
	Cksum *cos.Cksum

	BaseParams BaseParams

	Bck     cmn.Bck
	ObjName string

	Size uint64 // optional

	// Skip loading existing object's metadata in order to
	// compare its Checksum and update its existing Version (if exists);
	// can be used to reduce PUT latency when:
	// - we massively write a new content into a bucket, and/or
	// - we simply don't care.
	SkipVC bool
}

PUT, APPEND, PROMOTE (object)

type ReqParams 

type ReqParams struct {
	Query  url.Values
	Header http.Header

	BaseParams BaseParams

	Path string

	// Authentication
	User     string
	Password string

	Body []byte
}

ReqParams is used in constructing client-side API requests to aistore. Stores Query and Headers for providing arguments that are not used commonly in API requests 

See also: cmn.HreqArgs

func AllocRp 

func AllocRp() *ReqParams

func (*ReqParams) DoReqAny 

func (reqParams *ReqParams) DoReqAny(out any) (int, error)

Makes request via do(), decodes `resp.Body` into the `out` structure, closes the former, and returns the entire wrapped response (as well as `out`)

Returns an error if the response status >= 400.

func (*ReqParams) DoRequest 

func (reqParams *ReqParams) DoRequest() error

uses do() to make the request; if successful, checks, drains, and closes the response body

Source Files

View all Source files

Directories

Path Synopsis
Package apc: API messages and constants
 Package apc: API messages and constants
Package authn provides AuthN API over HTTP(S)
 Package authn provides AuthN API over HTTP(S)
Package env contains environment variables
 Package env contains environment variables

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.