aistore

module
v1.3.16 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 31, 2023 License: MIT

README

AIStore is a lightweight object storage system with the capability to linearly scale out with each added storage node and a special focus on petascale deep learning.

License Go Report Card

AIStore (AIS for short) is a built from scratch, lightweight storage stack tailored for AI apps. It's an elastic cluster that can grow and shrink at runtime and can be ad-hoc deployed, with or without Kubernetes, anywhere from a single Linux machine to a bare-metal cluster of any size.

AIS consistently shows balanced I/O distribution and linear scalability across arbitrary numbers of clustered nodes. The ability to scale linearly with each added disk was, and remains, one of the main incentives. Much of the development is also driven by the ideas to offload dataset transformations.

Features

  • Deploys anywhere. AIS clusters are immediately deployable on any commodity hardware, on any Linux machine(s).
  • Highly available control and data planes, end-to-end data protection, self-healing, n-way mirroring, erasure coding, and arbitrary number of extremely lightweight access points.
  • REST API. Comprehensive native HTTP-based API, as well as compliant Amazon S3 API to run unmodified S3 clients and apps.
  • Unified namespace across multiple remote backends including Amazon S3, Google Cloud, and Microsoft Azure.
  • Network of clusters. Any AIS cluster can attach any other AIS cluster, thus gaining immediate visibility and fast access to the respective hosted datasets.
  • Turn-key cache. Can be used as a standalone highly-available protected storage and/or LRU-based fast cache. Eviction watermarks, as well as numerous other management policies, are per-bucket configurable.
  • ETL offload. The capability to run I/O intensive custom data transformations close to data - offline (dataset to dataset) and inline (on-the-fly).
  • File datasets. AIS can be immediately populated from any file-based data source (local or remote, ad-hoc/on-demand or via asynchronus batch).
  • Small files. Sharding. To serialize small files, AIS supports TAR, TAR.GZ, ZIP, and MessagePack formats, and provides the entire spectrum of operations to make the corresponding sharding transparent to the apps.
  • Kubernetes. Provides for easy Kubernetes deployment via a separate GitHub repo and AIS/K8s Operator.
  • Command line management. Integrated powerful CLI for easy management and monitoring.
  • Access control. For security and fine-grained access control, AIS includes OAuth 2.0 compliant Authentication Server (AuthN). A single AuthN instance executes CLI requests over HTTPS and can serve multiple clusters.
  • Distributed shuffle extension for massively parallel resharding of very large datasets.
  • Batch jobs. APIs and CLI to start, stop, and monitor documented batch operations, such as prefetch, download, copy or transform datasets, and many more.

AIS runs natively on Kubernetes and features open format - thus, the freedom to copy or move your data from AIS at any time using the familiar Linux tar(1), scp(1), rsync(1) and similar.

For developers and data scientists, there's also:

For the original AIStore white paper and design philosophy, for introduction to large-scale deep learning and the most recently added features, please see AIStore Overview (where you can also find six alternative ways to work with existing datasets). Videos and animated presentations can be found at videos.

Finally, getting started with AIS takes only a few minutes.


Deployment options

AIS deployment options, as well as intended (development vs. production vs. first-time) usages, are all summarized here.

Since prerequisites boil down to, essentially, having Linux with a disk the deployment options range from all-in-one container to a petascale bare-metal cluster of any size, and from a single VM to multiple racks of high-end servers. But practical use cases require, of course, further consideration and may include:

Option Objective
Local playground AIS developers and development, Linux or Mac OS
Minimal production-ready deployment This option utilizes preinstalled docker image and is targeting first-time users or researchers (who could immediately start training their models on smaller datasets)
Easy automated GCP/GKE deployment Developers, first-time users, AI researchers
Large-scale production deployment Requires Kubernetes and is provided via a separate repository: ais-k8s

Further, there's the capability referred to as global namespace: given HTTP(S) connectivity, AIS clusters can be easily interconnected to "see" each other's datasets. Hence, the idea to start "small" to gradually and incrementally build high-performance shared capacity.

For detailed discussion on supported deployments, please refer to Getting Started.

For performance tuning and preparing AIS nodes for bare-metal deployment, see performance.

Installing from release binaries

Generally, AIStore (cluster) requires at least some sort of deployment procedure. There are standalone binaries, though, that can be built from source or, alternatively, installed directly from GitHub:

$ ./deploy/scripts/install_from_binaries.sh --help

The script installs aisloader and CLI from the most recent, or the previous, GitHub release. For CLI, it'll also enable auto-completions (which is strongly recommended).

PyTorch integration

AIS is one of the PyTorch Iterable Datapipes.

Specifically, TorchData library provides:

to list and, respectively, load data from AIStore.

Further references and usage examples - in our technical blog at https://aiatscale.org/blog:

Since AIS natively supports a number of remote backends, you can also use (PyTorch + AIS) to iterate over Amazon S3 and Google Cloud buckets, and more.

Reuse

This repo includes SGL and Slab allocator intended to optimize memory usage, Streams and Stream Bundles to multiplex messages over long-lived HTTP connections, and a few other sub-packages providing rather generic functionality.

With a little effort, they all could be extracted and used outside.

Guides and References

License

MIT

Author

Alex Aizman (NVIDIA)

Directories

Path Synopsis
3rdparty
glog
Package glog implements logging analogous to the Google-internal C++ INFO/ERROR/V setup.
Package glog implements logging analogous to the Google-internal C++ INFO/ERROR/V setup.
ais
Package ais provides core functionality for the AIStore object storage.
Package ais provides core functionality for the AIStore object storage.
backend
Package backend contains implementation of various backend providers.
Package backend contains implementation of various backend providers.
s3
Package s3 provides Amazon S3 compatibility layer
Package s3 provides Amazon S3 compatibility layer
test
Package integration contains AIS integration tests.
Package integration contains AIS integration tests.
api
Package api provides AIStore API over HTTP(S)
Package api provides AIStore API over HTTP(S)
apc
Package apc: API messages and constants
Package apc: API messages and constants
authn
Package authn provides AuthN API over HTTP(S)
Package authn provides AuthN API over HTTP(S)
env
Package env contains environment variables
Package env contains environment variables
bench
aisloader
Package aisloader
Package aisloader
aisloader/namegetter
Package namegetter is utility to generate filenames for aisloader PUT requests
Package namegetter is utility to generate filenames for aisloader PUT requests
aisloader/stats
Package stats provides various structs for collecting stats
Package stats provides various structs for collecting stats
disk/compare
Package main
Package main
frandread
Package frandread is a file-reading benchmark that makes a special effort to visit the files randomly and equally.
Package frandread is a file-reading benchmark that makes a special effort to visit the files randomly and equally.
hrw
Package hrw provides a way to benchmark different HRW variants.
Package hrw provides a way to benchmark different HRW variants.
http2
Package http2 puts a given number of files with a given size into AIStore.
Package http2 puts a given number of files with a given size into AIStore.
map
Package main
Package main
nstlvl
Package nstlvl is intended to measure impact (or lack of thereof) of POSIX directory nesting on random read performance.
Package nstlvl is intended to measure impact (or lack of thereof) of POSIX directory nesting on random read performance.
Package cluster provides local access to cluster-level metadata
Package cluster provides local access to cluster-level metadata
mock
Package mock provides a variety of mock implementations used for testing.
Package mock provides a variety of mock implementations used for testing.
cmd
aisloader
Package main for the `aisloader` executable.
Package main for the `aisloader` executable.
aisnode
Package main for the AIS node executable.
Package main for the AIS node executable.
aisnodeprofile
Package main for the AIS node executable.
Package main for the AIS node executable.
authn
Package authn is authentication server for AIStore.
Package authn is authentication server for AIStore.
authn/tok
Package tok provides AuthN token (structure and methods) for validation by AIS gateways
Package tok provides AuthN token (structure and methods) for validation by AIS gateways
xmeta
Package xmeta provides low-level tools to format or extract into plain text some of the AIS control structures.
Package xmeta provides low-level tools to format or extract into plain text some of the AIS control structures.
aisfs Module
cli Module
ishard Module
cmn
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
atomic
Package atomic provides simple wrappers around numerics to enforce atomic access.
Package atomic provides simple wrappers around numerics to enforce atomic access.
cos
Package cos provides common low-level types and utilities for all aistore projects
Package cos provides common low-level types and utilities for all aistore projects
debug
Package provides debug utilities
Package provides debug utilities
feat
Package feat: global runtime-configurable feature flags
Package feat: global runtime-configurable feature flags
fname
Package fname contains filename constants and common system directories
Package fname contains filename constants and common system directories
jsp
Package jsp (JSON persistence) provides utilities to store and load arbitrary JSON-encoded structures with optional checksumming and compression.
Package jsp (JSON persistence) provides utilities to store and load arbitrary JSON-encoded structures with optional checksumming and compression.
k8s
Package k8s provides utilities for communicating with Kubernetes cluster.
Package k8s provides utilities for communicating with Kubernetes cluster.
kvdb
Package kvdb provides a local key/value database server for AIS.
Package kvdb provides a local key/value database server for AIS.
mono
Package mono provides low-level monotonic time
Package mono provides low-level monotonic time
prob
Package prob implements fully features dynamic probabilistic filter.
Package prob implements fully features dynamic probabilistic filter.
xoshiro256
Package xoshiro256 implements the xoshiro256** RNG no-copyright
Package xoshiro256 implements the xoshiro256** RNG no-copyright
Package ec provides erasure coding (EC) based data protection for AIStore.
Package ec provides erasure coding (EC) based data protection for AIStore.
ext
dload
Package dload implements functionality to download resources into AIS cluster from external source.
Package dload implements functionality to download resources into AIS cluster from external source.
dsort
Package dsort provides distributed massively parallel resharding for very large datasets.
Package dsort provides distributed massively parallel resharding for very large datasets.
dsort/extract
Package extract provides provides functions for working with compressed files
Package extract provides provides functions for working with compressed files
dsort/filetype
Package filetype provides the implementation of custom content file type for dsort.
Package filetype provides the implementation of custom content file type for dsort.
etl
Package etl provides utilities to initialize and use transformation pods.
Package etl provides utilities to initialize and use transformation pods.
etl/runtime
Package runtime provides skeletons and static specifications for building ETL from scratch.
Package runtime provides skeletons and static specifications for building ETL from scratch.
fs
Package fs provides mountpath and FQN abstractions and methods to resolve/map stored content
Package fs provides mountpath and FQN abstractions and methods to resolve/map stored content
health
Package health provides a basic mountpath health monitor.
Package health provides a basic mountpath health monitor.
mpather
Package mpather provides per-mountpath concepts.
Package mpather provides per-mountpath concepts.
Package hk provides mechanism for registering cleanup functions which are invoked at specified intervals.
Package hk provides mechanism for registering cleanup functions which are invoked at specified intervals.
Package ios is a collection of interfaces to the local storage subsystem; the package includes OS-dependent implementations for those interfaces.
Package ios is a collection of interfaces to the local storage subsystem; the package includes OS-dependent implementations for those interfaces.
Package memsys provides memory management and slab/SGL allocation with io.Reader and io.Writer interfaces on top of scatter-gather lists of reusable buffers.
Package memsys provides memory management and slab/SGL allocation with io.Reader and io.Writer interfaces on top of scatter-gather lists of reusable buffers.
Package mirror provides local mirroring and replica management
Package mirror provides local mirroring and replica management
Package notifications provides interfaces for AIStore notifications
Package notifications provides interfaces for AIStore notifications
Package reb provides global cluster-wide rebalance upon adding/removing storage nodes.
Package reb provides global cluster-wide rebalance upon adding/removing storage nodes.
Package res provides local volume resilvering upon mountpath-attach and similar
Package res provides local volume resilvering upon mountpath-attach and similar
Package space provides storage cleanup and eviction functionality (the latter based on the least recently used cache replacement).
Package space provides storage cleanup and eviction functionality (the latter based on the least recently used cache replacement).
Package stats provides methods and functionality to register, track, log, and StatsD-notify statistics that, for the most part, include "counter" and "latency" kinds.
Package stats provides methods and functionality to register, track, log, and StatsD-notify statistics that, for the most part, include "counter" and "latency" kinds.
statsd
Package statsd provides a client to send basic statd metrics (timer, counter and gauge) to listening UDP StatsD server.
Package statsd provides a client to send basic statd metrics (timer, counter and gauge) to listening UDP StatsD server.
Package sys provides methods to read system information
Package sys provides methods to read system information
Package tools provides common tools and utilities for all unit and integration tests
Package tools provides common tools and utilities for all unit and integration tests
archive
Package archive provides common low-level utilities for testing archives
Package archive provides common low-level utilities for testing archives
cryptorand
Package cryptorand
Package cryptorand
docker
Packager docker provides common utilities for managing containerized AIS deployments
Packager docker provides common utilities for managing containerized AIS deployments
readers
Package readers provides implementation for common reader types
Package readers provides implementation for common reader types
tassert
Package tassert provides common asserts for tests
Package tassert provides common asserts for tests
tetl
Package tetl provides helpers for ETL.
Package tetl provides helpers for ETL.
tlog
Package tlog provides common logf and logln primitives for dev tools
Package tlog provides common logf and logln primitives for dev tools
trand
Package trand provides random string for dev tools and tests
Package trand provides random string for dev tools and tests
Package transport provides streaming object-based transport over http for intra-cluster continuous intra-cluster communications (see README for details and usage example).
Package transport provides streaming object-based transport over http for intra-cluster continuous intra-cluster communications (see README for details and usage example).
bundle
Package bundle provides multi-streaming transport with the functionality to dynamically (un)register receive endpoints, establish long-lived flows, and more.
Package bundle provides multi-streaming transport with the functionality to dynamically (un)register receive endpoints, establish long-lived flows, and more.
Package volume provides the volume abstraction and methods to configLoadVMD, store with redundancy, and validate the corresponding metadata.
Package volume provides the volume abstraction and methods to configLoadVMD, store with redundancy, and validate the corresponding metadata.
Package xact provides core functionality for the AIStore eXtended Actions (xactions).
Package xact provides core functionality for the AIStore eXtended Actions (xactions).
xreg
Package xreg provides registry and (renew, find) functions for AIS eXtended Actions (xactions).
Package xreg provides registry and (renew, find) functions for AIS eXtended Actions (xactions).
xs
Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.
Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL