README
¶
About:
AWE is a workload management system for bioinformatic workflow applications. AWE, together with Shock data management system, can be used to build an integrated platform for efficient data analysis and management which features following functionalities:
- Common workflow language support
- Multi cloud support
- Explicit task parallelization and convenient application integration
- Scalable, portable, and fault-tolerant workflow computation
- Integration of heterogeneous and geographically distributed computing resources
- Performance-aware, cost-efficient service management and resource management
- Reusable and reproducible data product management
AWE is designed as a distributed system that contains a centralized server and multiple distributed clients. The server receives job submissions and parses jobs into tasks, splits tasks into workunits, and manages workunits in a queue. The AWE clients, running on distributed, heterogeneous computing resources, keep checking out workunits from the server queue and dispatching the workunits on the local computing resources.
AWE uses the Shock data management system to handle input and output data (retrieval, storage, splitting, and merge). AWE uses a RESTful API for communication between AWE components and with outside components such as Shock, the job submitter, and the status monitor.
Related Links
| link | |
|---|---|
| AWE server | github.com/MG-RAST/AWE |
| AWE monitor | github.com/MG-RAST/awe-monitor |
| Shock | github.com/MG-RAST/Shock |
| Skyport2 | github.com/MG-RAST/Skyport2 |
Documentation
Documentation can be found on the AWE wiki pages:
https://github.com/MG-RAST/AWE/wiki
Papers to cite
W. Tang, J. Wilkening, N. Desai, W. Gerlach, A. Wilke, F. Meyer, "A scalable data analysis platform for metagenomics," in Proc. of IEEE International Conference on Big Data, 2013.[ieeexplore] [pdf]
W. Gerlach, W. Tang, K. Keegan, T. Harrison, A. Wilke, J. Bischof, M. D'Souza, S. Devoid, D. Murphy-Olson, N. Desai, F. Meyer, "Skyport – Container-Based Execution Environment Management for Multi-Cloud Scientific Workflows," in Proc. of the 5th International Workshop on Data Intensive Computing in the Clouds, 2014. [pdf]
AWE Discussion Group
For questions, bug reports or feature requests please use the awe-users mailing list:
https://groups.google.com/d/forum/awe-users (Email: awe-users@googlegroups.com)
Directories
¶
| Path | Synopsis |
|---|---|
|
lib
|
|
|
auth
Package auth implements http request authentication
|
Package auth implements http request authentication |
|
auth/basic
Package basic implements basic auth decoding and self contained user authentication
|
Package basic implements basic auth decoding and self contained user authentication |
|
auth/clientgroup
Package clientgroup implements clientgroup auth decoding and self-contained clientgroup authentication
|
Package clientgroup implements clientgroup auth decoding and self-contained clientgroup authentication |
|
auth/globus
Package globus implements Globus Online Nexus authentication
|
Package globus implements Globus Online Nexus authentication |
|
auth/oauth
Package oauth implements generic OAuth authentication
|
Package oauth implements generic OAuth authentication |
|
db
Package db to connect to mongodb
|
Package db to connect to mongodb |