nerd

command module
v1.0.0-alpha3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 2, 2018 License: Apache-2.0 Imports: 6 Imported by: 0

README

Nerd

Your personal nerd that takes care of running jobs on the Nerdalize cloud.

NOTE: This project is currently experimental and not functional.

Command Usage

# log into the scientific compute platform
$ nerd login
Successful login. You can now select a project using 'nerd project'

# list all projects
$ nerd project list
PROJECTS
nerdalize-video
nerdalize-weather

# set a project to work with
$ nerd project set nerdalize-video

# upload a piece of data that will acts as input to the program
$ ls ~/Desktop/videos
video1.mov
video2.mov

$ nerd dataset upload ~/Desktop/videos
Uploading dataset with ID 'd-96fac377'
314.38 MiB / 314.38 MiB [=============================] 100.00%

# start a workload
# we start 2 workers that use the jrottenberg/ffmpeg container to work on the input dataset
$ nerd workload start jrottenberg/ffmpeg
    --instances 2
    --input-dataset d-96fac377
Started workload with ID 'w-96fac375'

# create two tasks for this workload
# this will start the jrottenberg/ffmpeg container twice with the given arguments
# input dataset d-96fac377 will be available in /input, data in /output will be uploaded when the task has successfully executed
$ nerd task create w-96fac375 -- -i /input/video1.mov -acodec copy -vcodec copy /output/video1.avi
$ nerd task create w-96fac375 -- -i /input/video2.mov -acodec copy -vcodec copy /output/video2.avi

# get status of tasks
$ nerd task list w-96fac375
TASK ID     CMD                                                                 OUTPUT ID   STATUS    CREATED
t-14962176  [-i /input/video1.mov -acodec copy -vcodec copy /output/video1.avi] video1.avi  SUCCESS   1 minute ago
t-89491732  [-i /input/video2.mov -acodec copy -vcodec copy /output/video2.avi]             PENDING   1 minute ago

# when all tasks are done we can download the output
$ nerd workload download w-96fac375 ~/Desktop/videos_out
$ tree ~/Desktop/videos_out
~/Desktop/videos_out
├── 7691e3df0c824efc1007082057b9c867_14962176
│   └── video1.avi
└── f45371445a7b95f7352bf841c30e4f58_89491732
    └── video2.avi

Please note that each command has a --help option that shows how to use the command. Each command accepts at least the following options:

--config-file=  location of config file [$NERD_CONFIG_FILE]
--session-file= location of session file [$NERD_SESSION_FILE]
--debug=  show debug output [$NERD_DEBUG]
--output=[pretty|raw|json] Format of the output (default: pretty)

Power users

Config

The nerd command uses a config file located at ~/.nerd/config.json (location can be changed with the --config-file option) which can be used to customize nerd's behaviour. The structure of the config and the defaults are show below:

{
        "auth": {
                "api_endpoint": "http://auth.nerdalize.com/v1/", # URL of authentication server
                "public_key": "-----BEGIN PUBLIC KEY-----\nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEBthEmchVCtA3ZPXqiCXdj+7/ZFuhxRgx\ngrTxIHK+b0vEqKqA3O++ggD1GgjqtTfNLGUjLCE3KxyIN78TsK+HU4VVexTjlWXy\nWPtidD68xGD0JVPU1cSfu8iP0XzwgttG\n-----END PUBLIC KEY-----\n", # Public key used to verify JWT signature
                "client_id": "Po2PHR7YOIEQojfHy3zfOMulby9QGVSvcD9bO3S3", #OAuth client ID
                "oauth_localserver": "localhost:9876", #address of local oauth server
                "oauth_success_url": "https://cloud.nerdalize.com" #redirect URL after successful login
        },
        "logging": { # write all command output to a log file
          "enabled": false,
          "file_location": "~/.nerd/log"
        },
        "nerd_api_endpoint": "https://batch.nerdalize.com/v1/" # URL of nerdalize API (NCE)
}

Session details such as OAuth session, JWT, and current project are stored in ~/.nerd/session.json (can be changed using the --session-file option) The structure of session.json is show below:

{
  "oauth": {
  	"access_token": "", #oauth access token
  	"refresh_token": "", #oauth refresh token
  	"expiration": "", #expiration date + time
  	"scope": "", #oauth scope
  	"token_type": "" #Bearer
  },
  "jwt": {
    "token": "", #Current JWT
    "refresh_token": "" #used when JWT is refreshable
  },
  "project": {
    "name": "", #NLZ project name
    "aws_region": "" #AWS Region
  }
}

Docker

The nerd CLI can be dockerized. To build the docker container run:

docker build -t my-nerd .

You can now run the container like so:

docker run my-nerd <command>

If you want to use your local nerd config file (which contains your credentials), you can mount it:

docker run -v ~/.nerd:/root/.nerd my-nerd <command>

If you just want to set your credentials, you can also set it with an environment variable:

docker run -e NERD_JWT=put.jwt.here my-nerd <command>

Nerdalize SDK

Code in this repository can also be used as a Software Development Kit (SDK) to communicate with Nerdalize services. The SDK consists of two packages:

nerd/client
  • auth is a client to the Nerdalize authentication backend. It can be used to fetch new JWTs.
  • batch is a client to batch.nerdalize.com. It can be used to work with resources like queues, workers, and datasets.
nerd/service
  • datatransfer makes it possible to upload or download a dataset using one function call
  • working works on workload tasks

Documentation

The Go Gopher

There is no documentation for this package.

Directories

Path Synopsis
cmd
flex
main holds the flex volume executable, compiled separately
main holds the flex volume executable, compiled separately
crd
pkg/apis/stable.nerdalize.com/v1
Package v1 is the v1 version of the API.
Package v1 is the v1 version of the API.
aws
conf
Package conf gives the CLI access to the nerd config file.
Package conf gives the CLI access to the nerd config file.
jwt
pkg
populator
Package populator is a package that will help us to populate the kubernetes config file with the right credentials
Package populator is a package that will help us to populate the kubernetes config file with the right credentials
transfer
Package transfer provides primitives for uploading and downloading datasets
Package transfer provides primitives for uploading and downloading datasets
tool

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL