envd

README

envd

Development Environment for Data Scientists

envd is a development environment management tool for data scientists.

🐍 No docker, only python - Write python code to build the development environment, we help you take care of Docker.

📟 Built-in jupyter/vscode - Provision jupyter notebooks and vscode remote in the image, remote development is possible.

👨💻 Manage code and data - Help you manage the source code and dataset in the environment

⏱ Save time - Better cache management to save your time, keep the focus on the model, instead of dependencies

☁️ Local & cloud - Run the environment locally or in the cloud, without any code change

🐳 Container native - Leverage container technologies but no need to learn how to use them, we optimize it for you

🤟 Infrastructure as code - Describe your project in a declarative way, 100% reproducible

Install

From binary

TODO

From source code

git clone https://github.com/tensorchord/envd
go mod tidy
make
./bin/envd --version

Quickstart

Checkout the examples, and configure envd with the manifest build.envd:

vscode(plugins=[
    "ms-python.python",
])

base(os="ubuntu20.04", language="python3")
pip_package(name=[
    "tensorflow",
    "numpy",
])
cuda(version="11.6", cudnn="8")
shell("zsh")
jupyter(password="", port=8888)

Then you can run envd up to create the development environment.

TODO: illustrate that the cache will be persistent.

$ envd up
[+] ⌚ parse build.envd and download/cache dependencies 0.0s ✅ (finished)        
 => 💽 (cached) download oh-my-zsh                                            0.0s
 => 💽 (cached) download ms-python.python                                     0.0s
[+] 🐋 build envd environment 7.7s (24/25)                                        
 => 💽 (cached) (built-in packages) apt-get install curl openssh-client g     0.0s
 => 💽 (cached) create user group envd                                        0.0s
 => 💽 (cached) create user envd                                              0.0s
 => 💽 (cached) add user envd to sudoers                                      0.0s
 => 💽 (cached) (user-defined packages) apt-get install screenfetch           0.0s
 => 💽 (cached) install system packages                                       0.0s
 => 💽 (cached) pip install jupyter                                           0.0s
 => 💽 (cached) install PyPI packages                                         0.0s
 => 💽 (cached) install envd-ssh                                              0.0s
 => 💽 (cached) install vscode plugin ms-python.python                        0.0s
 => 💽 (cached) copy /oh-my-zsh /home/envd/.oh-my-zsh                         0.0s
 => 💽 (cached) mkfile /home/envd/install.sh                                  0.0s
 => 💽 (cached) install oh-my-zsh                                             0.0s
...
# You are in the docker container for dev
(envd 🐳)  ➜  mnist-dev git:(master) python3 ./main.py
...

Jupyter notebook service and sshd server are running inside the container. You can use jupyter or vscode remote-ssh extension to develop AI/ML models.

$ envd ls
NAME            JUPYTER                 SSH TARGET      GPU     STATUS  CONTAINER ID 
mnist           http://localhost:8888   mnist.envd      true    running 253f656b8c40

Features

Configure mirrors

envd supports PyPI mirror and apt source configuration. You can configure them in build.env or $HOME/.config/envd/config.envd to set up in all environments.

cat ~/.config/envd/config.envd
ubuntu_apt(source="""
deb https://mirror.sjtu.edu.cn/ubuntu focal main restricted
deb https://mirror.sjtu.edu.cn/ubuntu focal-updates main restricted
deb https://mirror.sjtu.edu.cn/ubuntu focal universe
deb https://mirror.sjtu.edu.cn/ubuntu focal-updates universe
deb https://mirror.sjtu.edu.cn/ubuntu focal multiverse
deb https://mirror.sjtu.edu.cn/ubuntu focal-updates multiverse
deb https://mirror.sjtu.edu.cn/ubuntu focal-backports main restricted universe multiverse
deb http://archive.canonical.com/ubuntu focal partner
deb https://mirror.sjtu.edu.cn/ubuntu focal-security main restricted universe multiverse
""")
pip_mirror(mirror = "https://mirror.sjtu.edu.cn/pypi/web/simple")
vscode(plugins = [
    "ms-python.python",
    "github.copilot"
])

Join Us

envd is backed by TensorChord and licensed under Apache-2.0. We are actively hiring engineers to build developer tools for machine learning practitioners in open source.

Contribute

We welcome all kinds of contributions from the open-source community, individuals, and partners.

• Join our discord community!
• To build from the source, check the contributing page.