llmos

README

LLMOS

LLMOS is an open-source, cloud-native infrastructure software tailored for managing AI applications and Large Language Models(LLMs).

Key Features

• Easy Installation: Seamlessly install on x86_64 or ARM64 architectures for an out-of-the-box user experience.
• Comprehensive LLM Lifecycle Management: A unified interface for both developers and non-developers to manage LLM infrastructure, ML Clusters, models and workloads.
• Easy to Use: Build models and AI applications in your own way, without the need to managing Kubernetes & infrastructure directly.
• Optimized for Edge & Branch Deployment: Enhance resource efficiency and simplify model deployment in edge and branch networks, while maintaining the capability to scale horizontally for larger workloads.

Quick Start

Installation Script

LLMOS can be installed to a bare-metal server or a virtual machine. To bootstrap a new cluster, follow the steps below:

curl -sfL https://get-llmos.1block.ai | sh -s - --cluster-init --token mytoken

To monitor installation logs, run journalctl -u llmos -f.

After installation, you may optionally add a worker node to the cluster with the following command:

curl -sfL https://get-llmos.1block.ai | LLMOS_SERVER=https://server-url:6443 LLMOS_TOKEN=mytoken sh -s -

Config Proxy

If your environment requires internet access through a proxy, set the HTTP_PROXY and HTTPS_PROXY environment variables before running the installation script:

export HTTP_PROXY=http://proxy.example.com:8080
export HTTPS_PROXY=http://proxy.example.com:8080
export NO_PROXY=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 # Replace the CIDRs with your own

Getting Started

After installing LLMOS, access the dashboard by navigating to https://<server-ip>:8443 in your web browser.

1. LLMOS will create a default admin user with a randomly generated password. To retrieve the password, run the following command on the cluster-init node:

   kubectl get secret --namespace llmos-system llmos-bootstrap-passwd -o go-template='{{.data.password|base64decode}}{{"\n"}}'
   

   
2. Upon logging in, you will be redirected to the setup page. Configure the following:
   • Set a new password for the admin user (strong passwords are recommended).
   • Configure the server URL that all other nodes in your cluster will use to connect.
3. After setup, you will be redirected to the home page where you can start using LLMOS.

More Examples

To learn more about using LLMOS, explore the following resources:

• Chat with LLMOS Models
• Creating a Machine Learning Cluster
• Creating a Jupyter Notebook

Documentation

Find more detailed documentation, visit here.

Community

If you're interested, please join us on Discord or participate in GitHub Discussions to discuss or contribute the project. We look forward to collaborating with you!

If you have any feedback or issues, feel free to file a GitHub issue.

License

Copyright (c) 2024 1Block.AI.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.