model-mesh-triton-adapter

README

Model Mesh Triton Adapter

This is an adapter which implements the internal model-mesh model management API for Triton Inference Server.

How to

1. Clone the repository

   $ git clone https://github.com/kserve/modelmesh-runtime-adapter.git
   $ cd modelmesh-runtime-adapter/model-mesh-triton-adapter
   

2. Pull Triton Serving Docker Image

   $ docker pull nvcr.io/nvidia/tritonserver:20.09-py3
   

3. Run Triton Serving Container with model data mounted

   By default, Triton Serving Docker expose Port 8000 for HTTP and Port 8001 for gRPC.

   Using following command to forward container's 8000 to your workstation's 8000 and container's 8001 to your workstation's 8001.

   $ docker run -p 8000:8000 -p 8001:8001 -v $(pwd)/examples/models:/models nvcr.io/nvidia/tritonserver:20.09-py3 tritonserver --model-store=/models --model-control-mode=explicit --strict-model-config=false --strict-readiness=false
   

4. Setup your Golang, gRPC and Protobuff Development Environment locally

   Follow this gRPC Go Quick Start Guide

5. Run Triton adapter with:

   $ export ROOT_MODEL_DIR=$(pwd)/examples/models
   $ export CONTAINER_MEM_REQ_BYTES=268435456 # 256MB
   $ go run main.go
   

6. Test adapter with this client from another terminal:

   $ go run triton/adapter_client/adapter_client.go
   

Documentation

Overview

Copyright 2021 IBM Corporation

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.