The simplest way to serve models in production

truss

Baseten

Releases

Truss examples

Truss

Learn model deployment

Reference docs

Contributor guides

GitHub

Roadmap

Issues

The simplest way to serve AI/ML models in production

Welcome to Truss

Create, deploy, and invoke an ML model server in less than 5 minutes

Quickstart

Get up and running quickly with Truss' developer workflow

User guide

Load a model that requires authentication with Hugging Face

Private Hugging Face model

Deploy a model with both Python and system dependencies

Model with system packages

LLM with streaming output

Deploy a model that makes use of pre-process

Pre/post-process methods

Enable fast cold starts for a model with private Hugging Face weights

Deploy Llama 2 with Caching

Serve LLM models with TGI

Serve models with VLLM

A guide to using secrets securely in your ML models

How to use secrets

A guide to configuring a base image for your truss

How to use base images

Accelerate cold starts by caching your weights

Caching model weights

A guide to setting concurrency for your model

How to configure concurrency

Deploy a Truss on Baseten

Deploy a Truss on SageMaker

Set up your development environment and create a Truss

Step 0: Create a Truss

Configure a remote host and push the base Truss

Step 1: Spin up a model server

Step 2: Call the model server

Patch the model server with local changes

Step 3: Enable live reload

Add required Python packages to the model server.

Step 4: Set Python requirements

Step 5: Implement model load

Add model inference and invoke the model server

Step 6: Implement model inference

Push your Truss to production-ready infrastructure

Step 7: Publish your Truss

Set your model resources, dependencies, and more

Config options

truss init

truss push

truss watch

truss predict

truss image

truss container

truss cleanup

Truss is MIT licensed and welcomes contributions

Truss reference

CLI reference

truss

Options

Main usage

truss init

truss push

truss watch

truss predict

Advanced usage

truss image

truss container

truss cleanup

Truss reference

CLI reference

​Options

​Main usage

truss init

truss push

truss watch

truss predict

​Advanced usage

truss image

truss container

truss cleanup

Options

Main usage

Advanced usage