How to use base images

Model serving enviroments will often be standardized as container images to avoid wrangling python, system, and other requirements needed to run your model on every deploy. Leverage your existing container artifacts by bringing your own base image to Truss.

While the default image for the truss server in adequate for most use cases, there may come a time when you require a custom base image for your truss. For example, maybe the python packages required for you project are not compatible with the ones installed by default on Truss. In a situation like this, you can create your own base image based on the default truss image.

​

Setting a base image in config.yaml

To specify a base image to build a truss container image from in your config.yaml configure a base_image.

base_image:
  image: <image_name:tag>
  python_executable_path: <path-to-python>

where python_executable_path is a path to a python executable with which to run your server.

​

Example usage

This example truss demonstrates how to properly configure a base image for Nvidia NeMo TitaNet:

base_image:
  image: nvcr.io/nvidia/nemo:23.03
  python_executable_path: /usr/bin/python
apply_library_patches: true
bundled_packages_dir: packages
data_dir: data
requirements:
- PySoundFile
live_reload: false
resources:
  accelerator: T4
  cpu: 2500m
  memory: 4512Mi
  use_gpu: true
secrets: {}
spec_version: '2.0'
system_packages:
- python3.8-venv

​

Configuring private base images with build time secrets

Secrets of the form DOCKER_REGISTRY_<REGISTRY_URL> will be supplied to your model build to authenticate image pulls from private container registries. For information on where to store secret values see the secrets guide.

For example, to configure docker credentials to a private dockerhub repository your config.yaml should include the following secret and placeholder:

secrets:
  DOCKER_REGISTRY_https://index.docker.io/v1/: null

along with a configured Baseten secret DOCKER_REGISTRY_https://index.docker.io/v1/ with a base64 encoded username:password secret value:

echo -n 'username:password' | base64

To add docker credentials for gcloud artifact registry provide an access token as the secret value. For example, to configure authentication for a repository in us-west2 your config.yaml should include the following secret and placeholder:

secrets:
  DOCKER_REGISTRY_us-west2-docker.pkg.dev: null

If you don’t have a base image but want to create one, the following section will guide you through that process.

​

Creating a custom base image

While it’s possible to create your own base image from scratch, it may be easier to use a truss server image as a starting point.

All of the base images used by truss can be found by going to docker hub. Each image tag is tied to a specific python version and may support a GPU. For example, the image with the tag 3.11-gpu-v0.7.16 is for Python 3.11 and has GPU support. On the other hand, image 3.9-v0.7.15 is for Python 3.9 and does not have GPU support. Based on your project requirements you can select the appropriate base image.

Next, we can write our Dockerfile.

FROM baseten/truss-server-base:3.11-gpu-v0.7.16
RUN pip uninstall cython -y
RUN pip install cython==0.29.30

In the example dockerfile above, we use the truss-server-base as our base image. This base image comes with some python dependencies installed. If you want to override the versions of these default python dependencies you can simply uninstall the package and reinstall it using the RUN pip install command. You can even add you own files or directories to this Dockerfile if required. Once you have your Dockerfile set up, you can build, tag, and push the image to your own docker registry.

​

Build, tag, and push your custom base image

To build the image run the command:

docker build -t my-custom-base-image:0.1 .

You can replace my-custom-base-image with the name for your docker image.

Next, to tag the image you can run the command:

docker tag my-custom-base-image:0.1 your-docker-username/my-custom-base-image:0.1

Lastly, to push the image to docker hub run the following command:

docker push your-docker-username/my-custom-base-image:0.1