Config options
Truss is configurable to its core. Every Truss must include a file config.yaml in its root directory, which is automatically generated when the Truss is created. However, configuration is optional. Every configurable value has a sensible default, and a completely empty config file is valid.
Config reference
Describe your model for documentation purposes.
Do not store secret values directly in environment variables (or anywhere in the config file). See the secrets arg for information on properly managing secrets.
Any environment variables can be provided here as key value pairs and are exposed to the environment that the model executes in. Many Python libraries can be customized using environment variables, so this field can be quite handy in those scenarios.
environment_variables:
ENVIRONMENT: Staging
DB_URL: https://my_database.example.com/
Set any additional metadata in this catch-all field. The entire contents of the config file are available to the model at runtime, so this is a good place to store any custom information that model needs. For example, scikit-learn models include a flag here that indicates whether the model supports returning probabilities alongside predictions.
model_metadata:
supports_predict_proba: true
The model’s name, for documentation purposes.
List the Python dependencies that the model depends on. The requirements should be provided in the pip requirements file format, but as a yaml list.
We strongly recommend pinning versions in your requirements.
requirements:
- scikit-learn==1.0.2
- threadpoolctl==3.0.0
- joblib==1.1.0
- numpy==1.20.3
- scipy==1.7.3
Specify model server runtime resources such as CPU, RAM and GPU.
resources:
cpu: "3"
memory: 14Gi
use_gpu: true
accelerator: A10G
This field can be used to specify the keys for such secrets and dummy default values. Never store actual secret values in the config. Dummy default values are instructive of what the actual values look like and thus act as documentation of the format.
A model may depend on certain secret values that can’t be bundled with the model and need to be bound securely at runtime. For example, a model may need to download information from s3 and may need access to AWS credentials for that.
secrets:
hf_access_token: "ACCESS TOKEN"
Specify any system packages that you would typically install using apt on a Debian operating system.
system_packages:
- ffmpeg
- libsm6
- libxext6
description: An instruction-following LLM Using Evol-Instruct.
environment_variables: {}
model_name: WizardLM
requirements:
- accelerate==0.20.3
- bitsandbytes==0.39.1
- peft==0.3.0
- protobuf==4.23.3
- sentencepiece==0.1.99
- torch==2.0.1
- transformers==4.30.2
resources:
cpu: "3"
memory: 14Gi
use_gpu: true
accelerator: A10G
secrets: {}
system_packages: []
description: An instruction-following LLM Using Evol-Instruct.
environment_variables: {}
model_name: WizardLM
requirements:
- accelerate==0.20.3
- bitsandbytes==0.39.1
- peft==0.3.0
- protobuf==4.23.3
- sentencepiece==0.1.99
- torch==2.0.1
- transformers==4.30.2
resources:
cpu: "3"
memory: 14Gi
use_gpu: true
accelerator: A10G
secrets: {}
system_packages: []