Optimizing Model Deployments with Triton Model Analyzer

How do you identify the batch size and number of model instances for the optimal inference performance? Triton Model Analyzer is an offline tool that can be used to evaluate 100’s of configurations to meet the latency, throughput & memory requirements of your application.
Get started with model analyzer here: github.com/tri...
#Triton #Inference #ModelAnalyzer #AI

[@razalminhas6349]

Awesome overview. Better than a 100 presentations to make you understand what Triton Inference Server vs. Model Analyzer are.

[@siddharthsharma4072]

I was facing issues when I followed the steps specified in the video. Seems like along with model repository we would need to bind the config volume e.g.
docker run -it --rm --gpus all -v /var/run/docker.sock:/var/run/docker.sock \
-v /home/ec2-user/SageMaker/workspace/model_repository:/models \
-v /home/ec2-user/SageMaker/workspace/output:/output \
-v /home/ec2-user/SageMaker/workspace/model_config:/config \
--net=host model-analyzer
if we don't bind config volume then model analyzer would throw error "can't find config ..."

[@qfz3711758]

Nice video👍👍👍

[@ibrahimgul9716]

hi , thanks for this video and i got error such as below. do u have any suggestion ?

[@ibrahimgul9716]

Model add_sub_config_3 load failed: [StatusCode.UNAVAILABLE] explicit model load / unload is not allowed if polling is enabled
[Model Analyzer]
[Model Analyzer] Creating model config: add_sub_config_4
[Model Analyzer] Enabling dynamic_batching
[Model Analyzer] Setting instance_group to [{'count': 5, 'kind': 'KIND_GPU'}]

[@nneeerrrd]

Echo 🤦‍♂️