chaiml-ssnew-v5-dpo-lr5_19068

developer_uid: chai_backend_admin

submission_id: chaiml-ssnew-v5-dpo-lr5_19068_v4

model_name: chaiml-ssnew-v5-dpo-lr5_19068_v4

model_group: ChaiML/ssnew-v5-dpo-lr5e

status: torndown

timestamp: 2026-02-10T20:41:30+00:00

num_battles: 10792

num_wins: 5244

celo_rating: 1297.61

family_friendly_score: 0.0

family_friendly_standard_error: 0.0

submission_type: basic

model_repo: ChaiML/ssnew-v5-dpo-lr5e6b01-lora-FP8

model_architecture: MistralForCausalLM

model_num_parameters: 24096691200.0

best_of: 8

max_input_tokens: 2048

max_output_tokens: 64

reward_model: default

display_name: chaiml-ssnew-v5-dpo-lr5_19068_v4

is_internal_developer: True

language_model: ChaiML/ssnew-v5-dpo-lr5e6b01-lora-FP8

model_size: 24B

ranking_group: single

us_pacific_date: 2026-02-07

win_ratio: 0.4859154929577465

generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['####', '\n', 'You:', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': "{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': 'You: {message}\n', 'response_template': '####\n{bot_name}:', 'truncate_by_message': True}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader
Waiting for job on chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader to finish
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: Using quantization_mode: none
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: Downloading snapshot of ChaiML/ssnew-v5-dpo-lr5e6b01-lora-FP8...
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: 
Fetching 14 files:   0%|          | 0/14 [00:00<?, ?it/s]
Fetching 14 files:   7%|▋         | 1/14 [00:00<00:03,  3.82it/s]
Fetching 14 files:  29%|██▊       | 4/14 [00:11<00:29,  2.96s/it]
Fetching 14 files: 100%|██████████| 14/14 [00:11<00:00,  1.27it/s]
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: Downloaded in 11.159s
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: Processed model ChaiML/ssnew-v5-dpo-lr5e6b01-lora-FP8 in 20.428s
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: creating bucket guanaco-vllm-models
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-ssnew-v5-dpo-lr5-19068-v4
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-ssnew-v5-dpo-lr5-19068-v4/.gitattributes
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-ssnew-v5-dpo-lr5-19068-v4/config.json
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-ssnew-v5-dpo-lr5-19068-v4/special_tokens_map.json
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-ssnew-v5-dpo-lr5-19068-v4/generation_config.json
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-ssnew-v5-dpo-lr5-19068-v4/model.safetensors.index.json
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-ssnew-v5-dpo-lr5-19068-v4/tokenizer_config.json
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-ssnew-v5-dpo-lr5-19068-v4/recipe.yaml
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-ssnew-v5-dpo-lr5-19068-v4/tokenizer.json
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: cp /dev/shm/model_output/model-00006-of-00006.safetensors s3://guanaco-vllm-models/chaiml-ssnew-v5-dpo-lr5-19068-v4/model-00006-of-00006.safetensors
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: cp /dev/shm/model_output/model-00005-of-00006.safetensors s3://guanaco-vllm-models/chaiml-ssnew-v5-dpo-lr5-19068-v4/model-00005-of-00006.safetensors
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: cp /dev/shm/model_output/model-00002-of-00006.safetensors s3://guanaco-vllm-models/chaiml-ssnew-v5-dpo-lr5-19068-v4/model-00002-of-00006.safetensors
chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader: cp /dev/shm/model_output/model-00003-of-00006.safetensors s3://guanaco-vllm-models/chaiml-ssnew-v5-dpo-lr5-19068-v4/model-00003-of-00006.safetensors
Job chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader completed after 127.57s with status: succeeded
Stopping job with name chaiml-ssnew-v5-dpo-lr5-19068-v4-uploader
Pipeline stage VLLMUploader completed in 132.57s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.56s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-ssnew-v5-dpo-lr5-19068-v4
Waiting for inference service chaiml-ssnew-v5-dpo-lr5-19068-v4 to be ready
admin requested tearing down of chaiml-gy-exp239-sft-gy_88241_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1419b998-274d-4f9e-a643-515c96561a99\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 07 Feb 2026 18:24:56 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
Inference service chaiml-ssnew-v5-dpo-lr5-19068-v4 ready after 422.134165763855s
Pipeline stage VLLMDeployer completed in 424.95s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.4900081157684326s
Received healthy response to inference request in 1.8822681903839111s
Received healthy response to inference request in 1.6385388374328613s
Received healthy response to inference request in 2.041226863861084s
Received healthy response to inference request in 2.593950033187866s
Received healthy response to inference request in 1.9433543682098389s
Received healthy response to inference request in 2.129180669784546s
Received healthy response to inference request in 1.789567232131958s
Received healthy response to inference request in 1.5140361785888672s
Received healthy response to inference request in 1.5358765125274658s
Received healthy response to inference request in 1.4051201343536377s
Received healthy response to inference request in 1.7709338665008545s
Received healthy response to inference request in 1.4329473972320557s
Received healthy response to inference request in 1.4228508472442627s
Received healthy response to inference request in 1.457221269607544s
Received healthy response to inference request in 1.4482495784759521s
Received healthy response to inference request in 1.3768196105957031s
Received healthy response to inference request in 1.779705286026001s
Received healthy response to inference request in 1.7815167903900146s
Received healthy response to inference request in 1.7927219867706299s
Received healthy response to inference request in 1.3872020244598389s
Received healthy response to inference request in 1.571000099182129s
Received healthy response to inference request in 1.739290475845337s
Received healthy response to inference request in 1.539438009262085s
Received healthy response to inference request in 1.3587236404418945s
Received healthy response to inference request in 1.5361409187316895s
Received healthy response to inference request in 1.5669920444488525s
Received healthy response to inference request in 1.4416508674621582s
Received healthy response to inference request in 1.3334705829620361s
Received healthy response to inference request in 1.5656969547271729s
30 requests
0 failed requests
5th percentile: 1.3668668270111084
10th percentile: 1.3861637830734252
20th percentile: 1.4309280872344972
30th percentile: 1.4545297622680664
40th percentile: 1.5271403789520264
50th percentile: 1.552567481994629
60th percentile: 1.5980155944824217
70th percentile: 1.7735652923583984
80th percentile: 1.7901981830596925
90th percentile: 1.9531416177749634
95th percentile: 2.0896014571189876
99th percentile: 2.459166917800904
mean time: 1.6421899795532227
Pipeline stage StressChecker completed in 62.05s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.62s
Shutdown handler de-registered
chaiml-ssnew-v5-dpo-lr5_19068_v4 status is now deployed due to DeploymentManager action
chaiml-ssnew-v5-dpo-lr5_19068_v4 status is now inactive due to auto deactivation removed underperforming models
chaiml-ssnew-v5-dpo-lr5_19068_v4 status is now torndown due to DeploymentManager action