developer_uid: chai_backend_admin
submission_id: chaiml-grpo-q235b-kimid_83709_v4
model_name: chaiml-grpo-q235b-kimid_83709_v4
model_group: ChaiML/grpo-q235b-kimid-
status: inactive
timestamp: 2026-03-02T19:57:16+00:00
num_battles: 16752
num_wins: 9071
celo_rating: 1322.98
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-q235b-kimid-v5a-merged-chai-rm-step-450
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-q235b-kimid_83709_v4
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-q235b-kimid-v5a-merged-chai-rm-step-450
model_size: 19B
ranking_group: single
us_pacific_date: 2026-03-02
win_ratio: 0.5414875835721108
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['####', '</think>', '</s>', '<|im_end|>', '<|user|>', '<|assistant|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-q235b-kimid-83709-v4-uploader
Waiting for job on chaiml-grpo-q235b-kimid-83709-v4-uploader to finish
chaiml-grpo-q235b-kimid-83709-v4-uploader: Using quantization_mode: w4a16
chaiml-grpo-q235b-kimid-83709-v4-uploader: Checking if ChaiML/grpo-q235b-kimid-v5a-merged-chai-rm-step-450-W4A16 already exists in ChaiML
chaiml-grpo-q235b-kimid-83709-v4-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-grpo-q235b-kimid-83709-v4-uploader: Downloading snapshot of ChaiML/grpo-q235b-kimid-v5a-merged-chai-rm-step-450-W4A16...
chaiml-grpo-q235b-kimid-83709-v4-uploader: Downloaded in 68.694s
chaiml-grpo-q235b-kimid-83709-v4-uploader: Processed model ChaiML/grpo-q235b-kimid-v5a-merged-chai-rm-step-450 in 69.246s
chaiml-grpo-q235b-kimid-83709-v4-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-q235b-kimid-83709-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-83709-v4-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-q235b-kimid-83709-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-q235b-kimid-83709-v4-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-q235b-kimid-83709-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-83709-v4-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-q235b-kimid-83709-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-83709-v4-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-q235b-kimid-83709-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-83709-v4-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-q235b-kimid-83709-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-83709-v4-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-q235b-kimid-83709-v4-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-q235b-kimid-83709-v4-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-q235b-kimid-83709-v4-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-q235b-kimid-83709-v4-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-q235b-kimid-83709-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-q235b-kimid-83709-v4-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default
chaiml-grpo-q235b-kimid-83709-v4-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/added_tokens.json
chaiml-grpo-q235b-kimid-83709-v4-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/tokenizer_config.json
chaiml-grpo-q235b-kimid-83709-v4-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/generation_config.json
chaiml-grpo-q235b-kimid-83709-v4-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/config.json
chaiml-grpo-q235b-kimid-83709-v4-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/chat_template.jinja
chaiml-grpo-q235b-kimid-83709-v4-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/special_tokens_map.json
chaiml-grpo-q235b-kimid-83709-v4-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/.gitattributes
chaiml-grpo-q235b-kimid-83709-v4-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/merges.txt
chaiml-grpo-q235b-kimid-83709-v4-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/quantization_config.json
chaiml-grpo-q235b-kimid-83709-v4-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/vocab.json
chaiml-grpo-q235b-kimid-83709-v4-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model.safetensors.index.json
chaiml-grpo-q235b-kimid-83709-v4-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/tokenizer.json
chaiml-grpo-q235b-kimid-83709-v4-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00027-of-00027.safetensors
chaiml-grpo-q235b-kimid-83709-v4-uploader: unable to retrieve container logs for containerd://4842ee1e7fdcfdaa2d0b7
Job chaiml-grpo-q235b-kimid-83709-v4-uploader completed after 143.54s with status: failed
Job failed chaiml-grpo-q235b-kimid-83709-v4-uploader: 
Stopping job with name chaiml-grpo-q235b-kimid-83709-v4-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-grpo-q235b-kimid-83709-v4-uploader
Waiting for job on chaiml-grpo-q235b-kimid-83709-v4-uploader to finish
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
Job failed chaiml-grpo-q235b-kimid-83709-v4-uploader: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Date': 'Mon, 02 Mar 2026 19:01:34 GMT', 'Content-Type': 'application/json', 'Content-Length': '272', 'Connection': 'keep-alive', 'audit-id': '91bad546-a85b-4854-98c6-77b38bb18ca0', 'Cache-Control': 'no-cache, private', 'x-kubernetes-pf-flowschema-uid': '757159af-7462-4610-a5a7-751d984daa90', 'x-kubernetes-pf-prioritylevel-uid': '5f6bf145-f133-455b-a330-c9bf3e5ea771', 'cf-cache-status': 'DYNAMIC', 'set-cookie': '__cf_bm=-1772478094.7431731-1.0.1.1-kfWdOO.8h0tFsS8_.RFbmOuHziD48bstPYc2vHbW90yPjRRXgfCBn1Rz.qlxtfS3hCbXCvJ3QFIzK8hpxws6fOADpfwMgTov82XJN7zPBjmI.14nCiMT4MiOMbOZ_urJ; HttpOnly; Secure; Path=/; Domain=coreweave.com; Expires=Mon, 02 Mar 2026 19:31:34 GMT', 'Server': 'cloudflare', 'CF-RAY': '9d62a69c2fe839ac-IAD'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch \"chaiml-grpo-q235b-kimid-83709-v4-uploader\" not found","reason":"NotFound","details":{"name":"chaiml-grpo-q235b-kimid-83709-v4-uploader","group":"batch","kind":"jobs"},"code":404}


Stopping job with name chaiml-grpo-q235b-kimid-83709-v4-uploader
%s, retrying in %s seconds...
Stopping job with name chaiml-grpo-q235b-kimid-83709-v4-uploader
%s, retrying in %s seconds...
Stopping job with name chaiml-grpo-q235b-kimid-83709-v4-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-grpo-q235b-kimid-83709-v4-uploader
Waiting for job on chaiml-grpo-q235b-kimid-83709-v4-uploader to finish
chaiml-grpo-q235b-kimid-83709-v4-uploader: Using quantization_mode: w4a16
chaiml-grpo-q235b-kimid-83709-v4-uploader: Checking if ChaiML/grpo-q235b-kimid-v5a-merged-chai-rm-step-450-W4A16 already exists in ChaiML
chaiml-grpo-q235b-kimid-83709-v4-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-grpo-q235b-kimid-83709-v4-uploader: Downloading snapshot of ChaiML/grpo-q235b-kimid-v5a-merged-chai-rm-step-450-W4A16...
chaiml-grpo-q235b-kimid-83709-v4-uploader: Downloaded in 56.979s
chaiml-grpo-q235b-kimid-83709-v4-uploader: Processed model ChaiML/grpo-q235b-kimid-v5a-merged-chai-rm-step-450 in 57.600s
chaiml-grpo-q235b-kimid-83709-v4-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-q235b-kimid-83709-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-83709-v4-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-q235b-kimid-83709-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-q235b-kimid-83709-v4-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-q235b-kimid-83709-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-83709-v4-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-q235b-kimid-83709-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-83709-v4-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-q235b-kimid-83709-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-83709-v4-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-q235b-kimid-83709-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-83709-v4-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-q235b-kimid-83709-v4-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-q235b-kimid-83709-v4-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-q235b-kimid-83709-v4-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-q235b-kimid-83709-v4-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-q235b-kimid-83709-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-q235b-kimid-83709-v4-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/.gitattributes": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/added_tokens.json": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/chat_template.jinja": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/config.json": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/generation_config.json": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/merges.txt": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00001-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00002-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00003-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00004-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00005-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00006-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00007-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00008-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00009-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00010-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00011-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00012-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00013-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00014-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00015-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00016-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00017-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00018-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00019-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00020-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00021-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00022-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00023-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00024-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00025-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00026-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model-00027-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/model.safetensors.index.json": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/quantization_config.json": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/special_tokens_map.json": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/tokenizer.json": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/tokenizer_config.json": object size matches
chaiml-grpo-q235b-kimid-83709-v4-uploader: DEBUG "sync /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-83709-v4/default/vocab.json": object size matches
Job chaiml-grpo-q235b-kimid-83709-v4-uploader completed after 115.39s with status: succeeded
Stopping job with name chaiml-grpo-q235b-kimid-83709-v4-uploader
Pipeline stage VLLMUploader completed in 514.67s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.23s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-q235b-kimid-83709-v4
Waiting for inference service chaiml-grpo-q235b-kimid-83709-v4 to be ready
Retrying (%r) after connection broken by '%r': %s
Inference service chaiml-grpo-q235b-kimid-83709-v4 ready after 401.62444257736206s
Pipeline stage VLLMDeployer completed in 402.36s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.2191569805145264s
Received healthy response to inference request in 2.1919734477996826s
Received healthy response to inference request in 2.2828152179718018s
Received healthy response to inference request in 2.0086119174957275s
Received healthy response to inference request in 1.818432331085205s
Received healthy response to inference request in 2.1019887924194336s
Received healthy response to inference request in 2.2218449115753174s
Received healthy response to inference request in 2.226229667663574s
Received healthy response to inference request in 2.135284185409546s
Received healthy response to inference request in 2.0635244846343994s
Received healthy response to inference request in 1.9167559146881104s
Received healthy response to inference request in 2.2117433547973633s
Received healthy response to inference request in 2.042443037033081s
Received healthy response to inference request in 2.1072065830230713s
Received healthy response to inference request in 2.0447874069213867s
Received healthy response to inference request in 2.158207654953003s
Received healthy response to inference request in 2.52671217918396s
Received healthy response to inference request in 2.053189516067505s
Received healthy response to inference request in 2.220147132873535s
Received healthy response to inference request in 2.322916269302368s
Received healthy response to inference request in 2.387645959854126s
Received healthy response to inference request in 2.1039376258850098s
Received healthy response to inference request in 2.0495097637176514s
Received healthy response to inference request in 2.248958110809326s
Received healthy response to inference request in 2.081650495529175s
Received healthy response to inference request in 2.309199333190918s
Received healthy response to inference request in 2.0220019817352295s
Received healthy response to inference request in 1.9411439895629883s
Received healthy response to inference request in 2.0863537788391113s
Received healthy response to inference request in 2.20746111869812s
30 requests
0 failed requests
5th percentile: 1.9277305483818055
10th percentile: 2.0018651247024537
20th percentile: 2.0443185329437257
30th percentile: 2.060423994064331
40th percentile: 2.095734786987305
50th percentile: 2.1212453842163086
60th percentile: 2.1981685161590576
70th percentile: 2.219454026222229
80th percentile: 2.2307753562927246
90th percentile: 2.310571026802063
95th percentile: 2.3585175991058347
99th percentile: 2.4863829755783082
mean time: 2.143727771441142
Pipeline stage StressChecker completed in 84.67s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.69s
Shutdown handler de-registered
chaiml-grpo-q235b-kimid_83709_v4 status is now deployed due to DeploymentManager action
chaiml-grpo-q235b-kimid_83709_v4 status is now inactive due to auto deactivation removed underperforming models