developer_uid: chai_backend_admin
submission_id: chaiml-grpo-glm46-combi_30639_v2
model_name: chaiml-grpo-glm46-combi_30639_v2
model_group: ChaiML/grpo-glm46-combin
status: inactive
timestamp: 2026-03-12T08:57:43+00:00
num_battles: 11457
num_wins: 6194
celo_rating: 1322.68
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm46-combine-rm82-lr5e6-step400
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm46-combi_30639_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm46-combine-rm82-lr5e6-step400
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-12
win_ratio: 0.5406301824212272
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['####', '<|im_end|>', '</s>', '<|assistant|>', '<|user|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}.<|user|>\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm46-combi-30639-v2-uploader
Waiting for job on chaiml-grpo-glm46-combi-30639-v2-uploader to finish
chaiml-grpo-glm46-combi-30639-v2-uploader: Using quantization_mode: fp8
chaiml-grpo-glm46-combi-30639-v2-uploader: Checking if ChaiML/grpo-glm46-combine-rm82-lr5e6-step400-FP8 already exists in ChaiML
chaiml-grpo-glm46-combi-30639-v2-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-grpo-glm46-combi-30639-v2-uploader: Downloading snapshot of ChaiML/grpo-glm46-combine-rm82-lr5e6-step400-FP8...
2026-03-12T06:39:45.966365+00:00 monitor updated for chaiml-grpo-glm46-combi_30639_v2
2026-03-12T06:40:46.052579+00:00 monitor updated for chaiml-grpo-glm46-combi_30639_v2
chaiml-grpo-glm46-combi-30639-v2-uploader: Downloaded in 124.159s
chaiml-grpo-glm46-combi-30639-v2-uploader: Processed model ChaiML/grpo-glm46-combine-rm82-lr5e6-step400 in 127.666s
chaiml-grpo-glm46-combi-30639-v2-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm46-combi-30639-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-30639-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm46-combi-30639-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm46-combi-30639-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm46-combi-30639-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-30639-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm46-combi-30639-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-30639-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm46-combi-30639-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-30639-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm46-combi-30639-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-30639-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm46-combi-30639-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm46-combi-30639-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm46-combi-30639-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm46-combi-30639-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm46-combi-30639-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm46-combi-30639-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/generation_config.json
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/.gitattributes
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/config.json
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/chat_template.jinja
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/tokenizer_config.json
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/recipe.yaml
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model.safetensors.index.json
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/tokenizer.json
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/special_tokens_map.json
2026-03-12T06:41:46.143324+00:00 monitor updated for chaiml-grpo-glm46-combi_30639_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm46-combi-19477-v1-uploader
Waiting for job on chaiml-grpo-glm46-combi-19477-v1-uploader to finish
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00071-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00029-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00064-of-00072.safetensors
2026-03-12T06:42:46.233820+00:00 monitor updated for chaiml-grpo-glm46-combi_30639_v2
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00021-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00031-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00066-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00006-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00028-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00030-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00032-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00019-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00011-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00046-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00051-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00005-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00059-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00022-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00052-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00027-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00069-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00043-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00010-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00018-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00070-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00045-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00001-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00049-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00047-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00026-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00067-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00017-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00014-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00056-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00044-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00003-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00061-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00060-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00048-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00050-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00037-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00025-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00002-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00040-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00009-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00004-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00013-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00016-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00053-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00038-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00041-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00068-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00042-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00033-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00063-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00024-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00015-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00054-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00036-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00065-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00058-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00012-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00035-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00039-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00008-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00055-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00057-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00062-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00034-of-00072.safetensors
chaiml-grpo-glm46-combi-30639-v2-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-30639-v2/default/model-00007-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v1-uploader: Using quantization_mode: fp8
chaiml-grpo-glm46-combi-19477-v1-uploader: Checking if ChaiML/grpo-glm46-combine-rm82-lr5e6-step500-FP8 already exists in ChaiML
chaiml-grpo-glm46-combi-19477-v1-uploader: Downloading snapshot of ChaiML/grpo-glm46-combine-rm82-lr5e6-step500...
Job chaiml-grpo-glm46-combi-30639-v2-uploader completed after 260.48s with status: succeeded
Stopping job with name chaiml-grpo-glm46-combi-30639-v2-uploader
Pipeline stage VLLMUploader completed in 261.10s
run pipeline stage %s
Running pipeline stage VLLMTemplater
2026-03-12T06:43:07.653921+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v1
Pipeline stage VLLMTemplater completed in 0.54s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm46-combi-30639-v2
Waiting for inference service chaiml-grpo-glm46-combi-30639-v2 to be ready
2026-03-12T06:43:46.374430+00:00 monitor updated for chaiml-grpo-glm46-combi_30639_v2
2026-03-12T06:44:07.802596+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v1
2026-03-12T06:44:46.517160+00:00 monitor updated for chaiml-grpo-glm46-combi_30639_v2
2026-03-12T06:45:07.951338+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v1
2026-03-12T06:45:46.669316+00:00 monitor updated for chaiml-grpo-glm46-combi_30639_v2
2026-03-12T06:46:08.097540+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v1
2026-03-12T06:46:46.817646+00:00 monitor updated for chaiml-grpo-glm46-combi_30639_v2
chaiml-grpo-glm46-combi-19477-v1-uploader: Downloaded in 231.101s
chaiml-grpo-glm46-combi-19477-v1-uploader: Loading /tmp/model_input...
chaiml-grpo-glm46-combi-19477-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-grpo-glm46-combi-19477-v1-uploader: `torch_dtype` is deprecated! Use `dtype` instead!
2026-03-12T06:47:08.248758+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v1
admin requested tearing down of chaiml-grpo-glm46-chai-_90872_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMDeleter
Checking if service chaiml-grpo-glm46-chai-90872-v1 is running
Tearing down inference service chaiml-grpo-glm46-chai-90872-v1
Service chaiml-grpo-glm46-chai-90872-v1 has been torndown
Pipeline stage VLLMDeleter completed in 0.99s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/chat_template.jinja from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/config.json from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/generation_config.json from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00001-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00002-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00003-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00004-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00005-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00006-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00007-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00008-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00009-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00010-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00011-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00012-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00013-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00014-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00015-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00016-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00017-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00018-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00019-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00020-of-00072.safetensors from bucket guanaco-vllm-models
chaiml-grpo-glm46-combi-19477-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00021-of-00072.safetensors from bucket guanaco-vllm-models
chaiml-grpo-glm46-combi-19477-v1-uploader: Applying quantization...
chaiml-grpo-glm46-combi-19477-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-grpo-glm46-combi-19477-v1-uploader: 2026-03-11T23:47:34.501675-0700 | reset | INFO - Compression lifecycle reset
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00022-of-00072.safetensors from bucket guanaco-vllm-models
chaiml-grpo-glm46-combi-19477-v1-uploader: 2026-03-11T23:47:34.576707-0700 | from_modifiers | INFO - Creating recipe from modifiers
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00023-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00024-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00025-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00026-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00027-of-00072.safetensors from bucket guanaco-vllm-models
2026-03-12T06:47:46.963245+00:00 monitor updated for chaiml-grpo-glm46-combi_30639_v2
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00028-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00029-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00030-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00031-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00032-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00033-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00034-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00035-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00036-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00037-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00038-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00039-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00040-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00041-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00042-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00043-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00044-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00045-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00046-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00047-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00048-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00049-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00050-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00051-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00052-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00053-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00054-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00055-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00056-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00057-of-00072.safetensors from bucket guanaco-vllm-models
2026-03-12T06:48:08.395917+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v1
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00058-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00059-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00060-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00061-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00062-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00063-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00064-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00065-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00066-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00067-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00068-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00069-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00070-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00071-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model-00072-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/model.safetensors.index.json from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/recipe.yaml from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/special_tokens_map.json from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/tokenizer.json from bucket guanaco-vllm-models
Deleting key chaiml-grpo-glm46-chai-90872-v1/default/tokenizer_config.json from bucket guanaco-vllm-models
Pipeline stage VLLMModelDeleter completed in 52.91s
Shutdown handler de-registered
chaiml-grpo-glm46-chai-_90872_v1 status is now torndown due to DeploymentManager action
chaiml-grpo-glm46-combi-19477-v1-uploader: 2026-03-11T23:48:31.587234-0700 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-grpo-glm46-combi-19477-v1-uploader: 2026-03-11T23:48:31.587717-0700 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
2026-03-12T06:48:47.214132+00:00 monitor updated for chaiml-grpo-glm46-combi_30639_v2
chaiml-grpo-glm46-combi-19477-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
2026-03-12T06:49:08.690381+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v1
Inference service chaiml-grpo-glm46-combi-30639-v2 ready after 361.5427556037903s
Pipeline stage VLLMDeployer completed in 362.46s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 9.025322675704956s
Received healthy response to inference request in 9.5512113571167s
Received healthy response to inference request in 1.9885339736938477s
Received healthy response to inference request in 9.259127616882324s
Received healthy response to inference request in 1.997434377670288s
2026-03-12T06:49:47.400822+00:00 monitor updated for chaiml-grpo-glm46-combi_30639_v2
Received healthy response to inference request in 8.97835087776184s
Received healthy response to inference request in 1.9937200546264648s
Received healthy response to inference request in 2.3307278156280518s
Received healthy response to inference request in 1.944077491760254s
Received healthy response to inference request in 1.9899547100067139s
Received healthy response to inference request in 1.9915063381195068s
Received healthy response to inference request in 2.1973934173583984s
Received healthy response to inference request in 1.9664437770843506s
2026-03-12T06:50:08.873636+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v1
Received healthy response to inference request in 2.041032314300537s
Received healthy response to inference request in 1.9754433631896973s
Received healthy response to inference request in 2.0073957443237305s
Received healthy response to inference request in 2.255277395248413s
Received healthy response to inference request in 2.028397798538208s
Received healthy response to inference request in 2.030850648880005s
Received healthy response to inference request in 1.9871211051940918s
Received healthy response to inference request in 2.136726140975952s
Received healthy response to inference request in 2.0912983417510986s
Received healthy response to inference request in 1.9997997283935547s
Received healthy response to inference request in 2.2217235565185547s
Received healthy response to inference request in 1.9999651908874512s
Received healthy response to inference request in 2.0316617488861084s
Received healthy response to inference request in 2.0918006896972656s
Received healthy response to inference request in 2.1420071125030518s
Received healthy response to inference request in 2.048006534576416s
Received healthy response to inference request in 2.0879735946655273s
30 requests
0 failed requests
5th percentile: 1.9704935908317567
10th percentile: 1.9859533309936523
20th percentile: 1.9911960124969483
30th percentile: 1.9990901231765748
40th percentile: 2.019996976852417
50th percentile: 2.0363470315933228
60th percentile: 2.089303493499756
70th percentile: 2.138310432434082
80th percentile: 2.2284343242645264
90th percentile: 8.983048057556152
95th percentile: 9.153915393352507
2026-03-12T06:50:47.573897+00:00 monitor updated for chaiml-grpo-glm46-combi_30639_v2
99th percentile: 9.466507072448731
mean time: 3.013009516398112
Pipeline stage StressChecker completed in 97.19s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.08s
Shutdown handler de-registered
chaiml-grpo-glm46-combi_30639_v2 status is now deployed due to DeploymentManager action
chaiml-grpo-glm46-combi_30639_v2 status is now inactive due to auto deactivation removed underperforming models