chaiml-grpo-glm50-combi_56686

developer_uid: chai_backend_admin
submission_id: chaiml-grpo-glm50-combi_56686_v1
model_name: chaiml-grpo-glm50-combi_56686_v1
model_group: ChaiML/grpo-glm50-combin
status: torndown
timestamp: 2026-04-03T09:21:59+00:00
num_battles: 10375
num_wins: 5868
celo_rating: 1356.03
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step200-int4-fix1
model_architecture: GlmMoeDsaForCausalLM
model_num_parameters: 31348144128.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm50-combi_56686_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step200-int4-fix1
model_size: 31B
ranking_group: single
us_pacific_date: 2026-03-31
win_ratio: 0.5655903614457831
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '####', '</s>', '<|assistant|>', '<|user|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm50-combi-56686-v1-uploader
Waiting for job on chaiml-grpo-glm50-combi-56686-v1-uploader to finish
chaiml-grpo-glm50-combi-56686-v1-uploader: {repo_id} is already quantized
chaiml-grpo-glm50-combi-56686-v1-uploader: Using quantization_mode: none
chaiml-grpo-glm50-combi-56686-v1-uploader: Downloading snapshot of ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step200-int4-fix1...
2026-03-31T02:11:36.724980+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
2026-03-31T02:12:36.876397+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
2026-03-31T02:13:36.998730+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
2026-03-31T02:14:37.164822+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
chaiml-grpo-glm50-combi-56686-v1-uploader: Downloaded in 211.918s
2026-03-31T02:15:37.253558+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
2026-03-31T02:16:37.439734+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
chaiml-grpo-glm50-combi-56686-v1-uploader: Processed model ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step200-int4-fix1 in 347.203s
chaiml-grpo-glm50-combi-56686-v1-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm50-combi-56686-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-56686-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm50-combi-56686-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm50-combi-56686-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm50-combi-56686-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-56686-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-56686-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-56686-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-56686-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-56686-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-56686-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-56686-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-56686-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm50-combi-56686-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm50-combi-56686-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm50-combi-56686-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm50-combi-56686-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm50-combi-56686-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/chat_template.jinja
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/.gitattributes
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/generation_config.json
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/config.json
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/tokenizer.json
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model.safetensors.index.json
2026-03-31T02:17:37.526056+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00075-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00075-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00004-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00004-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00021-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00021-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00018-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00018-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00044-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00044-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00055-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00055-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00035-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00035-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00024-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00024-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00041-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00041-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00057-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00057-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00038-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00038-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00033-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00033-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00007-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00007-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00056-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00056-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00005-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00005-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00011-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00011-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00036-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00036-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00045-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00045-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00037-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00037-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00016-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00016-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00012-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00012-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00030-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00030-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00013-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00013-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00010-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00010-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00020-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00020-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00002-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00002-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00028-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00028-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00019-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00019-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00031-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00031-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00048-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00048-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00032-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00032-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00074-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00074-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00043-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00043-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00008-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00008-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00046-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00046-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00027-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00027-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00001-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00001-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00015-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00015-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00051-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00051-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00053-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00053-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00009-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00009-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00025-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00025-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00060-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00060-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00017-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00017-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00014-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00014-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00022-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00022-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00047-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00047-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00042-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00042-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00062-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00062-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00066-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00066-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00052-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00052-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00040-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00040-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00072-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00072-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00034-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00034-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00067-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00067-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00039-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00039-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00061-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00061-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00003-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00003-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00006-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00006-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00026-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00026-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00070-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00070-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00071-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00071-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00063-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00063-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00023-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00023-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00050-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00050-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00069-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00069-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00054-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00054-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00058-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00058-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00049-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00049-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00059-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00059-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00029-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00029-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00073-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00073-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00064-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00064-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00068-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00068-of-00075.safetensors
chaiml-grpo-glm50-combi-56686-v1-uploader: cp /dev/shm/model_output/model-00065-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-56686-v1/default/model-00065-of-00075.safetensors
2026-03-31T02:18:37.619063+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
Job chaiml-grpo-glm50-combi-56686-v1-uploader completed after 494.15s with status: succeeded
Stopping job with name chaiml-grpo-glm50-combi-56686-v1-uploader
Pipeline stage VLLMUploader completed in 494.73s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.22s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.97s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm50-combi-56686-v1
Waiting for inference service chaiml-grpo-glm50-combi-56686-v1 to be ready
2026-03-31T02:19:37.725690+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
2026-03-31T02:20:37.850561+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
2026-03-31T02:21:38.443963+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
2026-03-31T02:22:38.615225+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
2026-03-31T02:23:38.747402+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
2026-03-31T02:24:38.922021+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
Failed to get request counts for guanaco-submitter. Falling back to default
2026-03-31T02:25:39.073059+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
2026-03-31T02:26:39.221615+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
Inference service chaiml-grpo-glm50-combi-56686-v1 ready after 471.25996136665344s
Pipeline stage VLLMDeployer completed in 472.21s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 11.477252006530762s
Received healthy response to inference request in 10.901255369186401s
Received healthy response to inference request in 2.539469003677368s
Received healthy response to inference request in 2.8131935596466064s
Received healthy response to inference request in 3.3983840942382812s
Received healthy response to inference request in 2.7972238063812256s
Received healthy response to inference request in 3.035301446914673s
Received healthy response to inference request in 3.0191690921783447s
Received healthy response to inference request in 2.7803125381469727s
Received healthy response to inference request in 2.72405743598938s
Received healthy response to inference request in 2.8544044494628906s
2026-03-31T02:27:39.313058+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
Received healthy response to inference request in 2.843510389328003s
Received healthy response to inference request in 2.7318179607391357s
Received healthy response to inference request in 3.2292139530181885s
Received healthy response to inference request in 2.8682353496551514s
Received healthy response to inference request in 3.277320623397827s
Received healthy response to inference request in 2.981687307357788s
Received healthy response to inference request in 2.9126927852630615s
Received healthy response to inference request in 2.980973482131958s
Received healthy response to inference request in 2.934044122695923s
Received healthy response to inference request in 2.87520170211792s
Received healthy response to inference request in 2.6983258724212646s
Received healthy response to inference request in 2.784043788909912s
Received healthy response to inference request in 2.81868052482605s
Received healthy response to inference request in 2.760061740875244s
Received healthy response to inference request in 3.2569191455841064s
Received healthy response to inference request in 3.1522295475006104s
Received healthy response to inference request in 2.876056671142578s
Received healthy response to inference request in 2.8437769412994385s
Received healthy response to inference request in 3.0478100776672363s
30 requests
0 failed requests
5th percentile: 2.7099050760269163
10th percentile: 2.73104190826416
20th percentile: 2.783297538757324
30th percentile: 2.817034435272217
40th percentile: 2.8501534461975098
50th percentile: 2.875629186630249
60th percentile: 2.952815866470337
70th percentile: 3.024008798599243
80th percentile: 3.1676264286041262
90th percentile: 3.289426970481873
95th percentile: 7.524963295459726
99th percentile: 11.310212981700898
mean time: 3.4737541596094768
Pipeline stage StressChecker completed in 110.88s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
2026-03-31T02:28:39.409170+00:00 monitor updated for chaiml-grpo-glm50-combi_56686_v1
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.89s
Shutdown handler de-registered
chaiml-grpo-glm50-combi_56686_v1 status is now deployed due to DeploymentManager action
chaiml-grpo-glm50-combi_56686_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-grpo-glm50-combi_56686_v1 status is now torndown due to DeploymentManager action