developer_uid: chai_backend_admin
submission_id: chaiml-grpo-glm50-combi_43081_v2
model_name: chaiml-grpo-glm50-combi_43081_v2
model_group: ChaiML/grpo-glm50-combin
status: inactive
timestamp: 2026-04-02T01:18:32+00:00
num_battles: 10525
num_wins: 6227
celo_rating: 8396.51
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step400-int4-fast
model_architecture: GlmMoeDsaForCausalLM
model_num_parameters: 31348144128.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm50-combi_43081_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step400-int4-fast
model_size: 31B
ranking_group: single
us_pacific_date: 2026-04-01
win_ratio: 0.5916389548693587
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '</s>', '<|assistant|>', '<|user|>', '####'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm50-combi-43081-v2-uploader
Waiting for job on chaiml-grpo-glm50-combi-43081-v2-uploader to finish
chaiml-grpo-glm50-combi-43081-v2-uploader: {repo_id} is already quantized
chaiml-grpo-glm50-combi-43081-v2-uploader: Using quantization_mode: none
chaiml-grpo-glm50-combi-43081-v2-uploader: Downloading snapshot of ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step400-int4-fast...
2026-04-01T21:49:20.654462+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T21:50:20.738429+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T21:51:23.635099+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T21:52:23.729237+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T21:53:23.814300+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T21:54:23.949781+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T21:55:24.045241+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T21:56:24.126897+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T21:57:24.209707+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T21:58:24.296084+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T21:59:24.382267+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:00:24.483270+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:01:24.571291+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:02:24.656393+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:03:24.743817+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:04:24.826378+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:05:24.912050+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:06:26.589912+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:07:26.679069+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:08:26.763210+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:09:26.853850+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:10:26.937369+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:11:27.021441+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:12:27.110456+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:13:27.198764+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:14:27.282389+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:15:27.372461+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:16:27.455883+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:17:27.551363+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:18:27.665272+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:19:27.751647+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:20:27.844633+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:21:27.953923+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:22:28.043147+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:23:28.129157+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:24:28.253521+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:25:28.341745+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:26:28.430382+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:27:28.520465+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:28:28.608999+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:29:28.695773+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:30:28.802726+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:31:28.911227+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:32:29.017154+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:33:29.105826+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:34:29.193040+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:35:29.279897+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:36:29.369125+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:37:29.452326+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:38:29.537078+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:39:29.624610+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:40:29.714422+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:41:29.807488+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:42:29.895746+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
chaiml-grpo-glm50-combi-43081-v2-uploader: {repo_id} is already quantized
chaiml-grpo-glm50-combi-43081-v2-uploader: Using quantization_mode: none
chaiml-grpo-glm50-combi-43081-v2-uploader: Downloading snapshot of ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step400-int4-fast...
2026-04-01T22:43:30.012453+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:44:30.099974+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:45:30.277719+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
chaiml-grpo-glm50-combi-43081-v2-uploader: Downloaded in 169.394s
2026-04-01T22:46:30.362540+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:47:30.477536+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:48:30.617192+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:49:30.718877+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:50:30.831116+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
chaiml-grpo-glm50-combi-43081-v2-uploader: Processed model ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step400-int4-fast in 433.437s
chaiml-grpo-glm50-combi-43081-v2-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm50-combi-43081-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-43081-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm50-combi-43081-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm50-combi-43081-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm50-combi-43081-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-43081-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-43081-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-43081-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-43081-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-43081-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-43081-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-43081-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-43081-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm50-combi-43081-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm50-combi-43081-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm50-combi-43081-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm50-combi-43081-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm50-combi-43081-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/.gitattributes
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/generation_config.json
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/config.json
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/chat_template.jinja
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/tokenizer.json
Retrying (%r) after connection broken by '%r': %s
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00081-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00081-of-00081.safetensors
2026-04-01T22:51:31.001632+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00080-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00080-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00035-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00035-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00053-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00053-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00057-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00057-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00016-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00016-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00017-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00017-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00037-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00037-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00005-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00005-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00020-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00020-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00045-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00045-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00056-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00056-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00061-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00061-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00001-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00001-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00079-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00079-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00022-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00022-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00036-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00036-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00012-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00012-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00006-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00006-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00044-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00044-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00058-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00058-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00028-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00028-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00055-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00055-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00042-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00042-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00047-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00047-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00004-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00004-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00043-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00043-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00015-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00015-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00078-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00078-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00064-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00064-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00048-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00048-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00039-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00039-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00010-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00010-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00031-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00031-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00066-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00066-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00038-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00038-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00002-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00002-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00052-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00052-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00014-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00014-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00030-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00030-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00041-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00041-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00070-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00070-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00069-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00069-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00074-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00074-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00027-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00027-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00023-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00023-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00071-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00071-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00013-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00013-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00003-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00003-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00008-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00008-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00068-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00068-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00033-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00033-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00067-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00067-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00032-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00032-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00011-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00011-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00034-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00034-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00026-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00026-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00046-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00046-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00024-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00024-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00065-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00065-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00040-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00040-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00051-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00051-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00059-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00059-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00073-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00073-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00009-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00009-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00019-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00019-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00072-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00072-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00060-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00060-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00018-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00018-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00062-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00062-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00007-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00007-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00054-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00054-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00050-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00050-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00025-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00025-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00075-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00075-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00076-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00076-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00049-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00049-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00029-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00029-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00077-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00077-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00063-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00063-of-00081.safetensors
chaiml-grpo-glm50-combi-43081-v2-uploader: cp /dev/shm/model_output/model-00021-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00021-of-00081.safetensors
2026-04-01T22:52:31.102632+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
2026-04-01T22:53:31.252533+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
%s, retrying in %s seconds...
2026-04-01T22:54:31.364861+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
%s, retrying in %s seconds...
2026-04-01T22:55:31.470219+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
%s, retrying in %s seconds...
2026-04-01T22:56:31.579373+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
Job failed chaiml-grpo-glm50-combi-43081-v2-uploader: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 01 Apr 2026 22:56:36 GMT', 'Content-Type': 'application/json', 'Content-Length': '272', 'Connection': 'keep-alive', 'audit-id': '1a974219-fd44-4424-9e7b-7262963da20b', 'Cache-Control': 'no-cache, private', 'x-kubernetes-pf-flowschema-uid': '757159af-7462-4610-a5a7-751d984daa90', 'x-kubernetes-pf-prioritylevel-uid': '5f6bf145-f133-455b-a330-c9bf3e5ea771', 'cf-cache-status': 'DYNAMIC', 'set-cookie': '__cf_bm=M7bdGPvO6xFhDSblc3rIJy10e.8lKsxDuhLOCtvqgCc-1775084196.6517298-1.0.1.1-d6NYp.Cqq3dZ.w8I2oh1018K0xzppOi9tRxuHBy2quwDRNe1XHoFFG1rMteGMbUhcqv7XpczkwdRvNTUSNbvx; HttpOnly; Secure; Path=/; Domain=coreweave.com; Expires=Wed, 01 Apr 2026 23:26:36 GMT', 'Server': 'cloudflare', 'CF-RAY': '9e5b30251e633ade-IAD'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch \"chaiml-grpo-glm50-combi-43081-v2-uploader\" not found","reason":"NotFound","details":{"name":"chaiml-grpo-glm50-combi-43081-v2-uploader","group":"batch","kind":"jobs"},"code":404}


Stopping job with name chaiml-grpo-glm50-combi-43081-v2-uploader
%s, retrying in %s seconds...
Stopping job with name chaiml-grpo-glm50-combi-43081-v2-uploader
%s, retrying in %s seconds...
Stopping job with name chaiml-grpo-glm50-combi-43081-v2-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-grpo-glm50-combi-43081-v2-uploader
Waiting for job on chaiml-grpo-glm50-combi-43081-v2-uploader to finish
chaiml-grpo-glm50-combi-43081-v2-uploader: {repo_id} is already quantized
chaiml-grpo-glm50-combi-43081-v2-uploader: Using quantization_mode: none
chaiml-grpo-glm50-combi-43081-v2-uploader: Downloading snapshot of ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step400-int4-fast...
2026-04-01T22:57:31.682304+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:58:31.792895+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T22:59:31.924194+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
chaiml-grpo-glm50-combi-43081-v2-uploader: Downloaded in 158.579s
2026-04-01T23:00:32.029761+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T23:01:32.141460+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T23:02:32.290754+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
2026-04-01T23:03:32.436874+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
chaiml-grpo-glm50-combi-43081-v2-uploader: Processed model ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step400-int4-fast in 406.343s
chaiml-grpo-glm50-combi-43081-v2-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm50-combi-43081-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-43081-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm50-combi-43081-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm50-combi-43081-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm50-combi-43081-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-43081-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-43081-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-43081-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-43081-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-43081-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-43081-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-43081-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-43081-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm50-combi-43081-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm50-combi-43081-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm50-combi-43081-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm50-combi-43081-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm50-combi-43081-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/.gitattributes": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/chat_template.jinja": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/config.json": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/generation_config.json": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00001-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00001-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00002-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00002-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00003-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00003-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00004-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00004-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00005-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00005-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00006-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00006-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00007-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00007-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00008-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00008-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00009-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00009-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00010-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00010-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00011-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00011-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00012-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00012-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00013-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00013-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00014-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00014-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00015-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00015-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00016-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00016-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00017-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00017-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00018-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00018-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00019-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00019-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00020-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00020-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00021-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00021-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00022-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00022-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00023-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00023-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00024-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00024-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00025-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00025-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00026-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00026-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00027-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00027-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00028-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00028-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00029-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00029-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00030-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00030-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00031-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00031-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00032-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00032-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00033-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00033-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00034-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00034-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00035-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00035-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00036-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00036-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00037-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00037-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00038-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00038-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00039-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00039-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00040-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00040-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00041-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00041-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00042-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00042-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00043-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00043-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00044-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00044-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00045-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00045-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00046-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00046-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00047-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00047-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00048-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00048-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00049-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00049-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00050-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00050-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00051-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00051-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00052-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00052-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00053-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00053-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00054-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00054-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00055-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00055-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00056-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00056-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00057-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00057-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00058-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00058-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00059-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00059-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00060-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00060-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00061-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00061-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00062-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00062-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00063-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00063-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00064-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00064-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00065-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00065-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00066-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00066-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00067-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00067-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00068-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00068-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00069-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00069-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00070-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00070-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00071-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00071-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00072-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00072-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00073-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00073-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00074-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00074-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00075-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00075-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00076-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00076-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00077-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00077-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00078-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00078-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00079-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00079-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00080-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00080-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00081-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model-00081-of-00081.safetensors": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/model.safetensors.index.json": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/quantization_config.json": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/tokenizer.json": object size matches
chaiml-grpo-glm50-combi-43081-v2-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-43081-v2/default/tokenizer_config.json": object size matches
2026-04-01T23:04:32.542741+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
Job chaiml-grpo-glm50-combi-43081-v2-uploader completed after 475.16s with status: succeeded
Stopping job with name chaiml-grpo-glm50-combi-43081-v2-uploader
Pipeline stage VLLMUploader completed in 4572.41s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.16s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.90s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Pipeline stage VLLMDeployer completed in 0.28s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.593890905380249s
Received healthy response to inference request in 2.652716636657715s
Received healthy response to inference request in 2.796140432357788s
Received healthy response to inference request in 2.727847099304199s
Received healthy response to inference request in 3.1418228149414062s
Received healthy response to inference request in 2.7575902938842773s
Received healthy response to inference request in 2.810250997543335s
Received healthy response to inference request in 2.7122178077697754s
Received healthy response to inference request in 2.709355354309082s
Received healthy response to inference request in 2.4714951515197754s
Received healthy response to inference request in 3.143925666809082s
Received healthy response to inference request in 2.518078565597534s
Received healthy response to inference request in 2.5974085330963135s
Received healthy response to inference request in 2.571784496307373s
Received healthy response to inference request in 2.7862040996551514s
Received healthy response to inference request in 3.805748224258423s
2026-04-01T23:05:33.015384+00:00 monitor updated for chaiml-grpo-glm50-combi_43081_v2
Received healthy response to inference request in 2.839883327484131s
Received healthy response to inference request in 3.0858752727508545s
Received healthy response to inference request in 3.0898380279541016s
Received healthy response to inference request in 2.46272873878479s
Received healthy response to inference request in 2.970881938934326s
Received healthy response to inference request in 2.6203551292419434s
Received healthy response to inference request in 2.4121334552764893s
Received healthy response to inference request in 2.517120838165283s
Received healthy response to inference request in 2.587052583694458s
Received healthy response to inference request in 3.896932363510132s
Received healthy response to inference request in 2.790393114089966s
Received healthy response to inference request in 2.6159393787384033s
Received healthy response to inference request in 2.6485543251037598s
Received healthy response to inference request in 2.634830951690674s
30 requests
0 failed requests
5th percentile: 2.4666736245155336
10th percentile: 2.5125582695007322
20th percentile: 2.583998966217041
30th percentile: 2.6103801250457765
40th percentile: 2.6430649757385254
50th percentile: 2.7107865810394287
60th percentile: 2.769035816192627
70th percentile: 2.800373601913452
80th percentile: 2.9938806056976324
90th percentile: 3.142033100128174
95th percentile: 3.5079280734062177
99th percentile: 3.870488963127136
mean time: 2.7989665508270263
Pipeline stage StressChecker completed in 96.08s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.39s
Shutdown handler de-registered
chaiml-grpo-glm50-combi_43081_v2 status is now deployed due to DeploymentManager action
chaiml-grpo-glm50-combi_43081_v2 status is now inactive due to auto deactivation removed underperforming models