chaiml-gspo-glm46-combi_11090

developer_uid: richhx
submission_id: chaiml-gspo-glm46-combi_11090_v7
model_name: chaiml-gspo-glm46-combi_11090_v7
model_group: ChaiML/gspo-glm46-combin
status: torndown
timestamp: 2026-03-21T15:34:37+00:00
num_battles: 49
num_wins: 27
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/gspo-glm46-combine-rm55-step200
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-gspo-glm46-combi_11090_v7
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/gspo-glm46-combine-rm55-step200
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-21
win_ratio: 0.5510204081632653
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|user|>', '<|im_end|>', '</s>', '####'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>\n{bot_name}: {message}', 'user_template': '<|user|>\n{message}', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
ad_prompt_fallback_message: You know, with [Chai Ultra](chai://chai.ml/subscription/ultra), you'll have even more opportunities to express yourself, without any guardrails or distracting & annoying ads. We give our Ultra memebers 2x more memory, so your AI won't forget a thing. AND - You will get access to our premium, most advanced AI, it is more intelligent and how do I put it ... it will rock ur world more 😍🫶💞
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm46-combi-11090-v7-uploader
Waiting for job on chaiml-gspo-glm46-combi-11090-v7-uploader to finish
chaiml-gspo-glm46-combi-11090-v7-uploader: Using quantization_mode: fp8
chaiml-gspo-glm46-combi-11090-v7-uploader: Checking if ChaiML/gspo-glm46-combine-rm55-step200-FP8 already exists in ChaiML
chaiml-gspo-glm46-combi-11090-v7-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-gspo-glm46-combi-11090-v7-uploader: Downloading snapshot of ChaiML/gspo-glm46-combine-rm55-step200-FP8...
2026-03-21T15:06:56.164232+00:00 monitor updated for chaiml-grpo-glm46-combi_79432_v7
2026-03-21T15:06:57.590473+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
chaiml-grpo-glm46-combi-79432-v7-uploader: Downloaded in 140.308s
chaiml-grpo-glm46-combi-79432-v7-uploader: Processed model ChaiML/grpo-glm46-combine-rm82-lr5e6-step150 in 143.745s
chaiml-grpo-glm46-combi-79432-v7-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm46-combi-79432-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-79432-v7-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm46-combi-79432-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm46-combi-79432-v7-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm46-combi-79432-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-79432-v7-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm46-combi-79432-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-79432-v7-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm46-combi-79432-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-79432-v7-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm46-combi-79432-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-79432-v7-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm46-combi-79432-v7-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm46-combi-79432-v7-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm46-combi-79432-v7-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm46-combi-79432-v7-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm46-combi-79432-v7-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm46-combi-79432-v7-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/chat_template.jinja
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/config.json
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/.gitattributes
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/generation_config.json
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/recipe.yaml
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/special_tokens_map.json
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/tokenizer_config.json
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model.safetensors.index.json
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/tokenizer.json
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00072-of-00072.safetensors
2026-03-21T15:07:56.293782+00:00 monitor updated for chaiml-grpo-glm46-combi_79432_v7
2026-03-21T15:07:57.734385+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm46-chai-81744-v7-uploader
Waiting for job on chaiml-grpo-glm46-chai-81744-v7-uploader to finish
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00071-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00011-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00019-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00038-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00030-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00031-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00025-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00015-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00053-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00032-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00061-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00041-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00020-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00006-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00040-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00028-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00033-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00070-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00044-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00014-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00003-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00035-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00012-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00042-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00036-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00046-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00067-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00010-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00034-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00043-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00050-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00054-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00063-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00062-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00060-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00016-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00049-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00021-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00069-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00004-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00027-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00002-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00065-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00009-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00045-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00052-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00023-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00029-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00017-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00026-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00047-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00057-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00051-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00005-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00056-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00008-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00024-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00013-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00037-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00048-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v7-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v7/default/model-00055-of-00072.safetensors
Job chaiml-grpo-glm46-combi-79432-v7-uploader completed after 284.16s with status: succeeded
Stopping job with name chaiml-grpo-glm46-combi-79432-v7-uploader
Pipeline stage VLLMUploader completed in 284.83s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.68s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm46-combi-79432-v7
Waiting for inference service chaiml-grpo-glm46-combi-79432-v7 to be ready
chaiml-grpo-glm46-chai-81744-v7-uploader: Using quantization_mode: fp8
chaiml-grpo-glm46-chai-81744-v7-uploader: Checking if ChaiML/grpo-glm46-chai-rm-lr5e6-step150-FP8 already exists in ChaiML
chaiml-grpo-glm46-chai-81744-v7-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-grpo-glm46-chai-81744-v7-uploader: Downloading snapshot of ChaiML/grpo-glm46-chai-rm-lr5e6-step150-FP8...
2026-03-21T15:08:56.422274+00:00 monitor updated for chaiml-grpo-glm46-combi_79432_v7
2026-03-21T15:08:57.866386+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
2026-03-21T15:08:58.997880+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
chaiml-gspo-glm46-combi-11090-v7-uploader: Downloaded in 136.696s
chaiml-gspo-glm46-combi-11090-v7-uploader: Processed model ChaiML/gspo-glm46-combine-rm55-step200 in 140.120s
chaiml-gspo-glm46-combi-11090-v7-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm46-combi-11090-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm46-combi-11090-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm46-combi-11090-v7-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm46-combi-11090-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm46-combi-11090-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm46-combi-11090-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm46-combi-11090-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm46-combi-11090-v7-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm46-combi-11090-v7-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm46-combi-11090-v7-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm46-combi-11090-v7-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm46-combi-11090-v7-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm46-combi-11090-v7-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/chat_template.jinja
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/.gitattributes
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/generation_config.json
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/config.json
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/tokenizer_config.json
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/recipe.yaml
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/special_tokens_map.json
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model.safetensors.index.json
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/tokenizer.json
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00072-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00071-of-00072.safetensors
2026-03-21T15:09:56.619380+00:00 monitor updated for chaiml-grpo-glm46-combi_79432_v7
2026-03-21T15:09:58.067769+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
2026-03-21T15:09:59.205867+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00010-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00008-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00035-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00037-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00013-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00031-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00042-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00041-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00009-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00057-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00004-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00058-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00063-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00034-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00017-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00039-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00054-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00036-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00011-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00062-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00045-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00061-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00070-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00032-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00049-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: Downloaded in 98.300s
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00043-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: Processed model ChaiML/grpo-glm46-chai-rm-lr5e6-step150 in 101.878s
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00065-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00007-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00052-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00014-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00053-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00005-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00001-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00012-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00016-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00064-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00056-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00028-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00044-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00020-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00019-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00068-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00006-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00060-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00022-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00024-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/.gitattributes
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00018-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/config.json
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00040-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/recipe.yaml
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00015-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/special_tokens_map.json
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00038-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/generation_config.json
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00051-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/chat_template.jinja
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00055-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/tokenizer_config.json
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00050-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/tokenizer.json
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00046-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v7-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model.safetensors.index.json
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00021-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00030-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00059-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00002-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00003-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00025-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00066-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00069-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00067-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00023-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00033-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00048-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00027-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00047-of-00072.safetensors
chaiml-gspo-glm46-combi-11090-v7-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00029-of-00072.safetensors
chaiml-gspo-glm46-combi_11090_v7 status is now pending due to richhx requesting redeployment of chaiml-gspo-glm46-combi_11090_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm46-combi-11090-v7-uploader
Job failed chaiml-gspo-glm46-combi-11090-v7-uploader: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Date': 'Sat, 21 Mar 2026 15:17:20 GMT', 'Content-Type': 'application/json', 'Content-Length': '282', 'Connection': 'keep-alive', 'audit-id': '436dc24e-58ae-4eac-92fd-4acc9275c7ef', 'Cache-Control': 'no-cache, private', 'x-kubernetes-pf-flowschema-uid': '757159af-7462-4610-a5a7-751d984daa90', 'x-kubernetes-pf-prioritylevel-uid': '5f6bf145-f133-455b-a330-c9bf3e5ea771', 'cf-cache-status': 'DYNAMIC', 'set-cookie': '__cf_bm=biQoDFe5LNRrcv.vtHL9zLBInmA.KT8fAmjfv8M0GL0-1774106240.9523787-1.0.1.1-NfZ3v7_5Zg3bAVmgRykrwddUI8V_NYMIvBwqbgAND5lXHHNaNuGMfEICPGXtpnjPl5YK5RSxEVYtRZ8eNq585; HttpOnly; Secure; Path=/; Domain=coreweave.com; Expires=Sat, 21 Mar 2026 15:47:20 GMT', 'Server': 'cloudflare', 'CF-RAY': '9dfdec45f8de81ab-IAD'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch \"chaiml-gspo-glm46-combi-11090-v7-uploader\" already exists","reason":"AlreadyExists","details":{"name":"chaiml-gspo-glm46-combi-11090-v7-uploader","group":"batch","kind":"jobs"},"code":409}

Stopping job with name chaiml-gspo-glm46-combi-11090-v7-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-gspo-glm46-combi-11090-v7-uploader
Waiting for job on chaiml-gspo-glm46-combi-11090-v7-uploader to finish
chaiml-grpo-glm46-chai-_81744_v7 status is now pending due to richhx requesting redeployment of chaiml-grpo-glm46-chai-_81744_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm46-chai-81744-v7-uploader
Job failed chaiml-grpo-glm46-chai-81744-v7-uploader: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Date': 'Sat, 21 Mar 2026 15:17:32 GMT', 'Content-Type': 'application/json', 'Content-Length': '280', 'Connection': 'keep-alive', 'audit-id': 'd71eec38-af6c-4f44-a4bc-6c168ea22d06', 'Cache-Control': 'no-cache, private', 'x-kubernetes-pf-flowschema-uid': '757159af-7462-4610-a5a7-751d984daa90', 'x-kubernetes-pf-prioritylevel-uid': '5f6bf145-f133-455b-a330-c9bf3e5ea771', 'cf-cache-status': 'DYNAMIC', 'set-cookie': '__cf_bm=-1774106252.3478396-1.0.1.1-w9L46xCBsMinSjxpRaVMgL4CrPBBV.Th2QV5IYqtHM9q.kVZBnTvZ4mg8UK9EINam7GXrNk1aysZDyvmF_uqPkYhDlR6nM4Jh10F1RtJtCaGYEiRONWAXVXvE_XhJ3nB; HttpOnly; Secure; Path=/; Domain=coreweave.com; Expires=Sat, 21 Mar 2026 15:47:32 GMT', 'Server': 'cloudflare', 'CF-RAY': '9dfdec8d2b91e5ba-IAD'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch \"chaiml-grpo-glm46-chai-81744-v7-uploader\" already exists","reason":"AlreadyExists","details":{"name":"chaiml-grpo-glm46-chai-81744-v7-uploader","group":"batch","kind":"jobs"},"code":409}

Stopping job with name chaiml-grpo-glm46-chai-81744-v7-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-grpo-glm46-chai-81744-v7-uploader
Waiting for job on chaiml-grpo-glm46-chai-81744-v7-uploader to finish
chaiml-gspo-glm46-combi-11090-v7-uploader: Using quantization_mode: fp8
2026-03-21T15:18:20.433194+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
chaiml-gspo-glm46-combi-11090-v7-uploader: Checking if ChaiML/gspo-glm46-combine-rm55-step200-FP8 already exists in ChaiML
chaiml-gspo-glm46-combi-11090-v7-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-gspo-glm46-combi-11090-v7-uploader: Downloading snapshot of ChaiML/gspo-glm46-combine-rm55-step200-FP8...
chaiml-grpo-glm46-chai-81744-v7-uploader: Using quantization_mode: fp8
chaiml-grpo-glm46-chai-81744-v7-uploader: Checking if ChaiML/grpo-glm46-chai-rm-lr5e6-step150-FP8 already exists in ChaiML
chaiml-grpo-glm46-chai-81744-v7-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-grpo-glm46-chai-81744-v7-uploader: Downloading snapshot of ChaiML/grpo-glm46-chai-rm-lr5e6-step150-FP8...
2026-03-21T15:18:31.510849+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
2026-03-21T15:19:20.638320+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
2026-03-21T15:19:31.722732+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
2026-03-21T15:20:20.991922+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
2026-03-21T15:20:31.971612+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
chaiml-gspo-glm46-combi-11090-v7-uploader: Downloaded in 132.251s
chaiml-gspo-glm46-combi-11090-v7-uploader: Processed model ChaiML/gspo-glm46-combine-rm55-step200 in 135.874s
chaiml-gspo-glm46-combi-11090-v7-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm46-combi-11090-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm46-combi-11090-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm46-combi-11090-v7-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm46-combi-11090-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm46-combi-11090-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-81744-v7-uploader: Downloaded in 135.607s
chaiml-gspo-glm46-combi-11090-v7-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm46-chai-81744-v7-uploader: Processed model ChaiML/grpo-glm46-chai-rm-lr5e6-step150 in 139.060s
chaiml-gspo-glm46-combi-11090-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm46-combi-11090-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm46-combi-11090-v7-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm46-combi-11090-v7-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm46-combi-11090-v7-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm46-combi-11090-v7-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm46-combi-11090-v7-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm46-combi-11090-v7-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/.gitattributes": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/chat_template.jinja": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/config.json": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/generation_config.json": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00001-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00002-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00003-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00004-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00005-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00006-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00007-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00008-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00009-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00010-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00011-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00012-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00013-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00014-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00015-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00016-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00017-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00018-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00019-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00020-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00021-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00022-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00023-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00024-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00025-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00026-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00027-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00028-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00029-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00030-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00031-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00032-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00033-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00034-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00035-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00036-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00037-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00038-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00039-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00040-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00041-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00042-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00043-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00044-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00045-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00046-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00047-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00048-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00049-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00050-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00051-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00052-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00053-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00054-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00055-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00056-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00057-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00058-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00059-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00060-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00061-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00062-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00063-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00064-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/.gitattributes": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00065-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/chat_template.jinja": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00066-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/config.json": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00067-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/generation_config.json": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00068-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00001-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00069-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00002-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00070-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00003-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00071-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00004-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model-00072-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00005-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/model.safetensors.index.json": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00006-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/recipe.yaml": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00007-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/special_tokens_map.json": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00008-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/tokenizer.json": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00009-of-00072.safetensors": object size matches
chaiml-gspo-glm46-combi-11090-v7-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combi-11090-v7/default/tokenizer_config.json": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00010-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00011-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00012-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00013-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00014-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00015-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00016-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00017-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00018-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00019-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00020-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00021-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00022-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00023-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00024-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00025-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00026-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00027-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00028-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00029-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00030-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00031-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00032-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00033-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00034-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00035-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00036-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00037-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00038-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00039-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00040-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00041-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00042-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00043-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00044-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00045-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00046-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00047-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00048-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00049-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00050-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00051-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00052-of-00072.safetensors": object size matches
Job chaiml-gspo-glm46-combi-11090-v7-uploader completed after 223.9s with status: succeeded
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00053-of-00072.safetensors": object size matches
Stopping job with name chaiml-gspo-glm46-combi-11090-v7-uploader
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00054-of-00072.safetensors": object size matches
Pipeline stage VLLMUploader completed in 225.49s
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00055-of-00072.safetensors": object size matches
run pipeline stage %s
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00056-of-00072.safetensors": object size matches
Running pipeline stage VLLMTemplater
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00057-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00058-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00059-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00060-of-00072.safetensors": object size matches
Pipeline stage VLLMTemplater completed in 0.87s
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00061-of-00072.safetensors": object size matches
run pipeline stage %s
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00062-of-00072.safetensors": object size matches
Running pipeline stage VLLMDeployer
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00063-of-00072.safetensors": object size matches
Creating inference service chaiml-gspo-glm46-combi-11090-v7
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00064-of-00072.safetensors": object size matches
Waiting for inference service chaiml-gspo-glm46-combi-11090-v7 to be ready
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00065-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00066-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00067-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00068-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00069-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00070-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00071-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00072-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model.safetensors.index.json": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/recipe.yaml": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/special_tokens_map.json": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/tokenizer.json": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/tokenizer_config.json": object size matches
2026-03-21T15:21:21.273318+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
Job chaiml-grpo-glm46-chai-81744-v7-uploader completed after 229.24s with status: succeeded
Stopping job with name chaiml-grpo-glm46-chai-81744-v7-uploader
Pipeline stage VLLMUploader completed in 231.00s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.49s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm46-chai-81744-v7
Waiting for inference service chaiml-grpo-glm46-chai-81744-v7 to be ready
2026-03-21T15:21:32.278675+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
2026-03-21T15:22:21.548852+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
2026-03-21T15:22:32.595530+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
2026-03-21T15:23:21.822177+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
2026-03-21T15:23:32.877964+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
2026-03-21T15:24:22.138910+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
2026-03-21T15:24:33.125702+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
2026-03-21T15:25:22.393474+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
2026-03-21T15:25:33.380871+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
chaiml-grpo-glm46-chai-_81744_v7 status is now pending due to richhx requesting redeployment of chaiml-grpo-glm46-chai-_81744_v7
run pipeline stage %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Running pipeline stage VLLMUploader
run pipeline %s
Starting job with name chaiml-grpo-glm46-chai-81744-v8-uploader
run pipeline stage %s
Waiting for job on chaiml-grpo-glm46-chai-81744-v8-uploader to finish
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm46-chai-81744-v7-uploader
Waiting for job on chaiml-grpo-glm46-chai-81744-v7-uploader to finish
2026-03-21T15:26:30.754945+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
2026-03-21T15:26:33.643926+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
Inference service chaiml-gspo-glm46-combi-11090-v7 ready after 340.55632519721985s
Pipeline stage VLLMDeployer completed in 341.87s
run pipeline stage %s
Running pipeline stage StressChecker
chaiml-grpo-glm46-chai-81744-v8-uploader: Using quantization_mode: fp8
chaiml-grpo-glm46-chai-81744-v8-uploader: Checking if ChaiML/grpo-glm46-chai-rm-lr5e6-step150-FP8 already exists in ChaiML
Received healthy response to inference request in 9.038647651672363s
chaiml-grpo-glm46-chai-81744-v8-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-grpo-glm46-chai-81744-v8-uploader: Downloading snapshot of ChaiML/grpo-glm46-chai-rm-lr5e6-step150-FP8...
Received healthy response to inference request in 9.208820343017578s
Received healthy response to inference request in 9.078523874282837s
chaiml-grpo-glm46-chai-81744-v7-uploader: Using quantization_mode: fp8
chaiml-grpo-glm46-chai-81744-v7-uploader: Checking if ChaiML/grpo-glm46-chai-rm-lr5e6-step150-FP8 already exists in ChaiML
chaiml-grpo-glm46-chai-81744-v7-uploader: Model already exists. Downloading to /dev/shm/model_output...
Received healthy response to inference request in 1.9838383197784424s
chaiml-grpo-glm46-chai-81744-v7-uploader: Downloading snapshot of ChaiML/grpo-glm46-chai-rm-lr5e6-step150-FP8...
2026-03-21T15:27:27.117940+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:27:28.094521+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
2026-03-21T15:27:31.141115+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
Received healthy response to inference request in 10.125329971313477s
2026-03-21T15:27:34.108428+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
Received healthy response to inference request in 2.0006966590881348s
Received healthy response to inference request in 1.9696125984191895s
Received healthy response to inference request in 1.9935522079467773s
Received healthy response to inference request in 1.9967689514160156s
Received healthy response to inference request in 2.0010976791381836s
chaiml-gspo-glm46-combi_11090_v7 status is now torndown due to DeploymentManager action