chaiml-gspo-glm47-chai-_95963

developer_uid: richhx
submission_id: chaiml-gspo-glm47-chai-_95963_v1
model_name: chaiml-gspo-glm47-chai-_95963_v1
model_group: ChaiML/gspo-glm47-chai-r
status: deployed
timestamp: 2026-03-27T20:07:38+00:00
num_battles: 10107
num_wins: 5889
celo_rating: 1350.26
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/gspo-glm47-chai-rm-kimi-data-step500-W4A16
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-gspo-glm47-chai-_95963_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/gspo-glm47-chai-rm-kimi-data-step500-W4A16
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-27
win_ratio: 0.5826654793707332
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '####', '</s>', '<|assistant|>', '<|user|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>\n{bot_name}: {message}', 'user_template': '<|user|>\n{message}', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-chai-95963-v1-uploader
Waiting for job on chaiml-gspo-glm47-chai-95963-v1-uploader to finish
'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 58b0b115-1852-4d17-87f5-c090b982ceb4)')' thrown while requesting HEAD https://huggingface.co/ChaiML/gspo-glm47-chai-rm-kimi-data-step500-FP8/resolve/main/config.json
Retrying in 1s [Retry 1/5].
chaiml-gspo-glm47-chai-95963-v1-uploader: {repo_id} is already quantized
chaiml-gspo-glm47-chai-95963-v1-uploader: Using quantization_mode: none
chaiml-gspo-glm47-chai-95963-v1-uploader: Downloading snapshot of ChaiML/gspo-glm47-chai-rm-kimi-data-step500-W4A16...
2026-03-27T19:53:01.012048+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
2026-03-27T19:54:01.235521+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
chaiml-gspo-glm47-chai-95963-v1-uploader: Traceback (most recent call last):
chaiml-gspo-glm47-chai-95963-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_http.py", line 761, in hf_raise_for_status
chaiml-gspo-glm47-chai-95963-v1-uploader:     response.raise_for_status()
chaiml-gspo-glm47-chai-95963-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpx/_models.py", line 829, in raise_for_status
chaiml-gspo-glm47-chai-95963-v1-uploader:     raise HTTPStatusError(message, request=request, response=self)
chaiml-gspo-glm47-chai-95963-v1-uploader: httpx.HTTPStatusError: Server error '504 Gateway Time-out' for url 'https://huggingface.co/api/models/ChaiML/gspo-glm47-chai-rm-kimi-data-step500-W4A16/paths-info/main'
chaiml-gspo-glm47-chai-95963-v1-uploader: For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/504
chaiml-gspo-glm47-chai-95963-v1-uploader: The above exception was the direct cause of the following exception:
chaiml-gspo-glm47-chai-95963-v1-uploader: Traceback (most recent call last):
chaiml-gspo-glm47-chai-95963-v1-uploader:   File "/code/uploading/compress.py", line 344, in <module>
chaiml-gspo-glm47-chai-95963-v1-uploader:     cli()
chaiml-gspo-glm47-chai-95963-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1485, in __call__
chaiml-gspo-glm47-chai-95963-v1-uploader:     return self.main(*args, **kwargs)
chaiml-gspo-glm47-chai-95963-v1-uploader:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-chai-95963-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1406, in main
chaiml-gspo-glm47-chai-95963-v1-uploader:     rv = self.invoke(ctx)
chaiml-gspo-glm47-chai-95963-v1-uploader:          ^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-chai-95963-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1873, in invoke
chaiml-gspo-glm47-chai-95963-v1-uploader:     return _process_result(sub_ctx.command.invoke(sub_ctx))
chaiml-gspo-glm47-chai-95963-v1-uploader:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-chai-95963-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1269, in invoke
chaiml-gspo-glm47-chai-95963-v1-uploader:     return ctx.invoke(self.callback, **ctx.params)
chaiml-gspo-glm47-chai-95963-v1-uploader:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-chai-95963-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 824, in invoke
chaiml-gspo-glm47-chai-95963-v1-uploader:     return callback(*args, **kwargs)
chaiml-gspo-glm47-chai-95963-v1-uploader:            ^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-chai-95963-v1-uploader:   File "/code/uploading/compress.py", line 57, in process
chaiml-gspo-glm47-chai-95963-v1-uploader:     hf_utils.download_snapshot_to_folder(repo_id, revision, hf_auth_token, download_path)
chaiml-gspo-glm47-chai-95963-v1-uploader:   File "/code/src/inference_service_common/hf_utils.py", line 55, in download_snapshot_to_folder
chaiml-gspo-glm47-chai-95963-v1-uploader:     if repo_has_model_safetensors(repo_id, revision, token):
chaiml-gspo-glm47-chai-95963-v1-uploader:        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-chai-95963-v1-uploader:   File "/code/src/inference_service_common/hf_utils.py", line 77, in repo_has_model_safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader:     files = [f.path for f in get_paths_info(repo_id, revision=revision, paths=["/"], token=token)]
chaiml-gspo-glm47-chai-95963-v1-uploader:                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-chai-95963-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
chaiml-gspo-glm47-chai-95963-v1-uploader:     return fn(*args, **kwargs)
chaiml-gspo-glm47-chai-95963-v1-uploader:            ^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-chai-95963-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/hf_api.py", line 3914, in get_paths_info
chaiml-gspo-glm47-chai-95963-v1-uploader:     hf_raise_for_status(response)
chaiml-gspo-glm47-chai-95963-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_http.py", line 889, in hf_raise_for_status
chaiml-gspo-glm47-chai-95963-v1-uploader:     raise _format(HfHubHTTPError, str(e), response) from e
chaiml-gspo-glm47-chai-95963-v1-uploader: huggingface_hub.errors.HfHubHTTPError: Server error '504 Gateway Time-out' for url 'https://huggingface.co/api/models/ChaiML/gspo-glm47-chai-rm-kimi-data-step500-W4A16/paths-info/main' (Amz CF ID: nC3emJ5V2NgecX_RuE9wvc0og-RqASrKl9yWElTgmkedkl0UStVgTA==)
chaiml-gspo-glm47-chai-95963-v1-uploader: For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/504
Job chaiml-gspo-glm47-chai-95963-v1-uploader completed after 172.39s with status: failed
Job failed chaiml-gspo-glm47-chai-95963-v1-uploader: 
Stopping job with name chaiml-gspo-glm47-chai-95963-v1-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-gspo-glm47-chai-95963-v1-uploader
Waiting for job on chaiml-gspo-glm47-chai-95963-v1-uploader to finish
2026-03-27T19:55:01.449550+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
chaiml-gspo-glm47-chai-95963-v1-uploader: {repo_id} is already quantized
chaiml-gspo-glm47-chai-95963-v1-uploader: Using quantization_mode: none
chaiml-gspo-glm47-chai-95963-v1-uploader: Downloading snapshot of ChaiML/gspo-glm47-chai-rm-kimi-data-step500-W4A16...
2026-03-27T19:56:01.673059+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
chaiml-gspo-glm47-chai-95963-v1-uploader: 'The read operation timed out' thrown while requesting HEAD https://huggingface.co/ChaiML/gspo-glm47-chai-rm-kimi-data-step500-W4A16/resolve/34eba47390896ab6088e12b6890e126a0c76b271/chat_template.jinja
chaiml-gspo-glm47-chai-95963-v1-uploader: Retrying in 1s [Retry 1/5].
chaiml-gspo-glm47-chai-95963-v1-uploader: 'The read operation timed out' thrown while requesting HEAD https://huggingface.co/ChaiML/gspo-glm47-chai-rm-kimi-data-step500-W4A16/resolve/34eba47390896ab6088e12b6890e126a0c76b271/model-00005-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: Retrying in 1s [Retry 1/5].
2026-03-27T19:57:02.739253+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
chaiml-gspo-glm47-chai-95963-v1-uploader: Downloaded in 138.610s
2026-03-27T19:58:02.971805+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
2026-03-27T19:59:03.241197+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
chaiml-gspo-glm47-chai-95963-v1-uploader: Processed model ChaiML/gspo-glm47-chai-rm-kimi-data-step500-W4A16 in 211.224s
chaiml-gspo-glm47-chai-95963-v1-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-chai-95963-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-95963-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-chai-95963-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-chai-95963-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-chai-95963-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-95963-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-chai-95963-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-95963-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-chai-95963-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-95963-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-chai-95963-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-95963-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-chai-95963-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-chai-95963-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-chai-95963-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-chai-95963-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-chai-95963-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-chai-95963-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/config.json
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/chat_template.jinja
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/tokenizer_config.json
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/quantization_config.json
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/generation_config.json
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/.gitattributes
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model.safetensors.index.json
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/tokenizer.json
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-chai-76408-v1-uploader
Waiting for job on chaiml-gspo-glm47-chai-76408-v1-uploader to finish
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00038-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00038-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00036-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00037-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00030-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00002-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00002-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00024-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00024-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00018-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00018-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00035-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00035-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00001-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00001-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00022-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00022-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00031-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00031-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00029-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00023-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00023-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00012-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00014-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00014-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00005-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00005-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00026-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00004-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00004-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00019-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00019-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00006-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00006-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00009-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00011-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00011-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00033-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00033-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00015-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00015-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00027-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00027-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00017-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00017-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00032-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00032-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00021-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00021-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00007-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00007-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00025-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00025-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00010-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00010-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00028-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00028-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00008-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00008-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00013-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00013-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00034-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00034-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00016-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00016-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00020-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00020-of-00038.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: {repo_id} is already quantized
chaiml-gspo-glm47-chai-76408-v1-uploader: Using quantization_mode: none
chaiml-gspo-glm47-chai-76408-v1-uploader: Downloading snapshot of ChaiML/gspo-glm47-chai-rm-kimi-data-step500-FP8...
2026-03-27T20:00:03.635206+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
2026-03-27T20:00:14.276244+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00003-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00003-of-00038.safetensors
Job chaiml-gspo-glm47-chai-95963-v1-uploader completed after 348.31s with status: succeeded
Stopping job with name chaiml-gspo-glm47-chai-95963-v1-uploader
Pipeline stage VLLMUploader completed in 529.54s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.46s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.58s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-chai-95963-v1
Waiting for inference service chaiml-gspo-glm47-chai-95963-v1 to be ready
2026-03-27T20:01:03.982704+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
2026-03-27T20:01:14.794647+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
2026-03-27T20:02:04.366523+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
2026-03-27T20:02:15.138496+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
chaiml-gspo-glm47-chai-76408-v1-uploader: 'The read operation timed out' thrown while requesting HEAD https://huggingface.co/ChaiML/gspo-glm47-chai-rm-kimi-data-step500-FP8/resolve/c41657dbe1763084c3fbe5f16b192ef824d3c91e/model-00040-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: Retrying in 1s [Retry 1/5].
chaiml-gspo-glm47-chai-76408-v1-uploader: Downloaded in 163.503s
2026-03-27T20:03:04.884348+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
2026-03-27T20:03:15.477270+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
2026-03-27T20:04:05.225697+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
2026-03-27T20:04:15.861276+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
2026-03-27T20:05:06.234519+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
chaiml-gspo-glm47-chai-76408-v1-uploader: Processed model ChaiML/gspo-glm47-chai-rm-kimi-data-step500-FP8 in 315.454s
chaiml-gspo-glm47-chai-76408-v1-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-chai-76408-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-76408-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-chai-76408-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-chai-76408-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-chai-76408-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-76408-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-chai-76408-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-76408-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-chai-76408-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-76408-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-chai-76408-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-76408-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-chai-76408-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
2026-03-27T20:05:16.264370+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
chaiml-gspo-glm47-chai-76408-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-chai-76408-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-chai-76408-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-chai-76408-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-chai-76408-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/generation_config.json
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/config.json
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/.gitattributes
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/chat_template.jinja
Inference service chaiml-gspo-glm47-chai-95963-v1 ready after 273.25780272483826s
Pipeline stage VLLMDeployer completed in 275.32s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 9.03371262550354s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00072-of-00072.safetensors
Received healthy response to inference request in 8.59642767906189s
Received healthy response to inference request in 2.17704701423645s
Received healthy response to inference request in 8.703476905822754s
Received healthy response to inference request in 2.176699638366699s
2026-03-27T20:06:06.653565+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
Received healthy response to inference request in 9.317871809005737s
2026-03-27T20:06:17.013261+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
Received healthy response to inference request in 9.36390733718872s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00007-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00009-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00045-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00005-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00036-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00038-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00039-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00018-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00003-of-00072.safetensors
Received healthy response to inference request in 2.1320817470550537s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00012-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00013-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00017-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00067-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00030-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00041-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00040-of-00072.safetensors
Received healthy response to inference request in 2.131870746612549s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00024-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00042-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00021-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00026-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00044-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00010-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00047-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00037-of-00072.safetensors
Received healthy response to inference request in 2.1966140270233154s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00046-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00014-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00034-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00048-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00015-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00029-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00052-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00032-of-00072.safetensors
Received healthy response to inference request in 2.097015857696533s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00004-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00016-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00027-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00054-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00069-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00020-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00022-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00031-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00025-of-00072.safetensors
Received healthy response to inference request in 2.1254215240478516s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00001-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00062-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00006-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00068-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00050-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00043-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00066-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00033-of-00072.safetensors
Received healthy response to inference request in 2.2825069427490234s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00019-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00035-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00063-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00011-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00070-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00064-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00065-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00060-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00023-of-00072.safetensors
Received healthy response to inference request in 2.3192391395568848s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00028-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00059-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00008-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00055-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00061-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00049-of-00072.safetensors
Received healthy response to inference request in 2.245492696762085s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00058-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00057-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00051-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00056-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00053-of-00072.safetensors
Received healthy response to inference request in 2.1251649856567383s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00002-of-00072.safetensors
Received healthy response to inference request in 2.2733314037323s
Received healthy response to inference request in 2.2194712162017822s
Received healthy response to inference request in 2.3096272945404053s
Received healthy response to inference request in 2.416706085205078s
Job chaiml-gspo-glm47-chai-76408-v1-uploader completed after 463.1s with status: succeeded
Stopping job with name chaiml-gspo-glm47-chai-76408-v1-uploader
Pipeline stage VLLMUploader completed in 465.01s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Received healthy response to inference request in 2.150949239730835s
Pipeline stage VLLMUploaderAMD completed in 0.59s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.57s
Received healthy response to inference request in 2.365492105484009s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-chai-76408-v1
Waiting for inference service chaiml-gspo-glm47-chai-76408-v1 to be ready
Received healthy response to inference request in 2.1242623329162598s
2026-03-27T20:07:07.000569+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
Received healthy response to inference request in 2.179020404815674s
Received healthy response to inference request in 2.346162796020508s
Received healthy response to inference request in 2.181541919708252s
2026-03-27T20:07:17.358329+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
Received healthy response to inference request in 2.6161746978759766s
Received healthy response to inference request in 2.1475462913513184s
Received healthy response to inference request in 2.2532219886779785s
Received healthy response to inference request in 2.2360966205596924s
30 requests
0 failed requests
5th percentile: 2.124668526649475
10th percentile: 2.1253958702087403
20th percentile: 2.1444533824920655
30th percentile: 2.176942801475525
40th percentile: 2.19058518409729
50th percentile: 2.2407946586608887
60th percentile: 2.2770016193389893
70th percentile: 2.3273162364959714
80th percentile: 2.4565998077392583
90th percentile: 8.736500477790832
95th percentile: 9.190000176429749
99th percentile: 9.350557034015656
mean time: 3.3614718357721967
Pipeline stage StressChecker completed in 122.99s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.29s
Shutdown handler de-registered
chaiml-gspo-glm47-chai-_95963_v1 status is now deployed due to DeploymentManager action