developer_uid: chai_backend_admin
submission_id: chaiml-gspo-glm47-think_23943_v1
model_name: chaiml-gspo-glm47-think_23943_v1
model_group: ChaiML/gspo-glm47-thinki
status: deployed
timestamp: 2026-04-13T07:29:47+00:00
num_battles: 2038
num_wins: 1133
celo_rating: 0.0
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/gspo-glm47-thinking-rm-kimi-pony-euqal-apr04-step500
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-gspo-glm47-think_23943_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/gspo-glm47-thinking-rm-kimi-pony-euqal-apr04-step500
model_size: 24B
ranking_group: single
us_pacific_date: 2026-04-13
win_ratio: 0.5559371933267909
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|user|>', '<|assistant|>', '####', '</s>', '<|im_end|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-think-23943-v1-uploader
Waiting for job on chaiml-gspo-glm47-think-23943-v1-uploader to finish
chaiml-gspo-glm47-think-23943-v1-uploader: Using quantization_mode: fp8
chaiml-gspo-glm47-think-23943-v1-uploader: Checking if ChaiML/gspo-glm47-thinking-rm-kimi-pony-euqal-apr04-step500-FP8 already exists in ChaiML
chaiml-gspo-glm47-think-23943-v1-uploader: Downloading snapshot of ChaiML/gspo-glm47-thinking-rm-kimi-pony-euqal-apr04-step500...
2026-04-13T06:47:11.335038+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T06:48:11.422199+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T06:49:11.510202+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T06:50:11.596442+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader: Downloaded in 206.811s
chaiml-gspo-glm47-think-23943-v1-uploader: Loading /tmp/model_input...
2026-04-13T06:51:11.685738+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T06:52:11.770047+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T06:53:11.856593+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T06:54:11.952227+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader: Applying quantization...
chaiml-gspo-glm47-think-23943-v1-uploader: 2026-04-13T06:55:09.9334 | __init__ | WARNING - Disabling tokenizer parallelism due to threading conflict between FastTokenizer and Datasets. Set TOKENIZERS_PARALLELISM=false to suppress this warning.
2026-04-13T06:55:12.066792+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader: 2026-04-13T06:55:11.3378 | reset | INFO - Compression lifecycle reset
chaiml-gspo-glm47-think-23943-v1-uploader: 2026-04-13T06:55:11.3454 | moe_calibration_context | INFO - Found 89 MoE modules to replace
chaiml-gspo-glm47-think-23943-v1-uploader: 2026-04-13T06:55:11.3488 | moe_calibration_context | INFO - Replaced 89 MoE modules for calibration
chaiml-gspo-glm47-think-23943-v1-uploader: 2026-04-13T06:55:11.3489 | moe_calibration_context | INFO - 89/89 modules will be restored after calibration
chaiml-gspo-glm47-think-23943-v1-uploader: 2026-04-13T06:55:11.3490 | from_modifiers | INFO - Creating recipe from modifiers
chaiml-gspo-glm47-think-23943-v1-uploader: 2026-04-13T06:55:11.4406 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-gspo-glm47-think-23943-v1-uploader: 2026-04-13T06:55:11.4408 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
chaiml-gspo-glm47-think-23943-v1-uploader: 2026-04-13T06:55:11.4637 | dispatch_model | WARNING - Forced to offload modules due to insufficient gpu resources
chaiml-gspo-glm47-think-23943-v1-uploader: 2026-04-13T06:55:35.1586 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
chaiml-gspo-glm47-think-23943-v1-uploader: 2026-04-13T06:55:35.1588 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-gspo-glm47-think-23943-v1-uploader: Saving to /tmp/model_output...
chaiml-gspo-glm47-think-23943-v1-uploader: /usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py:3344: UserWarning: Attempting to save a model with offloaded modules. Ensure that unallocated cpu memory exceeds the `shard_size` (50GB default)
chaiml-gspo-glm47-think-23943-v1-uploader:   warnings.warn(
2026-04-13T06:56:12.155406+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T06:57:12.254228+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T06:58:12.339662+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T06:59:12.431213+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T07:00:12.514071+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T07:01:12.598232+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T07:02:12.679985+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T07:03:12.765159+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader: Updating config in /tmp/model_output
chaiml-gspo-glm47-think-23943-v1-uploader: Pushing to ChaiML/gspo-glm47-thinking-rm-kimi-pony-euqal-apr04-step500-FP8
chaiml-gspo-glm47-think-23943-v1-uploader: Checking if ChaiML/gspo-glm47-thinking-rm-kimi-pony-euqal-apr04-step500-FP8 already exists in ChaiML
chaiml-gspo-glm47-think-23943-v1-uploader: Creating repo ChaiML/gspo-glm47-thinking-rm-kimi-pony-euqal-apr04-step500-FP8 and uploading /tmp/model_output to it
chaiml-gspo-glm47-think-23943-v1-uploader: Found 15 files larger than 20GB (recommended limit):
chaiml-gspo-glm47-think-23943-v1-uploader:   - model-00001-of-00015.safetensors: 45.2GB
chaiml-gspo-glm47-think-23943-v1-uploader:   - model-00002-of-00015.safetensors: 46.3GB
chaiml-gspo-glm47-think-23943-v1-uploader:   - model-00003-of-00015.safetensors: 46.3GB
chaiml-gspo-glm47-think-23943-v1-uploader:   - model-00004-of-00015.safetensors: 46.3GB
chaiml-gspo-glm47-think-23943-v1-uploader:   - model-00005-of-00015.safetensors: 46.3GB
chaiml-gspo-glm47-think-23943-v1-uploader:   ... and 10 more files
chaiml-gspo-glm47-think-23943-v1-uploader: Large files may slow down loading and processing.
chaiml-gspo-glm47-think-23943-v1-uploader: ---------- 2026-04-13 07:03:43 (0:00:00) ----------
chaiml-gspo-glm47-think-23943-v1-uploader: Files:   hashed 6/22 (4.1M/690.4G) | pre-uploaded: 0/0 (0.0/690.4G) (+22 unsure) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-think-23943-v1-uploader: Workers: hashing: 16 | get upload mode: 6 | pre-uploading: 0 | committing: 0 | waiting: 42
chaiml-gspo-glm47-think-23943-v1-uploader: ---------------------------------------------------
2026-04-13T07:04:12.851267+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader: 







chaiml-gspo-glm47-think-23943-v1-uploader: ---------- 2026-04-13 07:04:43 (0:01:00) ----------
chaiml-gspo-glm47-think-23943-v1-uploader: Files:   hashed 7/22 (24.0M/690.4G) | pre-uploaded: 1/1 (20.0M/690.4G) (+15 unsure) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-think-23943-v1-uploader: Workers: hashing: 15 | get upload mode: 0 | pre-uploading: 0 | committing: 0 | waiting: 49
chaiml-gspo-glm47-think-23943-v1-uploader: ---------------------------------------------------
2026-04-13T07:05:12.933418+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader:                              







chaiml-gspo-glm47-think-23943-v1-uploader: ---------- 2026-04-13 07:05:43 (0:02:00) ----------
chaiml-gspo-glm47-think-23943-v1-uploader: Files:   hashed 7/22 (24.0M/690.4G) | pre-uploaded: 1/1 (20.0M/690.4G) (+15 unsure) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-think-23943-v1-uploader: Workers: hashing: 15 | get upload mode: 0 | pre-uploading: 0 | committing: 0 | waiting: 49
chaiml-gspo-glm47-think-23943-v1-uploader: ---------------------------------------------------
2026-04-13T07:06:13.249932+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader:                              







chaiml-gspo-glm47-think-23943-v1-uploader: ---------- 2026-04-13 07:06:43 (0:03:00) ----------
chaiml-gspo-glm47-think-23943-v1-uploader: Files:   hashed 13/22 (274.0G/690.4G) | pre-uploaded: 1/7 (20.0M/690.4G) (+9 unsure) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-think-23943-v1-uploader: Workers: hashing: 9 | get upload mode: 0 | pre-uploading: 6 | committing: 0 | waiting: 49
chaiml-gspo-glm47-think-23943-v1-uploader: ---------------------------------------------------
2026-04-13T07:07:13.429535+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader:                              







chaiml-gspo-glm47-think-23943-v1-uploader: ---------- 2026-04-13 07:07:43 (0:04:00) ----------
chaiml-gspo-glm47-think-23943-v1-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 1/16 (20.0M/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-think-23943-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 15 | committing: 0 | waiting: 49
chaiml-gspo-glm47-think-23943-v1-uploader: ---------------------------------------------------
Failed to get request counts for guanaco-submitter. Falling back to default
2026-04-13T07:08:13.866215+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader:                              







chaiml-gspo-glm47-think-23943-v1-uploader: ---------- 2026-04-13 07:08:43 (0:05:00) ----------
chaiml-gspo-glm47-think-23943-v1-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 1/16 (20.0M/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-think-23943-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 15 | committing: 0 | waiting: 49
chaiml-gspo-glm47-think-23943-v1-uploader: ---------------------------------------------------
2026-04-13T07:09:14.030761+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader:                              







chaiml-gspo-glm47-think-23943-v1-uploader: ---------- 2026-04-13 07:09:43 (0:06:00) ----------
chaiml-gspo-glm47-think-23943-v1-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 8/16 (320.3G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-think-23943-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 8 | committing: 0 | waiting: 56
chaiml-gspo-glm47-think-23943-v1-uploader: ---------------------------------------------------
2026-04-13T07:10:14.121252+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader:                              







chaiml-gspo-glm47-think-23943-v1-uploader: ---------- 2026-04-13 07:10:43 (0:07:00) ----------
chaiml-gspo-glm47-think-23943-v1-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 9/16 (366.5G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-think-23943-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 7 | committing: 0 | waiting: 57
chaiml-gspo-glm47-think-23943-v1-uploader: ---------------------------------------------------
2026-04-13T07:11:14.253595+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader:                              







chaiml-gspo-glm47-think-23943-v1-uploader: ---------- 2026-04-13 07:11:43 (0:08:00) ----------
chaiml-gspo-glm47-think-23943-v1-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 13/16 (551.6G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-think-23943-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 3 | committing: 0 | waiting: 61
chaiml-gspo-glm47-think-23943-v1-uploader: ---------------------------------------------------
2026-04-13T07:12:14.349685+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader:                              







chaiml-gspo-glm47-think-23943-v1-uploader: ---------- 2026-04-13 07:12:43 (0:09:00) ----------
chaiml-gspo-glm47-think-23943-v1-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 16/16 (690.4G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-think-23943-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-think-23943-v1-uploader: ---------------------------------------------------
2026-04-13T07:13:14.911072+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader:                              







chaiml-gspo-glm47-think-23943-v1-uploader: ---------- 2026-04-13 07:13:43 (0:10:00) ----------
chaiml-gspo-glm47-think-23943-v1-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 16/16 (690.4G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-think-23943-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-think-23943-v1-uploader: ---------------------------------------------------
2026-04-13T07:14:15.033204+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader:                              







chaiml-gspo-glm47-think-23943-v1-uploader: ---------- 2026-04-13 07:14:43 (0:11:00) ----------
chaiml-gspo-glm47-think-23943-v1-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 16/16 (690.4G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-think-23943-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-think-23943-v1-uploader: ---------------------------------------------------
2026-04-13T07:15:15.131113+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader:                              







chaiml-gspo-glm47-think-23943-v1-uploader: ---------- 2026-04-13 07:15:43 (0:12:00) ----------
chaiml-gspo-glm47-think-23943-v1-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 16/16 (690.4G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-think-23943-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-think-23943-v1-uploader: ---------------------------------------------------
2026-04-13T07:16:15.323002+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader:                              







chaiml-gspo-glm47-think-23943-v1-uploader: ---------- 2026-04-13 07:16:43 (0:13:00) ----------
chaiml-gspo-glm47-think-23943-v1-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 16/16 (690.4G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-think-23943-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-think-23943-v1-uploader: ---------------------------------------------------
2026-04-13T07:17:15.474071+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader:                              Processed model ChaiML/gspo-glm47-thinking-rm-kimi-pony-euqal-apr04-step500 in 1858.406s
2026-04-13T07:18:15.568211+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-think-23943-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-think-23943-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-think-23943-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-think-23943-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-think-23943-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-think-23943-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-think-23943-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-think-23943-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-think-23943-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-think-23943-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-think-23943-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-think-23943-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-think-23943-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-think-23943-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-think-23943-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-think-23943-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-think-23943-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-think-23943-v1-uploader: uploading /tmp/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/generation_config.json
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/recipe.yaml
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/chat_template.jinja
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/config.json
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/tokenizer_config.json
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/model.safetensors.index.json
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/tokenizer.json
2026-04-13T07:19:15.663320+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T07:20:16.078770+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/model-00011-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/model-00011-of-00015.safetensors
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/model-00014-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/model-00014-of-00015.safetensors
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/model-00013-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/model-00013-of-00015.safetensors
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/model-00008-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/model-00008-of-00015.safetensors
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/model-00015-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/model-00015-of-00015.safetensors
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/model-00003-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/model-00003-of-00015.safetensors
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/model-00007-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/model-00007-of-00015.safetensors
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/model-00009-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/model-00009-of-00015.safetensors
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/model-00001-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/model-00001-of-00015.safetensors
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/model-00005-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/model-00005-of-00015.safetensors
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/model-00006-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/model-00006-of-00015.safetensors
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/model-00002-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/model-00002-of-00015.safetensors
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/model-00004-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/model-00004-of-00015.safetensors
chaiml-gspo-glm47-think-23943-v1-uploader: cp /tmp/model_output/model-00010-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-think-23943-v1/default/model-00010-of-00015.safetensors
2026-04-13T07:21:16.291353+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
Job chaiml-gspo-glm47-think-23943-v1-uploader completed after 2114.67s with status: succeeded
Stopping job with name chaiml-gspo-glm47-think-23943-v1-uploader
Pipeline stage VLLMUploader completed in 2115.17s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.14s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.22s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-think-23943-v1
Waiting for inference service chaiml-gspo-glm47-think-23943-v1 to be ready
2026-04-13T07:22:16.524042+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T07:23:16.726239+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T07:24:16.826600+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T07:25:16.948351+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T07:26:17.044152+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
2026-04-13T07:27:17.148065+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
Inference service chaiml-gspo-glm47-think-23943-v1 ready after 401.0070059299469s
Pipeline stage VLLMDeployer completed in 401.64s
run pipeline stage %s
Running pipeline stage StressChecker
2026-04-13T07:28:17.251249+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
Received healthy response to inference request in 9.552652835845947s
Received healthy response to inference request in 2.027484893798828s
Received healthy response to inference request in 9.486578702926636s
Received healthy response to inference request in 9.069741487503052s
Received healthy response to inference request in 2.039482593536377s
Received healthy response to inference request in 2.006399154663086s
Received healthy response to inference request in 2.1730618476867676s
Received healthy response to inference request in 1.973881483078003s
Received healthy response to inference request in 2.0483903884887695s
Received healthy response to inference request in 1.948486566543579s
Received healthy response to inference request in 2.113049030303955s
Received healthy response to inference request in 2.0604677200317383s
Received healthy response to inference request in 2.1434595584869385s
Received healthy response to inference request in 2.0311439037323s
Received healthy response to inference request in 2.070314407348633s
Received healthy response to inference request in 2.0345005989074707s
Received healthy response to inference request in 1.9877581596374512s
Received healthy response to inference request in 2.0143232345581055s
Received healthy response to inference request in 2.013653516769409s
Received healthy response to inference request in 2.1085822582244873s
2026-04-13T07:29:17.359385+00:00 monitor updated for chaiml-gspo-glm47-think_23943_v1
Received healthy response to inference request in 9.19091010093689s
Received healthy response to inference request in 2.5431535243988037s
Received healthy response to inference request in 2.8144900798797607s
Received healthy response to inference request in 2.2856760025024414s
Received healthy response to inference request in 2.1567609310150146s
Received healthy response to inference request in 2.0059702396392822s
Received healthy response to inference request in 1.9978923797607422s
Received healthy response to inference request in 2.0758328437805176s
Received healthy response to inference request in 2.0037314891815186s
Received healthy response to inference request in 2.7390296459198s
30 requests
0 failed requests
5th percentile: 1.9801259875297545
10th percentile: 1.9968789577484132
20th percentile: 2.006313371658325
30th percentile: 2.0235363960266115
40th percentile: 2.0374897956848144
50th percentile: 2.0653910636901855
60th percentile: 2.1103689670562744
70th percentile: 2.1616512060165403
80th percentile: 2.5823287487030036
90th percentile: 9.081858348846435
95th percentile: 9.353527832031249
99th percentile: 9.533491337299347
mean time: 3.0905619859695435
Pipeline stage StressChecker completed in 96.24s
Shutdown handler de-registered
chaiml-gspo-glm47-think_23943_v1 status is now deployed due to DeploymentManager action