developer_uid: chai_backend_admin
submission_id: chaiml-gspo-glm47-combi_63111_v3
model_name: chaiml-gspo-glm47-combi_63111_v3
model_group: ChaiML/gspo-glm47-combin
status: inactive
timestamp: 2026-03-19T03:18:47+00:00
num_battles: 11349
num_wins: 6314
celo_rating: 1332.62
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/gspo-glm47-combine-rm82-zy-data-step200
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-gspo-glm47-combi_63111_v3
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/gspo-glm47-combine-rm82-zy-data-step200
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-18
win_ratio: 0.5563485769671337
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['####', '<|user|>', '<|assistant|>', '</s>', '<|im_end|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>\n{bot_name}: {message}', 'user_template': '<|user|>\n{message}', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-combi-63111-v3-uploader
Waiting for job on chaiml-gspo-glm47-combi-63111-v3-uploader to finish
chaiml-gspo-glm47-combi-63111-v3-uploader: Using quantization_mode: fp8
chaiml-gspo-glm47-combi-63111-v3-uploader: Checking if ChaiML/gspo-glm47-combine-rm82-zy-data-step200-FP8 already exists in ChaiML
chaiml-gspo-glm47-combi-63111-v3-uploader: Downloading snapshot of ChaiML/gspo-glm47-combine-rm82-zy-data-step200...
2026-03-18T19:50:21.054580+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T19:51:21.146318+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T19:52:21.237812+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T19:53:21.323056+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T19:54:21.408208+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
chaiml-gspo-glm47-combi-63111-v3-uploader: Downloaded in 251.942s
chaiml-gspo-glm47-combi-63111-v3-uploader: Loading /tmp/model_input...
chaiml-gspo-glm47-combi-63111-v3-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-gspo-glm47-combi-63111-v3-uploader: `torch_dtype` is deprecated! Use `dtype` instead!
chaiml-gspo-glm47-combi-63111-v3-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
chaiml-gspo-glm47-combi-63111-v3-uploader: Applying quantization...
chaiml-gspo-glm47-combi-63111-v3-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-gspo-glm47-combi-63111-v3-uploader: 2026-03-18T12:55:07.732489-0700 | reset | INFO - Compression lifecycle reset
chaiml-gspo-glm47-combi-63111-v3-uploader: 2026-03-18T12:55:07.800908-0700 | from_modifiers | INFO - Creating recipe from modifiers
2026-03-18T19:55:21.500722+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
chaiml-gspo-glm47-combi-63111-v3-uploader: 2026-03-18T12:56:03.733191-0700 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-gspo-glm47-combi-63111-v3-uploader: 2026-03-18T12:56:03.733675-0700 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
2026-03-18T19:56:21.601331+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
chaiml-gspo-glm47-combi-63111-v3-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
2026-03-18T19:57:21.700922+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T19:58:21.793918+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T19:59:21.886878+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-18T20:00:21.990127+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:01:22.092482+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:02:22.190622+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:03:22.289608+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:04:22.376290+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:05:22.467421+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:06:22.571444+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:07:22.662690+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:08:22.750991+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:09:22.838351+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:10:22.924805+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:11:23.034390+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:12:23.128346+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:13:23.227993+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:14:23.327970+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
chaiml-gspo-glm47-combi-63111-v3-uploader: 2026-03-18T13:14:26.191939-0700 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
chaiml-gspo-glm47-combi-63111-v3-uploader: 2026-03-18T13:14:32.094830-0700 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-gspo-glm47-combi-63111-v3-uploader: Saving to /dev/shm/model_output...
chaiml-gspo-glm47-combi-63111-v3-uploader: 2026-03-18T13:14:32.129061-0700 | get_model_compressor | INFO - skip_sparsity_compression_stats set to True. Skipping sparsity compression statistic calculations. No sparsity compressor will be applied.
2026-03-18T20:15:23.422757+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:16:23.522962+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:17:23.620584+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:18:23.718477+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:19:23.810557+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:20:23.903782+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:21:23.995881+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:22:24.089954+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:23:24.189415+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
Retrying (%r) after connection broken by '%r': %s
2026-03-18T20:24:24.283272+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:25:24.374427+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:26:24.474629+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:27:24.575884+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:28:24.669953+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:29:24.770052+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:30:24.870626+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:31:33.402322+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:32:33.498008+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:33:33.593546+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:34:33.696236+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:35:33.791348+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:36:33.891826+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:37:33.993754+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:38:34.103859+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:39:34.198135+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:40:34.288414+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:41:34.384988+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:42:34.486787+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:43:34.583703+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:44:34.682098+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:45:34.783214+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:46:34.885317+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:47:35.265295+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:48:35.389310+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:49:35.489308+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:50:35.589466+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:51:35.687752+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:52:35.780995+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:53:35.887160+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:54:36.449105+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:55:36.546717+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T20:56:36.650312+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
chaiml-gspo-glm47-combi-63111-v3-uploader: Cleaning quantization config in /dev/shm/model_output
chaiml-gspo-glm47-combi-63111-v3-uploader: Pushing to ChaiML/gspo-glm47-combine-rm82-zy-data-step200-FP8
chaiml-gspo-glm47-combi-63111-v3-uploader: Checking if ChaiML/gspo-glm47-combine-rm82-zy-data-step200-FP8 already exists in ChaiML
chaiml-gspo-glm47-combi-63111-v3-uploader: ---------- 2026-03-18 13:57:17 (0:00:00) ----------
chaiml-gspo-glm47-combi-63111-v3-uploader: Files:   hashed 8/80 (28.2M/354.7G) | pre-uploaded: 0/1 (0.0/354.7G) (+72 unsure) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-gspo-glm47-combi-63111-v3-uploader: Workers: hashing: 72 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 53
chaiml-gspo-glm47-combi-63111-v3-uploader: ---------------------------------------------------
2026-03-18T20:57:36.739244+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
chaiml-gspo-glm47-combi-63111-v3-uploader: 







chaiml-gspo-glm47-combi-63111-v3-uploader: ---------- 2026-03-18 13:58:17 (0:01:00) ----------
chaiml-gspo-glm47-combi-63111-v3-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 13/73 (55.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-gspo-glm47-combi-63111-v3-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 60 | committing: 0 | waiting: 66
chaiml-gspo-glm47-combi-63111-v3-uploader: ---------------------------------------------------
2026-03-18T20:58:36.837895+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
chaiml-gspo-glm47-combi-63111-v3-uploader:                              







chaiml-gspo-glm47-combi-63111-v3-uploader: ---------- 2026-03-18 13:59:17 (0:02:00) ----------
chaiml-gspo-glm47-combi-63111-v3-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 37/73 (174.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-gspo-glm47-combi-63111-v3-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 36 | committing: 0 | waiting: 90
chaiml-gspo-glm47-combi-63111-v3-uploader: ---------------------------------------------------
2026-03-18T20:59:36.934391+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
chaiml-gspo-glm47-combi-63111-v3-uploader:                              







chaiml-gspo-glm47-combi-63111-v3-uploader: ---------- 2026-03-18 14:00:17 (0:03:00) ----------
chaiml-gspo-glm47-combi-63111-v3-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 63/73 (304.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-gspo-glm47-combi-63111-v3-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 10 | committing: 0 | waiting: 116
chaiml-gspo-glm47-combi-63111-v3-uploader: ---------------------------------------------------
2026-03-18T21:00:37.029417+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
chaiml-gspo-glm47-combi-63111-v3-uploader:                              







chaiml-gspo-glm47-combi-63111-v3-uploader: ---------- 2026-03-18 14:01:17 (0:04:00) ----------
chaiml-gspo-glm47-combi-63111-v3-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-gspo-glm47-combi-63111-v3-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-gspo-glm47-combi-63111-v3-uploader: ---------------------------------------------------
2026-03-18T21:01:37.644360+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
chaiml-gspo-glm47-combi-63111-v3-uploader:                              







chaiml-gspo-glm47-combi-63111-v3-uploader: ---------- 2026-03-18 14:02:17 (0:05:00) ----------
chaiml-gspo-glm47-combi-63111-v3-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 50/80 (204.8G/354.7G) | ignored: 0
chaiml-gspo-glm47-combi-63111-v3-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-gspo-glm47-combi-63111-v3-uploader: ---------------------------------------------------
2026-03-18T21:02:37.748345+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
chaiml-gspo-glm47-combi-63111-v3-uploader:                              Processed model ChaiML/gspo-glm47-combine-rm82-zy-data-step200 in 4392.344s
2026-03-18T21:03:37.842305+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
chaiml-gspo-glm47-combi-63111-v3-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-combi-63111-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-63111-v3-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-combi-63111-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-combi-63111-v3-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-combi-63111-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-63111-v3-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-63111-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-63111-v3-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-63111-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-63111-v3-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-63111-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-63111-v3-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-63111-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-combi-63111-v3-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-combi-63111-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-combi-63111-v3-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-combi-63111-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-combi-63111-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/chat_template.jinja
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/generation_config.json
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/recipe.yaml
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/tokenizer_config.json
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/config.json
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/special_tokens_map.json
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model.safetensors.index.json
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/tokenizer.json
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00072-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00071-of-00072.safetensors
2026-03-18T21:04:37.947318+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00026-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00057-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00020-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00031-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00038-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00067-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00054-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00040-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00021-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00055-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00016-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00018-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00009-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00028-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00042-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00056-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00036-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00049-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00035-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00046-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00005-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00017-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00045-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00041-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00002-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00011-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00068-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00024-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00006-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00047-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00001-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00007-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00058-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00015-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00039-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00034-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00061-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00050-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00069-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00008-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00066-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00010-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00027-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00043-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00013-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00022-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00062-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00070-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00060-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00051-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00052-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00048-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00003-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00004-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00059-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00030-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00012-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00029-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00014-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00063-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00019-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00033-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00032-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00053-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00037-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00065-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00064-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00044-of-00072.safetensors
chaiml-gspo-glm47-combi-63111-v3-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v3/default/model-00023-of-00072.safetensors
Job chaiml-gspo-glm47-combi-63111-v3-uploader completed after 4556.72s with status: succeeded
Stopping job with name chaiml-gspo-glm47-combi-63111-v3-uploader
Pipeline stage VLLMUploader completed in 4557.24s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.95s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-combi-63111-v3
Waiting for inference service chaiml-gspo-glm47-combi-63111-v3 to be ready
2026-03-18T21:05:38.061287+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T21:06:38.170553+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T21:07:38.294337+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T21:08:38.437077+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T21:09:38.562434+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T21:10:38.703780+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T21:11:38.815115+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T21:12:38.957196+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T21:13:39.065564+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T21:14:39.200896+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T21:15:39.339544+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T21:16:39.455603+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T21:17:39.577684+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T21:18:39.718315+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
2026-03-18T21:19:39.847022+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
Inference service chaiml-gspo-glm47-combi-63111-v3 ready after 862.4275586605072s
Pipeline stage VLLMDeployer completed in 862.92s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 9.511130094528198s
Received healthy response to inference request in 1.9924366474151611s
Received healthy response to inference request in 1.9944095611572266s
Received healthy response to inference request in 2.105564594268799s
Received healthy response to inference request in 1.9801993370056152s
Received healthy response to inference request in 2.0625619888305664s
Received healthy response to inference request in 2.0751447677612305s
Received healthy response to inference request in 1.9787037372589111s
Received healthy response to inference request in 1.96836256980896s
Received healthy response to inference request in 2.159243583679199s
Received healthy response to inference request in 1.9759752750396729s
Received healthy response to inference request in 1.9639437198638916s
Received healthy response to inference request in 1.990999698638916s
Received healthy response to inference request in 2.28367018699646s
Received healthy response to inference request in 2.0000457763671875s
Received healthy response to inference request in 2.1136951446533203s
Received healthy response to inference request in 2.075932502746582s
Received healthy response to inference request in 2.118267774581909s
Received healthy response to inference request in 2.2501306533813477s
Received healthy response to inference request in 2.009978771209717s
Received healthy response to inference request in 2.069430351257324s
Received healthy response to inference request in 2.0198967456817627s
Received healthy response to inference request in 1.9924867153167725s
2026-03-18T21:20:39.953806+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v3
Received healthy response to inference request in 2.095282554626465s
Received healthy response to inference request in 2.186985731124878s
Received healthy response to inference request in 2.039522647857666s
Received healthy response to inference request in 2.058481454849243s
Received healthy response to inference request in 2.0852794647216797s
Received healthy response to inference request in 2.0710015296936035s
Received healthy response to inference request in 1.9808862209320068s
30 requests
0 failed requests
5th percentile: 1.9717882871627808
10th percentile: 1.9784308910369872
20th percentile: 1.9889770030975342
30th percentile: 1.9938327074050903
40th percentile: 2.0159295558929444
50th percentile: 2.060521721839905
60th percentile: 2.072658824920654
70th percentile: 2.0882803916931154
80th percentile: 2.1146096706390383
90th percentile: 2.193300223350525
95th percentile: 2.2685773968696594
99th percentile: 7.415166721344001
mean time: 2.306988326708476
Pipeline stage StressChecker completed in 73.21s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.78s
Shutdown handler de-registered
chaiml-gspo-glm47-combi_63111_v3 status is now deployed due to DeploymentManager action
chaiml-gspo-glm47-combi_63111_v3 status is now inactive due to auto deactivation removed underperforming models