developer_uid: chai_backend_admin
submission_id: chaiml-gspo-glm47-combi_24742_v2
model_name: chaiml-gspo-glm47-combi_24742_v2
model_group: ChaiML/gspo-glm47-combin
status: deployed
timestamp: 2026-04-07T21:30:06+00:00
num_battles: 7866
num_wins: 4656
celo_rating: 1369.45
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/gspo-glm47-combine-rm82-mega-data-step1100
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-gspo-glm47-combi_24742_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/gspo-glm47-combine-rm82-mega-data-step1100
model_size: 24B
ranking_group: single
us_pacific_date: 2026-04-07
win_ratio: 0.5919145690312738
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|user|>', '####', '</s>', '<|assistant|>', '<|im_end|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-combi-24742-v2-uploader
Waiting for job on chaiml-gspo-glm47-combi-24742-v2-uploader to finish
2026-04-07T20:45:31.568093+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader: Using quantization_mode: fp8
chaiml-gspo-glm47-combi-24742-v2-uploader: Checking if ChaiML/gspo-glm47-combine-rm82-mega-data-step1100-FP8 already exists in ChaiML
chaiml-gspo-glm47-combi-24742-v2-uploader: Downloading snapshot of ChaiML/gspo-glm47-combine-rm82-mega-data-step1100...
2026-04-07T20:46:31.653386+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T20:47:31.740034+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T20:48:31.825782+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-07T20:49:31.918121+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader: Downloaded in 249.344s
chaiml-gspo-glm47-combi-24742-v2-uploader: Loading /tmp/model_input...
2026-04-07T20:50:32.049542+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T20:51:32.139289+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T20:52:32.233288+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T20:53:32.327671+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader: Applying quantization...
chaiml-gspo-glm47-combi-24742-v2-uploader: 2026-04-07T20:54:15.767388+0000 | __init__ | WARNING - Disabling tokenizer parallelism due to threading conflict between FastTokenizer and Datasets. Set TOKENIZERS_PARALLELISM=false to suppress this warning.
chaiml-gspo-glm47-combi-24742-v2-uploader: 2026-04-07T20:54:16.965477+0000 | reset | INFO - Compression lifecycle reset
chaiml-gspo-glm47-combi-24742-v2-uploader: 2026-04-07T20:54:16.973417+0000 | moe_calibration_context | INFO - Found 89 MoE modules to replace
chaiml-gspo-glm47-combi-24742-v2-uploader: 2026-04-07T20:54:16.978012+0000 | moe_calibration_context | INFO - Replaced 89 MoE modules for calibration
chaiml-gspo-glm47-combi-24742-v2-uploader: 2026-04-07T20:54:16.978090+0000 | moe_calibration_context | INFO - 89/89 modules will be restored after calibration
chaiml-gspo-glm47-combi-24742-v2-uploader: 2026-04-07T20:54:16.978159+0000 | from_modifiers | INFO - Creating recipe from modifiers
chaiml-gspo-glm47-combi-24742-v2-uploader: 2026-04-07T20:54:17.068674+0000 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-gspo-glm47-combi-24742-v2-uploader: 2026-04-07T20:54:17.068930+0000 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
chaiml-gspo-glm47-combi-24742-v2-uploader: 2026-04-07T20:54:17.109140+0000 | dispatch_model | WARNING - Forced to offload modules due to insufficient gpu resources
2026-04-07T20:54:32.411385+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader: 2026-04-07T20:54:41.954839+0000 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
chaiml-gspo-glm47-combi-24742-v2-uploader: 2026-04-07T20:54:41.955123+0000 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-gspo-glm47-combi-24742-v2-uploader: Saving to /tmp/model_output...
chaiml-gspo-glm47-combi-24742-v2-uploader: /usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py:3344: UserWarning: Attempting to save a model with offloaded modules. Ensure that unallocated cpu memory exceeds the `shard_size` (50GB default)
chaiml-gspo-glm47-combi-24742-v2-uploader:   warnings.warn(
2026-04-07T20:55:32.497095+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T20:56:43.699301+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T20:57:43.794093+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T20:58:43.883740+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T20:59:43.969441+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T21:00:44.064836+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T21:01:44.182632+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader: Updating config in /tmp/model_output
chaiml-gspo-glm47-combi-24742-v2-uploader: Pushing to ChaiML/gspo-glm47-combine-rm82-mega-data-step1100-FP8
chaiml-gspo-glm47-combi-24742-v2-uploader: Checking if ChaiML/gspo-glm47-combine-rm82-mega-data-step1100-FP8 already exists in ChaiML
chaiml-gspo-glm47-combi-24742-v2-uploader: Creating repo ChaiML/gspo-glm47-combine-rm82-mega-data-step1100-FP8 and uploading /tmp/model_output to it
chaiml-gspo-glm47-combi-24742-v2-uploader: Found 15 files larger than 20GB (recommended limit):
chaiml-gspo-glm47-combi-24742-v2-uploader:   - model-00001-of-00015.safetensors: 45.2GB
chaiml-gspo-glm47-combi-24742-v2-uploader:   - model-00002-of-00015.safetensors: 46.3GB
chaiml-gspo-glm47-combi-24742-v2-uploader:   - model-00003-of-00015.safetensors: 46.3GB
chaiml-gspo-glm47-combi-24742-v2-uploader:   - model-00004-of-00015.safetensors: 46.3GB
chaiml-gspo-glm47-combi-24742-v2-uploader:   - model-00005-of-00015.safetensors: 46.3GB
chaiml-gspo-glm47-combi-24742-v2-uploader:   ... and 10 more files
chaiml-gspo-glm47-combi-24742-v2-uploader: Large files may slow down loading and processing.
chaiml-gspo-glm47-combi-24742-v2-uploader: ---------- 2026-04-07 21:02:35 (0:00:00) ----------
chaiml-gspo-glm47-combi-24742-v2-uploader: Files:   hashed 7/22 (24.0M/690.4G) | pre-uploaded: 0/0 (0.0/690.4G) (+19 unsure) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-combi-24742-v2-uploader: Workers: hashing: 15 | get upload mode: 6 | pre-uploading: 0 | committing: 0 | waiting: 43
chaiml-gspo-glm47-combi-24742-v2-uploader: ---------------------------------------------------
2026-04-07T21:02:44.268563+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader: 







chaiml-gspo-glm47-combi-24742-v2-uploader: ---------- 2026-04-07 21:03:35 (0:01:00) ----------
chaiml-gspo-glm47-combi-24742-v2-uploader: Files:   hashed 8/22 (43.8G/690.4G) | pre-uploaded: 1/2 (20.0M/690.4G) (+14 unsure) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-combi-24742-v2-uploader: Workers: hashing: 14 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 49
chaiml-gspo-glm47-combi-24742-v2-uploader: ---------------------------------------------------
2026-04-07T21:03:44.359901+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader:                              







chaiml-gspo-glm47-combi-24742-v2-uploader: ---------- 2026-04-07 21:04:35 (0:02:00) ----------
chaiml-gspo-glm47-combi-24742-v2-uploader: Files:   hashed 8/22 (43.8G/690.4G) | pre-uploaded: 1/2 (20.0M/690.4G) (+14 unsure) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-combi-24742-v2-uploader: Workers: hashing: 14 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 49
chaiml-gspo-glm47-combi-24742-v2-uploader: ---------------------------------------------------
2026-04-07T21:04:44.452458+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader:                              







chaiml-gspo-glm47-combi-24742-v2-uploader: ---------- 2026-04-07 21:05:35 (0:03:00) ----------
chaiml-gspo-glm47-combi-24742-v2-uploader: Files:   hashed 11/22 (182.6G/690.4G) | pre-uploaded: 1/5 (20.0M/690.4G) (+11 unsure) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-combi-24742-v2-uploader: Workers: hashing: 11 | get upload mode: 0 | pre-uploading: 4 | committing: 0 | waiting: 49
chaiml-gspo-glm47-combi-24742-v2-uploader: ---------------------------------------------------
2026-04-07T21:05:44.556560+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader:                              







chaiml-gspo-glm47-combi-24742-v2-uploader: ---------- 2026-04-07 21:06:35 (0:04:00) ----------
chaiml-gspo-glm47-combi-24742-v2-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 2/16 (43.8G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-combi-24742-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 14 | committing: 0 | waiting: 50
chaiml-gspo-glm47-combi-24742-v2-uploader: ---------------------------------------------------
2026-04-07T21:06:44.938739+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader:                              







chaiml-gspo-glm47-combi-24742-v2-uploader: ---------- 2026-04-07 21:07:35 (0:05:00) ----------
chaiml-gspo-glm47-combi-24742-v2-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 2/16 (43.8G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-combi-24742-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 14 | committing: 0 | waiting: 50
chaiml-gspo-glm47-combi-24742-v2-uploader: ---------------------------------------------------
2026-04-07T21:07:45.046534+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader:                              







chaiml-gspo-glm47-combi-24742-v2-uploader: ---------- 2026-04-07 21:08:35 (0:06:00) ----------
chaiml-gspo-glm47-combi-24742-v2-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 8/16 (321.4G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-combi-24742-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 8 | committing: 0 | waiting: 56
chaiml-gspo-glm47-combi-24742-v2-uploader: ---------------------------------------------------
2026-04-07T21:08:45.140300+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader:                              







chaiml-gspo-glm47-combi-24742-v2-uploader: ---------- 2026-04-07 21:09:35 (0:07:00) ----------
2026-04-07T21:09:45.241238+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 10/16 (413.9G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-combi-24742-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 6 | committing: 0 | waiting: 58
chaiml-gspo-glm47-combi-24742-v2-uploader: ---------------------------------------------------
chaiml-gspo-glm47-combi-24742-v2-uploader:                              







chaiml-gspo-glm47-combi-24742-v2-uploader: ---------- 2026-04-07 21:10:35 (0:08:00) ----------
chaiml-gspo-glm47-combi-24742-v2-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 11/16 (460.2G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-combi-24742-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 5 | committing: 0 | waiting: 59
chaiml-gspo-glm47-combi-24742-v2-uploader: ---------------------------------------------------
2026-04-07T21:10:45.460782+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader:                              







chaiml-gspo-glm47-combi-24742-v2-uploader: ---------- 2026-04-07 21:11:35 (0:09:00) ----------
chaiml-gspo-glm47-combi-24742-v2-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 16/16 (690.4G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-combi-24742-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-combi-24742-v2-uploader: ---------------------------------------------------
2026-04-07T21:11:45.559121+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T21:12:45.660732+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader:                              







chaiml-gspo-glm47-combi-24742-v2-uploader: ---------- 2026-04-07 21:13:35 (0:11:00) ----------
chaiml-gspo-glm47-combi-24742-v2-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 16/16 (690.4G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-combi-24742-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-combi-24742-v2-uploader: ---------------------------------------------------
2026-04-07T21:13:45.766028+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader:                              







chaiml-gspo-glm47-combi-24742-v2-uploader: ---------- 2026-04-07 21:14:35 (0:12:00) ----------
chaiml-gspo-glm47-combi-24742-v2-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 16/16 (690.4G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-combi-24742-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-combi-24742-v2-uploader: ---------------------------------------------------
2026-04-07T21:14:45.923663+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader:                              







chaiml-gspo-glm47-combi-24742-v2-uploader: ---------- 2026-04-07 21:15:35 (0:13:00) ----------
chaiml-gspo-glm47-combi-24742-v2-uploader: Files:   hashed 22/22 (690.4G/690.4G) | pre-uploaded: 16/16 (690.4G/690.4G) | committed: 0/22 (0.0/690.4G) | ignored: 0
chaiml-gspo-glm47-combi-24742-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-combi-24742-v2-uploader: ---------------------------------------------------
2026-04-07T21:15:46.119075+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader:                              Processed model ChaiML/gspo-glm47-combine-rm82-mega-data-step1100 in 1829.651s
2026-04-07T21:16:46.243577+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-combi-24742-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-24742-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-combi-24742-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-combi-24742-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-combi-24742-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-24742-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-24742-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-24742-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-24742-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-24742-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-24742-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-24742-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-24742-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-combi-24742-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-combi-24742-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-combi-24742-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-combi-24742-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-combi-24742-v2-uploader: uploading /tmp/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/recipe.yaml
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/chat_template.jinja
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/config.json
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/generation_config.json
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/tokenizer_config.json
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/model.safetensors.index.json
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/tokenizer.json
2026-04-07T21:17:46.394214+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T21:18:46.504256+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/model-00002-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/model-00002-of-00015.safetensors
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/model-00001-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/model-00001-of-00015.safetensors
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/model-00013-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/model-00013-of-00015.safetensors
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/model-00006-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/model-00006-of-00015.safetensors
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/model-00011-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/model-00011-of-00015.safetensors
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/model-00007-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/model-00007-of-00015.safetensors
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/model-00009-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/model-00009-of-00015.safetensors
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/model-00003-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/model-00003-of-00015.safetensors
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/model-00014-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/model-00014-of-00015.safetensors
2026-04-07T21:19:46.599175+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/model-00004-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/model-00004-of-00015.safetensors
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/model-00005-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/model-00005-of-00015.safetensors
chaiml-gspo-glm47-combi-24742-v2-uploader: cp /tmp/model_output/model-00010-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v2/default/model-00010-of-00015.safetensors
Job chaiml-gspo-glm47-combi-24742-v2-uploader completed after 2145.07s with status: succeeded
Stopping job with name chaiml-gspo-glm47-combi-24742-v2-uploader
Pipeline stage VLLMUploader completed in 2145.68s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.13s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.02s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-combi-24742-v2
Waiting for inference service chaiml-gspo-glm47-combi-24742-v2 to be ready
2026-04-07T21:20:46.711800+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T21:21:46.839636+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T21:22:46.948452+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T21:23:47.048316+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T21:24:47.158269+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
Retrying (%r) after connection broken by '%r': %s
2026-04-07T21:25:47.443570+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T21:26:50.635611+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
2026-04-07T21:27:50.746100+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
Inference service chaiml-gspo-glm47-combi-24742-v2 ready after 501.3005588054657s
Pipeline stage VLLMDeployer completed in 501.89s
run pipeline stage %s
Running pipeline stage StressChecker
2026-04-07T21:28:50.900538+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
Received healthy response to inference request in 9.85748839378357s
Received healthy response to inference request in 2.132714033126831s
Received healthy response to inference request in 2.0193357467651367s
Received healthy response to inference request in 2.3637518882751465s
Received healthy response to inference request in 2.2967381477355957s
Received healthy response to inference request in 9.751967191696167s
Received healthy response to inference request in 2.160940170288086s
Received healthy response to inference request in 2.5560736656188965s
Received healthy response to inference request in 2.021204948425293s
Received healthy response to inference request in 2.00907826423645s
Received healthy response to inference request in 2.109811782836914s
Received healthy response to inference request in 2.0120980739593506s
Received healthy response to inference request in 2.012178659439087s
Received healthy response to inference request in 2.042282819747925s
Received healthy response to inference request in 1.9965338706970215s
Received healthy response to inference request in 2.3780879974365234s
Received healthy response to inference request in 2.1458945274353027s
Received healthy response to inference request in 2.054675817489624s
Received healthy response to inference request in 2.2574899196624756s
Received healthy response to inference request in 2.3701939582824707s
Received healthy response to inference request in 2.0526857376098633s
Received healthy response to inference request in 2.4910967350006104s
Received healthy response to inference request in 2.057922840118408s
2026-04-07T21:29:51.144338+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v2
Received healthy response to inference request in 1.9965193271636963s
Received healthy response to inference request in 2.085628032684326s
Received healthy response to inference request in 2.0121021270751953s
Received healthy response to inference request in 2.2976233959198s
Received healthy response to inference request in 2.2290031909942627s
Received healthy response to inference request in 1.9965219497680664s
Received healthy response to inference request in 2.0940122604370117s
30 requests
0 failed requests
5th percentile: 1.9965273141860962
10th percentile: 2.0078238248825073
20th percentile: 2.0121633529663088
30th percentile: 2.0359594583511353
40th percentile: 2.0566240310668946
50th percentile: 2.101912021636963
60th percentile: 2.151912784576416
70th percentile: 2.2692643880844114
80th percentile: 2.3650403022766113
90th percentile: 2.497594428062439
95th percentile: 6.513815104961375
99th percentile: 9.826887245178222
mean time: 2.66205518245697
Pipeline stage StressChecker completed in 84.11s
Shutdown handler de-registered
chaiml-gspo-glm47-combi_24742_v2 status is now deployed due to DeploymentManager action