developer_uid: chai_backend_admin
submission_id: chaiml-grpo-glm46-chai-rm_718_v1
model_name: chaiml-grpo-glm46-chai-rm_718_v1
model_group: ChaiML/grpo-glm46-chai-r
status: torndown
timestamp: 2026-03-12T00:41:58+00:00
num_battles: 11173
num_wins: 5834
celo_rating: 1309.53
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm46-chai-rm-lr5e6-step200
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm46-chai-rm_718_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm46-chai-rm-lr5e6-step200
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-08
win_ratio: 0.5221516155016558
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '<|user|>', '</s>', '<|assistant|>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}.<|user|>\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm46-chai-rm-718-v1-uploader
Waiting for job on chaiml-grpo-glm46-chai-rm-718-v1-uploader to finish
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Using quantization_mode: fp8
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Checking if ChaiML/grpo-glm46-chai-rm-lr5e6-step200-FP8 already exists in ChaiML
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Downloading snapshot of ChaiML/grpo-glm46-chai-rm-lr5e6-step200...
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Downloaded in 236.432s
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Loading /tmp/model_input...
chaiml-grpo-glm46-chai-rm-718-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-grpo-glm46-chai-rm-718-v1-uploader: `torch_dtype` is deprecated! Use `dtype` instead!
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Applying quantization...
chaiml-grpo-glm46-chai-rm-718-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-grpo-glm46-chai-rm-718-v1-uploader: 2026-03-08T15:07:15.392232-0700 | reset | INFO - Compression lifecycle reset
chaiml-grpo-glm46-chai-rm-718-v1-uploader: 2026-03-08T15:07:15.459772-0700 | from_modifiers | INFO - Creating recipe from modifiers
chaiml-grpo-glm46-chai-rm-718-v1-uploader: 2026-03-08T15:08:12.355873-0700 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-grpo-glm46-chai-rm-718-v1-uploader: 2026-03-08T15:08:12.356368-0700 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
Failed to get response for submission chaiml-7b07-69d4-linear_27339_v6: ('http://guanaco-model-mesh-load-balancer-v2.model-mesh.kchai-google-us-east4.chaiverse.com/models/chaiml-7b07-69d4-linear_27339_v6/predict', '{"detail":"1 validation error for RuntimeResponse\\npredictions\\n  Field required [type=missing, input_value={\'detail\': \\"1 validation ...tic.dev/2.12/v/missing\\"}, input_type=dict]\\n    For further information visit https://errors.pydantic.dev/2.12/v/missing"}')
chaiml-grpo-glm46-chai-rm-718-v1-uploader: 2026-03-08T15:18:14.333452-0700 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
chaiml-grpo-glm46-chai-rm-718-v1-uploader: 2026-03-08T15:18:20.986194-0700 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Saving to /dev/shm/model_output...
chaiml-grpo-glm46-chai-rm-718-v1-uploader: 2026-03-08T15:18:21.014461-0700 | get_model_compressor | INFO - skip_sparsity_compression_stats set to True. Skipping sparsity compression statistic calculations. No sparsity compressor will be applied.
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Cleaning quantization config in /dev/shm/model_output
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Pushing to ChaiML/grpo-glm46-chai-rm-lr5e6-step200-FP8
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Checking if ChaiML/grpo-glm46-chai-rm-lr5e6-step200-FP8 already exists in ChaiML
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Creating repo ChaiML/grpo-glm46-chai-rm-lr5e6-step200-FP8 and uploading /dev/shm/model_output to it
chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------- 2026-03-08 16:00:50 (0:00:00) ----------
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Files:   hashed 8/80 (28.2M/354.7G) | pre-uploaded: 0/1 (0.0/354.7G) (+72 unsure) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Workers: hashing: 72 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 53
chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-rm-718-v1-uploader: 







chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------- 2026-03-08 16:01:50 (0:01:00) ----------
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 3/73 (5.1G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 70 | committing: 0 | waiting: 56
chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-rm-718-v1-uploader:                              







chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------- 2026-03-08 16:02:50 (0:02:00) ----------
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 10/73 (40.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 63 | committing: 0 | waiting: 63
chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-rm-718-v1-uploader:                              







chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------- 2026-03-08 16:03:50 (0:03:00) ----------
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 13/73 (55.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 60 | committing: 0 | waiting: 66
chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-rm-718-v1-uploader:                              







chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------- 2026-03-08 16:04:50 (0:04:00) ----------
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 20/73 (90.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 53 | committing: 0 | waiting: 73
chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-rm-718-v1-uploader:                              







chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------- 2026-03-08 16:05:50 (0:05:00) ----------
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 27/73 (125.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 46 | committing: 0 | waiting: 80
chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-rm-718-v1-uploader:                              







chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------- 2026-03-08 16:06:50 (0:06:00) ----------
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 35/73 (164.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 38 | committing: 0 | waiting: 88
chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-rm-718-v1-uploader:                              







chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------- 2026-03-08 16:07:50 (0:07:00) ----------
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 43/73 (204.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 30 | committing: 0 | waiting: 96
chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-rm-718-v1-uploader:                              







chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------- 2026-03-08 16:08:50 (0:08:00) ----------
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 50/73 (239.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 23 | committing: 0 | waiting: 103
chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-rm-718-v1-uploader:                              







chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------- 2026-03-08 16:09:50 (0:09:00) ----------
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 53/73 (254.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 20 | committing: 0 | waiting: 106
chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-rm-718-v1-uploader:                              







chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------- 2026-03-08 16:10:50 (0:10:00) ----------
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 60/73 (289.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 13 | committing: 0 | waiting: 113
chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-rm-718-v1-uploader:                              







chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------- 2026-03-08 16:11:50 (0:11:00) ----------
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 68/73 (329.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 5 | committing: 0 | waiting: 121
chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-rm-718-v1-uploader:                              







chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------- 2026-03-08 16:12:50 (0:12:00) ----------
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-rm-718-v1-uploader:                              







chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------- 2026-03-08 16:13:50 (0:13:00) ----------
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 50/80 (204.8G/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-grpo-glm46-chai-rm-718-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-rm-718-v1-uploader:                              Processed model ChaiML/grpo-glm46-chai-rm-lr5e6-step200 in 4322.411s
chaiml-grpo-glm46-chai-rm-718-v1-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm46-chai-rm-718-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-rm-718-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm46-chai-rm-718-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm46-chai-rm-718-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm46-chai-rm-718-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-rm-718-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm46-chai-rm-718-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-rm-718-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm46-chai-rm-718-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-rm-718-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm46-chai-rm-718-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-rm-718-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm46-chai-rm-718-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm46-chai-rm-718-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm46-chai-rm-718-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm46-chai-rm-718-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm46-chai-rm-718-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm46-chai-rm-718-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/chat_template.jinja
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/generation_config.json
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/config.json
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/recipe.yaml
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/special_tokens_map.json
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/tokenizer_config.json
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model.safetensors.index.json
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/tokenizer.json
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00072-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00071-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00001-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00019-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00046-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00003-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00018-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00041-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00024-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00037-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00055-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00017-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00026-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00010-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00059-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00042-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00008-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00029-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00048-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00012-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00060-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00067-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00032-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00005-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00014-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00025-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00050-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00020-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00028-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00021-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00065-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00033-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00034-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00045-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00011-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00062-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00013-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00068-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00044-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00007-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00057-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00022-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00047-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00009-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00015-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00049-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00002-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00054-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00006-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00035-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00043-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00051-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00031-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00058-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00052-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00069-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00036-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00040-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00004-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00030-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00061-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00016-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00064-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00053-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00056-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00066-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00038-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00039-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00027-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00063-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00023-of-00072.safetensors
chaiml-grpo-glm46-chai-rm-718-v1-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-rm-718-v1/default/model-00070-of-00072.safetensors
Job chaiml-grpo-glm46-chai-rm-718-v1-uploader completed after 4475.88s with status: succeeded
Stopping job with name chaiml-grpo-glm46-chai-rm-718-v1-uploader
Pipeline stage VLLMUploader completed in 4476.54s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.64s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm46-chai-rm-718-v1
Waiting for inference service chaiml-grpo-glm46-chai-rm-718-v1 to be ready
Inference service chaiml-grpo-glm46-chai-rm-718-v1 ready after 551.6593148708344s
Pipeline stage VLLMDeployer completed in 552.33s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.4609739780426025s
Received healthy response to inference request in 2.1861045360565186s
Received healthy response to inference request in 2.155271053314209s
Received healthy response to inference request in 2.1740622520446777s
Received healthy response to inference request in 2.1958870887756348s
Received healthy response to inference request in 2.15120005607605s
Received healthy response to inference request in 2.1397712230682373s
Received healthy response to inference request in 3.1038074493408203s
Received healthy response to inference request in 2.2665321826934814s
Received healthy response to inference request in 2.243168830871582s
Received healthy response to inference request in 2.3057894706726074s
Received healthy response to inference request in 2.343893527984619s
Received healthy response to inference request in 2.323500394821167s
Received healthy response to inference request in 2.3169877529144287s
Received healthy response to inference request in 2.215811014175415s
Received healthy response to inference request in 2.208009719848633s
Received healthy response to inference request in 2.234881639480591s
Received healthy response to inference request in 2.337887763977051s
Received healthy response to inference request in 2.618896722793579s
Received healthy response to inference request in 2.2931320667266846s
Received healthy response to inference request in 2.2617552280426025s
Received healthy response to inference request in 2.1692864894866943s
Received healthy response to inference request in 2.2384817600250244s
Received healthy response to inference request in 2.126864433288574s
Received healthy response to inference request in 2.1492667198181152s
Received healthy response to inference request in 2.206563949584961s
Received healthy response to inference request in 2.2315025329589844s
Received healthy response to inference request in 2.6796164512634277s
Received healthy response to inference request in 2.2085180282592773s
Received healthy response to inference request in 2.166149854660034s
30 requests
0 failed requests
5th percentile: 2.1440441966056825
10th percentile: 2.1510067224502563
20th percentile: 2.1686591625213625
30th percentile: 2.1929523229598997
40th percentile: 2.2083147048950194
50th percentile: 2.2331920862197876
60th percentile: 2.2506033897399904
70th percentile: 2.2969292879104612
80th percentile: 2.3263778686523438
90th percentile: 2.4767662525177006
95th percentile: 2.6522925734519958
99th percentile: 2.980792059898377
mean time: 2.290452472368876
Pipeline stage StressChecker completed in 72.97s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.37s
Shutdown handler de-registered
chaiml-grpo-glm46-chai-rm_718_v1 status is now deployed due to DeploymentManager action
chaiml-grpo-glm46-chai-rm_718_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-grpo-glm46-chai-rm_718_v1 status is now torndown due to DeploymentManager action