developer_uid: chai_backend_admin
submission_id: chaiml-grpo-glm46-chai-_81744_v1
model_name: chaiml-grpo-glm46-chai-_81744_v1
model_group: ChaiML/grpo-glm46-chai-r
status: inactive
timestamp: 2026-03-09T01:57:10+00:00
num_battles: 12772
num_wins: 6821
celo_rating: 1314.76
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm46-chai-rm-lr5e6-step150
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm46-chai-_81744_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm46-chai-rm-lr5e6-step150
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-08
win_ratio: 0.5340588787973692
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['####', '<|im_end|>', '<|user|>', '<|assistant|>', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}.<|user|>\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm46-chai-81744-v1-uploader
Waiting for job on chaiml-grpo-glm46-chai-81744-v1-uploader to finish
chaiml-grpo-glm46-chai-81744-v1-uploader: Using quantization_mode: fp8
chaiml-grpo-glm46-chai-81744-v1-uploader: Checking if ChaiML/grpo-glm46-chai-rm-lr5e6-step150-FP8 already exists in ChaiML
chaiml-grpo-glm46-chai-81744-v1-uploader: Downloading snapshot of ChaiML/grpo-glm46-chai-rm-lr5e6-step150...
chaiml-grpo-glm46-chai-81744-v1-uploader: Downloaded in 235.416s
chaiml-grpo-glm46-chai-81744-v1-uploader: Loading /tmp/model_input...
chaiml-grpo-glm46-chai-81744-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-grpo-glm46-chai-81744-v1-uploader: `torch_dtype` is deprecated! Use `dtype` instead!
chaiml-grpo-glm46-chai-81744-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
chaiml-grpo-glm46-chai-81744-v1-uploader: Applying quantization...
chaiml-grpo-glm46-chai-81744-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-grpo-glm46-chai-81744-v1-uploader: 2026-03-08T16:39:57.114344-0700 | reset | INFO - Compression lifecycle reset
chaiml-grpo-glm46-chai-81744-v1-uploader: 2026-03-08T16:39:57.182509-0700 | from_modifiers | INFO - Creating recipe from modifiers
chaiml-grpo-glm46-chai-81744-v1-uploader: 2026-03-08T16:40:53.227720-0700 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-grpo-glm46-chai-81744-v1-uploader: 2026-03-08T16:40:53.228205-0700 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
chaiml-grpo-glm46-chai-81744-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
chaiml-grpo-glm46-chai-81744-v1-uploader: 2026-03-08T16:50:26.060307-0700 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
chaiml-grpo-glm46-chai-81744-v1-uploader: 2026-03-08T16:50:32.562672-0700 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-grpo-glm46-chai-81744-v1-uploader: Saving to /dev/shm/model_output...
chaiml-grpo-glm46-chai-81744-v1-uploader: 2026-03-08T16:50:32.589270-0700 | get_model_compressor | INFO - skip_sparsity_compression_stats set to True. Skipping sparsity compression statistic calculations. No sparsity compressor will be applied.
Unable to record family friendly update due to error: Invalid JSON input: Expecting value: line 1 column 1 (char 0)
chaiml-grpo-glm46-chai-81744-v1-uploader: Cleaning quantization config in /dev/shm/model_output
chaiml-grpo-glm46-chai-81744-v1-uploader: Pushing to ChaiML/grpo-glm46-chai-rm-lr5e6-step150-FP8
chaiml-grpo-glm46-chai-81744-v1-uploader: Checking if ChaiML/grpo-glm46-chai-rm-lr5e6-step150-FP8 already exists in ChaiML
chaiml-grpo-glm46-chai-81744-v1-uploader: Creating repo ChaiML/grpo-glm46-chai-rm-lr5e6-step150-FP8 and uploading /dev/shm/model_output to it
chaiml-grpo-glm46-chai-81744-v1-uploader: ---------- 2026-03-08 17:33:02 (0:00:00) ----------
chaiml-grpo-glm46-chai-81744-v1-uploader: Files:   hashed 8/80 (28.2M/354.7G) | pre-uploaded: 0/1 (0.0/354.7G) (+72 unsure) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-81744-v1-uploader: Workers: hashing: 72 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 53
chaiml-grpo-glm46-chai-81744-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-81744-v1-uploader: 







chaiml-grpo-glm46-chai-81744-v1-uploader: ---------- 2026-03-08 17:34:02 (0:01:00) ----------
chaiml-grpo-glm46-chai-81744-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 3/73 (5.1G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-81744-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 70 | committing: 0 | waiting: 56
chaiml-grpo-glm46-chai-81744-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-81744-v1-uploader:                              







chaiml-grpo-glm46-chai-81744-v1-uploader: ---------- 2026-03-08 17:35:02 (0:02:00) ----------
chaiml-grpo-glm46-chai-81744-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 11/73 (45.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-81744-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 62 | committing: 0 | waiting: 64
chaiml-grpo-glm46-chai-81744-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-81744-v1-uploader:                              







chaiml-grpo-glm46-chai-81744-v1-uploader: ---------- 2026-03-08 17:36:02 (0:03:00) ----------
chaiml-grpo-glm46-chai-81744-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 11/73 (45.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-81744-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 62 | committing: 0 | waiting: 64
chaiml-grpo-glm46-chai-81744-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-81744-v1-uploader:                              







chaiml-grpo-glm46-chai-81744-v1-uploader: ---------- 2026-03-08 17:37:02 (0:04:00) ----------
chaiml-grpo-glm46-chai-81744-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 19/73 (85.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-81744-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 54 | committing: 0 | waiting: 72
chaiml-grpo-glm46-chai-81744-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-81744-v1-uploader:                              







chaiml-grpo-glm46-chai-81744-v1-uploader: ---------- 2026-03-08 17:38:02 (0:05:00) ----------
chaiml-grpo-glm46-chai-81744-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 27/73 (125.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-81744-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 46 | committing: 0 | waiting: 80
chaiml-grpo-glm46-chai-81744-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-81744-v1-uploader:                              







chaiml-grpo-glm46-chai-81744-v1-uploader: ---------- 2026-03-08 17:39:02 (0:06:00) ----------
chaiml-grpo-glm46-chai-81744-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 35/73 (164.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-81744-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 38 | committing: 0 | waiting: 88
chaiml-grpo-glm46-chai-81744-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-81744-v1-uploader:                              







chaiml-grpo-glm46-chai-81744-v1-uploader: ---------- 2026-03-08 17:40:02 (0:07:00) ----------
chaiml-grpo-glm46-chai-81744-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 43/73 (204.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-81744-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 30 | committing: 0 | waiting: 96
chaiml-grpo-glm46-chai-81744-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-81744-v1-uploader:                              







chaiml-grpo-glm46-chai-81744-v1-uploader: ---------- 2026-03-08 17:41:02 (0:08:00) ----------
chaiml-grpo-glm46-chai-81744-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 47/73 (224.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-81744-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 26 | committing: 0 | waiting: 100
chaiml-grpo-glm46-chai-81744-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-81744-v1-uploader:                              







chaiml-grpo-glm46-chai-81744-v1-uploader: ---------- 2026-03-08 17:42:02 (0:09:00) ----------
chaiml-grpo-glm46-chai-81744-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 52/73 (249.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-81744-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 21 | committing: 0 | waiting: 105
chaiml-grpo-glm46-chai-81744-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-81744-v1-uploader:                              







chaiml-grpo-glm46-chai-81744-v1-uploader: ---------- 2026-03-08 17:43:02 (0:10:00) ----------
chaiml-grpo-glm46-chai-81744-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 59/73 (284.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-81744-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 14 | committing: 0 | waiting: 112
chaiml-grpo-glm46-chai-81744-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-81744-v1-uploader:                              







chaiml-grpo-glm46-chai-81744-v1-uploader: ---------- 2026-03-08 17:44:02 (0:11:00) ----------
chaiml-grpo-glm46-chai-81744-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 67/73 (324.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-81744-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 6 | committing: 0 | waiting: 120
chaiml-grpo-glm46-chai-81744-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-81744-v1-uploader:                              







chaiml-grpo-glm46-chai-81744-v1-uploader: ---------- 2026-03-08 17:45:02 (0:12:00) ----------
chaiml-grpo-glm46-chai-81744-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-81744-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-grpo-glm46-chai-81744-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-81744-v1-uploader:                              







chaiml-grpo-glm46-chai-81744-v1-uploader: ---------- 2026-03-08 17:46:02 (0:13:00) ----------
chaiml-grpo-glm46-chai-81744-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 50/80 (204.8G/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-81744-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-grpo-glm46-chai-81744-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-81744-v1-uploader:                              Processed model ChaiML/grpo-glm46-chai-rm-lr5e6-step150 in 4303.958s
chaiml-grpo-glm46-chai-81744-v1-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm46-chai-81744-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-81744-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm46-chai-81744-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm46-chai-81744-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm46-chai-81744-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-81744-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm46-chai-81744-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-81744-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm46-chai-81744-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-81744-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm46-chai-81744-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-81744-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm46-chai-81744-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm46-chai-81744-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm46-chai-81744-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm46-chai-81744-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm46-chai-81744-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm46-chai-81744-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/config.json
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/special_tokens_map.json
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/tokenizer_config.json
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/recipe.yaml
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/generation_config.json
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/chat_template.jinja
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model.safetensors.index.json
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/tokenizer.json
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00072-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00071-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00050-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00041-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00020-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00001-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00011-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00049-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00067-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00014-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00008-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00038-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00062-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00006-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00025-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00028-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00007-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00002-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00044-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00035-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00056-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00060-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00026-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00052-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00015-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00003-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00069-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00061-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00030-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00029-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00055-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00059-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00068-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00036-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00058-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00070-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00024-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00032-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00017-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00043-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00010-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00066-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00023-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00051-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00009-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00018-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00021-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00005-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00033-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00048-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00027-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00013-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00031-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00054-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00019-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00047-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00012-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00063-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00053-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00039-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00046-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00040-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00064-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00057-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00037-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00004-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00045-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00065-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00022-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00016-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00042-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v1-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v1/default/model-00034-of-00072.safetensors
Job chaiml-grpo-glm46-chai-81744-v1-uploader completed after 4464.25s with status: succeeded
Stopping job with name chaiml-grpo-glm46-chai-81744-v1-uploader
Pipeline stage VLLMUploader completed in 4464.82s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.88s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm46-chai-81744-v1
Waiting for inference service chaiml-grpo-glm46-chai-81744-v1 to be ready
Inference service chaiml-grpo-glm46-chai-81744-v1 ready after 511.4813663959503s
Pipeline stage VLLMDeployer completed in 512.69s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.1974937915802s
Received healthy response to inference request in 2.145380735397339s
Received healthy response to inference request in 2.138472080230713s
Received healthy response to inference request in 2.2306365966796875s
Received healthy response to inference request in 2.1358518600463867s
Received healthy response to inference request in 2.312675714492798s
Received healthy response to inference request in 2.4103357791900635s
Received healthy response to inference request in 2.2771689891815186s
Received healthy response to inference request in 2.1434881687164307s
Received healthy response to inference request in 2.3716652393341064s
Received healthy response to inference request in 2.180540084838867s
Received healthy response to inference request in 2.2561042308807373s
Received healthy response to inference request in 2.2695069313049316s
Received healthy response to inference request in 2.4046590328216553s
Received healthy response to inference request in 2.0952060222625732s
Received healthy response to inference request in 2.321467399597168s
Received healthy response to inference request in 2.1681973934173584s
Received healthy response to inference request in 2.1355185508728027s
Received healthy response to inference request in 2.1819560527801514s
Received healthy response to inference request in 2.107311248779297s
Received healthy response to inference request in 2.326429843902588s
Received healthy response to inference request in 2.082119941711426s
Received healthy response to inference request in 2.134185791015625s
Received healthy response to inference request in 2.1730520725250244s
Received healthy response to inference request in 2.440120220184326s
Received healthy response to inference request in 2.186476469039917s
Received healthy response to inference request in 2.3478140830993652s
Received healthy response to inference request in 2.2621593475341797s
Received healthy response to inference request in 2.1664817333221436s
Received healthy response to inference request in 2.134260654449463s
30 requests
0 failed requests
5th percentile: 2.100653374195099
10th percentile: 2.131498336791992
20th percentile: 2.13578519821167
30th percentile: 2.1448129653930663
40th percentile: 2.171110200881958
50th percentile: 2.184216260910034
60th percentile: 2.2408236503601073
70th percentile: 2.2718055486679076
80th percentile: 2.322459888458252
90th percentile: 2.3749646186828612
95th percentile: 2.40778124332428
99th percentile: 2.43148273229599
mean time: 2.224557868639628
Pipeline stage StressChecker completed in 71.60s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.81s
Shutdown handler de-registered
chaiml-grpo-glm46-chai-_81744_v1 status is now deployed due to DeploymentManager action
chaiml-grpo-glm46-chai-_81744_v1 status is now inactive due to auto deactivation removed underperforming models