developer_uid: chai_backend_admin
submission_id: chaiml-grpo-glm46-combi_95250_v1
model_name: chaiml-grpo-glm46-combi_95250_v1
model_group: ChaiML/grpo-glm46-combin
status: torndown
timestamp: 2026-03-11T20:51:59+00:00
num_battles: 12863
num_wins: 6775
celo_rating: 1312.34
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm46-combine-rm82-lr5e6-step100
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm46-combi_95250_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm46-combine-rm82-lr5e6-step100
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-08
win_ratio: 0.5267045012827489
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '####', '<|im_end|>', '<|user|>', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}.<|user|>\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm46-combi-95250-v1-uploader
Waiting for job on chaiml-grpo-glm46-combi-95250-v1-uploader to finish
chaiml-grpo-glm46-combi-95250-v1-uploader: Using quantization_mode: fp8
chaiml-grpo-glm46-combi-95250-v1-uploader: Checking if ChaiML/grpo-glm46-combine-rm82-lr5e6-step100-FP8 already exists in ChaiML
chaiml-grpo-glm46-combi-95250-v1-uploader: Downloading snapshot of ChaiML/grpo-glm46-combine-rm82-lr5e6-step100...
chaiml-grpo-glm46-combi-95250-v1-uploader: Downloaded in 241.486s
chaiml-grpo-glm46-combi-95250-v1-uploader: Loading /tmp/model_input...
chaiml-grpo-glm46-combi-95250-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-grpo-glm46-combi-95250-v1-uploader: `torch_dtype` is deprecated! Use `dtype` instead!
chaiml-grpo-glm46-combi-95250-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
chaiml-grpo-glm46-combi-95250-v1-uploader: Applying quantization...
chaiml-grpo-glm46-combi-95250-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-grpo-glm46-combi-95250-v1-uploader: 2026-03-08T11:26:53.246864-0700 | reset | INFO - Compression lifecycle reset
chaiml-grpo-glm46-combi-95250-v1-uploader: 2026-03-08T11:26:53.314742-0700 | from_modifiers | INFO - Creating recipe from modifiers
chaiml-grpo-glm46-combi-95250-v1-uploader: 2026-03-08T11:27:48.773221-0700 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-grpo-glm46-combi-95250-v1-uploader: 2026-03-08T11:27:48.773695-0700 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
chaiml-grpo-glm46-combi-95250-v1-uploader: 2026-03-08T11:38:05.052785-0700 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
chaiml-grpo-glm46-combi-95250-v1-uploader: 2026-03-08T11:38:11.426221-0700 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-grpo-glm46-combi-95250-v1-uploader: Saving to /dev/shm/model_output...
chaiml-grpo-glm46-combi-95250-v1-uploader: 2026-03-08T11:38:11.453739-0700 | get_model_compressor | INFO - skip_sparsity_compression_stats set to True. Skipping sparsity compression statistic calculations. No sparsity compressor will be applied.
chaiml-grpo-glm46-combi-95250-v1-uploader: Cleaning quantization config in /dev/shm/model_output
chaiml-grpo-glm46-combi-95250-v1-uploader: Pushing to ChaiML/grpo-glm46-combine-rm82-lr5e6-step100-FP8
chaiml-grpo-glm46-combi-95250-v1-uploader: Checking if ChaiML/grpo-glm46-combine-rm82-lr5e6-step100-FP8 already exists in ChaiML
chaiml-grpo-glm46-combi-95250-v1-uploader: Creating repo ChaiML/grpo-glm46-combine-rm82-lr5e6-step100-FP8 and uploading /dev/shm/model_output to it
chaiml-grpo-glm46-combi-95250-v1-uploader: ---------- 2026-03-08 12:20:21 (0:00:00) ----------
chaiml-grpo-glm46-combi-95250-v1-uploader: Files:   hashed 8/80 (28.2M/354.7G) | pre-uploaded: 0/1 (0.0/354.7G) (+72 unsure) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-95250-v1-uploader: Workers: hashing: 72 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 53
chaiml-grpo-glm46-combi-95250-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-95250-v1-uploader: 







chaiml-grpo-glm46-combi-95250-v1-uploader: ---------- 2026-03-08 12:21:21 (0:01:00) ----------
chaiml-grpo-glm46-combi-95250-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 3/73 (5.1G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-95250-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 70 | committing: 0 | waiting: 56
chaiml-grpo-glm46-combi-95250-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-95250-v1-uploader:                              







chaiml-grpo-glm46-combi-95250-v1-uploader: ---------- 2026-03-08 12:22:21 (0:02:00) ----------
chaiml-grpo-glm46-combi-95250-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 11/73 (45.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-95250-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 62 | committing: 0 | waiting: 64
chaiml-grpo-glm46-combi-95250-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-95250-v1-uploader:                              







chaiml-grpo-glm46-combi-95250-v1-uploader: ---------- 2026-03-08 12:23:21 (0:03:00) ----------
chaiml-grpo-glm46-combi-95250-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 17/73 (75.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-95250-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 56 | committing: 0 | waiting: 70
chaiml-grpo-glm46-combi-95250-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-95250-v1-uploader:                              







chaiml-grpo-glm46-combi-95250-v1-uploader: ---------- 2026-03-08 12:24:21 (0:04:00) ----------
chaiml-grpo-glm46-combi-95250-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 19/73 (85.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-95250-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 54 | committing: 0 | waiting: 72
chaiml-grpo-glm46-combi-95250-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-95250-v1-uploader:                              







chaiml-grpo-glm46-combi-95250-v1-uploader: ---------- 2026-03-08 12:25:21 (0:05:00) ----------
chaiml-grpo-glm46-combi-95250-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 27/73 (124.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-95250-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 46 | committing: 0 | waiting: 80
chaiml-grpo-glm46-combi-95250-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-95250-v1-uploader:                              







chaiml-grpo-glm46-combi-95250-v1-uploader: ---------- 2026-03-08 12:26:21 (0:06:00) ----------
chaiml-grpo-glm46-combi-95250-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 35/73 (164.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-95250-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 38 | committing: 0 | waiting: 88
chaiml-grpo-glm46-combi-95250-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-95250-v1-uploader:                              







chaiml-grpo-glm46-combi-95250-v1-uploader: ---------- 2026-03-08 12:27:21 (0:07:00) ----------
chaiml-grpo-glm46-combi-95250-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 43/73 (204.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-95250-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 30 | committing: 0 | waiting: 96
chaiml-grpo-glm46-combi-95250-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-95250-v1-uploader:                              







chaiml-grpo-glm46-combi-95250-v1-uploader: ---------- 2026-03-08 12:28:21 (0:08:00) ----------
chaiml-grpo-glm46-combi-95250-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 45/73 (214.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-95250-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 28 | committing: 0 | waiting: 98
chaiml-grpo-glm46-combi-95250-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-95250-v1-uploader:                              







chaiml-grpo-glm46-combi-95250-v1-uploader: ---------- 2026-03-08 12:29:21 (0:09:00) ----------
chaiml-grpo-glm46-combi-95250-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 51/73 (244.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-95250-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 22 | committing: 0 | waiting: 104
chaiml-grpo-glm46-combi-95250-v1-uploader: ---------------------------------------------------
Unable to record family friendly update due to error: Invalid JSON input: Expecting value: line 1 column 1 (char 0)
chaiml-grpo-glm46-combi-95250-v1-uploader:                              







chaiml-grpo-glm46-combi-95250-v1-uploader: ---------- 2026-03-08 12:30:21 (0:10:00) ----------
chaiml-grpo-glm46-combi-95250-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 59/73 (284.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-95250-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 14 | committing: 0 | waiting: 112
chaiml-grpo-glm46-combi-95250-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-95250-v1-uploader:                              







chaiml-grpo-glm46-combi-95250-v1-uploader: ---------- 2026-03-08 12:31:21 (0:11:00) ----------
chaiml-grpo-glm46-combi-95250-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 67/73 (324.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-95250-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 6 | committing: 0 | waiting: 120
chaiml-grpo-glm46-combi-95250-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-95250-v1-uploader:                              







chaiml-grpo-glm46-combi-95250-v1-uploader: ---------- 2026-03-08 12:32:21 (0:12:00) ----------
chaiml-grpo-glm46-combi-95250-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-95250-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-grpo-glm46-combi-95250-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-95250-v1-uploader:                              







chaiml-grpo-glm46-combi-95250-v1-uploader: ---------- 2026-03-08 12:33:21 (0:13:00) ----------
chaiml-grpo-glm46-combi-95250-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 50/80 (204.9G/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-95250-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-grpo-glm46-combi-95250-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-95250-v1-uploader:                              Processed model ChaiML/grpo-glm46-combine-rm82-lr5e6-step100 in 4367.533s
chaiml-grpo-glm46-combi-95250-v1-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm46-combi-95250-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-95250-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm46-combi-95250-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm46-combi-95250-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm46-combi-95250-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-95250-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm46-combi-95250-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-95250-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm46-combi-95250-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-95250-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm46-combi-95250-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-95250-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm46-combi-95250-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm46-combi-95250-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm46-combi-95250-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm46-combi-95250-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm46-combi-95250-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm46-combi-95250-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/tokenizer_config.json
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/config.json
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/chat_template.jinja
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model.safetensors.index.json
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/recipe.yaml
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/tokenizer.json
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/generation_config.json
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/special_tokens_map.json
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00072-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00071-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00048-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00039-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00037-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00003-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00025-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00023-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00008-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00002-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00045-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00067-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00005-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00046-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00068-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00009-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00027-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00016-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00004-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00021-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00062-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00026-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00013-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00029-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00036-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00015-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00017-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00035-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00044-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00010-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00001-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00059-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00032-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00042-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00052-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00063-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00070-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00014-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00012-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00031-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00006-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00061-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00020-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00057-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00050-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00051-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00054-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00069-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00022-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00041-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00058-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00034-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00007-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00019-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00049-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00011-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00043-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00065-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00033-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00028-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00030-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00018-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00047-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00066-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00055-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00040-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00038-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00024-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00053-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00060-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00064-of-00072.safetensors
chaiml-grpo-glm46-combi-95250-v1-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-95250-v1/default/model-00056-of-00072.safetensors
Job chaiml-grpo-glm46-combi-95250-v1-uploader completed after 4521.2s with status: succeeded
Stopping job with name chaiml-grpo-glm46-combi-95250-v1-uploader
Pipeline stage VLLMUploader completed in 4522.13s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.97s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm46-combi-95250-v1
Waiting for inference service chaiml-grpo-glm46-combi-95250-v1 to be ready
Inference service chaiml-grpo-glm46-combi-95250-v1 ready after 552.0043408870697s
Pipeline stage VLLMDeployer completed in 552.58s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.3197858333587646s
Received healthy response to inference request in 2.386530637741089s
Received healthy response to inference request in 2.145657539367676s
Received healthy response to inference request in 2.1234889030456543s
Received healthy response to inference request in 2.217270612716675s
Received healthy response to inference request in 2.477803945541382s
Received healthy response to inference request in 2.7399253845214844s
Received healthy response to inference request in 2.229193925857544s
Received healthy response to inference request in 2.1717774868011475s
Received healthy response to inference request in 2.3101351261138916s
Received healthy response to inference request in 2.4238882064819336s
Received healthy response to inference request in 2.3987913131713867s
Received healthy response to inference request in 2.1765472888946533s
Received healthy response to inference request in 2.512061595916748s
Received healthy response to inference request in 2.129345178604126s
Received healthy response to inference request in 2.1289122104644775s
Received healthy response to inference request in 2.1132545471191406s
Received healthy response to inference request in 2.246896266937256s
Received healthy response to inference request in 2.2499923706054688s
Received healthy response to inference request in 2.2846169471740723s
Received healthy response to inference request in 2.241652488708496s
Received healthy response to inference request in 2.2320029735565186s
Received healthy response to inference request in 2.212899684906006s
Received healthy response to inference request in 2.1872122287750244s
Received healthy response to inference request in 2.0902469158172607s
Received healthy response to inference request in 2.1870687007904053s
Received healthy response to inference request in 2.3033440113067627s
Received healthy response to inference request in 2.1204118728637695s
Received healthy response to inference request in 2.2578957080841064s
Received healthy response to inference request in 2.272381544113159s
30 requests
0 failed requests
5th percentile: 2.1164753437042236
10th percentile: 2.123181200027466
20th percentile: 2.1423950672149656
30th percentile: 2.1839122772216797
40th percentile: 2.2155222415924074
50th percentile: 2.2368277311325073
60th percentile: 2.2531537055969237
70th percentile: 2.2902350664138793
80th percentile: 2.33313479423523
90th percentile: 2.4292797803878785
95th percentile: 2.496645653247833
99th percentile: 2.673844885826111
mean time: 2.2630330483118692
Pipeline stage StressChecker completed in 73.19s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.85s
Shutdown handler de-registered
chaiml-grpo-glm46-combi_95250_v1 status is now deployed due to DeploymentManager action
chaiml-grpo-glm46-combi_95250_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-grpo-glm46-combi_95250_v1 status is now torndown due to DeploymentManager action