developer_uid: chai_backend_admin
submission_id: chaiml-grpo-glm46-combi_16187_v1
model_name: chaiml-grpo-glm46-combi_16187_v1
model_group: ChaiML/grpo-glm46-combin
status: torndown
timestamp: 2026-03-11T21:21:30+00:00
num_battles: 12194
num_wins: 6299
celo_rating: 1300.01
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm46-combine-rm82-lr5e6-step50
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm46-combi_16187_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm46-combine-rm82-lr5e6-step50
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-08
win_ratio: 0.5165655240282105
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|user|>', '<|im_end|>', '</s>', '<|assistant|>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}.<|user|>\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm46-combi-16187-v1-uploader
Waiting for job on chaiml-grpo-glm46-combi-16187-v1-uploader to finish
chaiml-grpo-glm46-combi-16187-v1-uploader: Using quantization_mode: fp8
chaiml-grpo-glm46-combi-16187-v1-uploader: Checking if ChaiML/grpo-glm46-combine-rm82-lr5e6-step50-FP8 already exists in ChaiML
chaiml-grpo-glm46-combi-16187-v1-uploader: Downloading snapshot of ChaiML/grpo-glm46-combine-rm82-lr5e6-step50...
chaiml-grpo-glm46-combi-16187-v1-uploader: Downloaded in 241.417s
chaiml-grpo-glm46-combi-16187-v1-uploader: Loading /tmp/model_input...
chaiml-grpo-glm46-combi-16187-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-grpo-glm46-combi-16187-v1-uploader: `torch_dtype` is deprecated! Use `dtype` instead!
chaiml-grpo-glm46-combi-16187-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
chaiml-grpo-glm46-combi-16187-v1-uploader: Applying quantization...
chaiml-grpo-glm46-combi-16187-v1-uploader: 2026-03-08T12:00:39.287513-0700 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-grpo-glm46-combi-16187-v1-uploader: 2026-03-08T12:00:39.287995-0700 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
chaiml-grpo-glm46-combi-16187-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
chaiml-grpo-glm46-combi-16187-v1-uploader: 2026-03-08T12:20:24.167289-0700 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
chaiml-grpo-glm46-combi-16187-v1-uploader: 2026-03-08T12:20:30.285300-0700 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-grpo-glm46-combi-16187-v1-uploader: Saving to /dev/shm/model_output...
chaiml-grpo-glm46-combi-16187-v1-uploader: 2026-03-08T12:20:30.313311-0700 | get_model_compressor | INFO - skip_sparsity_compression_stats set to True. Skipping sparsity compression statistic calculations. No sparsity compressor will be applied.
Failed to get response for submission chaiml-grpo-glm46-combi_95250_v1: ('http://chaiml-grpo-glm46-combi-95250-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-combi_95250_v1: ('http://chaiml-grpo-glm46-combi-95250-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
chaiml-grpo-glm46-combi-16187-v1-uploader: Cleaning quantization config in /dev/shm/model_output
chaiml-grpo-glm46-combi-16187-v1-uploader: Pushing to ChaiML/grpo-glm46-combine-rm82-lr5e6-step50-FP8
chaiml-grpo-glm46-combi-16187-v1-uploader: Checking if ChaiML/grpo-glm46-combine-rm82-lr5e6-step50-FP8 already exists in ChaiML
chaiml-grpo-glm46-combi-16187-v1-uploader: Creating repo ChaiML/grpo-glm46-combine-rm82-lr5e6-step50-FP8 and uploading /dev/shm/model_output to it
chaiml-grpo-glm46-combi-16187-v1-uploader: ---------- 2026-03-08 13:02:20 (0:00:00) ----------
chaiml-grpo-glm46-combi-16187-v1-uploader: Files:   hashed 8/80 (28.2M/354.7G) | pre-uploaded: 0/1 (0.0/354.7G) (+72 unsure) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-16187-v1-uploader: Workers: hashing: 72 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 53
chaiml-grpo-glm46-combi-16187-v1-uploader: ---------------------------------------------------
Failed to get response for submission chaiml-grpo-glm46-combi_95250_v1: ('http://chaiml-grpo-glm46-combi-95250-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-combi_95250_v1: HTTPConnectionPool(host='chaiml-grpo-glm46-combi-95250-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
chaiml-grpo-glm46-combi-16187-v1-uploader: 







chaiml-grpo-glm46-combi-16187-v1-uploader: ---------- 2026-03-08 13:03:20 (0:01:00) ----------
chaiml-grpo-glm46-combi-16187-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 3/73 (5.1G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-16187-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 70 | committing: 0 | waiting: 56
chaiml-grpo-glm46-combi-16187-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-16187-v1-uploader:                              







chaiml-grpo-glm46-combi-16187-v1-uploader: ---------- 2026-03-08 13:04:20 (0:02:00) ----------
chaiml-grpo-glm46-combi-16187-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 11/73 (45.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-16187-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 62 | committing: 0 | waiting: 64
chaiml-grpo-glm46-combi-16187-v1-uploader: ---------------------------------------------------
Failed to get response for submission chaiml-grpo-glm46-combi_95250_v1: ('http://chaiml-grpo-glm46-combi-95250-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-combi_95250_v1: ('http://chaiml-grpo-glm46-combi-95250-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
chaiml-grpo-glm46-combi-16187-v1-uploader:                              







chaiml-grpo-glm46-combi-16187-v1-uploader: ---------- 2026-03-08 13:05:20 (0:03:00) ----------
chaiml-grpo-glm46-combi-16187-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 18/73 (80.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-16187-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 55 | committing: 0 | waiting: 71
chaiml-grpo-glm46-combi-16187-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-16187-v1-uploader:                              







chaiml-grpo-glm46-combi-16187-v1-uploader: ---------- 2026-03-08 13:07:20 (0:05:00) ----------
chaiml-grpo-glm46-combi-16187-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 27/73 (125.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-16187-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 46 | committing: 0 | waiting: 80
chaiml-grpo-glm46-combi-16187-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-16187-v1-uploader:                              







chaiml-grpo-glm46-combi-16187-v1-uploader: ---------- 2026-03-08 13:08:20 (0:06:00) ----------
chaiml-grpo-glm46-combi-16187-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 35/73 (164.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-16187-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 38 | committing: 0 | waiting: 88
chaiml-grpo-glm46-combi-16187-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-16187-v1-uploader:                              







chaiml-grpo-glm46-combi-16187-v1-uploader: ---------- 2026-03-08 13:09:20 (0:07:00) ----------
chaiml-grpo-glm46-combi-16187-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 43/73 (204.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-16187-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 30 | committing: 0 | waiting: 96
chaiml-grpo-glm46-combi-16187-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-16187-v1-uploader:                              







chaiml-grpo-glm46-combi-16187-v1-uploader: ---------- 2026-03-08 13:10:20 (0:08:00) ----------
chaiml-grpo-glm46-combi-16187-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 50/73 (239.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-16187-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 23 | committing: 0 | waiting: 103
chaiml-grpo-glm46-combi-16187-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-16187-v1-uploader:                              







chaiml-grpo-glm46-combi-16187-v1-uploader: ---------- 2026-03-08 13:11:20 (0:09:00) ----------
chaiml-grpo-glm46-combi-16187-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 51/73 (244.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-16187-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 22 | committing: 0 | waiting: 104
chaiml-grpo-glm46-combi-16187-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-16187-v1-uploader:                              







chaiml-grpo-glm46-combi-16187-v1-uploader: ---------- 2026-03-08 13:12:20 (0:10:00) ----------
chaiml-grpo-glm46-combi-16187-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 59/73 (284.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-16187-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 14 | committing: 0 | waiting: 112
chaiml-grpo-glm46-combi-16187-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-16187-v1-uploader:                              







chaiml-grpo-glm46-combi-16187-v1-uploader: ---------- 2026-03-08 13:13:20 (0:11:00) ----------
chaiml-grpo-glm46-combi-16187-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 67/73 (324.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-16187-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 6 | committing: 0 | waiting: 120
chaiml-grpo-glm46-combi-16187-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-16187-v1-uploader:                              







chaiml-grpo-glm46-combi-16187-v1-uploader: ---------- 2026-03-08 13:14:20 (0:12:00) ----------
chaiml-grpo-glm46-combi-16187-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-16187-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-grpo-glm46-combi-16187-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-16187-v1-uploader:                              







chaiml-grpo-glm46-combi-16187-v1-uploader: ---------- 2026-03-08 13:15:20 (0:13:00) ----------
chaiml-grpo-glm46-combi-16187-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 50/80 (204.9G/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-16187-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-grpo-glm46-combi-16187-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-16187-v1-uploader:                              Processed model ChaiML/grpo-glm46-combine-rm82-lr5e6-step50 in 4872.633s
chaiml-grpo-glm46-combi-16187-v1-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm46-combi-16187-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-16187-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm46-combi-16187-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm46-combi-16187-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm46-combi-16187-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-16187-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm46-combi-16187-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-16187-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm46-combi-16187-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-16187-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm46-combi-16187-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-16187-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm46-combi-16187-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm46-combi-16187-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm46-combi-16187-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm46-combi-16187-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm46-combi-16187-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm46-combi-16187-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/chat_template.jinja
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/config.json
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/generation_config.json
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/special_tokens_map.json
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/tokenizer_config.json
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/recipe.yaml
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model.safetensors.index.json
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/tokenizer.json
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00072-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00071-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00040-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00063-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00054-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00020-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00025-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00009-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00034-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00051-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00005-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00032-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00050-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00041-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00028-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00011-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00012-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00057-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00016-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00058-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00035-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00059-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00027-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00052-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00023-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00042-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00047-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00033-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00062-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00003-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00060-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00017-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00044-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00010-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00037-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00068-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00069-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00001-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00022-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00002-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00039-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00026-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00024-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00031-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00064-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00006-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00036-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00008-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00067-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00007-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00013-of-00072.safetensors
chaiml-grpo-glm46-combi-16187-v1-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-16187-v1/default/model-00053-of-00072.safetensors
Job chaiml-grpo-glm46-combi-16187-v1-uploader completed after 5037.86s with status: succeeded
Stopping job with name chaiml-grpo-glm46-combi-16187-v1-uploader
Pipeline stage VLLMUploader completed in 5038.45s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.26s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm46-combi-16187-v1
Waiting for inference service chaiml-grpo-glm46-combi-16187-v1 to be ready
Inference service chaiml-grpo-glm46-combi-16187-v1 ready after 561.2108664512634s
Pipeline stage VLLMDeployer completed in 561.78s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.4009201526641846s
Received healthy response to inference request in 2.3277058601379395s
Received healthy response to inference request in 2.191312074661255s
Received healthy response to inference request in 2.118347406387329s
Received healthy response to inference request in 2.158820629119873s
Received healthy response to inference request in 2.2418899536132812s
Received healthy response to inference request in 2.098198652267456s
Received healthy response to inference request in 2.2190909385681152s
Received healthy response to inference request in 2.62947940826416s
Received healthy response to inference request in 2.118652105331421s
Received healthy response to inference request in 2.086409568786621s
Received healthy response to inference request in 2.211158037185669s
Received healthy response to inference request in 2.102259874343872s
Received healthy response to inference request in 2.1105446815490723s
Received healthy response to inference request in 2.1048972606658936s
Received healthy response to inference request in 2.2974483966827393s
Received healthy response to inference request in 2.641362428665161s
Received healthy response to inference request in 2.15167498588562s
Received healthy response to inference request in 2.088827610015869s
Received healthy response to inference request in 2.2186036109924316s
Received healthy response to inference request in 2.3794307708740234s
Received healthy response to inference request in 2.105989456176758s
Received healthy response to inference request in 2.1112256050109863s
Received healthy response to inference request in 2.1319637298583984s
Received healthy response to inference request in 2.2970471382141113s
Received healthy response to inference request in 2.125448703765869s
Received healthy response to inference request in 2.214747667312622s
Received healthy response to inference request in 2.14121675491333s
Received healthy response to inference request in 2.3095309734344482s
Received healthy response to inference request in 2.2512502670288086s
30 requests
0 failed requests
5th percentile: 2.0930445790290833
10th percentile: 2.1018537521362304
20th percentile: 2.1096336364746096
30th percentile: 2.1185606956481933
40th percentile: 2.1375155448913574
50th percentile: 2.175066351890564
60th percentile: 2.2162900447845457
70th percentile: 2.2446980476379395
80th percentile: 2.299864912033081
90th percentile: 2.3815797090530397
95th percentile: 2.5266277432441706
99th percentile: 2.6379163527488707
mean time: 2.219515156745911
Pipeline stage StressChecker completed in 70.55s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.22s
Shutdown handler de-registered
chaiml-grpo-glm46-combi_16187_v1 status is now deployed due to DeploymentManager action
chaiml-grpo-glm46-combi_16187_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-grpo-glm46-combi_16187_v1 status is now torndown due to DeploymentManager action