developer_uid: chai_backend_admin
submission_id: chaiml-grpo-glm46-combi_79432_v1
model_name: chaiml-grpo-glm46-combi_79432_v1
model_group: ChaiML/grpo-glm46-combin
status: torndown
timestamp: 2026-03-11T10:31:38+00:00
num_battles: 11550
num_wins: 6202
celo_rating: 1321.72
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm46-combine-rm82-lr5e6-step150
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm46-combi_79432_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm46-combine-rm82-lr5e6-step150
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-08
win_ratio: 0.536969696969697
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|user|>', '<|im_end|>', '</s>', '<|assistant|>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}.<|user|>\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm46-combi-79432-v1-uploader
Waiting for job on chaiml-grpo-glm46-combi-79432-v1-uploader to finish
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v3: ('http://chaiml-pony-d3-g46-pv2-64172-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
chaiml-grpo-glm46-combi-79432-v1-uploader: Using quantization_mode: fp8
chaiml-grpo-glm46-combi-79432-v1-uploader: Checking if ChaiML/grpo-glm46-combine-rm82-lr5e6-step150-FP8 already exists in ChaiML
chaiml-grpo-glm46-combi-79432-v1-uploader: Downloading snapshot of ChaiML/grpo-glm46-combine-rm82-lr5e6-step150...
chaiml-grpo-glm46-combi-79432-v1-uploader: Downloaded in 225.532s
chaiml-grpo-glm46-combi-79432-v1-uploader: Loading /tmp/model_input...
chaiml-grpo-glm46-combi-79432-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-grpo-glm46-combi-79432-v1-uploader: `torch_dtype` is deprecated! Use `dtype` instead!
chaiml-grpo-glm46-combi-79432-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
chaiml-grpo-glm46-combi-79432-v1-uploader: Applying quantization...
chaiml-grpo-glm46-combi-79432-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-grpo-glm46-combi-79432-v1-uploader: 2026-03-07T23:22:23.588195-0800 | reset | INFO - Compression lifecycle reset
chaiml-grpo-glm46-combi-79432-v1-uploader: 2026-03-07T23:22:23.658233-0800 | from_modifiers | INFO - Creating recipe from modifiers
chaiml-grpo-glm46-combi-79432-v1-uploader: 2026-03-07T23:23:20.817038-0800 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-grpo-glm46-combi-79432-v1-uploader: 2026-03-07T23:23:20.817563-0800 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
chaiml-grpo-glm46-combi-79432-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v3: ('http://chaiml-pony-d3-g46-pv2-64172-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v3: ('http://chaiml-pony-d3-g46-pv2-64172-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
chaiml-grpo-glm46-combi-79432-v1-uploader: 2026-03-07T23:35:31.341852-0800 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
chaiml-grpo-glm46-combi-79432-v1-uploader: 2026-03-07T23:35:38.204274-0800 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-grpo-glm46-combi-79432-v1-uploader: Saving to /dev/shm/model_output...
chaiml-grpo-glm46-combi-79432-v1-uploader: 2026-03-07T23:35:38.233110-0800 | get_model_compressor | INFO - skip_sparsity_compression_stats set to True. Skipping sparsity compression statistic calculations. No sparsity compressor will be applied.
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v3: HTTPConnectionPool(host='chaiml-pony-d3-g46-pv2-64172-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v3: ('http://chaiml-pony-d3-g46-pv2-64172-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v3: HTTPConnectionPool(host='chaiml-pony-d3-g46-pv2-64172-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v3: ('http://chaiml-pony-d3-g46-pv2-64172-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v3: ('http://chaiml-pony-d3-g46-pv2-64172-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v3: ('http://chaiml-pony-d3-g46-pv2-64172-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v3: ('http://chaiml-pony-d3-g46-pv2-64172-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v3: ('http://chaiml-pony-d3-g46-pv2-64172-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v3: ('http://chaiml-pony-d3-g46-pv2-64172-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
chaiml-grpo-glm46-combi-79432-v1-uploader: Cleaning quantization config in /dev/shm/model_output
chaiml-grpo-glm46-combi-79432-v1-uploader: Pushing to ChaiML/grpo-glm46-combine-rm82-lr5e6-step150-FP8
chaiml-grpo-glm46-combi-79432-v1-uploader: Checking if ChaiML/grpo-glm46-combine-rm82-lr5e6-step150-FP8 already exists in ChaiML
chaiml-grpo-glm46-combi-79432-v1-uploader: Creating repo ChaiML/grpo-glm46-combine-rm82-lr5e6-step150-FP8 and uploading /dev/shm/model_output to it
chaiml-grpo-glm46-combi-79432-v1-uploader: ---------- 2026-03-08 00:17:16 (0:00:00) ----------
chaiml-grpo-glm46-combi-79432-v1-uploader: Files:   hashed 8/80 (28.2M/354.7G) | pre-uploaded: 0/1 (0.0/354.7G) (+72 unsure) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-79432-v1-uploader: Workers: hashing: 72 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 53
chaiml-grpo-glm46-combi-79432-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-79432-v1-uploader: 







chaiml-grpo-glm46-combi-79432-v1-uploader: ---------- 2026-03-08 00:18:16 (0:01:00) ----------
chaiml-grpo-glm46-combi-79432-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 3/73 (5.1G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-79432-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 70 | committing: 0 | waiting: 56
chaiml-grpo-glm46-combi-79432-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-79432-v1-uploader:                              







chaiml-grpo-glm46-combi-79432-v1-uploader: ---------- 2026-03-08 00:19:16 (0:02:00) ----------
chaiml-grpo-glm46-combi-79432-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 11/73 (45.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-79432-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 62 | committing: 0 | waiting: 64
chaiml-grpo-glm46-combi-79432-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-79432-v1-uploader:                              







chaiml-grpo-glm46-combi-79432-v1-uploader: ---------- 2026-03-08 00:20:16 (0:03:00) ----------
chaiml-grpo-glm46-combi-79432-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 18/73 (80.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-79432-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 55 | committing: 0 | waiting: 71
chaiml-grpo-glm46-combi-79432-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-79432-v1-uploader:                              







chaiml-grpo-glm46-combi-79432-v1-uploader: ---------- 2026-03-08 00:21:16 (0:04:00) ----------
chaiml-grpo-glm46-combi-79432-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 19/73 (85.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-79432-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 54 | committing: 0 | waiting: 72
chaiml-grpo-glm46-combi-79432-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-79432-v1-uploader:                              







chaiml-grpo-glm46-combi-79432-v1-uploader: ---------- 2026-03-08 00:22:16 (0:05:00) ----------
chaiml-grpo-glm46-combi-79432-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 27/73 (125.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-79432-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 46 | committing: 0 | waiting: 80
chaiml-grpo-glm46-combi-79432-v1-uploader: ---------------------------------------------------
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v3: ('http://chaiml-pony-d3-g46-pv2-64172-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v3: ('http://chaiml-pony-d3-g46-pv2-64172-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v3: ('http://chaiml-pony-d3-g46-pv2-64172-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
chaiml-grpo-glm46-combi-79432-v1-uploader:                              







chaiml-grpo-glm46-combi-79432-v1-uploader: ---------- 2026-03-08 00:23:16 (0:06:00) ----------
chaiml-grpo-glm46-combi-79432-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 35/73 (164.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-79432-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 38 | committing: 0 | waiting: 88
chaiml-grpo-glm46-combi-79432-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-79432-v1-uploader:                              







chaiml-grpo-glm46-combi-79432-v1-uploader: ---------- 2026-03-08 00:24:16 (0:07:00) ----------
chaiml-grpo-glm46-combi-79432-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 43/73 (204.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-79432-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 30 | committing: 0 | waiting: 96
chaiml-grpo-glm46-combi-79432-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-79432-v1-uploader:                              







chaiml-grpo-glm46-combi-79432-v1-uploader: ---------- 2026-03-08 00:25:16 (0:08:00) ----------
chaiml-grpo-glm46-combi-79432-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 51/73 (244.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-79432-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 22 | committing: 0 | waiting: 104
chaiml-grpo-glm46-combi-79432-v1-uploader: ---------------------------------------------------
Retrying (%r) after connection broken by '%r': %s
chaiml-grpo-glm46-combi-79432-v1-uploader:                              







chaiml-grpo-glm46-combi-79432-v1-uploader: ---------- 2026-03-08 00:26:16 (0:09:00) ----------
chaiml-grpo-glm46-combi-79432-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 57/73 (274.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-79432-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 16 | committing: 0 | waiting: 110
chaiml-grpo-glm46-combi-79432-v1-uploader: ---------------------------------------------------
Retrying (%r) after connection broken by '%r': %s
chaiml-grpo-glm46-combi-79432-v1-uploader:                              







chaiml-grpo-glm46-combi-79432-v1-uploader: ---------- 2026-03-08 00:27:16 (0:10:00) ----------
chaiml-grpo-glm46-combi-79432-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 61/73 (294.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-79432-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 12 | committing: 0 | waiting: 114
chaiml-grpo-glm46-combi-79432-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-79432-v1-uploader:                              







chaiml-grpo-glm46-combi-79432-v1-uploader: ---------- 2026-03-08 00:28:16 (0:11:00) ----------
chaiml-grpo-glm46-combi-79432-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-79432-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-grpo-glm46-combi-79432-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-79432-v1-uploader:                              







chaiml-grpo-glm46-combi-79432-v1-uploader: ---------- 2026-03-08 00:29:16 (0:12:00) ----------
chaiml-grpo-glm46-combi-79432-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-79432-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-grpo-glm46-combi-79432-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-79432-v1-uploader:                              







chaiml-grpo-glm46-combi-79432-v1-uploader: ---------- 2026-03-08 00:30:17 (0:13:00) ----------
chaiml-grpo-glm46-combi-79432-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 50/80 (204.9G/354.7G) | ignored: 0
chaiml-grpo-glm46-combi-79432-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-grpo-glm46-combi-79432-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-combi-79432-v1-uploader:                              Processed model ChaiML/grpo-glm46-combine-rm82-lr5e6-step150 in 4380.411s
chaiml-grpo-glm46-combi-79432-v1-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm46-combi-79432-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-79432-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm46-combi-79432-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm46-combi-79432-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm46-combi-79432-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-79432-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm46-combi-79432-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-79432-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm46-combi-79432-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-79432-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm46-combi-79432-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-79432-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm46-combi-79432-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm46-combi-79432-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm46-combi-79432-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm46-combi-79432-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm46-combi-79432-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm46-combi-79432-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/chat_template.jinja
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/config.json
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/generation_config.json
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/recipe.yaml
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/special_tokens_map.json
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/tokenizer_config.json
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model.safetensors.index.json
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/tokenizer.json
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00072-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00071-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00015-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00044-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00014-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00061-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00051-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00050-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00006-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00009-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00008-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00001-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00046-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00045-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00021-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00040-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00037-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00063-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00047-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00055-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00056-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00060-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00068-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00018-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00025-of-00072.safetensors
chaiml-grpo-glm46-combi-79432-v1-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-79432-v1/default/model-00066-of-00072.safetensors
Job chaiml-grpo-glm46-combi-79432-v1-uploader completed after 4541.04s with status: succeeded
Stopping job with name chaiml-grpo-glm46-combi-79432-v1-uploader
Pipeline stage VLLMUploader completed in 4541.57s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.92s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm46-combi-79432-v1
Waiting for inference service chaiml-grpo-glm46-combi-79432-v1 to be ready
Inference service chaiml-grpo-glm46-combi-79432-v1 ready after 551.7170944213867s
Pipeline stage VLLMDeployer completed in 552.29s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.3421807289123535s
Received healthy response to inference request in 3.64538836479187s
Received healthy response to inference request in 4.455213785171509s
Received healthy response to inference request in 2.183856248855591s
Received healthy response to inference request in 3.791494607925415s
Received healthy response to inference request in 3.8071188926696777s
Received healthy response to inference request in 4.268755912780762s
Received healthy response to inference request in 3.1676971912384033s
Received healthy response to inference request in 2.1961958408355713s
Received healthy response to inference request in 3.9328746795654297s
Received healthy response to inference request in 3.400808095932007s
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v6: ('http://chaiml-pony-d3-g46-pv2-64172-v6-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Received healthy response to inference request in 4.018600225448608s
Received healthy response to inference request in 4.411339282989502s
Received healthy response to inference request in 4.9896862506866455s
Received healthy response to inference request in 3.9422101974487305s
Received healthy response to inference request in 3.001424551010132s
Received healthy response to inference request in 3.6907317638397217s
Received healthy response to inference request in 2.19240140914917s
Received healthy response to inference request in 2.803391695022583s
Received healthy response to inference request in 5.22694993019104s
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v5: HTTPConnectionPool(host='chaiml-pony-d3-g46-pv2-64172-v5-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Received healthy response to inference request in 4.8714799880981445s
Received healthy response to inference request in 2.323253631591797s
Received healthy response to inference request in 8.487486362457275s
Failed to get response for submission chaiml-pony-d3-g46-pv2-_64172_v5: ('http://chaiml-pony-d3-g46-pv2-64172-v5-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Received healthy response to inference request in 5.13721489906311s
Received healthy response to inference request in 5.274026155471802s
Received healthy response to inference request in 3.786804676055908s
Received healthy response to inference request in 7.03693962097168s
Received healthy response to inference request in 6.075695037841797s
Received healthy response to inference request in 4.653975248336792s
Received healthy response to inference request in 6.006698131561279s
30 requests
0 failed requests
5th percentile: 2.1941089034080505
10th percentile: 2.3105478525161742
20th percentile: 3.134442663192749
30th percentile: 3.6771287441253664
40th percentile: 3.800869178771973
50th percentile: 3.9804052114486694
60th percentile: 4.369844150543213
70th percentile: 4.719226670265197
80th percentile: 5.155161905288696
90th percentile: 6.013597822189332
95th percentile: 6.604379558563229
99th percentile: 8.066827807426455
mean time: 4.23739644686381
Pipeline stage StressChecker completed in 132.60s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.77s
Shutdown handler de-registered
chaiml-grpo-glm46-combi_79432_v1 status is now deployed due to DeploymentManager action
chaiml-grpo-glm46-combi_79432_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-grpo-glm46-combi_79432_v1 status is now torndown due to DeploymentManager action