developer_uid: chai_backend_admin
submission_id: chaiml-grpo-glm46-chai-_90872_v1
model_name: chaiml-grpo-glm46-chai-_90872_v1
model_group: ChaiML/grpo-glm46-chai-r
status: inactive
timestamp: 2026-03-09T06:46:55+00:00
num_battles: 12624
num_wins: 6704
celo_rating: 1313.29
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm46-chai-rm-lr5e6-step50
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm46-chai-_90872_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm46-chai-rm-lr5e6-step50
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-08
win_ratio: 0.5310519645120405
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|user|>', '</s>', '<|assistant|>', '####', '<|im_end|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}.<|user|>\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm46-chai-90872-v1-uploader
Waiting for job on chaiml-grpo-glm46-chai-90872-v1-uploader to finish
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
chaiml-grpo-glm46-chai-90872-v1-uploader: Using quantization_mode: fp8
chaiml-grpo-glm46-chai-90872-v1-uploader: Checking if ChaiML/grpo-glm46-chai-rm-lr5e6-step50-FP8 already exists in ChaiML
chaiml-grpo-glm46-chai-90872-v1-uploader: Downloading snapshot of ChaiML/grpo-glm46-chai-rm-lr5e6-step50...
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: HTTPConnectionPool(host='chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
chaiml-grpo-glm46-chai-90872-v1-uploader: Downloaded in 222.045s
chaiml-grpo-glm46-chai-90872-v1-uploader: Loading /tmp/model_input...
chaiml-grpo-glm46-chai-90872-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-grpo-glm46-chai-90872-v1-uploader: `torch_dtype` is deprecated! Use `dtype` instead!
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
chaiml-grpo-glm46-chai-90872-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
chaiml-grpo-glm46-chai-90872-v1-uploader: Applying quantization...
chaiml-grpo-glm46-chai-90872-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-grpo-glm46-chai-90872-v1-uploader: 2026-03-08T20:40:12.428081-0700 | reset | INFO - Compression lifecycle reset
chaiml-grpo-glm46-chai-90872-v1-uploader: 2026-03-08T20:40:12.494019-0700 | from_modifiers | INFO - Creating recipe from modifiers
chaiml-grpo-glm46-chai-90872-v1-uploader: 2026-03-08T20:41:08.811366-0700 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-grpo-glm46-chai-90872-v1-uploader: 2026-03-08T20:41:08.811869-0700 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
chaiml-grpo-glm46-chai-90872-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: HTTPConnectionPool(host='chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
chaiml-grpo-glm46-chai-90872-v1-uploader: 2026-03-08T20:52:31.605221-0700 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
chaiml-grpo-glm46-chai-90872-v1-uploader: 2026-03-08T20:52:37.813926-0700 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-grpo-glm46-chai-90872-v1-uploader: Saving to /dev/shm/model_output...
chaiml-grpo-glm46-chai-90872-v1-uploader: 2026-03-08T20:52:37.841160-0700 | get_model_compressor | INFO - skip_sparsity_compression_stats set to True. Skipping sparsity compression statistic calculations. No sparsity compressor will be applied.
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'upstream connect error or disconnect/reset before headers. reset reason: connection termination')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: HTTPConnectionPool(host='chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Failed to get response for submission chaiml-grpo-glm46-chai-_46509_v1: ('http://chaiml-grpo-glm46-chai-46509-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Unable to record family friendly update due to error: Invalid JSON input: Expecting value: line 1 column 1 (char 0)
chaiml-grpo-glm46-chai-90872-v1-uploader: Cleaning quantization config in /dev/shm/model_output
chaiml-grpo-glm46-chai-90872-v1-uploader: Pushing to ChaiML/grpo-glm46-chai-rm-lr5e6-step50-FP8
chaiml-grpo-glm46-chai-90872-v1-uploader: Checking if ChaiML/grpo-glm46-chai-rm-lr5e6-step50-FP8 already exists in ChaiML
chaiml-grpo-glm46-chai-90872-v1-uploader: Creating repo ChaiML/grpo-glm46-chai-rm-lr5e6-step50-FP8 and uploading /dev/shm/model_output to it
chaiml-grpo-glm46-chai-90872-v1-uploader: ---------- 2026-03-08 21:36:08 (0:00:00) ----------
chaiml-grpo-glm46-chai-90872-v1-uploader: Files:   hashed 8/80 (28.2M/354.7G) | pre-uploaded: 0/1 (0.0/354.7G) (+72 unsure) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-90872-v1-uploader: Workers: hashing: 72 | get upload mode: 1 | pre-uploading: 1 | committing: 0 | waiting: 52
chaiml-grpo-glm46-chai-90872-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-90872-v1-uploader: 







chaiml-grpo-glm46-chai-90872-v1-uploader: ---------- 2026-03-08 21:37:08 (0:01:00) ----------
chaiml-grpo-glm46-chai-90872-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 3/73 (5.1G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-90872-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 70 | committing: 0 | waiting: 56
chaiml-grpo-glm46-chai-90872-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-90872-v1-uploader:                              







chaiml-grpo-glm46-chai-90872-v1-uploader: ---------- 2026-03-08 21:38:08 (0:02:00) ----------
chaiml-grpo-glm46-chai-90872-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 11/73 (45.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-90872-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 62 | committing: 0 | waiting: 64
chaiml-grpo-glm46-chai-90872-v1-uploader: ---------------------------------------------------
Failed to get response for submission chaiml-4d70-fd43-linear_51732_v5: ('http://guanaco-model-mesh-load-balancer-v2.model-mesh.kchai-google-us-east4.chaiverse.com/models/chaiml-4d70-fd43-linear_51732_v5/predict', 'upstream connect error or disconnect/reset before headers. reset reason: connection termination')
chaiml-grpo-glm46-chai-90872-v1-uploader:                              







chaiml-grpo-glm46-chai-90872-v1-uploader: ---------- 2026-03-08 21:39:08 (0:03:00) ----------
chaiml-grpo-glm46-chai-90872-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 18/73 (80.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-90872-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 55 | committing: 0 | waiting: 71
chaiml-grpo-glm46-chai-90872-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-90872-v1-uploader:                              







chaiml-grpo-glm46-chai-90872-v1-uploader: ---------- 2026-03-08 21:40:08 (0:04:00) ----------
chaiml-grpo-glm46-chai-90872-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 19/73 (85.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-90872-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 54 | committing: 0 | waiting: 72
chaiml-grpo-glm46-chai-90872-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-90872-v1-uploader:                              







chaiml-grpo-glm46-chai-90872-v1-uploader: ---------- 2026-03-08 21:41:09 (0:05:00) ----------
chaiml-grpo-glm46-chai-90872-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 27/73 (124.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-90872-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 46 | committing: 0 | waiting: 80
chaiml-grpo-glm46-chai-90872-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-90872-v1-uploader:                              







chaiml-grpo-glm46-chai-90872-v1-uploader: ---------- 2026-03-08 21:42:09 (0:06:00) ----------
chaiml-grpo-glm46-chai-90872-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 35/73 (164.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-90872-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 38 | committing: 0 | waiting: 88
chaiml-grpo-glm46-chai-90872-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-90872-v1-uploader:                              







chaiml-grpo-glm46-chai-90872-v1-uploader: ---------- 2026-03-08 21:43:09 (0:07:00) ----------
chaiml-grpo-glm46-chai-90872-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 43/73 (204.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-90872-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 30 | committing: 0 | waiting: 96
chaiml-grpo-glm46-chai-90872-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-90872-v1-uploader:                              







chaiml-grpo-glm46-chai-90872-v1-uploader: ---------- 2026-03-08 21:44:09 (0:08:00) ----------
chaiml-grpo-glm46-chai-90872-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 51/73 (244.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-90872-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 22 | committing: 0 | waiting: 104
chaiml-grpo-glm46-chai-90872-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-90872-v1-uploader:                              







chaiml-grpo-glm46-chai-90872-v1-uploader: ---------- 2026-03-08 21:45:09 (0:09:00) ----------
chaiml-grpo-glm46-chai-90872-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 59/73 (284.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-90872-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 14 | committing: 0 | waiting: 112
chaiml-grpo-glm46-chai-90872-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-90872-v1-uploader:                              







chaiml-grpo-glm46-chai-90872-v1-uploader: ---------- 2026-03-08 21:46:09 (0:10:00) ----------
chaiml-grpo-glm46-chai-90872-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 66/73 (319.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-90872-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 7 | committing: 0 | waiting: 119
chaiml-grpo-glm46-chai-90872-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-90872-v1-uploader:                              







chaiml-grpo-glm46-chai-90872-v1-uploader: ---------- 2026-03-08 21:47:09 (0:11:00) ----------
chaiml-grpo-glm46-chai-90872-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-90872-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-grpo-glm46-chai-90872-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-90872-v1-uploader:                              







chaiml-grpo-glm46-chai-90872-v1-uploader: ---------- 2026-03-08 21:48:09 (0:12:00) ----------
chaiml-grpo-glm46-chai-90872-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-90872-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-grpo-glm46-chai-90872-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-90872-v1-uploader:                              







chaiml-grpo-glm46-chai-90872-v1-uploader: ---------- 2026-03-08 21:49:09 (0:13:00) ----------
chaiml-grpo-glm46-chai-90872-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 50/80 (204.8G/354.7G) | ignored: 0
chaiml-grpo-glm46-chai-90872-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-grpo-glm46-chai-90872-v1-uploader: ---------------------------------------------------
chaiml-grpo-glm46-chai-90872-v1-uploader:                              Processed model ChaiML/grpo-glm46-chai-rm-lr5e6-step50 in 4428.067s
chaiml-grpo-glm46-chai-90872-v1-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm46-chai-90872-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-90872-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm46-chai-90872-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm46-chai-90872-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm46-chai-90872-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-90872-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm46-chai-90872-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-90872-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm46-chai-90872-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-90872-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm46-chai-90872-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-90872-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm46-chai-90872-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm46-chai-90872-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm46-chai-90872-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm46-chai-90872-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm46-chai-90872-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm46-chai-90872-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/config.json
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/chat_template.jinja
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/tokenizer_config.json
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/generation_config.json
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/recipe.yaml
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model.safetensors.index.json
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/special_tokens_map.json
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/tokenizer.json
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00072-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00071-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00044-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00031-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00010-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00048-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00060-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00004-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00056-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00062-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00026-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00016-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00049-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00030-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00015-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00025-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00050-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00033-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00021-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00070-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00027-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00022-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00002-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00032-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00014-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00006-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00003-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00042-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00029-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00023-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00005-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00013-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00068-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00008-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00001-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00063-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00051-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00052-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00040-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00059-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00019-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00045-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00065-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00036-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00066-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00007-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00064-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00039-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00061-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00054-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00058-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00020-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00028-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00017-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00053-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00035-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00069-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00024-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00057-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00046-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00011-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00055-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00067-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00047-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00009-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00018-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00037-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00041-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00043-of-00072.safetensors
chaiml-grpo-glm46-chai-90872-v1-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-90872-v1/default/model-00012-of-00072.safetensors
Job chaiml-grpo-glm46-chai-90872-v1-uploader completed after 4589.26s with status: succeeded
Stopping job with name chaiml-grpo-glm46-chai-90872-v1-uploader
Pipeline stage VLLMUploader completed in 4589.80s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.93s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm46-chai-90872-v1
Waiting for inference service chaiml-grpo-glm46-chai-90872-v1 to be ready
Inference service chaiml-grpo-glm46-chai-90872-v1 ready after 562.5101697444916s
Pipeline stage VLLMDeployer completed in 563.01s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.385666847229004s
Received healthy response to inference request in 2.2533938884735107s
Received healthy response to inference request in 2.2871346473693848s
Received healthy response to inference request in 2.1871325969696045s
Received healthy response to inference request in 2.135712146759033s
Received healthy response to inference request in 2.147090435028076s
Received healthy response to inference request in 2.227323055267334s
Received healthy response to inference request in 2.2293503284454346s
Received healthy response to inference request in 2.214061975479126s
Received healthy response to inference request in 2.1839053630828857s
Received healthy response to inference request in 2.2596492767333984s
Received healthy response to inference request in 2.124074697494507s
Received healthy response to inference request in 2.502326250076294s
Received healthy response to inference request in 2.2071526050567627s
Received healthy response to inference request in 2.5402615070343018s
Received healthy response to inference request in 2.1839919090270996s
Received healthy response to inference request in 2.2365646362304688s
Received healthy response to inference request in 2.196314811706543s
Received healthy response to inference request in 2.224907875061035s
Received healthy response to inference request in 2.164797782897949s
Received healthy response to inference request in 2.661526679992676s
Received healthy response to inference request in 2.3009254932403564s
Received healthy response to inference request in 2.197744607925415s
Received healthy response to inference request in 2.1823532581329346s
Received healthy response to inference request in 2.117539644241333s
Received healthy response to inference request in 2.109997034072876s
Received healthy response to inference request in 2.1394362449645996s
Received healthy response to inference request in 2.2456068992614746s
Received healthy response to inference request in 2.3690192699432373s
Received healthy response to inference request in 2.3284780979156494s
30 requests
0 failed requests
5th percentile: 2.1204804182052612
10th percentile: 2.1345484018325807
20th percentile: 2.1612563133239746
30th percentile: 2.1839659452438354
40th percentile: 2.197172689437866
50th percentile: 2.2194849252700806
60th percentile: 2.232236051559448
70th percentile: 2.255270504951477
80th percentile: 2.306436014175415
90th percentile: 2.397332787513733
95th percentile: 2.523190641403198
99th percentile: 2.6263597798347473
mean time: 2.2514479955037436
Pipeline stage StressChecker completed in 73.25s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.87s
Shutdown handler de-registered
chaiml-grpo-glm46-chai-_90872_v1 status is now deployed due to DeploymentManager action
chaiml-grpo-glm46-chai-_90872_v1 status is now inactive due to auto deactivation removed underperforming models