developer_uid: richhx
submission_id: chaiml-grpo-glm46-chai-_81744_v8
model_name: chaiml-grpo-glm46-chai-_81744_v8
model_group: ChaiML/grpo-glm46-chai-r
status: deployed
timestamp: 2026-03-21T15:55:02+00:00
num_battles: 3320
num_wins: 1838
celo_rating: 1331.93
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm46-chai-rm-lr5e6-step150
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm46-chai-_81744_v8
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm46-chai-rm-lr5e6-step150
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-21
win_ratio: 0.5536144578313253
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|im_end|>', '<|user|>', '####', '</s>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>\n{bot_name}: {message}', 'user_template': '<|user|>\n{message}', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
chaiml-grpo-glm46-chai-_81744_v7 status is now pending due to richhx requesting redeployment of chaiml-grpo-glm46-chai-_81744_v7
run pipeline stage %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Running pipeline stage VLLMUploader
run pipeline %s
Starting job with name chaiml-grpo-glm46-chai-81744-v8-uploader
run pipeline stage %s
Waiting for job on chaiml-grpo-glm46-chai-81744-v8-uploader to finish
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm46-chai-81744-v7-uploader
Waiting for job on chaiml-grpo-glm46-chai-81744-v7-uploader to finish
2026-03-21T15:26:30.754945+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
2026-03-21T15:26:33.643926+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
Inference service chaiml-gspo-glm46-combi-11090-v7 ready after 340.55632519721985s
Pipeline stage VLLMDeployer completed in 341.87s
run pipeline stage %s
Running pipeline stage StressChecker
chaiml-grpo-glm46-chai-81744-v8-uploader: Using quantization_mode: fp8
chaiml-grpo-glm46-chai-81744-v8-uploader: Checking if ChaiML/grpo-glm46-chai-rm-lr5e6-step150-FP8 already exists in ChaiML
Received healthy response to inference request in 9.038647651672363s
chaiml-grpo-glm46-chai-81744-v8-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-grpo-glm46-chai-81744-v8-uploader: Downloading snapshot of ChaiML/grpo-glm46-chai-rm-lr5e6-step150-FP8...
Received healthy response to inference request in 9.208820343017578s
Received healthy response to inference request in 9.078523874282837s
chaiml-grpo-glm46-chai-81744-v7-uploader: Using quantization_mode: fp8
chaiml-grpo-glm46-chai-81744-v7-uploader: Checking if ChaiML/grpo-glm46-chai-rm-lr5e6-step150-FP8 already exists in ChaiML
chaiml-grpo-glm46-chai-81744-v7-uploader: Model already exists. Downloading to /dev/shm/model_output...
Received healthy response to inference request in 1.9838383197784424s
chaiml-grpo-glm46-chai-81744-v7-uploader: Downloading snapshot of ChaiML/grpo-glm46-chai-rm-lr5e6-step150-FP8...
2026-03-21T15:27:27.117940+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:27:28.094521+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
2026-03-21T15:27:31.141115+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
Received healthy response to inference request in 10.125329971313477s
2026-03-21T15:27:34.108428+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
Received healthy response to inference request in 2.0006966590881348s
Received healthy response to inference request in 1.9696125984191895s
Received healthy response to inference request in 1.9935522079467773s
Received healthy response to inference request in 1.9967689514160156s
Received healthy response to inference request in 2.0010976791381836s
Received healthy response to inference request in 1.9791412353515625s
Received healthy response to inference request in 1.9962663650512695s
Received healthy response to inference request in 2.0064189434051514s
Received healthy response to inference request in 2.186549663543701s
Received healthy response to inference request in 2.088719129562378s
Received healthy response to inference request in 2.1658735275268555s
Received healthy response to inference request in 2.0128238201141357s
Received healthy response to inference request in 2.1172831058502197s
Received healthy response to inference request in 1.989224910736084s
Received healthy response to inference request in 2.013917922973633s
Received healthy response to inference request in 2.0737595558166504s
Received healthy response to inference request in 2.175530433654785s
Received healthy response to inference request in 2.0622477531433105s
Received healthy response to inference request in 2.141507148742676s
Received healthy response to inference request in 1.9857430458068848s
Received healthy response to inference request in 2.0186400413513184s
Received healthy response to inference request in 1.9892330169677734s
Received healthy response to inference request in 2.015737533569336s
2026-03-21T15:28:27.941121+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:28:28.488718+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
Received healthy response to inference request in 2.026859760284424s
2026-03-21T15:28:31.562679+00:00 monitor updated for chaiml-gspo-glm46-combi_11090_v7
Received healthy response to inference request in 2.1880719661712646s
30 requests
0 failed requests
5th percentile: 1.9812549233436585
10th percentile: 1.9855525732040404
20th percentile: 1.9926883697509765
30th percentile: 1.999518346786499
2026-03-21T15:28:34.545833+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
40th percentile: 2.010261869430542
50th percentile: 2.017188787460327
60th percentile: 2.0668524742126464
70th percentile: 2.1245503187179566
80th percentile: 2.1777342796325683
90th percentile: 9.04263527393341
95th percentile: 9.150186932086944
99th percentile: 9.859542179107667
mean time: 3.0210145711898804
Pipeline stage StressChecker completed in 107.91s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.99s
Shutdown handler de-registered
chaiml-gspo-glm46-combi_11090_v7 status is now deployed due to DeploymentManager action
chaiml-grpo-glm46-chai-81744-v7-uploader: Downloaded in 86.814s
chaiml-grpo-glm46-chai-81744-v7-uploader: Processed model ChaiML/grpo-glm46-chai-rm-lr5e6-step150 in 90.198s
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-81744-v7-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-81744-v7-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-81744-v7-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-chai-81744-v7-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm46-chai-81744-v7-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
Inference service chaiml-grpo-glm46-chai-81744-v7 ready after 451.26661014556885s
chaiml-grpo-glm46-chai-81744-v7-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
Pipeline stage VLLMDeployer completed in 452.53s
chaiml-grpo-glm46-chai-81744-v7-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
run pipeline stage %s
chaiml-grpo-glm46-chai-81744-v7-uploader: Bucket 's3://guanaco-vllm-models/' created
Running pipeline stage StressChecker
chaiml-grpo-glm46-chai-81744-v7-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/.gitattributes": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/chat_template.jinja": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/config.json": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/generation_config.json": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00001-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00002-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00003-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00004-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00005-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00006-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00007-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00008-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00009-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00010-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00011-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00012-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v8-uploader: Downloaded in 124.677s
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00013-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v8-uploader: Processed model ChaiML/grpo-glm46-chai-rm-lr5e6-step150 in 128.170s
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00014-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00015-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00016-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00017-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00018-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00019-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00020-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00021-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00022-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00023-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00024-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00025-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00026-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00027-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00028-of-00072.safetensors": object size matches
Received healthy response to inference request in 9.108803987503052s
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00029-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00030-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00031-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00032-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00033-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00034-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00035-of-00072.safetensors": object size matches
Received healthy response to inference request in 2.0280117988586426s
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00036-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00037-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00038-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00039-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00040-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00041-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00042-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00043-of-00072.safetensors": object size matches
Received healthy response to inference request in 2.006671667098999s
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00044-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00045-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00046-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00047-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/recipe.yaml
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00048-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/special_tokens_map.json
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00049-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/tokenizer_config.json
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00050-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model.safetensors.index.json
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00051-of-00072.safetensors": object size matches
Received healthy response to inference request in 2.047217607498169s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/tokenizer.json
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00052-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00053-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00054-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00055-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00056-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00057-of-00072.safetensors": object size matches
Received healthy response to inference request in 2.0475754737854004s
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00058-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00059-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00060-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00061-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00062-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00063-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00064-of-00072.safetensors": object size matches
Received healthy response to inference request in 2.037482738494873s
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00065-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00066-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00067-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00068-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00069-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00070-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00071-of-00072.safetensors": object size matches
Received healthy response to inference request in 2.019712448120117s
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model-00072-of-00072.safetensors": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/model.safetensors.index.json": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/recipe.yaml": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/special_tokens_map.json": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/tokenizer.json": object size matches
chaiml-grpo-glm46-chai-81744-v7-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v7/default/tokenizer_config.json": object size matches
Received healthy response to inference request in 2.07246470451355s
Received healthy response to inference request in 2.030858039855957s
2026-03-21T15:29:28.374103+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
Received healthy response to inference request in 2.1163623332977295s
2026-03-21T15:29:28.987166+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
Received healthy response to inference request in 2.0391225814819336s
Received healthy response to inference request in 2.0188839435577393s
Job chaiml-grpo-glm46-chai-81744-v7-uploader completed after 183.7s with status: succeeded
Stopping job with name chaiml-grpo-glm46-chai-81744-v7-uploader
Pipeline stage VLLMUploader completed in 185.61s
run pipeline stage %s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00072-of-00072.safetensors
2026-03-21T15:29:35.228570+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
Running pipeline stage VLLMTemplater
Received healthy response to inference request in 2.2166507244110107s
Pipeline stage VLLMTemplater completed in 0.81s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Pipeline stage VLLMDeployer completed in 0.52s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.102433919906616s
Received healthy response to inference request in 2.420449733734131s
Received healthy response to inference request in 2.590345621109009s
Received healthy response to inference request in 2.405822515487671s
Received healthy response to inference request in 2.400960683822632s
Received healthy response to inference request in 2.374340534210205s
Received healthy response to inference request in 2.4401824474334717s
Received healthy response to inference request in 2.3864004611968994s
Received healthy response to inference request in 2.5155396461486816s
Received healthy response to inference request in 2.3214848041534424s
Received healthy response to inference request in 2.339210271835327s
Received healthy response to inference request in 2.30895733833313s
Received healthy response to inference request in 2.3060295581817627s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00071-of-00072.safetensors
Received healthy response to inference request in 2.2782771587371826s
Received healthy response to inference request in 2.3160369396209717s
Received healthy response to inference request in 2.286897897720337s
Received healthy response to inference request in 2.4460554122924805s
Received healthy response to inference request in 2.481485366821289s
Received healthy response to inference request in 2.385533094406128s
Received healthy response to inference request in 2.46691632270813s
Received healthy response to inference request in 2.438882350921631s
Received healthy response to inference request in 2.4105937480926514s
Received healthy response to inference request in 2.385930299758911s
Received healthy response to inference request in 2.4073455333709717s
Received healthy response to inference request in 2.4344189167022705s
Received healthy response to inference request in 2.35856294631958s
Received healthy response to inference request in 2.3480422496795654s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00035-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00026-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00012-of-00072.safetensors
Received healthy response to inference request in 2.4871885776519775s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00030-of-00072.safetensors
Received healthy response to inference request in 2.5293054580688477s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00006-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00050-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00028-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00014-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00039-of-00072.safetensors
Received healthy response to inference request in 2.396519660949707s
Received healthy response to inference request in 2.4538276195526123s
2026-03-21T15:30:29.469153+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
2026-03-21T15:30:28.734026+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00034-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00017-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00022-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00025-of-00072.safetensors
2026-03-21T15:30:35.793957+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v7
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00023-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00049-of-00072.safetensors
Received healthy response to inference request in 2.400561809539795s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00031-of-00072.safetensors
Received healthy response to inference request in 2.587103843688965s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00051-of-00072.safetensors
30 requests
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00032-of-00072.safetensors
0 failed requests
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00048-of-00072.safetensors
5th percentile: 2.0192567706108093
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00065-of-00072.safetensors
10th percentile: 2.02718186378479
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00004-of-00072.safetensors
Received healthy response to inference request in 2.154668092727661s
20th percentile: 2.0387946128845216
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00029-of-00072.safetensors
30th percentile: 2.064997935295105
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00046-of-00072.safetensors
40th percentile: 2.1765353679656982
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00021-of-00072.safetensors
50th percentile: 2.3276236057281494
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00041-of-00072.safetensors
60th percentile: 2.3856919765472413
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00042-of-00072.safetensors
70th percentile: 2.435757946968079
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00067-of-00072.safetensors
Received healthy response to inference request in 2.036328077316284s
80th percentile: 2.447609853744507
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00040-of-00072.safetensors
90th percentile: 2.5350852966308595
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00015-of-00072.safetensors
95th percentile: 2.588886821269989
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00010-of-00072.safetensors
99th percentile: 7.218451061248785
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00055-of-00072.safetensors
mean time: 2.493655212720235
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00044-of-00072.safetensors
Pipeline stage StressChecker completed in 106.60s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00011-of-00072.safetensors
Received healthy response to inference request in 1.9879858493804932s
run pipeline stage %s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00008-of-00072.safetensors
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00003-of-00072.safetensors
run_pipeline:run_in_cloud %s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00019-of-00072.safetensors
starting trigger_guanaco_pipeline args=%s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00043-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00057-of-00072.safetensors
triggered trigger_guanaco_pipeline args=%s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00069-of-00072.safetensors
Received healthy response to inference request in 2.045806646347046s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.17s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00020-of-00072.safetensors
Shutdown handler de-registered
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00002-of-00072.safetensors
chaiml-grpo-glm46-chai-_81744_v7 status is now deployed due to DeploymentManager action
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00016-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00064-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00033-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00013-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00056-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00068-of-00072.safetensors
Received healthy response to inference request in 2.0721986293792725s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00027-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00060-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00059-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00061-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00053-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00001-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00070-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00005-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00038-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00024-of-00072.safetensors
Received healthy response to inference request in 2.040605306625366s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00052-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00037-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00036-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00047-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00062-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00058-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00066-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00063-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00018-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00045-of-00072.safetensors
Received healthy response to inference request in 1.9682025909423828s
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00007-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00009-of-00072.safetensors
chaiml-grpo-glm46-chai-81744-v8-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-chai-81744-v8/default/model-00054-of-00072.safetensors
Received healthy response to inference request in 1.9678800106048584s
Received healthy response to inference request in 2.3773205280303955s
Received healthy response to inference request in 2.3239028453826904s
Received healthy response to inference request in 2.254167318344116s
Job chaiml-grpo-glm46-chai-81744-v8-uploader completed after 276.02s with status: succeeded
Stopping job with name chaiml-grpo-glm46-chai-81744-v8-uploader
Pipeline stage VLLMUploader completed in 277.84s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Received healthy response to inference request in 1.9806163311004639s
Pipeline stage VLLMTemplater completed in 0.59s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm46-chai-81744-v8
Waiting for inference service chaiml-grpo-glm46-chai-81744-v8 to be ready
Received healthy response to inference request in 1.986952781677246s
Received healthy response to inference request in 2.6070845127105713s
30 requests
0 failed requests
5th percentile: 1.9737887740135194
10th percentile: 1.9863191366195678
20th percentile: 2.03974986076355
30th percentile: 2.1299272537231446
40th percentile: 2.2834496021270754
50th percentile: 2.3226938247680664
60th percentile: 2.375532531738281
70th percentile: 2.397732305526733
80th percentile: 2.407995176315308
90th percentile: 2.4683732271194456
95th percentile: 2.484622132778168
99th percentile: 2.5723146915435793
mean time: 2.266517464319865
Pipeline stage StressChecker completed in 99.68s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.30s
Shutdown handler de-registered
chaiml-grpo-glm46-chai-_81744_v7 status is now deployed due to DeploymentManager action
2026-03-21T15:31:34.735765+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:32:34.870545+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:33:35.026936+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:34:35.165864+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:35:35.306567+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:36:35.447580+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:37:35.569228+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:38:35.718688+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:39:35.828114+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:40:35.970184+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:41:36.082777+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:42:36.188823+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:43:36.320686+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:44:36.433395+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:45:36.551102+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:46:36.725846+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:47:36.842311+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:48:36.958463+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
2026-03-21T15:49:37.072293+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
Inference service chaiml-grpo-glm46-chai-81744-v8 ready after 1124.1934988498688s
Pipeline stage VLLMDeployer completed in 1125.15s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 9.357688665390015s
Received healthy response to inference request in 2.037404775619507s
Received healthy response to inference request in 1.9997522830963135s
Received healthy response to inference request in 2.1160757541656494s
Received healthy response to inference request in 2.101181983947754s
Received healthy response to inference request in 2.0488977432250977s
Received healthy response to inference request in 1.9943480491638184s
Received healthy response to inference request in 2.017087697982788s
Received healthy response to inference request in 2.0480568408966064s
Received healthy response to inference request in 2.027055263519287s
Received healthy response to inference request in 1.9990644454956055s
Received healthy response to inference request in 2.109405755996704s
Received healthy response to inference request in 2.154538154602051s
Received healthy response to inference request in 2.006319522857666s
Received healthy response to inference request in 1.9956886768341064s
Received healthy response to inference request in 2.109961748123169s
Received healthy response to inference request in 2.0856235027313232s
2026-03-21T15:50:37.187660+00:00 monitor updated for chaiml-grpo-glm46-chai-_81744_v8
Received healthy response to inference request in 2.0308303833007812s
Received healthy response to inference request in 2.0341620445251465s
Received healthy response to inference request in 2.003141164779663s
Received healthy response to inference request in 2.0091192722320557s
Received healthy response to inference request in 2.010530710220337s
Received healthy response to inference request in 1.9703748226165771s
Received healthy response to inference request in 2.0116689205169678s
Received healthy response to inference request in 2.1107306480407715s
Received healthy response to inference request in 2.119797706604004s
Received healthy response to inference request in 2.044821262359619s
Received healthy response to inference request in 1.9851679801940918s
Received healthy response to inference request in 2.1833372116088867s
Received healthy response to inference request in 2.0086512565612793s
30 requests
0 failed requests
5th percentile: 1.9892990112304687
10th percentile: 1.9955546140670777
20th percentile: 2.002463388442993
30th percentile: 2.0089788675308227
40th percentile: 2.01492018699646
50th percentile: 2.032496213912964
60th percentile: 2.046115493774414
70th percentile: 2.0902910470962524
80th percentile: 2.1101155281066895
90th percentile: 2.123271751403809
95th percentile: 2.1703776359558105
99th percentile: 7.2771267437934934
mean time: 2.291016141573588
Pipeline stage StressChecker completed in 72.86s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.77s
Shutdown handler de-registered
chaiml-grpo-glm46-chai-_81744_v8 status is now deployed due to DeploymentManager action
chaiml-grpo-glm46-chai-_81744_v8 status is now inactive due to system request