rirv938-simp-grpo-cp624_37437

developer_uid: rirv938

submission_id: rirv938-simp-grpo-cp624_37437_v1

model_name: rirv938-simp-grpo-cp624_37437_v1

model_group: rirv938/simp_grpo_cp624_

status: torndown

timestamp: 2025-07-01T02:06:25+00:00

num_battles: 6971

num_wins: 3590

celo_rating: 1299.27

family_friendly_score: 0.48519999999999996

family_friendly_standard_error: 0.007067969439662285

submission_type: basic

model_repo: rirv938/simp_grpo_cp624_98ff_b3_r2_12m_US_merged

model_architecture: MistralForCausalLM

model_num_parameters: 24096691200.0

best_of: 8

max_input_tokens: 768

max_output_tokens: 64

reward_model: default

latencies: [{'batch_size': 1, 'throughput': 0.5481477642334119, 'latency_mean': 1.8242257702350617, 'latency_p50': 1.8101227283477783, 'latency_p90': 2.0238024473190306}, {'batch_size': 3, 'throughput': 1.142199917043191, 'latency_mean': 2.6123016560077668, 'latency_p50': 2.583051562309265, 'latency_p90': 2.896562647819519}, {'batch_size': 5, 'throughput': 1.4792154460782974, 'latency_mean': 3.355816172361374, 'latency_p50': 3.3588075637817383, 'latency_p90': 3.70848491191864}, {'batch_size': 6, 'throughput': 1.6064362361970632, 'latency_mean': 3.698586273193359, 'latency_p50': 3.699658513069153, 'latency_p90': 4.147879314422608}, {'batch_size': 8, 'throughput': 1.7760308501553217, 'latency_mean': 4.468008675575256, 'latency_p50': 4.511534810066223, 'latency_p90': 5.042016100883484}, {'batch_size': 10, 'throughput': 1.886519168307118, 'latency_mean': 5.239479876756668, 'latency_p50': 5.269908666610718, 'latency_p90': 5.886461305618286}]

gpu_counts: {'NVIDIA A100-SXM4-80GB': 1}

display_name: rirv938-simp-grpo-cp624_37437_v1

ineligible_reason: num_battles<10000

is_internal_developer: True

language_model: rirv938/simp_grpo_cp624_98ff_b3_r2_12m_US_merged

model_size: 24B

ranking_group: single

throughput_3p7s: 1.62

us_pacific_date: 2025-06-30

win_ratio: 0.5149906756562903

generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.45, 'frequency_penalty': 0.45, 'stopping_words': ['You:', '</s>', '\n', '###'], 'max_input_tokens': 768, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-simp-grpo-cp624-37437-v1-mkmlizer
Waiting for job on rirv938-simp-grpo-cp624-37437-v1-mkmlizer to finish
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║                                                                     ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║                                                                     ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║  Version: 0.29.3                                                    ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║  https://mk1.ai                                                     ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║                                                                     ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║  belonging to:                                                      ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║                                                                     ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║  Chai Research Corp.                                                ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ║                                                                     ║
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: Downloaded to shared memory in 198.428s
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: Checking if rirv938/simp_grpo_cp624_98ff_b3_r2_12m_US_merged already exists in ChaiML
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: Creating repo ChaiML/simp_grpo_cp624_98ff_b3_r2_12m_US_merged and uploading /tmp/tmph69fwfdi to it
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: 
  0%|          | 0/22 [00:00<?, ?it/s]
  5%|▍         | 1/22 [00:06<02:13,  6.35s/it]
  9%|▉         | 2/22 [00:11<01:54,  5.73s/it]
 14%|█▎        | 3/22 [00:16<01:44,  5.48s/it]
 18%|█▊        | 4/22 [00:22<01:37,  5.42s/it]
 23%|██▎       | 5/22 [00:30<01:50,  6.51s/it]
 27%|██▋       | 6/22 [00:39<01:59,  7.47s/it]
 32%|███▏      | 7/22 [00:45<01:40,  6.70s/it]
 36%|███▋      | 8/22 [00:50<01:26,  6.21s/it]
 41%|████      | 9/22 [00:56<01:21,  6.23s/it]
 45%|████▌     | 10/22 [01:01<01:09,  5.82s/it]
 50%|█████     | 11/22 [01:06<01:02,  5.70s/it]
 55%|█████▍    | 12/22 [01:14<01:03,  6.40s/it]
 59%|█████▉    | 13/22 [01:21<00:57,  6.36s/it]
 64%|██████▎   | 14/22 [01:29<00:56,  7.08s/it]
 68%|██████▊   | 15/22 [01:35<00:45,  6.56s/it]
 73%|███████▎  | 16/22 [01:40<00:36,  6.14s/it]
 77%|███████▋  | 17/22 [01:45<00:29,  5.80s/it]
 82%|████████▏ | 18/22 [01:52<00:24,  6.24s/it]
 86%|████████▋ | 19/22 [01:57<00:17,  5.93s/it]
 91%|█████████ | 20/22 [02:03<00:11,  5.75s/it]
 95%|█████████▌| 21/22 [02:11<00:06,  6.54s/it]
100%|██████████| 22/22 [02:13<00:00,  5.07s/it]
100%|██████████| 22/22 [02:13<00:00,  6.05s/it]
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmph69fwfdi, device:0
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: quantized model in 75.324s
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: Processed model rirv938/simp_grpo_cp624_98ff_b3_r2_12m_US_merged in 527.965s
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: creating bucket guanaco-mkml-models
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-simp-grpo-cp624-37437-v1
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-simp-grpo-cp624-37437-v1/config.json
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-simp-grpo-cp624-37437-v1/special_tokens_map.json
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-simp-grpo-cp624-37437-v1/tokenizer_config.json
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-simp-grpo-cp624-37437-v1/tokenizer.json
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/rirv938-simp-grpo-cp624-37437-v1/flywheel_model.1.safetensors
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-simp-grpo-cp624-37437-v1/flywheel_model.0.safetensors
rirv938-simp-grpo-cp624-37437-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/363 [00:00<00:17, 20.40it/s]
Loading 0:   2%|▏         | 6/363 [00:00<00:34, 10.35it/s]
Loading 0:   3%|▎         | 10/363 [00:00<00:21, 16.18it/s]
Loading 0:   4%|▎         | 13/363 [00:01<00:34, 10.12it/s]
Loading 0:   4%|▍         | 15/363 [00:01<00:40,  8.60it/s]
Loading 0:   5%|▌         | 19/363 [00:01<00:27, 12.55it/s]
Loading 0:   6%|▌         | 21/363 [00:01<00:26, 12.87it/s]
Loading 0:   6%|▋         | 23/363 [00:02<00:33, 10.13it/s]
Loading 0:   8%|▊         | 28/363 [00:02<00:21, 15.54it/s]
Loading 0:   9%|▉         | 32/363 [00:02<00:17, 18.57it/s]
Loading 0:  10%|▉         | 35/363 [00:02<00:27, 12.07it/s]
Loading 0:  10%|█         | 37/363 [00:03<00:29, 11.03it/s]
Loading 0:  11%|█         | 39/363 [00:03<00:28, 11.54it/s]
Loading 0:  11%|█▏        | 41/363 [00:03<00:33,  9.49it/s]
Loading 0:  13%|█▎        | 46/363 [00:03<00:21, 14.59it/s]
Loading 0:  14%|█▍        | 50/363 [00:03<00:17, 17.86it/s]
Loading 0:  15%|█▍        | 53/363 [00:04<00:25, 12.04it/s]
Loading 0:  15%|█▌        | 55/363 [00:04<00:26, 11.53it/s]
Loading 0:  16%|█▌        | 57/363 [00:04<00:25, 12.16it/s]
Loading 0:  16%|█▋        | 59/363 [00:04<00:30,  9.90it/s]
Loading 0:  18%|█▊        | 64/363 [00:05<00:19, 14.98it/s]
Loading 0:  19%|█▊        | 68/363 [00:05<00:16, 18.05it/s]
Loading 0:  20%|█▉        | 71/363 [00:05<00:23, 12.29it/s]
Loading 0:  20%|██        | 73/363 [00:05<00:24, 11.70it/s]
Loading 0:  21%|██        | 75/363 [00:05<00:23, 12.25it/s]
Loading 0:  21%|██        | 77/363 [00:06<00:29,  9.63it/s]
Loading 0:  23%|██▎       | 82/363 [00:06<00:19, 14.53it/s]
Loading 0:  24%|██▎       | 86/363 [00:06<00:15, 17.66it/s]
Loading 0:  25%|██▍       | 89/363 [00:07<00:22, 11.97it/s]
Loading 0:  25%|██▌       | 91/363 [00:07<00:23, 11.53it/s]
Loading 0:  26%|██▌       | 95/363 [00:07<00:18, 14.70it/s]
Loading 0:  27%|██▋       | 99/363 [00:07<00:15, 17.25it/s]
Loading 0:  28%|██▊       | 102/363 [00:07<00:17, 15.11it/s]
Loading 0:  29%|██▊       | 104/363 [00:08<00:19, 13.46it/s]
Loading 0:  29%|██▉       | 106/363 [00:08<00:25, 10.14it/s]
Loading 0:  30%|██▉       | 108/363 [00:08<00:29,  8.61it/s]
Loading 0:  31%|███       | 111/363 [00:08<00:23, 10.81it/s]
Loading 0:  31%|███       | 113/363 [00:09<00:27,  9.16it/s]
Loading 0:  33%|███▎      | 118/363 [00:09<00:17, 14.09it/s]
Loading 0:  34%|███▎      | 122/363 [00:09<00:13, 17.39it/s]
Loading 0:  34%|███▍      | 125/363 [00:09<00:19, 11.93it/s]
Loading 0:  35%|███▍      | 127/363 [00:10<00:20, 11.34it/s]
Loading 0:  36%|███▌      | 129/363 [00:10<00:19, 11.77it/s]
Loading 0:  36%|███▌      | 131/363 [00:10<00:24,  9.51it/s]
Loading 0:  37%|███▋      | 136/363 [00:10<00:15, 14.53it/s]
Loading 0:  39%|███▊      | 140/363 [00:10<00:13, 16.75it/s]
Loading 0:  39%|███▉      | 143/363 [00:11<00:18, 11.59it/s]
Loading 0:  40%|███▉      | 145/363 [00:11<00:19, 11.15it/s]
Loading 0:  40%|████      | 147/363 [00:11<00:18, 11.86it/s]
Loading 0:  41%|████      | 149/363 [00:12<00:22,  9.46it/s]
Loading 0:  42%|████▏     | 154/363 [00:12<00:14, 14.52it/s]
Loading 0:  44%|████▎     | 158/363 [00:12<00:11, 17.21it/s]
Loading 0:  44%|████▍     | 161/363 [00:12<00:17, 11.69it/s]
Loading 0:  45%|████▍     | 163/363 [00:13<00:17, 11.23it/s]
Loading 0:  45%|████▌     | 165/363 [00:13<00:16, 11.96it/s]
Loading 0:  46%|████▌     | 167/363 [00:13<00:20,  9.41it/s]
Loading 0:  47%|████▋     | 172/363 [00:13<00:13, 14.26it/s]
Loading 0:  48%|████▊     | 176/363 [00:13<00:10, 17.25it/s]
Loading 0:  49%|████▉     | 179/363 [00:14<00:15, 11.88it/s]
Loading 0:  50%|████▉     | 181/363 [00:14<00:15, 11.45it/s]
Loading 0:  50%|█████     | 183/363 [00:14<00:15, 11.94it/s]
Loading 0:  51%|█████     | 185/363 [00:14<00:18,  9.59it/s]
Loading 0:  52%|█████▏    | 190/363 [00:15<00:11, 14.61it/s]
Loading 0:  53%|█████▎    | 194/363 [00:15<00:09, 17.46it/s]
Loading 0:  54%|█████▍    | 197/363 [00:15<00:13, 12.04it/s]
Loading 0:  55%|█████▍    | 199/363 [00:15<00:14, 11.51it/s]
Loading 0:  55%|█████▌    | 200/363 [00:34<00:14, 11.51it/s]
Loading 0:  55%|█████▌    | 201/363 [00:34<05:46,  2.14s/it]
Loading 0:  56%|█████▌    | 202/363 [00:34<04:58,  1.86s/it]
Loading 0:  56%|█████▌    | 204/363 [00:34<03:36,  1.36s/it]
Loading 0:  57%|█████▋    | 207/363 [00:34<02:14,  1.16it/s]
Loading 0:  58%|█████▊    | 209/363 [00:34<01:39,  1.55it/s]
Loading 0:  58%|█████▊    | 212/363 [00:34<01:05,  2.30it/s]
Loading 0:  59%|█████▉    | 214/363 [00:35<00:54,  2.73it/s]
Loading 0:  60%|█████▉    | 216/363 [00:35<00:45,  3.21it/s]
Loading 0:  60%|██████    | 219/363 [00:35<00:30,  4.67it/s]
Loading 0:  61%|██████    | 221/363 [00:36<00:28,  5.01it/s]
Loading 0:  62%|██████▏   | 226/363 [00:36<00:15,  8.60it/s]
Loading 0:  63%|██████▎   | 230/363 [00:36<00:11, 11.59it/s]
Loading 0:  64%|██████▍   | 233/363 [00:36<00:13,  9.71it/s]
Loading 0:  65%|██████▍   | 235/363 [00:36<00:13,  9.66it/s]
Loading 0:  65%|██████▌   | 237/363 [00:37<00:11, 10.56it/s]
Loading 0:  66%|██████▌   | 239/363 [00:37<00:13,  9.02it/s]
Loading 0:  67%|██████▋   | 244/363 [00:37<00:08, 13.89it/s]
Loading 0:  68%|██████▊   | 248/363 [00:37<00:06, 16.56it/s]
Loading 0:  69%|██████▉   | 251/363 [00:38<00:09, 11.84it/s]
Loading 0:  70%|██████▉   | 253/363 [00:38<00:09, 11.57it/s]
Loading 0:  70%|███████   | 255/363 [00:38<00:08, 12.28it/s]
Loading 0:  71%|███████   | 257/363 [00:38<00:10,  9.95it/s]
Loading 0:  72%|███████▏  | 262/363 [00:38<00:06, 15.16it/s]
Loading 0:  73%|███████▎  | 266/363 [00:39<00:05, 18.18it/s]
Loading 0:  74%|███████▍  | 269/363 [00:39<00:07, 12.44it/s]
Loading 0:  75%|███████▍  | 271/363 [00:39<00:07, 11.98it/s]
Loading 0:  75%|███████▌  | 273/363 [00:39<00:07, 12.67it/s]
Loading 0:  76%|███████▌  | 275/363 [00:40<00:08, 10.13it/s]
Loading 0:  77%|███████▋  | 280/363 [00:40<00:05, 15.46it/s]
Loading 0:  78%|███████▊  | 284/363 [00:40<00:04, 18.35it/s]
Loading 0:  79%|███████▉  | 287/363 [00:40<00:05, 12.77it/s]
Loading 0:  80%|███████▉  | 289/363 [00:41<00:06, 12.10it/s]
Loading 0:  80%|████████  | 291/363 [00:41<00:05, 12.57it/s]
Loading 0:  81%|████████  | 293/363 [00:41<00:07,  9.73it/s]
Loading 0:  82%|████████▏ | 298/363 [00:41<00:04, 14.89it/s]
Loading 0:  83%|████████▎ | 302/363 [00:41<00:03, 17.80it/s]
Loading 0:  84%|████████▍ | 305/363 [00:42<00:04, 11.85it/s]
Loading 0:  85%|████████▍ | 307/363 [00:42<00:04, 11.30it/s]
Loading 0:  85%|████████▌ | 309/363 [00:42<00:04, 11.82it/s]
Loading 0:  86%|████████▌ | 311/363 [00:42<00:05,  9.48it/s]
Loading 0:  87%|████████▋ | 316/363 [00:43<00:03, 14.16it/s]
Loading 0:  88%|████████▊ | 320/363 [00:43<00:02, 17.03it/s]
Loading 0:  89%|████████▉ | 323/363 [00:43<00:03, 11.86it/s]
Loading 0:  90%|████████▉ | 325/363 [00:43<00:03, 11.64it/s]
Loading 0:  90%|█████████ | 327/363 [00:44<00:02, 12.55it/s]
Loading 0:  91%|█████████ | 329/363 [00:44<00:03, 10.31it/s]
Loading 0:  92%|█████████▏| 334/363 [00:44<00:01, 15.40it/s]
Loading 0:  93%|█████████▎| 338/363 [00:44<00:01, 18.61it/s]
Loading 0:  94%|█████████▍| 341/363 [00:45<00:01, 12.30it/s]
Loading 0:  94%|█████████▍| 343/363 [00:45<00:01, 11.49it/s]
Loading 0:  95%|█████████▌| 345/363 [00:45<00:01, 12.19it/s]
Loading 0:  96%|█████████▌| 347/363 [00:45<00:01,  9.67it/s]
Loading 0:  97%|█████████▋| 352/363 [00:45<00:00, 14.70it/s]
Loading 0:  98%|█████████▊| 356/363 [00:46<00:00, 17.83it/s]
Loading 0:  99%|█████████▉| 359/363 [00:46<00:00,  8.77it/s]
Loading 0:  99%|█████████▉| 361/363 [00:47<00:00,  8.05it/s]
                                                            
Job rirv938-simp-grpo-cp624-37437-v1-mkmlizer completed after 567.98s with status: succeeded
Stopping job with name rirv938-simp-grpo-cp624-37437-v1-mkmlizer
Pipeline stage MKMLizer completed in 568.56s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-simp-grpo-cp624-37437-v1
Waiting for inference service rirv938-simp-grpo-cp624-37437-v1 to be ready
Inference service rirv938-simp-grpo-cp624-37437-v1 ready after 210.87485456466675s
Pipeline stage MKMLDeployer completed in 211.47s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.6198456287384033s
Received healthy response to inference request in 2.060497283935547s
Received healthy response to inference request in 1.9952290058135986s
Received healthy response to inference request in 2.0105154514312744s
{"detail":"HTTPConnectionPool(host='rirv938-simp-grpo-cp624-37437-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x78912c5df890>, 'Connection to rirv938-simp-grpo-cp624-37437-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com timed out. (connect timeout=12.0)'))"}
Received unhealthy response to inference request!
5 requests
1 failed requests
5th percentile: 1.9982862949371338
10th percentile: 2.001343584060669
20th percentile: 2.0074581623077394
30th percentile: 2.020511817932129
40th percentile: 2.040504550933838
50th percentile: 2.060497283935547
60th percentile: 2.2842366218566896
70th percentile: 2.507975959777832
80th percentile: 4.539570093154909
90th percentile: 8.379019021987915
95th percentile: 10.298743486404417
99th percentile: 11.834523057937622
mean time: 4.180911064147949
%s, retrying in %s seconds...
Received healthy response to inference request in 2.3180882930755615s
Received healthy response to inference request in 1.8954603672027588s
Received healthy response to inference request in 2.022575616836548s
Received healthy response to inference request in 1.9226348400115967s
Received healthy response to inference request in 1.8908236026763916s
5 requests
0 failed requests
5th percentile: 1.891750955581665
10th percentile: 1.8926783084869385
20th percentile: 1.8945330142974854
30th percentile: 1.9008952617645263
40th percentile: 1.9117650508880615
50th percentile: 1.9226348400115967
60th percentile: 1.962611150741577
70th percentile: 2.0025874614715575
80th percentile: 2.0816781520843506
90th percentile: 2.199883222579956
95th percentile: 2.258985757827759
99th percentile: 2.306267786026001
mean time: 2.0099165439605713
Pipeline stage StressChecker completed in 34.25s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.72s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.77s
Shutdown handler de-registered
rirv938-simp-grpo-cp624_37437_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-simp-grpo-cp624-37437-v1-profiler
Waiting for inference service rirv938-simp-grpo-cp624-37437-v1-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3212.13s
Shutdown handler de-registered
rirv938-simp-grpo-cp624_37437_v1 status is now inactive due to auto deactivation removed underperforming models
rirv938-simp-grpo-cp624_37437_v1 status is now torndown due to DeploymentManager action