chaiml-exp-grpo-cp312-9_36146

developer_uid: richhx

submission_id: chaiml-exp-grpo-cp312-9_36146_v7

model_name: chaiml-exp-grpo-cp312-9_36146_v7

model_group: ChaiML/exp_grpo_cp312_96

status: torndown

timestamp: 2025-06-24T22:01:08+00:00

num_battles: 7190

num_wins: 3893

celo_rating: 1312.7

family_friendly_score: 0.5306

family_friendly_standard_error: 0.00705781325907678

submission_type: basic

model_repo: ChaiML/exp_grpo_cp312_96ff_b6_r2_less_ff_merged

model_architecture: MistralForCausalLM

model_num_parameters: 24096691200.0

best_of: 8

max_input_tokens: 1024

max_output_tokens: 64

reward_model: default

latencies: [{'batch_size': 1, 'throughput': 0.5190373914716053, 'latency_mean': 1.9265186381340027, 'latency_p50': 1.9299342632293701, 'latency_p90': 2.1375203371047973}, {'batch_size': 3, 'throughput': 1.0495541739946213, 'latency_mean': 2.8459734582901, 'latency_p50': 2.8318768739700317, 'latency_p90': 3.1618891954421997}, {'batch_size': 5, 'throughput': 1.3353979857656415, 'latency_mean': 3.72079851269722, 'latency_p50': 3.7721697092056274, 'latency_p90': 4.125406885147095}, {'batch_size': 6, 'throughput': 1.4424111410089202, 'latency_mean': 4.133251509666443, 'latency_p50': 4.090632200241089, 'latency_p90': 4.598502922058105}, {'batch_size': 8, 'throughput': 1.5728495946637748, 'latency_mean': 5.047424978017807, 'latency_p50': 5.02520215511322, 'latency_p90': 5.721230626106262}, {'batch_size': 10, 'throughput': 1.6681500021737274, 'latency_mean': 5.950609290599823, 'latency_p50': 5.955088496208191, 'latency_p90': 6.703561115264892}]

gpu_counts: {'NVIDIA A100-SXM4-80GB': 1}

display_name: chaiml-exp-grpo-cp312-9_36146_v7

ineligible_reason: num_battles<10000

is_internal_developer: True

language_model: ChaiML/exp_grpo_cp312_96ff_b6_r2_less_ff_merged

model_size: 24B

ranking_group: single

throughput_3p7s: 1.34

us_pacific_date: 2025-06-24

win_ratio: 0.5414464534075104

generation_params: {'temperature': 0.9, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.45, 'frequency_penalty': 0.45, 'stopping_words': ['</s>', '\n', '###', 'You:'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer
Waiting for job on chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer to finish
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║                                                                     ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║                                                                     ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║  Version: 0.29.3                                                    ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║                                                                     ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║  belonging to:                                                      ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║                                                                     ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ║                                                                     ║
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission chaiml-giyu-disabled-hu_13897_v2: ('http://chaiml-giyu-disabled-hu-13897-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-giyu-disabled-hu_13897_v2: ('http://chaiml-giyu-disabled-hu-13897-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: Downloaded to shared memory in 152.270s
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: Checking if ChaiML/exp_grpo_cp312_96ff_b6_r2_less_ff_merged already exists in ChaiML
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp738q9c9o, device:0
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: quantized model in 56.729s
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: Processed model ChaiML/exp_grpo_cp312_96ff_b6_r2_less_ff_merged in 208.999s
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-exp-grpo-cp312-9-36146-v7
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-exp-grpo-cp312-9-36146-v7/config.json
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-exp-grpo-cp312-9-36146-v7/special_tokens_map.json
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-exp-grpo-cp312-9-36146-v7/tokenizer_config.json
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-exp-grpo-cp312-9-36146-v7/tokenizer.json
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-exp-grpo-cp312-9-36146-v7/flywheel_model.1.safetensors
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-exp-grpo-cp312-9-36146-v7/flywheel_model.0.safetensors
chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/363 [00:00<00:13, 25.81it/s]
Loading 0:   2%|▏         | 6/363 [00:00<00:27, 12.78it/s]
Loading 0:   3%|▎         | 11/363 [00:00<00:17, 20.32it/s]
Loading 0:   4%|▍         | 14/363 [00:01<00:30, 11.63it/s]
Loading 0:   4%|▍         | 16/363 [00:01<00:29, 11.63it/s]
Loading 0:   6%|▌         | 21/363 [00:01<00:22, 15.25it/s]
Loading 0:   6%|▋         | 23/363 [00:01<00:27, 12.23it/s]
Loading 0:   8%|▊         | 28/363 [00:01<00:19, 17.38it/s]
Loading 0:   9%|▉         | 32/363 [00:01<00:16, 20.29it/s]
Loading 0:  10%|▉         | 35/363 [00:02<00:24, 13.55it/s]
Loading 0:  10%|█         | 37/363 [00:02<00:24, 13.12it/s]
Loading 0:  11%|█         | 39/363 [00:02<00:23, 13.86it/s]
Loading 0:  11%|█▏        | 41/363 [00:02<00:28, 11.41it/s]
Loading 0:  13%|█▎        | 46/363 [00:03<00:17, 17.61it/s]
Loading 0:  14%|█▍        | 50/363 [00:03<00:14, 21.77it/s]
Loading 0:  15%|█▍        | 53/363 [00:03<00:21, 14.56it/s]
Loading 0:  15%|█▌        | 56/363 [00:03<00:19, 15.36it/s]
Loading 0:  16%|█▋        | 59/363 [00:04<00:25, 12.01it/s]
Loading 0:  18%|█▊        | 64/363 [00:04<00:17, 16.82it/s]
Loading 0:  19%|█▊        | 68/363 [00:04<00:14, 20.37it/s]
Loading 0:  20%|█▉        | 71/363 [00:04<00:20, 14.49it/s]
Loading 0:  20%|██        | 74/363 [00:04<00:19, 15.08it/s]
Loading 0:  21%|██        | 77/363 [00:05<00:24, 11.89it/s]
Loading 0:  23%|██▎       | 82/363 [00:05<00:17, 16.44it/s]
Loading 0:  24%|██▎       | 86/363 [00:05<00:14, 19.56it/s]
Loading 0:  25%|██▍       | 89/363 [00:05<00:19, 14.16it/s]
Loading 0:  25%|██▌       | 92/363 [00:06<00:18, 14.62it/s]
Loading 0:  26%|██▋       | 96/363 [00:06<00:14, 18.47it/s]
Loading 0:  27%|██▋       | 99/363 [00:06<00:13, 20.25it/s]
Loading 0:  28%|██▊       | 102/363 [00:06<00:14, 18.48it/s]
Loading 0:  29%|██▉       | 105/363 [00:06<00:18, 13.69it/s]
Loading 0:  29%|██▉       | 107/363 [00:07<00:20, 12.74it/s]
Loading 0:  30%|███       | 109/363 [00:07<00:20, 12.54it/s]
Loading 0:  31%|███       | 111/363 [00:07<00:18, 13.47it/s]
Loading 0:  31%|███       | 113/363 [00:07<00:22, 11.18it/s]
Loading 0:  33%|███▎      | 118/363 [00:07<00:13, 17.53it/s]
Loading 0:  34%|███▎      | 122/363 [00:07<00:11, 21.68it/s]
Loading 0:  34%|███▍      | 125/363 [00:08<00:16, 14.83it/s]
Loading 0:  35%|███▌      | 128/363 [00:08<00:15, 15.57it/s]
Loading 0:  36%|███▌      | 131/363 [00:08<00:18, 12.22it/s]
Loading 0:  37%|███▋      | 136/363 [00:08<00:12, 17.52it/s]
Loading 0:  39%|███▊      | 140/363 [00:08<00:10, 20.93it/s]
Loading 0:  39%|███▉      | 143/363 [00:09<00:14, 15.09it/s]
Loading 0:  40%|████      | 146/363 [00:09<00:13, 15.73it/s]
Loading 0:  41%|████      | 149/363 [00:09<00:17, 12.48it/s]
Loading 0:  42%|████▏     | 154/363 [00:09<00:11, 17.65it/s]
Loading 0:  44%|████▎     | 158/363 [00:10<00:09, 21.24it/s]
Loading 0:  45%|████▍     | 162/363 [00:10<00:14, 13.74it/s]
Loading 0:  45%|████▌     | 165/363 [00:10<00:12, 15.60it/s]
Loading 0:  46%|████▋     | 168/363 [00:10<00:13, 14.25it/s]
Loading 0:  47%|████▋     | 172/363 [00:11<00:10, 17.94it/s]
Loading 0:  48%|████▊     | 176/363 [00:11<00:08, 21.66it/s]
Loading 0:  49%|████▉     | 179/363 [00:11<00:11, 15.49it/s]
Loading 0:  50%|█████     | 182/363 [00:11<00:11, 16.20it/s]
Loading 0:  51%|█████     | 185/363 [00:12<00:14, 12.70it/s]
Loading 0:  52%|█████▏    | 190/363 [00:12<00:09, 17.74it/s]
Loading 0:  53%|█████▎    | 194/363 [00:12<00:07, 21.30it/s]
Loading 0:  54%|█████▍    | 197/363 [00:12<00:10, 15.39it/s]
Loading 0:  55%|█████▌    | 200/363 [00:12<00:10, 15.98it/s]
Loading 0:  56%|█████▌    | 203/363 [00:27<03:40,  1.38s/it]
Loading 0:  57%|█████▋    | 207/363 [00:27<02:23,  1.09it/s]
Loading 0:  58%|█████▊    | 210/363 [00:27<01:44,  1.47it/s]
Loading 0:  59%|█████▊    | 213/363 [00:27<01:17,  1.93it/s]
Loading 0:  60%|█████▉    | 216/363 [00:28<00:59,  2.46it/s]
Loading 0:  60%|██████    | 219/363 [00:28<00:43,  3.35it/s]
Loading 0:  61%|██████    | 222/363 [00:28<00:33,  4.22it/s]
Loading 0:  62%|██████▏   | 226/363 [00:28<00:22,  6.17it/s]
Loading 0:  63%|██████▎   | 230/363 [00:28<00:15,  8.53it/s]
Loading 0:  64%|██████▍   | 233/363 [00:29<00:15,  8.51it/s]
Loading 0:  65%|██████▌   | 236/363 [00:29<00:12,  9.99it/s]
Loading 0:  66%|██████▌   | 239/363 [00:29<00:13,  9.42it/s]
Loading 0:  67%|██████▋   | 244/363 [00:29<00:08, 13.90it/s]
Loading 0:  68%|██████▊   | 248/363 [00:30<00:06, 17.40it/s]
Loading 0:  69%|██████▉   | 251/363 [00:30<00:08, 13.83it/s]
Loading 0:  70%|██████▉   | 254/363 [00:30<00:07, 14.73it/s]
Loading 0:  71%|███████   | 257/363 [00:30<00:08, 12.14it/s]
Loading 0:  72%|███████▏  | 262/363 [00:31<00:05, 17.00it/s]
Loading 0:  73%|███████▎  | 266/363 [00:31<00:04, 20.43it/s]
Loading 0:  74%|███████▍  | 269/363 [00:31<00:06, 15.21it/s]
Loading 0:  75%|███████▍  | 272/363 [00:31<00:05, 15.90it/s]
Loading 0:  76%|███████▌  | 275/363 [00:32<00:07, 12.53it/s]
Loading 0:  77%|███████▋  | 280/363 [00:32<00:04, 17.58it/s]
Loading 0:  78%|███████▊  | 284/363 [00:32<00:03, 21.15it/s]
Loading 0:  79%|███████▉  | 287/363 [00:32<00:04, 15.48it/s]
Loading 0:  80%|███████▉  | 290/363 [00:32<00:04, 16.11it/s]
Loading 0:  81%|████████  | 293/363 [00:33<00:05, 12.72it/s]
Loading 0:  82%|████████▏ | 298/363 [00:33<00:03, 17.82it/s]
Loading 0:  83%|████████▎ | 302/363 [00:33<00:02, 21.35it/s]
Loading 0:  84%|████████▍ | 305/363 [00:33<00:03, 15.53it/s]
Loading 0:  85%|████████▍ | 308/363 [00:33<00:03, 16.16it/s]
Loading 0:  86%|████████▌ | 311/363 [00:34<00:04, 12.69it/s]
Loading 0:  87%|████████▋ | 316/363 [00:34<00:02, 17.70it/s]
Loading 0:  88%|████████▊ | 320/363 [00:34<00:02, 21.10it/s]
Loading 0:  89%|████████▉ | 323/363 [00:34<00:02, 15.47it/s]
Loading 0:  90%|████████▉ | 326/363 [00:34<00:02, 16.07it/s]
Loading 0:  91%|█████████ | 329/363 [00:35<00:02, 12.66it/s]
Loading 0:  92%|█████████▏| 334/363 [00:35<00:01, 17.81it/s]
Loading 0:  93%|█████████▎| 338/363 [00:35<00:01, 21.34it/s]
Loading 0:  94%|█████████▍| 341/363 [00:35<00:01, 15.58it/s]
Loading 0:  95%|█████████▍| 344/363 [00:36<00:01, 16.14it/s]
Loading 0:  96%|█████████▌| 347/363 [00:36<00:01, 12.77it/s]
Loading 0:  97%|█████████▋| 352/363 [00:36<00:00, 17.90it/s]
Loading 0:  98%|█████████▊| 356/363 [00:36<00:00, 20.73it/s]
Loading 0:  99%|█████████▉| 359/363 [00:37<00:00, 11.29it/s]
Loading 0: 100%|█████████▉| 362/363 [00:37<00:00, 11.38it/s]
                                                            
Job chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer completed after 237.86s with status: succeeded
Stopping job with name chaiml-exp-grpo-cp312-9-36146-v7-mkmlizer
Pipeline stage MKMLizer completed in 238.35s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-exp-grpo-cp312-9-36146-v7
Waiting for inference service chaiml-exp-grpo-cp312-9-36146-v7 to be ready
Failed to get response for submission chaiml-giyu-disabled-hu_13897_v2: ('http://chaiml-giyu-disabled-hu-13897-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-giyu-disabled-hu_13897_v2: ('http://chaiml-giyu-disabled-hu-13897-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-giyu-disabled-hu_13897_v2: ('http://chaiml-giyu-disabled-hu-13897-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Inference service chaiml-exp-grpo-cp312-9-36146-v7 ready after 130.49137663841248s
Pipeline stage MKMLDeployer completed in 131.25s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.522092580795288s
Received healthy response to inference request in 2.4874796867370605s
Failed to get response for submission chaiml-giyu-disabled-hu_13897_v2: ('http://chaiml-giyu-disabled-hu-13897-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Received healthy response to inference request in 2.119551420211792s
Received healthy response to inference request in 2.0632877349853516s
Received healthy response to inference request in 1.9537315368652344s
5 requests
0 failed requests
5th percentile: 1.975642776489258
10th percentile: 1.9975540161132812
20th percentile: 2.041376495361328
30th percentile: 2.0745404720306397
40th percentile: 2.0970459461212156
50th percentile: 2.119551420211792
60th percentile: 2.2667227268218992
70th percentile: 2.413894033432007
80th percentile: 2.494402265548706
90th percentile: 2.508247423171997
95th percentile: 2.5151700019836425
99th percentile: 2.520708065032959
mean time: 2.2292285919189454
Pipeline stage StressChecker completed in 12.30s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.78s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.86s
Shutdown handler de-registered
chaiml-exp-grpo-cp312-9_36146_v7 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3236.05s
Shutdown handler de-registered
chaiml-exp-grpo-cp312-9_36146_v7 status is now inactive due to auto deactivation removed underperforming models
chaiml-exp-grpo-cp312-9_36146_v7 status is now protected due to ABTestQueueItem
chaiml-exp-grpo-cp312-9_36146_v7 status is now protected due to ABTestQueueItem
chaiml-exp-grpo-cp312-9_36146_v7 status is now protected due to ABTestQueueItem
chaiml-exp-grpo-cp312-9_36146_v7 status is now protected due to ABTestQueueItem
chaiml-exp-grpo-cp312-9_36146_v7 status is now protected due to ABTestQueueItem
chaiml-exp-grpo-cp312-9_36146_v7 status is now protected due to ABTestQueueItem
chaiml-exp-grpo-cp312-9_36146_v7 status is now protected due to ABTestQueueItem
chaiml-exp-grpo-cp312-9_36146_v7 status is now protected due to ABTestQueueItem
chaiml-exp-grpo-cp312-9_36146_v7 status is now protected due to ABTestQueueItem
chaiml-exp-grpo-cp312-9_36146_v7 status is now protected due to ABTestQueueItem
chaiml-exp-grpo-cp312-9_36146_v7 status is now torndown due to DeploymentManager action