chaiml-mistral-24b-grpo_99461

developer_uid: rirv938

submission_id: chaiml-mistral-24b-grpo_99461_v1

model_name: chaiml-mistral-24b-grpo_99461_v1

model_group: ChaiML/mistral_24b_grpo_

status: torndown

timestamp: 2025-05-01T18:49:45+00:00

num_battles: 8096

num_wins: 4312

celo_rating: 1328.43

family_friendly_score: 0.5926

family_friendly_standard_error: 0.0069487443470025575

submission_type: basic

model_repo: ChaiML/mistral_24b_grpo_cp592_92ff

model_architecture: MistralForCausalLM

model_num_parameters: 24096691200.0

best_of: 8

max_input_tokens: 768

max_output_tokens: 64

reward_model: default

latencies: [{'batch_size': 1, 'throughput': 0.5501818601626098, 'latency_mean': 1.8175215637683868, 'latency_p50': 1.8045289516448975, 'latency_p90': 2.0375624656677247}, {'batch_size': 3, 'throughput': 1.1615995168273494, 'latency_mean': 2.5687725746631624, 'latency_p50': 2.5625134706497192, 'latency_p90': 2.8204426527023316}, {'batch_size': 5, 'throughput': 1.5087001807301887, 'latency_mean': 3.295857892036438, 'latency_p50': 3.3116711378097534, 'latency_p90': 3.615669274330139}, {'batch_size': 6, 'throughput': 1.6674150574178417, 'latency_mean': 3.5696474111080168, 'latency_p50': 3.562259316444397, 'latency_p90': 3.9775128841400145}, {'batch_size': 8, 'throughput': 1.8359936773030772, 'latency_mean': 4.312337701320648, 'latency_p50': 4.298784494400024, 'latency_p90': 4.841775441169739}, {'batch_size': 10, 'throughput': 1.9505630197382182, 'latency_mean': 5.071480650901794, 'latency_p50': 5.119187712669373, 'latency_p90': 5.735112023353577}]

gpu_counts: {'NVIDIA A100-SXM4-80GB': 1}

display_name: chaiml-mistral-24b-grpo_99461_v1

is_internal_developer: True

language_model: ChaiML/mistral_24b_grpo_cp592_92ff

model_size: 24B

ranking_group: single

throughput_3p7s: 1.72

us_pacific_date: 2025-05-01

win_ratio: 0.532608695652174

generation_params: {'temperature': 0.9, 'top_p': 0.9, 'min_p': 0.2, 'top_k': 80, 'presence_penalty': 0.5, 'frequency_penalty': 0.5, 'stopping_words': ['###', 'You:', '\n', '</s>'], 'max_input_tokens': 768, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-mistral-24b-grpo-99461-v1-mkmlizer
Waiting for job on chaiml-mistral-24b-grpo-99461-v1-mkmlizer to finish
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║     _____            __           __                                ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║       /___/                                                         ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║                                                                     ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║  Version: 0.12.8                                                    ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║                                                                     ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║  belonging to:                                                      ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║                                                                     ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ║                                                                     ║
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: Downloaded to shared memory in 141.695s
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmprj0eyoy8, device:0
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission chaiml-anthropic-grpo-4_66492_v2: HTTPConnectionPool(host='chaiml-anthropic-grpo-4-66492-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: quantized model in 62.005s
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: Processed model ChaiML/mistral_24b_grpo_cp592_92ff in 203.732s
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: creating bucket guanaco-mkml-models
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-mistral-24b-grpo-99461-v1
Failed to get response for submission chaiml-anthropic-grpo-4_66492_v2: HTTPConnectionPool(host='chaiml-anthropic-grpo-4-66492-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-mistral-24b-grpo-99461-v1/special_tokens_map.json
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-mistral-24b-grpo-99461-v1/tokenizer_config.json
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-mistral-24b-grpo-99461-v1/tokenizer.json
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-mistral-24b-grpo-99461-v1/flywheel_model.1.safetensors
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-mistral-24b-grpo-99461-v1/flywheel_model.0.safetensors
chaiml-mistral-24b-grpo-99461-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 4/363 [00:00<00:16, 21.74it/s]
Loading 0:   2%|▏         | 7/363 [00:00<00:19, 18.57it/s]
Loading 0:   3%|▎         | 11/363 [00:00<00:15, 23.04it/s]
Loading 0:   4%|▍         | 14/363 [00:00<00:24, 14.14it/s]
Loading 0:   4%|▍         | 16/363 [00:01<00:24, 13.89it/s]
Loading 0:   6%|▌         | 21/363 [00:01<00:19, 17.73it/s]
Loading 0:   6%|▋         | 23/363 [00:01<00:23, 14.29it/s]
Loading 0:   8%|▊         | 29/363 [00:01<00:15, 21.97it/s]
Loading 0:   9%|▉         | 33/363 [00:01<00:16, 19.67it/s]
Loading 0:  10%|▉         | 36/363 [00:02<00:21, 15.21it/s]
Loading 0:  11%|█         | 39/363 [00:02<00:18, 17.38it/s]
Loading 0:  12%|█▏        | 42/363 [00:02<00:20, 15.48it/s]
Loading 0:  13%|█▎        | 46/363 [00:02<00:16, 19.45it/s]
Loading 0:  14%|█▍        | 50/363 [00:02<00:13, 22.81it/s]
Loading 0:  15%|█▍        | 53/363 [00:03<00:18, 16.39it/s]
Loading 0:  15%|█▌        | 56/363 [00:03<00:17, 17.12it/s]
Loading 0:  16%|█▋        | 59/363 [00:03<00:22, 13.30it/s]
Loading 0:  18%|█▊        | 64/363 [00:03<00:16, 18.42it/s]
Loading 0:  19%|█▊        | 68/363 [00:03<00:13, 21.42it/s]
Loading 0:  20%|█▉        | 71/363 [00:04<00:18, 15.58it/s]
Loading 0:  20%|██        | 74/363 [00:04<00:17, 16.54it/s]
Loading 0:  21%|██        | 77/363 [00:04<00:21, 13.27it/s]
Loading 0:  23%|██▎       | 82/363 [00:04<00:15, 18.52it/s]
Loading 0:  24%|██▎       | 86/363 [00:04<00:12, 21.75it/s]
Loading 0:  25%|██▍       | 89/363 [00:05<00:17, 15.83it/s]
Loading 0:  25%|██▌       | 92/363 [00:05<00:16, 16.41it/s]
Loading 0:  26%|██▋       | 96/363 [00:05<00:13, 20.21it/s]
Loading 0:  27%|██▋       | 99/363 [00:05<00:11, 22.06it/s]
Loading 0:  28%|██▊       | 102/363 [00:05<00:12, 20.27it/s]
Loading 0:  29%|██▉       | 105/363 [00:06<00:16, 15.44it/s]
Loading 0:  29%|██▉       | 107/363 [00:06<00:18, 14.16it/s]
Loading 0:  30%|███       | 109/363 [00:06<00:18, 13.69it/s]
Loading 0:  31%|███       | 111/363 [00:06<00:17, 14.70it/s]
Loading 0:  31%|███       | 113/363 [00:06<00:20, 12.27it/s]
Loading 0:  33%|███▎      | 118/363 [00:06<00:12, 19.30it/s]
Loading 0:  34%|███▎      | 122/363 [00:06<00:10, 23.23it/s]
Loading 0:  34%|███▍      | 125/363 [00:07<00:14, 16.14it/s]
Loading 0:  35%|███▌      | 128/363 [00:07<00:13, 17.07it/s]
Loading 0:  36%|███▌      | 131/363 [00:07<00:17, 13.45it/s]
Loading 0:  37%|███▋      | 136/363 [00:07<00:12, 18.92it/s]
Loading 0:  39%|███▊      | 140/363 [00:08<00:09, 22.46it/s]
Loading 0:  39%|███▉      | 143/363 [00:08<00:13, 16.48it/s]
Loading 0:  40%|████      | 146/363 [00:08<00:12, 17.17it/s]
Loading 0:  41%|████      | 149/363 [00:08<00:15, 13.57it/s]
Loading 0:  42%|████▏     | 154/363 [00:08<00:11, 18.84it/s]
Loading 0:  44%|████▎     | 158/363 [00:09<00:09, 22.34it/s]
Loading 0:  45%|████▍     | 162/363 [00:09<00:13, 14.54it/s]
Loading 0:  45%|████▌     | 165/363 [00:09<00:11, 16.66it/s]
Loading 0:  46%|████▋     | 168/363 [00:09<00:12, 15.28it/s]
Loading 0:  47%|████▋     | 172/363 [00:09<00:09, 19.12it/s]
Loading 0:  48%|████▊     | 176/363 [00:10<00:08, 22.87it/s]
Loading 0:  50%|████▉     | 180/363 [00:10<00:12, 14.82it/s]
Loading 0:  51%|█████     | 184/363 [00:10<00:10, 16.29it/s]
Loading 0:  52%|█████▏    | 187/363 [00:10<00:10, 16.69it/s]
Loading 0:  53%|█████▎    | 191/363 [00:11<00:08, 20.48it/s]
Loading 0:  54%|█████▎    | 195/363 [00:11<00:08, 19.21it/s]
Loading 0:  55%|█████▍    | 198/363 [00:11<00:11, 14.83it/s]
Loading 0:  55%|█████▌    | 201/363 [00:25<03:29,  1.30s/it]
Loading 0:  56%|█████▌    | 203/363 [00:26<02:50,  1.07s/it]
Loading 0:  57%|█████▋    | 207/363 [00:26<01:47,  1.46it/s]
Loading 0:  58%|█████▊    | 210/363 [00:26<01:17,  1.97it/s]
Loading 0:  59%|█████▊    | 213/363 [00:26<00:57,  2.59it/s]
Loading 0:  59%|█████▉    | 215/363 [00:26<00:47,  3.09it/s]
Loading 0:  60%|█████▉    | 217/363 [00:26<00:39,  3.74it/s]
Loading 0:  60%|██████    | 219/363 [00:26<00:30,  4.67it/s]
Loading 0:  61%|██████    | 221/363 [00:27<00:27,  5.22it/s]
Loading 0:  62%|██████▏   | 226/363 [00:27<00:15,  8.97it/s]
Loading 0:  63%|██████▎   | 230/363 [00:27<00:11, 12.07it/s]
Loading 0:  64%|██████▍   | 233/363 [00:27<00:12, 10.31it/s]
Loading 0:  65%|██████▍   | 235/363 [00:28<00:11, 10.67it/s]
Loading 0:  65%|██████▌   | 237/363 [00:28<00:10, 11.91it/s]
Loading 0:  66%|██████▌   | 239/363 [00:28<00:11, 10.47it/s]
Loading 0:  67%|██████▋   | 244/363 [00:28<00:07, 15.97it/s]
Loading 0:  68%|██████▊   | 248/363 [00:28<00:05, 19.51it/s]
Loading 0:  69%|██████▉   | 251/363 [00:29<00:08, 13.96it/s]
Loading 0:  70%|██████▉   | 254/363 [00:29<00:07, 15.09it/s]
Loading 0:  71%|███████   | 256/363 [00:29<00:07, 13.50it/s]
Loading 0:  71%|███████   | 258/363 [00:29<00:08, 13.01it/s]
Loading 0:  72%|███████▏  | 262/363 [00:29<00:05, 17.27it/s]
Loading 0:  73%|███████▎  | 266/363 [00:29<00:04, 21.23it/s]
Loading 0:  74%|███████▍  | 269/363 [00:30<00:06, 14.60it/s]
Loading 0:  75%|███████▍  | 272/363 [00:30<00:06, 15.09it/s]
Loading 0:  75%|███████▌  | 274/363 [00:30<00:06, 13.64it/s]
Loading 0:  76%|███████▌  | 276/363 [00:30<00:06, 13.14it/s]
Loading 0:  77%|███████▋  | 280/363 [00:30<00:04, 17.39it/s]
Loading 0:  78%|███████▊  | 284/363 [00:31<00:03, 20.23it/s]
Loading 0:  79%|███████▉  | 287/363 [00:31<00:05, 14.46it/s]
Loading 0:  80%|███████▉  | 289/363 [00:31<00:05, 14.12it/s]
Loading 0:  80%|████████  | 291/363 [00:31<00:04, 14.92it/s]
Loading 0:  81%|████████  | 293/363 [00:31<00:05, 11.97it/s]
Loading 0:  82%|████████▏ | 298/363 [00:32<00:03, 18.08it/s]
Loading 0:  83%|████████▎ | 302/363 [00:32<00:02, 22.08it/s]
Loading 0:  84%|████████▍ | 305/363 [00:32<00:03, 15.18it/s]
Loading 0:  85%|████████▍ | 308/363 [00:32<00:03, 15.74it/s]
Loading 0:  86%|████████▌ | 311/363 [00:33<00:04, 12.41it/s]
Loading 0:  87%|████████▋ | 316/363 [00:33<00:02, 17.21it/s]
Loading 0:  88%|████████▊ | 320/363 [00:33<00:02, 20.34it/s]
Loading 0:  89%|████████▉ | 323/363 [00:33<00:02, 14.77it/s]
Loading 0:  90%|████████▉ | 326/363 [00:33<00:02, 15.48it/s]
Loading 0:  91%|█████████ | 329/363 [00:34<00:02, 12.33it/s]
Loading 0:  92%|█████████▏| 334/363 [00:34<00:01, 17.16it/s]
Loading 0:  93%|█████████▎| 338/363 [00:34<00:01, 20.39it/s]
Loading 0:  94%|█████████▍| 341/363 [00:34<00:01, 15.24it/s]
Loading 0:  95%|█████████▍| 344/363 [00:34<00:01, 15.82it/s]
Loading 0:  96%|█████████▌| 347/363 [00:35<00:01, 12.54it/s]
Loading 0:  97%|█████████▋| 352/363 [00:35<00:00, 17.19it/s]
Loading 0:  98%|█████████▊| 356/363 [00:35<00:00, 20.22it/s]
Loading 0:  99%|█████████▉| 359/363 [00:42<00:02,  1.58it/s]
Loading 0:  99%|█████████▉| 361/363 [00:42<00:01,  1.87it/s]
                                                            
Job chaiml-mistral-24b-grpo-99461-v1-mkmlizer completed after 237.36s with status: succeeded
Stopping job with name chaiml-mistral-24b-grpo-99461-v1-mkmlizer
Pipeline stage MKMLizer completed in 237.83s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-mistral-24b-grpo-99461-v1
Waiting for inference service chaiml-mistral-24b-grpo-99461-v1 to be ready
Failed to get response for submission chaiml-anthropic-grpo-4_66492_v2: HTTPConnectionPool(host='chaiml-anthropic-grpo-4-66492-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service chaiml-mistral-24b-grpo-99461-v1 ready after 150.4840214252472s
Pipeline stage MKMLDeployer completed in 150.99s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.4429595470428467s
Received healthy response to inference request in 2.232837200164795s
Failed to get response for submission chaiml-rinen-grpo-40k-c_51339_v1: HTTPConnectionPool(host='chaiml-rinen-grpo-40k-c-51339-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Received healthy response to inference request in 2.4377307891845703s
Received healthy response to inference request in 2.6128368377685547s
Received healthy response to inference request in 2.164224147796631s
5 requests
0 failed requests
5th percentile: 2.177946758270264
10th percentile: 2.1916693687438964
20th percentile: 2.219114589691162
30th percentile: 2.27381591796875
40th percentile: 2.35577335357666
50th percentile: 2.4377307891845703
60th percentile: 2.439822292327881
70th percentile: 2.4419137954711916
80th percentile: 2.4769350051879884
90th percentile: 2.5448859214782713
95th percentile: 2.578861379623413
99th percentile: 2.6060417461395264
mean time: 2.3781177043914794
Pipeline stage StressChecker completed in 13.22s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.66s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.69s
Shutdown handler de-registered
chaiml-mistral-24b-grpo_99461_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3179.61s
Shutdown handler de-registered
chaiml-mistral-24b-grpo_99461_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-mistral-24b-grpo_99461_v1 status is now torndown due to DeploymentManager action