rirv938-mistral-24b-grp_82678

developer_uid: rirv938

submission_id: rirv938-mistral-24b-grp_82678_v2

model_name: rirv938-mistral-24b-grp_82678_v2

model_group: rirv938/mistral_24b_grpo

status: torndown

timestamp: 2025-04-29T17:19:46+00:00

num_battles: 6301

num_wins: 3357

celo_rating: 1330.77

family_friendly_score: 0.5656

family_friendly_standard_error: 0.007009944935589723

submission_type: basic

model_repo: rirv938/mistral_24b_grpo_40k_cp1184_92ff_merged

model_architecture: MistralForCausalLM

model_num_parameters: 24096691200.0

best_of: 8

max_input_tokens: 768

max_output_tokens: 64

reward_model: default

latencies: [{'batch_size': 1, 'throughput': 0.5574631386739425, 'latency_mean': 1.7937846076488495, 'latency_p50': 1.7767367362976074, 'latency_p90': 2.004782962799072}, {'batch_size': 4, 'throughput': 1.3769672747048358, 'latency_mean': 2.885166893005371, 'latency_p50': 2.897214889526367, 'latency_p90': 3.2314123153686523}, {'batch_size': 5, 'throughput': 1.5513008933434496, 'latency_mean': 3.195073699951172, 'latency_p50': 3.1933467388153076, 'latency_p90': 3.5352396965026855}, {'batch_size': 8, 'throughput': 1.853121638588651, 'latency_mean': 4.270884137153626, 'latency_p50': 4.28747284412384, 'latency_p90': 4.793134546279907}, {'batch_size': 10, 'throughput': 1.99848795849854, 'latency_mean': 4.964040117263794, 'latency_p50': 4.9481329917907715, 'latency_p90': 5.618371534347534}, {'batch_size': 12, 'throughput': 2.0761871808941263, 'latency_mean': 5.694347257614136, 'latency_p50': 5.656128525733948, 'latency_p90': 6.472790288925171}, {'batch_size': 15, 'throughput': 2.1482484987090085, 'latency_mean': 6.917988175153733, 'latency_p50': 6.932480573654175, 'latency_p90': 7.864267563819885}]

gpu_counts: {'NVIDIA A100-SXM4-80GB': 1}

display_name: rirv938-mistral-24b-grp_82678_v2

is_internal_developer: True

language_model: rirv938/mistral_24b_grpo_40k_cp1184_92ff_merged

model_size: 24B

ranking_group: single

throughput_3p7s: 1.73

us_pacific_date: 2025-04-29

win_ratio: 0.532772575781622

generation_params: {'temperature': 0.9, 'top_p': 0.9, 'min_p': 0.2, 'top_k': 80, 'presence_penalty': 0.5, 'frequency_penalty': 0.5, 'stopping_words': ['</s>', '\n', 'You:', '###'], 'max_input_tokens': 768, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-mistral-24b-grp-82678-v2-mkmlizer
Waiting for job on rirv938-mistral-24b-grp-82678-v2-mkmlizer to finish
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║     _____            __           __                                ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║       /___/                                                         ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║                                                                     ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║  Version: 0.12.8                                                    ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║  https://mk1.ai                                                     ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║                                                                     ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║  belonging to:                                                      ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║                                                                     ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║  Chai Research Corp.                                                ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ║                                                                     ║
rirv938-mistral-24b-grp-82678-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-mistral-24b-grp-82678-v2-mkmlizer: Downloaded to shared memory in 176.044s
rirv938-mistral-24b-grp-82678-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpame7nyui, device:0
rirv938-mistral-24b-grp-82678-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-mistral-24b-grp-82678-v2-mkmlizer: quantized model in 62.232s
rirv938-mistral-24b-grp-82678-v2-mkmlizer: Processed model rirv938/mistral_24b_grpo_40k_cp1184_92ff_merged in 238.277s
rirv938-mistral-24b-grp-82678-v2-mkmlizer: creating bucket guanaco-mkml-models
rirv938-mistral-24b-grp-82678-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-mistral-24b-grp-82678-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-mistral-24b-grp-82678-v2
rirv938-mistral-24b-grp-82678-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-mistral-24b-grp-82678-v2/config.json
rirv938-mistral-24b-grp-82678-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-mistral-24b-grp-82678-v2/special_tokens_map.json
rirv938-mistral-24b-grp-82678-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-mistral-24b-grp-82678-v2/tokenizer_config.json
rirv938-mistral-24b-grp-82678-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-mistral-24b-grp-82678-v2/tokenizer.json
rirv938-mistral-24b-grp-82678-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-mistral-24b-grp-82678-v2/flywheel_model.0.safetensors
rirv938-mistral-24b-grp-82678-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/rirv938-mistral-24b-grp-82678-v2/flywheel_model.1.safetensors
rirv938-mistral-24b-grp-82678-v2-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/363 [00:00<00:12, 29.31it/s]
Loading 0:   2%|▏         | 6/363 [00:00<00:25, 14.18it/s]
Loading 0:   3%|▎         | 11/363 [00:00<00:15, 22.32it/s]
Loading 0:   4%|▍         | 14/363 [00:00<00:24, 14.22it/s]
Loading 0:   4%|▍         | 16/363 [00:01<00:25, 13.72it/s]
Loading 0:   6%|▌         | 21/363 [00:01<00:19, 17.42it/s]
Loading 0:   6%|▋         | 23/363 [00:01<00:24, 13.80it/s]
Loading 0:   8%|▊         | 29/363 [00:01<00:15, 21.34it/s]
Loading 0:   9%|▉         | 33/363 [00:01<00:17, 19.40it/s]
Loading 0:  10%|▉         | 36/363 [00:02<00:22, 14.66it/s]
Loading 0:  11%|█         | 39/363 [00:02<00:19, 16.85it/s]
Loading 0:  12%|█▏        | 42/363 [00:02<00:20, 15.30it/s]
Loading 0:  13%|█▎        | 46/363 [00:02<00:16, 19.34it/s]
Loading 0:  14%|█▍        | 50/363 [00:02<00:13, 23.23it/s]
Loading 0:  15%|█▍        | 53/363 [00:03<00:18, 16.56it/s]
Loading 0:  15%|█▌        | 56/363 [00:03<00:17, 17.15it/s]
Loading 0:  16%|█▋        | 59/363 [00:03<00:22, 13.22it/s]
Loading 0:  18%|█▊        | 64/363 [00:03<00:16, 18.42it/s]
Loading 0:  19%|█▊        | 68/363 [00:03<00:13, 22.07it/s]
Loading 0:  20%|█▉        | 71/363 [00:04<00:18, 16.17it/s]
Loading 0:  20%|██        | 74/363 [00:04<00:17, 16.77it/s]
Loading 0:  21%|██        | 77/363 [00:04<00:21, 13.04it/s]
Loading 0:  23%|██▎       | 82/363 [00:04<00:15, 18.33it/s]
Loading 0:  24%|██▍       | 87/363 [00:05<00:14, 19.19it/s]
Loading 0:  25%|██▍       | 90/363 [00:05<00:18, 14.77it/s]
Loading 0:  26%|██▋       | 96/363 [00:05<00:12, 21.06it/s]
Loading 0:  28%|██▊       | 101/363 [00:05<00:11, 22.77it/s]
Loading 0:  29%|██▉       | 105/363 [00:06<00:15, 16.89it/s]
Loading 0:  30%|██▉       | 108/363 [00:06<00:18, 13.85it/s]
Loading 0:  31%|███       | 111/363 [00:06<00:16, 15.69it/s]
Loading 0:  31%|███▏      | 114/363 [00:06<00:17, 14.45it/s]
Loading 0:  33%|███▎      | 118/363 [00:06<00:13, 18.09it/s]
Loading 0:  34%|███▍      | 123/363 [00:07<00:12, 18.92it/s]
Loading 0:  35%|███▍      | 126/363 [00:07<00:16, 14.56it/s]
Loading 0:  36%|███▌      | 129/363 [00:07<00:14, 16.57it/s]
Loading 0:  36%|███▋      | 132/363 [00:07<00:15, 14.82it/s]
Loading 0:  37%|███▋      | 136/363 [00:07<00:12, 18.28it/s]
Loading 0:  39%|███▊      | 140/363 [00:08<00:10, 21.42it/s]
Loading 0:  39%|███▉      | 143/363 [00:08<00:14, 15.56it/s]
Loading 0:  40%|████      | 146/363 [00:08<00:13, 16.15it/s]
Loading 0:  41%|████      | 149/363 [00:08<00:16, 12.75it/s]
Loading 0:  42%|████▏     | 154/363 [00:09<00:11, 18.00it/s]
Loading 0:  44%|████▎     | 158/363 [00:09<00:09, 21.52it/s]
Loading 0:  44%|████▍     | 161/363 [00:09<00:12, 15.65it/s]
Loading 0:  45%|████▌     | 164/363 [00:09<00:12, 16.29it/s]
Loading 0:  46%|████▌     | 167/363 [00:10<00:15, 12.77it/s]
Loading 0:  47%|████▋     | 172/363 [00:10<00:10, 17.72it/s]
Loading 0:  48%|████▊     | 176/363 [00:10<00:08, 20.92it/s]
Loading 0:  49%|████▉     | 179/363 [00:10<00:11, 15.56it/s]
Loading 0:  50%|█████     | 182/363 [00:10<00:11, 16.17it/s]
Loading 0:  51%|█████     | 185/363 [00:11<00:13, 12.91it/s]
Loading 0:  52%|█████▏    | 190/363 [00:11<00:09, 18.16it/s]
Loading 0:  53%|█████▎    | 194/363 [00:11<00:07, 21.77it/s]
Loading 0:  55%|█████▍    | 198/363 [00:11<00:11, 14.23it/s]
Loading 0:  55%|█████▌    | 201/363 [00:26<03:22,  1.25s/it]
Loading 0:  56%|█████▌    | 203/363 [00:26<02:46,  1.04s/it]
Loading 0:  57%|█████▋    | 207/363 [00:26<01:46,  1.47it/s]
Loading 0:  58%|█████▊    | 210/363 [00:26<01:17,  1.97it/s]
Loading 0:  59%|█████▊    | 213/363 [00:27<00:58,  2.58it/s]
Loading 0:  59%|█████▉    | 215/363 [00:27<00:48,  3.06it/s]
Loading 0:  60%|█████▉    | 217/363 [00:27<00:39,  3.72it/s]
Loading 0:  60%|██████    | 219/363 [00:27<00:31,  4.63it/s]
Loading 0:  61%|██████    | 221/363 [00:27<00:27,  5.20it/s]
Loading 0:  62%|██████▏   | 226/363 [00:27<00:15,  9.11it/s]
Loading 0:  63%|██████▎   | 230/363 [00:28<00:10, 12.57it/s]
Loading 0:  64%|██████▍   | 233/363 [00:28<00:11, 11.31it/s]
Loading 0:  65%|██████▌   | 236/363 [00:28<00:10, 12.66it/s]
Loading 0:  66%|██████▌   | 239/363 [00:28<00:11, 10.95it/s]
Loading 0:  67%|██████▋   | 244/363 [00:29<00:07, 15.87it/s]
Loading 0:  68%|██████▊   | 248/363 [00:29<00:05, 19.49it/s]
Loading 0:  69%|██████▉   | 251/363 [00:29<00:07, 14.84it/s]
Loading 0:  70%|██████▉   | 254/363 [00:29<00:07, 15.56it/s]
Loading 0:  71%|███████   | 257/363 [00:30<00:08, 12.50it/s]
Loading 0:  72%|███████▏  | 262/363 [00:30<00:05, 17.62it/s]
Loading 0:  73%|███████▎  | 266/363 [00:30<00:04, 21.05it/s]
Loading 0:  74%|███████▍  | 269/363 [00:30<00:06, 15.50it/s]
Loading 0:  75%|███████▍  | 272/363 [00:30<00:05, 16.18it/s]
Loading 0:  76%|███████▌  | 275/363 [00:31<00:06, 12.93it/s]
Loading 0:  77%|███████▋  | 280/363 [00:31<00:04, 18.22it/s]
Loading 0:  78%|███████▊  | 284/363 [00:31<00:03, 21.88it/s]
Loading 0:  79%|███████▉  | 288/363 [00:31<00:05, 14.34it/s]
Loading 0:  80%|████████  | 291/363 [00:31<00:04, 16.28it/s]
Loading 0:  81%|████████  | 294/363 [00:32<00:04, 14.51it/s]
Loading 0:  82%|████████▏ | 299/363 [00:32<00:03, 19.61it/s]
Loading 0:  83%|████████▎ | 302/363 [00:32<00:02, 21.45it/s]
Loading 0:  84%|████████▍ | 305/363 [00:32<00:03, 15.71it/s]
Loading 0:  85%|████████▍ | 308/363 [00:32<00:03, 16.36it/s]
Loading 0:  86%|████████▌ | 311/363 [00:33<00:03, 13.18it/s]
Loading 0:  87%|████████▋ | 316/363 [00:33<00:02, 18.65it/s]
Loading 0:  88%|████████▊ | 320/363 [00:33<00:01, 22.18it/s]
Loading 0:  89%|████████▉ | 324/363 [00:33<00:02, 14.48it/s]
Loading 0:  90%|█████████ | 327/363 [00:34<00:02, 16.54it/s]
Loading 0:  91%|█████████ | 330/363 [00:34<00:02, 15.19it/s]
Loading 0:  92%|█████████▏| 335/363 [00:34<00:01, 20.49it/s]
Loading 0:  93%|█████████▎| 339/363 [00:34<00:01, 19.77it/s]
Loading 0:  94%|█████████▍| 342/363 [00:34<00:01, 15.03it/s]
Loading 0:  95%|█████████▌| 345/363 [00:35<00:01, 17.16it/s]
Loading 0:  96%|█████████▌| 348/363 [00:35<00:00, 15.27it/s]
Loading 0:  97%|█████████▋| 353/363 [00:35<00:00, 20.65it/s]
Loading 0:  98%|█████████▊| 357/363 [00:42<00:03,  1.72it/s]
Loading 0:  99%|█████████▉| 360/363 [00:42<00:01,  2.21it/s]
Loading 0: 100%|█████████▉| 362/363 [00:42<00:00,  2.62it/s]
                                                            
Job rirv938-mistral-24b-grp-82678-v2-mkmlizer completed after 269.57s with status: succeeded
Stopping job with name rirv938-mistral-24b-grp-82678-v2-mkmlizer
Pipeline stage MKMLizer completed in 270.06s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-mistral-24b-grp-82678-v2
Waiting for inference service rirv938-mistral-24b-grp-82678-v2 to be ready
Inference service rirv938-mistral-24b-grp-82678-v2 ready after 140.46401000022888s
Pipeline stage MKMLDeployer completed in 140.93s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.682149648666382s
Received healthy response to inference request in 2.032391309738159s
Failed to get response for submission blend_ruhas_2025-04-28: HTTPConnectionPool(host='blend-tajit-2025-04-28-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Received healthy response to inference request in 2.1320595741271973s
Received healthy response to inference request in 2.1620664596557617s
Received healthy response to inference request in 2.1306514739990234s
5 requests
0 failed requests
5th percentile: 2.052043342590332
10th percentile: 2.0716953754425047
20th percentile: 2.1109994411468507
30th percentile: 2.130933094024658
40th percentile: 2.1314963340759276
50th percentile: 2.1320595741271973
60th percentile: 2.144062328338623
70th percentile: 2.156065082550049
80th percentile: 2.266083097457886
90th percentile: 2.4741163730621336
95th percentile: 2.5781330108642577
99th percentile: 2.661346321105957
mean time: 2.2278636932373046
Pipeline stage StressChecker completed in 12.44s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.67s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.83s
Shutdown handler de-registered
rirv938-mistral-24b-grp_82678_v2 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-mistral-24b-grp-82678-v2-profiler
Waiting for inference service rirv938-mistral-24b-grp-82678-v2-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2896.21s
Shutdown handler de-registered
rirv938-mistral-24b-grp_82678_v2 status is now inactive due to auto deactivation removed underperforming models
rirv938-mistral-24b-grp_82678_v2 status is now torndown due to DeploymentManager action