developer_uid: Jellywibble
submission_id: nitral-ai-captain-bmo-12b_v14
model_name: nitral-ai-captain-bmo-12b_v14
model_group: Nitral-AI/Captain_BMO-12
status: inactive
timestamp: 2024-11-30T18:03:51+00:00
num_battles: 12942
num_wins: 6745
celo_rating: 1276.79
family_friendly_score: 0.5728
family_friendly_standard_error: 0.006995715260071696
submission_type: basic
model_repo: Nitral-AI/Captain_BMO-12B
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 4
max_input_tokens: 768
max_output_tokens: 64
latencies: [{'batch_size': 1, 'throughput': 0.6686335108220461, 'latency_mean': 1.495523191690445, 'latency_p50': 1.4885475635528564, 'latency_p90': 1.6581653833389283}, {'batch_size': 4, 'throughput': 1.6675818352988288, 'latency_mean': 2.389960322380066, 'latency_p50': 2.3998807668685913, 'latency_p90': 2.6336602687835695}, {'batch_size': 5, 'throughput': 1.829983675167636, 'latency_mean': 2.716525400876999, 'latency_p50': 2.748122453689575, 'latency_p90': 3.05793182849884}, {'batch_size': 8, 'throughput': 2.1877932421195214, 'latency_mean': 3.6153977525234224, 'latency_p50': 3.6074235439300537, 'latency_p90': 4.105613207817077}, {'batch_size': 10, 'throughput': 2.3199693113648125, 'latency_mean': 4.271664638519287, 'latency_p50': 4.276527047157288, 'latency_p90': 4.830790710449219}, {'batch_size': 12, 'throughput': 2.400021714216235, 'latency_mean': 4.948734129667282, 'latency_p50': 4.950570464134216, 'latency_p90': 5.606878089904785}, {'batch_size': 15, 'throughput': 2.4607112834563654, 'latency_mean': 6.038047381639481, 'latency_p50': 6.041696071624756, 'latency_p90': 6.85889081954956}]
gpu_counts: {'NVIDIA RTX A5000': 1}
display_name: nitral-ai-captain-bmo-12b_v14
is_internal_developer: True
language_model: Nitral-AI/Captain_BMO-12B
model_size: 13B
ranking_group: single
throughput_3p7s: 2.23
us_pacific_date: 2024-11-30
win_ratio: 0.5211713800030907
generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '####', 'Bot:', 'User:', 'You:', '<|im_end|>', '<|eot_id|>'], 'max_input_tokens': 768, 'best_of': 4, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name nitral-ai-captain-bmo-12b-v14-mkmlizer
Waiting for job on nitral-ai-captain-bmo-12b-v14-mkmlizer to finish
nitral-ai-captain-bmo-12b-v14-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║     _____            __           __                                ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║       /___/                                                         ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║                                                                     ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║  Version: 0.11.12                                                   ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║  https://mk1.ai                                                     ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║                                                                     ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║  The license key for the current software has been verified as      ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║  belonging to:                                                      ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║                                                                     ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║  Chai Research Corp.                                                ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ║                                                                     ║
nitral-ai-captain-bmo-12b-v14-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
nitral-ai-captain-bmo-12b-v14-mkmlizer: Downloaded to shared memory in 50.740s
nitral-ai-captain-bmo-12b-v14-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpsapwnwek, device:0
nitral-ai-captain-bmo-12b-v14-mkmlizer: Saving flywheel model at /dev/shm/model_cache
nitral-ai-captain-bmo-12b-v14-mkmlizer: quantized model in 38.593s
nitral-ai-captain-bmo-12b-v14-mkmlizer: Processed model Nitral-AI/Captain_BMO-12B in 89.333s
nitral-ai-captain-bmo-12b-v14-mkmlizer: creating bucket guanaco-mkml-models
nitral-ai-captain-bmo-12b-v14-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nitral-ai-captain-bmo-12b-v14-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/nitral-ai-captain-bmo-12b-v14
nitral-ai-captain-bmo-12b-v14-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/nitral-ai-captain-bmo-12b-v14/config.json
nitral-ai-captain-bmo-12b-v14-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/nitral-ai-captain-bmo-12b-v14/special_tokens_map.json
nitral-ai-captain-bmo-12b-v14-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/nitral-ai-captain-bmo-12b-v14/tokenizer_config.json
nitral-ai-captain-bmo-12b-v14-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/nitral-ai-captain-bmo-12b-v14/tokenizer.json
nitral-ai-captain-bmo-12b-v14-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/nitral-ai-captain-bmo-12b-v14/flywheel_model.0.safetensors
nitral-ai-captain-bmo-12b-v14-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/363 [00:06<19:02,  3.17s/it]
Loading 0:   2%|▏         | 6/363 [00:06<05:04,  1.17it/s]
Loading 0:   3%|▎         | 11/363 [00:06<02:13,  2.63it/s]
Loading 0:   4%|▍         | 15/363 [00:06<01:25,  4.09it/s]
Loading 0:   6%|▌         | 22/363 [00:06<00:44,  7.60it/s]
Loading 0:   7%|▋         | 26/363 [00:07<00:34,  9.70it/s]
Loading 0:   8%|▊         | 30/363 [00:07<00:26, 12.44it/s]
Loading 0:   9%|▉         | 34/363 [00:07<00:22, 14.51it/s]
Loading 0:  10%|█         | 38/363 [00:07<00:18, 17.74it/s]
Loading 0:  12%|█▏        | 42/363 [00:07<00:22, 14.54it/s]
Loading 0:  13%|█▎        | 47/363 [00:07<00:16, 19.22it/s]
Loading 0:  14%|█▍        | 51/363 [00:08<00:15, 20.61it/s]
Loading 0:  15%|█▌        | 56/363 [00:08<00:12, 25.55it/s]
Loading 0:  17%|█▋        | 60/363 [00:08<00:12, 24.77it/s]
Loading 0:  18%|█▊        | 65/363 [00:08<00:10, 29.33it/s]
Loading 0:  19%|█▉        | 69/363 [00:08<00:10, 28.04it/s]
Loading 0:  20%|██        | 74/363 [00:08<00:08, 32.21it/s]
Loading 0:  21%|██▏       | 78/363 [00:08<00:09, 29.49it/s]
Loading 0:  23%|██▎       | 83/363 [00:08<00:08, 33.32it/s]
Loading 0:  24%|██▍       | 87/363 [00:09<00:08, 30.73it/s]
Loading 0:  25%|██▌       | 92/363 [00:09<00:07, 34.58it/s]
Loading 0:  26%|██▋       | 96/363 [00:09<00:08, 31.26it/s]
Loading 0:  28%|██▊       | 101/363 [00:09<00:07, 35.50it/s]
Loading 0:  29%|██▉       | 105/363 [00:09<00:07, 33.86it/s]
Loading 0:  31%|███       | 112/363 [00:09<00:06, 40.50it/s]
Loading 0:  32%|███▏      | 117/363 [00:09<00:06, 40.72it/s]
Loading 0:  34%|███▎      | 122/363 [00:10<00:08, 28.99it/s]
Loading 0:  35%|███▍      | 126/363 [00:10<00:07, 31.07it/s]
Loading 0:  36%|███▌      | 131/363 [00:10<00:06, 34.19it/s]
Loading 0:  38%|███▊      | 137/363 [00:10<00:06, 36.51it/s]
Loading 0:  39%|███▉      | 141/363 [00:10<00:06, 35.49it/s]
Loading 0:  41%|████      | 148/363 [00:10<00:05, 42.37it/s]
Loading 0:  42%|████▏     | 153/363 [00:10<00:05, 41.43it/s]
Loading 0:  44%|████▎     | 158/363 [00:10<00:04, 42.60it/s]
Loading 0:  45%|████▌     | 164/363 [00:11<00:04, 41.97it/s]
Loading 0:  47%|████▋     | 169/363 [00:11<00:04, 39.09it/s]
Loading 0:  48%|████▊     | 174/363 [00:11<00:04, 40.76it/s]
Loading 0:  49%|████▉     | 179/363 [00:11<00:05, 34.69it/s]
Loading 0:  50%|█████     | 183/363 [00:11<00:05, 33.77it/s]
Loading 0:  52%|█████▏    | 187/363 [00:11<00:05, 30.74it/s]
Loading 0:  53%|█████▎    | 191/363 [00:11<00:05, 32.48it/s]
Loading 0:  54%|█████▎    | 195/363 [00:12<00:05, 28.79it/s]
Loading 0:  55%|█████▌    | 200/363 [00:12<00:05, 31.88it/s]
Loading 0:  56%|█████▌    | 204/363 [00:12<00:07, 21.10it/s]
Loading 0:  58%|█████▊    | 211/363 [00:12<00:05, 28.00it/s]
Loading 0:  59%|█████▉    | 215/363 [00:12<00:05, 28.26it/s]
Loading 0:  60%|██████    | 219/363 [00:13<00:04, 29.90it/s]
Loading 0:  61%|██████▏   | 223/363 [00:13<00:05, 27.38it/s]
Loading 0:  63%|██████▎   | 227/363 [00:13<00:04, 28.70it/s]
Loading 0:  64%|██████▎   | 231/363 [00:13<00:04, 26.94it/s]
Loading 0:  65%|██████▌   | 236/363 [00:13<00:04, 31.27it/s]
Loading 0:  66%|██████▌   | 240/363 [00:13<00:04, 28.35it/s]
Loading 0:  67%|██████▋   | 245/363 [00:13<00:03, 31.75it/s]
Loading 0:  69%|██████▊   | 249/363 [00:14<00:03, 29.59it/s]
Loading 0:  70%|██████▉   | 254/363 [00:14<00:03, 33.34it/s]
Loading 0:  71%|███████   | 258/363 [00:14<00:03, 29.87it/s]
Loading 0:  72%|███████▏  | 263/363 [00:14<00:02, 33.59it/s]
Loading 0:  74%|███████▎  | 267/363 [00:14<00:03, 29.83it/s]
Loading 0:  75%|███████▍  | 272/363 [00:14<00:02, 32.75it/s]
Loading 0:  76%|███████▌  | 276/363 [00:14<00:02, 29.40it/s]
Loading 0:  77%|███████▋  | 281/363 [00:15<00:02, 33.75it/s]
Loading 0:  79%|███████▊  | 285/363 [00:15<00:03, 22.58it/s]
Loading 0:  80%|████████  | 292/363 [00:15<00:02, 30.61it/s]
Loading 0:  82%|████████▏ | 296/363 [00:15<00:02, 31.26it/s]
Loading 0:  83%|████████▎ | 302/363 [00:15<00:01, 31.89it/s]
Loading 0:  84%|████████▍ | 306/363 [00:15<00:01, 33.15it/s]
Loading 0:  86%|████████▌ | 311/363 [00:16<00:01, 36.07it/s]
Loading 0:  87%|████████▋ | 317/363 [00:16<00:01, 38.11it/s]
Loading 0:  89%|████████▊ | 322/363 [00:16<00:01, 38.54it/s]
Loading 0:  90%|█████████ | 328/363 [00:16<00:00, 42.85it/s]
Loading 0:  92%|█████████▏| 333/363 [00:16<00:00, 43.93it/s]
Loading 0:  93%|█████████▎| 338/363 [00:16<00:00, 44.41it/s]
Loading 0:  95%|█████████▍| 344/363 [00:16<00:00, 44.19it/s]
Loading 0:  96%|█████████▌| 349/363 [00:16<00:00, 43.26it/s]
Loading 0:  98%|█████████▊| 355/363 [00:16<00:00, 47.49it/s]
Loading 0:  99%|█████████▉| 360/363 [00:17<00:00, 46.00it/s]
                                                            

Job nitral-ai-captain-bmo-12b-v14-mkmlizer completed after 136.7s with status: succeeded
Stopping job with name nitral-ai-captain-bmo-12b-v14-mkmlizer
Pipeline stage MKMLizer completed in 137.30s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.18s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service nitral-ai-captain-bmo-12b-v14
Waiting for inference service nitral-ai-captain-bmo-12b-v14 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service nitral-ai-captain-bmo-12b-v14 ready after 140.53254389762878s
Pipeline stage MKMLDeployer completed in 141.08s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.1155846118927s
Received healthy response to inference request in 1.3692176342010498s
Received healthy response to inference request in 1.5388026237487793s
Received healthy response to inference request in 1.6056976318359375s
Received healthy response to inference request in 1.8446290493011475s
5 requests
0 failed requests
5th percentile: 1.4031346321105957
10th percentile: 1.4370516300201417
20th percentile: 1.5048856258392334
30th percentile: 1.5521816253662108
40th percentile: 1.5789396286010742
50th percentile: 1.6056976318359375
60th percentile: 1.7012701988220216
70th percentile: 1.7968427658081054
80th percentile: 1.898820161819458
90th percentile: 2.007202386856079
95th percentile: 2.0613934993743896
99th percentile: 2.104746389389038
mean time: 1.6947863101959229
Pipeline stage StressChecker completed in 9.63s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.46s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.28s
Shutdown handler de-registered
nitral-ai-captain-bmo-12b_v14 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2742.86s
Shutdown handler de-registered
nitral-ai-captain-bmo-12b_v14 status is now inactive due to auto deactivation removed underperforming models