hastagaras-mn-base-12b-e_9769

developer_uid: Hastagaras

submission_id: hastagaras-mn-base-12b-e_9769_v1

model_name: test

model_group: Hastagaras/MN-Base-12B-E

status: inactive

timestamp: 2024-11-29T07:35:28+00:00

num_battles: 15148

num_wins: 7167

celo_rating: 1245.73

family_friendly_score: 0.573

family_friendly_standard_error: 0.006995298421082549

submission_type: basic

model_repo: Hastagaras/MN-Base-12B-Experiment-LMH-ET-10.20.30

model_architecture: MistralForCausalLM

model_num_parameters: 12772070400.0

best_of: 8

max_input_tokens: 1024

max_output_tokens: 64

latencies: [{'batch_size': 1, 'throughput': 0.6154463686634699, 'latency_mean': 1.6247795975208283, 'latency_p50': 1.6327276229858398, 'latency_p90': 1.7716447591781617}, {'batch_size': 3, 'throughput': 1.1431873699347257, 'latency_mean': 2.6193230986595153, 'latency_p50': 2.630236864089966, 'latency_p90': 2.9591696500778197}, {'batch_size': 5, 'throughput': 1.380427306104375, 'latency_mean': 3.6098186337947844, 'latency_p50': 3.6283698081970215, 'latency_p90': 4.043443608283996}, {'batch_size': 6, 'throughput': 1.4447813544342605, 'latency_mean': 4.122050507068634, 'latency_p50': 4.137305021286011, 'latency_p90': 4.653046822547912}, {'batch_size': 8, 'throughput': 1.517871704519896, 'latency_mean': 5.23478373169899, 'latency_p50': 5.224444031715393, 'latency_p90': 5.897012495994567}, {'batch_size': 10, 'throughput': 1.5637263756743658, 'latency_mean': 6.34233766913414, 'latency_p50': 6.365033268928528, 'latency_p90': 7.225018048286438}]

gpu_counts: {'NVIDIA RTX A5000': 1}

display_name: test

is_internal_developer: False

language_model: Hastagaras/MN-Base-12B-Experiment-LMH-ET-10.20.30

model_size: 13B

ranking_group: single

throughput_3p7s: 1.4

us_pacific_date: 2024-11-28

win_ratio: 0.4731317665698442

generation_params: {'temperature': 0.85, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '[SUFFIX]'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': "[PREFIX]system[MIDDLE]You're {bot_name}\n{memory}\n", 'prompt_template': '{prompt}[SUFFIX]', 'bot_template': '[PREFIX]model[MIDDLE]{bot_name}: {message}[SUFFIX]', 'user_template': '[PREFIX]user[MIDDLE]{user_name}: {message}[SUFFIX]', 'response_template': '[PREFIX]model[MIDDLE]{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name hastagaras-mn-base-12b-e-9769-v1-mkmlizer
Waiting for job on hastagaras-mn-base-12b-e-9769-v1-mkmlizer to finish
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║     _____            __           __                                ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║       /___/                                                         ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║                                                                     ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║  https://mk1.ai                                                     ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║                                                                     ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║  belonging to:                                                      ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║                                                                     ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║  Chai Research Corp.                                                ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ║                                                                     ║
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: Downloaded to shared memory in 49.062s
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpkf7icv42, device:0
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: quantized model in 36.700s
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: Processed model Hastagaras/MN-Base-12B-Experiment-LMH-ET-10.20.30 in 85.763s
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: creating bucket guanaco-mkml-models
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/hastagaras-mn-base-12b-e-9769-v1
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/hastagaras-mn-base-12b-e-9769-v1/config.json
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/hastagaras-mn-base-12b-e-9769-v1/special_tokens_map.json
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/hastagaras-mn-base-12b-e-9769-v1/tokenizer_config.json
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/hastagaras-mn-base-12b-e-9769-v1/tokenizer.json
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/hastagaras-mn-base-12b-e-9769-v1/flywheel_model.0.safetensors
hastagaras-mn-base-12b-e-9769-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 30.33it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 50.82it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 47.91it/s]
Loading 0:   7%|▋         | 25/363 [00:00<00:06, 48.61it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 49.89it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:06, 47.00it/s]
Loading 0:  12%|█▏        | 42/363 [00:00<00:07, 45.78it/s]
Loading 0:  13%|█▎        | 49/363 [00:01<00:06, 50.96it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 46.65it/s]
Loading 0:  17%|█▋        | 60/363 [00:01<00:06, 46.72it/s]
Loading 0:  18%|█▊        | 65/363 [00:01<00:10, 29.44it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:08, 35.61it/s]
Loading 0:  21%|██        | 77/363 [00:01<00:07, 38.10it/s]
Loading 0:  23%|██▎       | 82/363 [00:02<00:08, 33.09it/s]
Loading 0:  25%|██▍       | 89/363 [00:02<00:06, 40.60it/s]
Loading 0:  26%|██▌       | 94/363 [00:02<00:06, 41.56it/s]
Loading 0:  27%|██▋       | 99/363 [00:02<00:06, 42.11it/s]
Loading 0:  29%|██▉       | 105/363 [00:02<00:06, 41.69it/s]
Loading 0:  31%|███       | 112/363 [00:02<00:05, 46.20it/s]
Loading 0:  32%|███▏      | 117/363 [00:02<00:05, 44.49it/s]
Loading 0:  34%|███▎      | 122/363 [00:02<00:05, 44.01it/s]
Loading 0:  35%|███▍      | 127/363 [00:03<00:06, 35.94it/s]
Loading 0:  37%|███▋      | 135/363 [00:03<00:05, 43.75it/s]
Loading 0:  39%|███▉      | 141/363 [00:03<00:05, 42.25it/s]
Loading 0:  40%|████      | 146/363 [00:03<00:07, 30.37it/s]
Loading 0:  41%|████▏     | 150/363 [00:03<00:07, 28.99it/s]
Loading 0:  43%|████▎     | 156/363 [00:03<00:06, 33.33it/s]
Loading 0:  44%|████▍     | 160/363 [00:04<00:06, 33.48it/s]
Loading 0:  45%|████▌     | 165/363 [00:04<00:05, 36.86it/s]
Loading 0:  47%|████▋     | 169/363 [00:04<00:05, 36.50it/s]
Loading 0:  48%|████▊     | 174/363 [00:04<00:04, 38.74it/s]
Loading 0:  49%|████▉     | 179/363 [00:04<00:04, 39.17it/s]
Loading 0:  51%|█████     | 184/363 [00:04<00:04, 40.54it/s]
Loading 0:  52%|█████▏    | 189/363 [00:04<00:04, 42.80it/s]
Loading 0:  53%|█████▎    | 194/363 [00:04<00:04, 35.47it/s]
Loading 0:  55%|█████▌    | 201/363 [00:05<00:03, 42.90it/s]
Loading 0:  57%|█████▋    | 206/363 [00:05<00:03, 42.16it/s]
Loading 0:  58%|█████▊    | 211/363 [00:05<00:03, 40.77it/s]
Loading 0:  60%|█████▉    | 216/363 [00:05<00:03, 42.07it/s]
Loading 0:  61%|██████    | 221/363 [00:05<00:03, 42.34it/s]
Loading 0:  62%|██████▏   | 226/363 [00:05<00:05, 25.93it/s]
Loading 0:  63%|██████▎   | 230/363 [00:06<00:05, 26.23it/s]
Loading 0:  65%|██████▌   | 237/363 [00:06<00:03, 33.72it/s]
Loading 0:  67%|██████▋   | 242/363 [00:06<00:03, 35.01it/s]
Loading 0:  68%|██████▊   | 247/363 [00:06<00:03, 36.25it/s]
Loading 0:  69%|██████▉   | 252/363 [00:06<00:02, 38.55it/s]
Loading 0:  71%|███████   | 257/363 [00:06<00:03, 33.05it/s]
Loading 0:  73%|███████▎  | 264/363 [00:06<00:02, 39.14it/s]
Loading 0:  74%|███████▍  | 269/363 [00:06<00:02, 39.64it/s]
Loading 0:  75%|███████▌  | 274/363 [00:07<00:02, 39.87it/s]
Loading 0:  77%|███████▋  | 279/363 [00:07<00:02, 41.54it/s]
Loading 0:  78%|███████▊  | 284/363 [00:07<00:02, 34.07it/s]
Loading 0:  80%|████████  | 291/363 [00:07<00:01, 40.62it/s]
Loading 0:  82%|████████▏ | 296/363 [00:07<00:01, 40.00it/s]
Loading 0:  83%|████████▎ | 301/363 [00:07<00:01, 41.47it/s]
Loading 0:  84%|████████▍ | 306/363 [00:14<00:23,  2.45it/s]
Loading 0:  85%|████████▌ | 310/363 [00:14<00:16,  3.18it/s]
Loading 0:  87%|████████▋ | 314/363 [00:14<00:11,  4.18it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:06,  6.24it/s]
Loading 0:  90%|████████▉ | 326/363 [00:15<00:04,  8.74it/s]
Loading 0:  91%|█████████ | 331/363 [00:15<00:02, 11.16it/s]
Loading 0:  93%|█████████▎| 338/363 [00:15<00:01, 15.95it/s]
Loading 0:  95%|█████████▍| 344/363 [00:15<00:00, 19.61it/s]
Loading 0:  96%|█████████▌| 349/363 [00:15<00:00, 22.58it/s]
Loading 0:  98%|█████████▊| 356/363 [00:15<00:00, 28.75it/s]
Loading 0: 100%|█████████▉| 362/363 [00:15<00:00, 31.53it/s]
                                                            
Job hastagaras-mn-base-12b-e-9769-v1-mkmlizer completed after 114.46s with status: succeeded
Stopping job with name hastagaras-mn-base-12b-e-9769-v1-mkmlizer
Pipeline stage MKMLizer completed in 115.01s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service hastagaras-mn-base-12b-e-9769-v1
Waiting for inference service hastagaras-mn-base-12b-e-9769-v1 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service hastagaras-mn-base-12b-e-9769-v1 ready after 140.4789080619812s
Pipeline stage MKMLDeployer completed in 141.02s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.1923136711120605s
Received healthy response to inference request in 1.868908405303955s
Received healthy response to inference request in 1.9027905464172363s
Received healthy response to inference request in 1.6078033447265625s
Received healthy response to inference request in 2.001307964324951s
5 requests
0 failed requests
5th percentile: 1.660024356842041
10th percentile: 1.7122453689575194
20th percentile: 1.8166873931884766
30th percentile: 1.8756848335266114
40th percentile: 1.8892376899719239
50th percentile: 1.9027905464172363
60th percentile: 1.9421975135803222
70th percentile: 1.9816044807434081
80th percentile: 2.039509105682373
90th percentile: 2.1159113883972167
95th percentile: 2.154112529754639
99th percentile: 2.1846734428405763
mean time: 1.9146247863769532
Pipeline stage StressChecker completed in 10.96s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Connection pool is full, discarding connection: %s. Connection pool size: %s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.47s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.34s
Shutdown handler de-registered
hastagaras-mn-base-12b-e_9769_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.09s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service hastagaras-mn-base-12b-e-9769-v1-profiler
Waiting for inference service hastagaras-mn-base-12b-e-9769-v1-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3215.33s
Shutdown handler de-registered
hastagaras-mn-base-12b-e_9769_v1 status is now inactive due to auto deactivation removed underperforming models