chaiml-snugstable1-5-12_76771

developer_uid: zonemercy

submission_id: chaiml-snugstable1-5-12_76771_v9

model_name: tempv1-2

model_group: ChaiML/SnugStable1.5-12b

status: torndown

timestamp: 2025-01-08T09:22:29+00:00

num_battles: 5376

num_wins: 2833

celo_rating: 1289.8

family_friendly_score: 0.597

family_friendly_standard_error: 0.006936728335461898

submission_type: basic

model_repo: ChaiML/SnugStable1.5-12b0-l1e5r7b4

model_architecture: MistralForCausalLM

model_num_parameters: 12772070400.0

best_of: 8

max_input_tokens: 768

max_output_tokens: 64

reward_model: default

latencies: [{'batch_size': 1, 'throughput': 0.6574889152056638, 'latency_mean': 1.520849496126175, 'latency_p50': 1.5098381042480469, 'latency_p90': 1.6889514207839966}, {'batch_size': 3, 'throughput': 1.262550629814562, 'latency_mean': 2.3726762104034425, 'latency_p50': 2.37555730342865, 'latency_p90': 2.6056356906890867}, {'batch_size': 5, 'throughput': 1.588771931472642, 'latency_mean': 3.1254941022396086, 'latency_p50': 3.081817150115967, 'latency_p90': 3.531702399253845}, {'batch_size': 6, 'throughput': 1.6780181460788477, 'latency_mean': 3.5507429993152617, 'latency_p50': 3.552335500717163, 'latency_p90': 3.984095501899719}, {'batch_size': 8, 'throughput': 1.7653046109263804, 'latency_mean': 4.506830245256424, 'latency_p50': 4.508641719818115, 'latency_p90': 5.11854829788208}, {'batch_size': 10, 'throughput': 1.825486729570002, 'latency_mean': 5.433750205039978, 'latency_p50': 5.426156282424927, 'latency_p90': 6.0998001337051395}]

gpu_counts: {'NVIDIA RTX A5000': 1}

display_name: tempv1-2

is_internal_developer: True

language_model: ChaiML/SnugStable1.5-12b0-l1e5r7b4

model_size: 13B

ranking_group: single

throughput_3p7s: 1.71

us_pacific_date: 2025-01-08

win_ratio: 0.5269717261904762

generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '<|eot_id|>', 'Bot:', '</s>', '\n', 'You:', 'User:', '####'], 'max_input_tokens': 768, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': True}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-snugstable1-5-12-76771-v9-mkmlizer
Waiting for job on chaiml-snugstable1-5-12-76771-v9-mkmlizer to finish
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║     _____            __           __                                ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║       /___/                                                         ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║                                                                     ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║  Version: 0.11.12                                                   ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║                                                                     ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║  belonging to:                                                      ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║                                                                     ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║  Expiration: 2025-04-15 23:59:59                                    ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ║                                                                     ║
chaiml-snugstable1-5-12-76771-v9-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-snugstable1-5-12-76771-v9-mkmlizer: quantized model in 38.714s
chaiml-snugstable1-5-12-76771-v9-mkmlizer: Processed model ChaiML/SnugStable1.5-12b0-l1e5r7b4 in 77.695s
chaiml-snugstable1-5-12-76771-v9-mkmlizer: creating bucket guanaco-mkml-models
chaiml-snugstable1-5-12-76771-v9-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-snugstable1-5-12-76771-v9-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-snugstable1-5-12-76771-v9
chaiml-snugstable1-5-12-76771-v9-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-snugstable1-5-12-76771-v9/special_tokens_map.json
chaiml-snugstable1-5-12-76771-v9-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-snugstable1-5-12-76771-v9/config.json
chaiml-snugstable1-5-12-76771-v9-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-snugstable1-5-12-76771-v9/tokenizer_config.json
chaiml-snugstable1-5-12-76771-v9-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-snugstable1-5-12-76771-v9/flywheel_model.0.safetensors
chaiml-snugstable1-5-12-76771-v9-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:13, 25.83it/s]
Loading 0:   3%|▎         | 12/363 [00:00<00:08, 41.67it/s]
Loading 0:   5%|▍         | 17/363 [00:00<00:09, 38.35it/s]
Loading 0:   6%|▌         | 22/363 [00:00<00:08, 39.22it/s]
Loading 0:   7%|▋         | 27/363 [00:00<00:08, 40.61it/s]
Loading 0:   9%|▉         | 32/363 [00:00<00:09, 33.25it/s]
Loading 0:  11%|█         | 39/363 [00:01<00:07, 41.18it/s]
Loading 0:  12%|█▏        | 44/363 [00:01<00:07, 40.12it/s]
Loading 0:  13%|█▎        | 49/363 [00:01<00:07, 39.67it/s]
Loading 0:  15%|█▍        | 54/363 [00:01<00:07, 40.15it/s]
Loading 0:  16%|█▋        | 59/363 [00:01<00:07, 40.94it/s]
Loading 0:  18%|█▊        | 64/363 [00:01<00:12, 23.32it/s]
Loading 0:  19%|█▉        | 69/363 [00:02<00:10, 27.45it/s]
Loading 0:  20%|██        | 73/363 [00:02<00:11, 25.90it/s]
Loading 0:  21%|██▏       | 78/363 [00:02<00:09, 30.08it/s]
Loading 0:  23%|██▎       | 82/363 [00:02<00:09, 28.84it/s]
Loading 0:  24%|██▍       | 87/363 [00:02<00:08, 33.25it/s]
Loading 0:  25%|██▌       | 91/363 [00:02<00:08, 31.00it/s]
Loading 0:  26%|██▋       | 96/363 [00:02<00:07, 34.03it/s]
Loading 0:  28%|██▊       | 100/363 [00:03<00:08, 31.06it/s]
Loading 0:  29%|██▉       | 105/363 [00:03<00:07, 35.12it/s]
Loading 0:  30%|███       | 110/363 [00:03<00:07, 35.60it/s]
Loading 0:  31%|███▏      | 114/363 [00:03<00:07, 33.15it/s]
Loading 0:  33%|███▎      | 118/363 [00:03<00:07, 30.73it/s]
Loading 0:  34%|███▍      | 123/363 [00:03<00:06, 34.96it/s]
Loading 0:  35%|███▍      | 127/363 [00:03<00:07, 32.15it/s]
Loading 0:  36%|███▋      | 132/363 [00:03<00:06, 36.27it/s]
Loading 0:  37%|███▋      | 136/363 [00:04<00:07, 32.36it/s]
Loading 0:  39%|███▉      | 142/363 [00:04<00:08, 26.72it/s]
Loading 0:  40%|████      | 146/363 [00:04<00:07, 27.66it/s]
Loading 0:  41%|████▏     | 150/363 [00:04<00:07, 27.09it/s]
Loading 0:  43%|████▎     | 155/363 [00:04<00:06, 31.83it/s]
Loading 0:  44%|████▍     | 159/363 [00:04<00:06, 30.36it/s]
Loading 0:  45%|████▌     | 165/363 [00:05<00:05, 35.49it/s]
Loading 0:  47%|████▋     | 169/363 [00:05<00:05, 33.28it/s]
Loading 0:  48%|████▊     | 174/363 [00:05<00:05, 36.00it/s]
Loading 0:  49%|████▉     | 178/363 [00:05<00:05, 33.76it/s]
Loading 0:  50%|█████     | 183/363 [00:05<00:05, 35.53it/s]
Loading 0:  52%|█████▏    | 187/363 [00:05<00:05, 33.69it/s]
Loading 0:  53%|█████▎    | 192/363 [00:05<00:04, 36.24it/s]
Loading 0:  54%|█████▍    | 196/363 [00:05<00:04, 34.28it/s]
Loading 0:  55%|█████▌    | 201/363 [00:06<00:04, 36.29it/s]
Loading 0:  56%|█████▋    | 205/363 [00:06<00:04, 34.33it/s]
Loading 0:  58%|█████▊    | 210/363 [00:06<00:04, 36.54it/s]
Loading 0:  59%|█████▉    | 214/363 [00:06<00:04, 34.43it/s]
Loading 0:  60%|██████    | 218/363 [00:06<00:04, 33.60it/s]
Loading 0:  61%|██████▏   | 223/363 [00:06<00:05, 25.43it/s]
Loading 0:  62%|██████▏   | 226/363 [00:06<00:05, 24.88it/s]
Loading 0:  63%|██████▎   | 230/363 [00:07<00:05, 25.16it/s]
Loading 0:  65%|██████▌   | 237/363 [00:07<00:03, 33.15it/s]
Loading 0:  66%|██████▋   | 241/363 [00:07<00:03, 31.86it/s]
Loading 0:  68%|██████▊   | 246/363 [00:07<00:03, 34.30it/s]
Loading 0:  69%|██████▉   | 250/363 [00:07<00:03, 32.98it/s]
Loading 0:  70%|██████▉   | 254/363 [00:07<00:03, 33.50it/s]
Loading 0:  71%|███████   | 258/363 [00:07<00:03, 30.44it/s]
Loading 0:  73%|███████▎  | 264/363 [00:08<00:02, 35.81it/s]
Loading 0:  74%|███████▍  | 268/363 [00:08<00:02, 34.54it/s]
Loading 0:  75%|███████▌  | 273/363 [00:08<00:02, 37.33it/s]
Loading 0:  76%|███████▋  | 277/363 [00:08<00:02, 35.51it/s]
Loading 0:  78%|███████▊  | 282/363 [00:08<00:02, 37.41it/s]
Loading 0:  79%|███████▉  | 286/363 [00:08<00:02, 35.42it/s]
Loading 0:  80%|████████  | 291/363 [00:08<00:01, 37.70it/s]
Loading 0:  81%|████████▏ | 295/363 [00:08<00:01, 35.66it/s]
Loading 0:  82%|████████▏ | 299/363 [00:09<00:01, 35.61it/s]
Loading 0:  84%|████████▎ | 304/363 [00:15<00:28,  2.10it/s]
Loading 0:  85%|████████▍ | 307/363 [00:16<00:21,  2.63it/s]
Loading 0:  86%|████████▌ | 312/363 [00:16<00:13,  3.87it/s]
Loading 0:  88%|████████▊ | 320/363 [00:16<00:06,  6.63it/s]
Loading 0:  90%|████████▉ | 325/363 [00:16<00:04,  8.77it/s]
Loading 0:  91%|█████████ | 330/363 [00:16<00:03, 10.83it/s]
Loading 0:  93%|█████████▎| 337/363 [00:16<00:01, 15.59it/s]
Loading 0:  94%|█████████▍| 342/363 [00:16<00:01, 18.60it/s]
Loading 0:  96%|█████████▌| 347/363 [00:16<00:00, 21.83it/s]
Loading 0:  97%|█████████▋| 352/363 [00:17<00:00, 24.31it/s]
Loading 0:  98%|█████████▊| 356/363 [00:17<00:00, 25.82it/s]
Loading 0:  99%|█████████▉| 360/363 [00:17<00:00, 27.80it/s]
                                                            
Job chaiml-snugstable1-5-12-76771-v9-mkmlizer completed after 124.71s with status: succeeded
Stopping job with name chaiml-snugstable1-5-12-76771-v9-mkmlizer
Pipeline stage MKMLizer completed in 125.21s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-snugstable1-5-12-76771-v9
Waiting for inference service chaiml-snugstable1-5-12-76771-v9 to be ready
Failed to get response for submission mistralai-mistral-nem_93303_v257: HTTPConnectionPool(host='mistralai-mistral-nem-93303-v257-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service chaiml-snugstable1-5-12-76771-v9 ready after 371.83159160614014s
Pipeline stage MKMLDeployer completed in 372.39s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.8730063438415527s
Received healthy response to inference request in 1.9550962448120117s
Failed to get response for submission mistralai-mistral-nem_93303_v257: HTTPConnectionPool(host='mistralai-mistral-nem-93303-v257-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Received healthy response to inference request in 1.6914186477661133s
Received healthy response to inference request in 2.087245225906372s
Received healthy response to inference request in 1.0627760887145996s
5 requests
0 failed requests
5th percentile: 1.1885046005249023
10th percentile: 1.3142331123352051
20th percentile: 1.5656901359558106
30th percentile: 1.7277361869812011
40th percentile: 1.800371265411377
50th percentile: 1.8730063438415527
60th percentile: 1.9058423042297363
70th percentile: 1.93867826461792
80th percentile: 1.9815260410308837
90th percentile: 2.034385633468628
95th percentile: 2.0608154296875
99th percentile: 2.0819592666625977
mean time: 1.73390851020813
Pipeline stage StressChecker completed in 10.45s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.71s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.63s
Shutdown handler de-registered
chaiml-snugstable1-5-12_76771_v9 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.09s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service chaiml-snugstable1-5-12-76771-v9-profiler
Waiting for inference service chaiml-snugstable1-5-12-76771-v9-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2519.56s
Shutdown handler de-registered
chaiml-snugstable1-5-12_76771_v9 status is now inactive due to auto deactivation removed underperforming models
chaiml-snugstable1-5-12_76771_v9 status is now torndown due to DeploymentManager action