developer_uid: chai_backend_admin
submission_id: chaiml-nemo-20241010-ti_5991_v14
model_name: chaiml-nemo-20241010-ti_5991_v14
model_group: ChaiML/nemo-20241010_tie
status: torndown
timestamp: 2024-10-12T06:00:51+00:00
num_battles: 6201
num_wins: 3158
celo_rating: 1258.19
family_friendly_score: 0.5418839360807401
family_friendly_standard_error: 0.006431971805130002
submission_type: basic
model_repo: ChaiML/nemo-20241010_tier_merge_v4-albert
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 4
max_input_tokens: 1024
max_output_tokens: 64
display_name: chaiml-nemo-20241010-ti_5991_v14
is_internal_developer: True
language_model: ChaiML/nemo-20241010_tier_merge_v4-albert
model_size: 13B
ranking_group: single
us_pacific_date: 2024-10-11
win_ratio: 0.5092726979519432
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|eot_id|>', '<|end_of_text|>', 'You:'], 'max_input_tokens': 1024, 'best_of': 4, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '### Response:\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-nemo-20241010-ti-5991-v14-mkmlizer
Waiting for job on chaiml-nemo-20241010-ti-5991-v14-mkmlizer to finish
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║     _____            __           __                                ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║       /___/                                                         ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║                                                                     ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║  Version: 0.11.12                                                   ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║                                                                     ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║  belonging to:                                                      ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║                                                                     ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ║                                                                     ║
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: Downloaded to shared memory in 60.264s
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpxc96nkfs, device:0
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: quantized model in 36.527s
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: Processed model ChaiML/nemo-20241010_tier_merge_v4-albert in 96.791s
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: creating bucket guanaco-mkml-models
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-nemo-20241010-ti-5991-v14
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-nemo-20241010-ti-5991-v14/config.json
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-nemo-20241010-ti-5991-v14/special_tokens_map.json
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-nemo-20241010-ti-5991-v14/tokenizer_config.json
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-nemo-20241010-ti-5991-v14/tokenizer.json
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-nemo-20241010-ti-5991-v14/flywheel_model.0.safetensors
chaiml-nemo-20241010-ti-5991-v14-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/363 [00:06<18:14,  3.03s/it]
Loading 0:   2%|▏         | 6/363 [00:06<04:52,  1.22it/s]
Loading 0:   3%|▎         | 11/363 [00:06<02:08,  2.74it/s]
Loading 0:   4%|▍         | 15/363 [00:06<01:21,  4.29it/s]
Loading 0:   6%|▌         | 22/363 [00:06<00:42,  8.03it/s]
Loading 0:   7%|▋         | 27/363 [00:06<00:30, 10.98it/s]
Loading 0:   9%|▉         | 32/363 [00:06<00:22, 14.42it/s]
Loading 0:  10%|█         | 38/363 [00:06<00:17, 18.59it/s]
Loading 0:  12%|█▏        | 43/363 [00:07<00:18, 17.54it/s]
Loading 0:  14%|█▍        | 50/363 [00:07<00:13, 23.74it/s]
Loading 0:  15%|█▌        | 56/363 [00:07<00:11, 26.94it/s]
Loading 0:  17%|█▋        | 61/363 [00:07<00:10, 29.31it/s]
Loading 0:  18%|█▊        | 67/363 [00:07<00:08, 34.47it/s]
Loading 0:  20%|█▉        | 72/363 [00:07<00:08, 35.75it/s]
Loading 0:  21%|██        | 77/363 [00:08<00:07, 36.52it/s]
Loading 0:  23%|██▎       | 82/363 [00:08<00:07, 39.22it/s]
Loading 0:  24%|██▍       | 87/363 [00:08<00:07, 34.56it/s]
Loading 0:  26%|██▌       | 94/363 [00:08<00:06, 41.19it/s]
Loading 0:  27%|██▋       | 99/363 [00:08<00:06, 41.29it/s]
Loading 0:  29%|██▊       | 104/363 [00:08<00:06, 42.34it/s]
Loading 0:  30%|███       | 109/363 [00:08<00:05, 44.18it/s]
Loading 0:  31%|███▏      | 114/363 [00:09<00:06, 35.97it/s]
Loading 0:  33%|███▎      | 121/363 [00:09<00:07, 30.55it/s]
Loading 0:  34%|███▍      | 125/363 [00:09<00:07, 31.12it/s]
Loading 0:  36%|███▌      | 130/363 [00:09<00:06, 34.80it/s]
Loading 0:  37%|███▋      | 134/363 [00:09<00:06, 34.93it/s]
Loading 0:  39%|███▊      | 140/363 [00:09<00:05, 39.00it/s]
Loading 0:  40%|████      | 146/363 [00:09<00:05, 38.75it/s]
Loading 0:  42%|████▏     | 151/363 [00:10<00:05, 39.66it/s]
Loading 0:  43%|████▎     | 157/363 [00:10<00:04, 43.21it/s]
Loading 0:  45%|████▍     | 162/363 [00:10<00:04, 42.21it/s]
Loading 0:  46%|████▌     | 167/363 [00:10<00:04, 42.05it/s]
Loading 0:  48%|████▊     | 173/363 [00:10<00:04, 40.90it/s]
Loading 0:  49%|████▉     | 178/363 [00:10<00:04, 40.56it/s]
Loading 0:  51%|█████     | 184/363 [00:10<00:04, 44.29it/s]
Loading 0:  52%|█████▏    | 189/363 [00:10<00:03, 43.61it/s]
Loading 0:  53%|█████▎    | 194/363 [00:11<00:03, 44.86it/s]
Loading 0:  55%|█████▌    | 200/363 [00:11<00:03, 42.02it/s]
Loading 0:  56%|█████▋    | 205/363 [00:11<00:05, 30.31it/s]
Loading 0:  58%|█████▊    | 211/363 [00:11<00:04, 35.94it/s]
Loading 0:  60%|█████▉    | 216/363 [00:11<00:03, 37.90it/s]
Loading 0:  61%|██████    | 221/363 [00:11<00:03, 39.53it/s]
Loading 0:  62%|██████▏   | 226/363 [00:11<00:03, 40.78it/s]
Loading 0:  64%|██████▎   | 231/363 [00:12<00:03, 35.48it/s]
Loading 0:  66%|██████▌   | 238/363 [00:12<00:02, 42.83it/s]
Loading 0:  67%|██████▋   | 243/363 [00:12<00:02, 43.08it/s]
Loading 0:  68%|██████▊   | 248/363 [00:12<00:02, 42.87it/s]
Loading 0:  70%|██████▉   | 253/363 [00:12<00:02, 44.09it/s]
Loading 0:  71%|███████   | 258/363 [00:12<00:02, 35.92it/s]
Loading 0:  73%|███████▎  | 265/363 [00:12<00:02, 42.92it/s]
Loading 0:  74%|███████▍  | 270/363 [00:12<00:02, 42.78it/s]
Loading 0:  76%|███████▌  | 275/363 [00:13<00:02, 42.86it/s]
Loading 0:  77%|███████▋  | 280/363 [00:13<00:01, 43.41it/s]
Loading 0:  79%|███████▊  | 285/363 [00:13<00:02, 26.06it/s]
Loading 0:  80%|████████  | 292/363 [00:13<00:02, 33.41it/s]
Loading 0:  82%|████████▏ | 297/363 [00:13<00:01, 35.77it/s]
Loading 0:  83%|████████▎ | 302/363 [00:13<00:01, 37.69it/s]
Loading 0:  85%|████████▍ | 307/363 [00:13<00:01, 39.69it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:01, 33.68it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:01, 42.25it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:00, 40.92it/s]
Loading 0:  91%|█████████ | 331/363 [00:14<00:00, 40.64it/s]
Loading 0:  93%|█████████▎| 337/363 [00:14<00:00, 44.57it/s]
Loading 0:  94%|█████████▍| 342/363 [00:14<00:00, 43.23it/s]
Loading 0:  96%|█████████▌| 347/363 [00:14<00:00, 43.60it/s]
Loading 0:  97%|█████████▋| 352/363 [00:15<00:00, 45.04it/s]
Loading 0:  98%|█████████▊| 357/363 [00:15<00:00, 36.92it/s]
                                                            

Job chaiml-nemo-20241010-ti-5991-v14-mkmlizer completed after 124.83s with status: succeeded
Stopping job with name chaiml-nemo-20241010-ti-5991-v14-mkmlizer
Pipeline stage MKMLizer completed in 125.32s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-nemo-20241010-ti-5991-v14
Waiting for inference service chaiml-nemo-20241010-ti-5991-v14 to be ready
Inference service chaiml-nemo-20241010-ti-5991-v14 ready after 160.5605595111847s
Pipeline stage MKMLDeployer completed in 161.02s
run pipeline stage %s
Running pipeline stage StressChecker
('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Received unhealthy response to inference request!
Received healthy response to inference request in 2.3122661113739014s
Received healthy response to inference request in 1.4385921955108643s
Received healthy response to inference request in 1.613325834274292s
Received healthy response to inference request in 1.8038711547851562s
5 requests
1 failed requests
5th percentile: 0.38289837837219237
10th percentile: 0.6468218326568603
20th percentile: 1.1746687412261965
30th percentile: 1.4735389232635498
40th percentile: 1.5434323787689208
50th percentile: 1.613325834274292
60th percentile: 1.6895439624786377
70th percentile: 1.7657620906829834
80th percentile: 1.9055501461029054
90th percentile: 2.1089081287384035
95th percentile: 2.2105871200561524
99th percentile: 2.2919303131103517
mean time: 1.4574060440063477
%s, retrying in %s seconds...
Received healthy response to inference request in 1.65085768699646s
Received healthy response to inference request in 1.1226425170898438s
Received healthy response to inference request in 1.6587307453155518s
Received healthy response to inference request in 1.5897495746612549s
Received healthy response to inference request in 1.5765111446380615s
5 requests
0 failed requests
5th percentile: 1.2134162425994872
10th percentile: 1.304189968109131
20th percentile: 1.485737419128418
30th percentile: 1.5791588306427002
40th percentile: 1.5844542026519775
50th percentile: 1.5897495746612549
60th percentile: 1.614192819595337
70th percentile: 1.638636064529419
80th percentile: 1.6524322986602784
90th percentile: 1.655581521987915
95th percentile: 1.6571561336517333
99th percentile: 1.6584158229827881
mean time: 1.5196983337402343
Pipeline stage StressChecker completed in 17.46s
Shutdown handler de-registered
chaiml-nemo-20241010-ti_5991_v14 status is now deployed due to DeploymentManager action
chaiml-nemo-20241010-ti_5991_v14 status is now inactive due to auto deactivation removed underperforming models
chaiml-nemo-20241010-ti_5991_v14 status is now torndown due to DeploymentManager action