chaiml-nemo-20241010-tie_9990

developer_uid: azuruce

submission_id: chaiml-nemo-20241010-tie_9990_v2

model_name: chaiml-nemo-20241010-tie_9990_v2

model_group: ChaiML/nemo-20241010_tie

status: torndown

timestamp: 2024-10-10T21:26:03+00:00

num_battles: 6014

num_wins: 3182

celo_rating: 1274.5

family_friendly_score: 0.5528021568290025

family_friendly_standard_error: 0.006503156224466569

submission_type: basic

model_repo: ChaiML/nemo-20241010_tier_merge_v1-albert

model_architecture: MistralForCausalLM

model_num_parameters: 12772070400.0

best_of: 8

max_input_tokens: 1024

max_output_tokens: 64

display_name: chaiml-nemo-20241010-tie_9990_v2

is_internal_developer: True

language_model: ChaiML/nemo-20241010_tier_merge_v1-albert

model_size: 13B

ranking_group: single

us_pacific_date: 2024-10-10

win_ratio: 0.5290987695377453

generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.05, 'top_k': 1000, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|eot_id|>', '<|end_of_text|>', 'You:'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '### Response:\n{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-nemo-20241010-tie-9990-v2-mkmlizer
Waiting for job on chaiml-nemo-20241010-tie-9990-v2-mkmlizer to finish
chaiml-nemo-20241010-tie-9990-v2-mkmlizer: Downloaded to shared memory in 42.931s
chaiml-nemo-20241010-tie-9990-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpp7o_bofd, device:0
chaiml-nemo-20241010-tie-9990-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-nemo-20241010-tie-9990-v2-mkmlizer: quantized model in 37.005s
chaiml-nemo-20241010-tie-9990-v2-mkmlizer: Processed model ChaiML/nemo-20241010_tier_merge_v1-albert in 79.936s
chaiml-nemo-20241010-tie-9990-v2-mkmlizer: creating bucket guanaco-mkml-models
chaiml-nemo-20241010-tie-9990-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-nemo-20241010-tie-9990-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-nemo-20241010-tie-9990-v2
chaiml-nemo-20241010-tie-9990-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-nemo-20241010-tie-9990-v2/config.json
chaiml-nemo-20241010-tie-9990-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-nemo-20241010-tie-9990-v2/special_tokens_map.json
chaiml-nemo-20241010-tie-9990-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-nemo-20241010-tie-9990-v2/tokenizer_config.json
chaiml-nemo-20241010-tie-9990-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-nemo-20241010-tie-9990-v2/tokenizer.json
chaiml-nemo-20241010-tie-9990-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-nemo-20241010-tie-9990-v2/flywheel_model.0.safetensors
chaiml-nemo-20241010-tie-9990-v2-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/363 [00:06<18:10,  3.02s/it]
Loading 0:   2%|▏         | 6/363 [00:06<04:52,  1.22it/s]
Loading 0:   3%|▎         | 11/363 [00:06<02:08,  2.74it/s]
Loading 0:   4%|▍         | 15/363 [00:06<01:21,  4.27it/s]
Loading 0:   6%|▌         | 22/363 [00:06<00:42,  7.96it/s]
Loading 0:   7%|▋         | 27/363 [00:06<00:30, 10.89it/s]
Loading 0:   9%|▉         | 32/363 [00:06<00:23, 14.38it/s]
Loading 0:  10%|█         | 37/363 [00:06<00:17, 18.17it/s]
Loading 0:  12%|█▏        | 42/363 [00:07<00:21, 15.18it/s]
Loading 0:  13%|█▎        | 49/363 [00:07<00:14, 21.15it/s]
Loading 0:  15%|█▍        | 54/363 [00:07<00:12, 24.24it/s]
Loading 0:  16%|█▋        | 59/363 [00:07<00:11, 27.61it/s]
Loading 0:  18%|█▊        | 64/363 [00:07<00:09, 30.71it/s]
Loading 0:  19%|█▉        | 69/363 [00:08<00:10, 28.15it/s]
Loading 0:  21%|██        | 76/363 [00:08<00:08, 35.26it/s]
Loading 0:  22%|██▏       | 81/363 [00:08<00:07, 35.93it/s]
Loading 0:  24%|██▎       | 86/363 [00:08<00:07, 36.49it/s]
Loading 0:  25%|██▌       | 91/363 [00:08<00:06, 39.02it/s]
Loading 0:  26%|██▋       | 96/363 [00:08<00:07, 34.14it/s]
Loading 0:  28%|██▊       | 103/363 [00:08<00:06, 41.49it/s]
Loading 0:  30%|██▉       | 108/363 [00:08<00:06, 41.92it/s]
Loading 0:  31%|███       | 113/363 [00:09<00:06, 40.19it/s]
Loading 0:  33%|███▎      | 118/363 [00:09<00:05, 41.60it/s]
Loading 0:  34%|███▍      | 123/363 [00:09<00:09, 25.32it/s]
Loading 0:  36%|███▌      | 130/363 [00:09<00:07, 32.45it/s]
Loading 0:  37%|███▋      | 135/363 [00:09<00:06, 34.69it/s]
Loading 0:  39%|███▊      | 140/363 [00:09<00:06, 36.57it/s]
Loading 0:  40%|███▉      | 145/363 [00:10<00:05, 39.01it/s]
Loading 0:  41%|████▏     | 150/363 [00:10<00:06, 34.83it/s]
Loading 0:  43%|████▎     | 157/363 [00:10<00:04, 42.40it/s]
Loading 0:  45%|████▍     | 162/363 [00:10<00:04, 41.26it/s]
Loading 0:  46%|████▌     | 167/363 [00:10<00:04, 40.82it/s]
Loading 0:  47%|████▋     | 172/363 [00:10<00:04, 42.19it/s]
Loading 0:  49%|████▉     | 177/363 [00:10<00:05, 34.64it/s]
Loading 0:  51%|█████     | 184/363 [00:11<00:04, 42.30it/s]
Loading 0:  52%|█████▏    | 189/363 [00:11<00:04, 42.93it/s]
Loading 0:  53%|█████▎    | 194/363 [00:11<00:03, 44.60it/s]
Loading 0:  55%|█████▍    | 199/363 [00:11<00:03, 45.59it/s]
Loading 0:  56%|█████▌    | 204/363 [00:11<00:05, 27.31it/s]
Loading 0:  58%|█████▊    | 211/363 [00:11<00:04, 34.53it/s]
Loading 0:  60%|█████▉    | 216/363 [00:11<00:04, 35.26it/s]
Loading 0:  61%|██████    | 221/363 [00:12<00:03, 36.92it/s]
Loading 0:  62%|██████▏   | 226/363 [00:12<00:03, 37.98it/s]
Loading 0:  64%|██████▎   | 231/363 [00:12<00:03, 34.76it/s]
Loading 0:  66%|██████▌   | 238/363 [00:12<00:02, 42.25it/s]
Loading 0:  67%|██████▋   | 243/363 [00:12<00:02, 41.70it/s]
Loading 0:  68%|██████▊   | 248/363 [00:12<00:02, 43.07it/s]
Loading 0:  70%|██████▉   | 253/363 [00:12<00:02, 43.92it/s]
Loading 0:  71%|███████   | 258/363 [00:13<00:02, 36.54it/s]
Loading 0:  73%|███████▎  | 265/363 [00:13<00:02, 43.24it/s]
Loading 0:  74%|███████▍  | 270/363 [00:13<00:02, 43.33it/s]
Loading 0:  76%|███████▌  | 275/363 [00:13<00:02, 43.92it/s]
Loading 0:  77%|███████▋  | 280/363 [00:13<00:01, 45.34it/s]
Loading 0:  79%|███████▊  | 285/363 [00:13<00:02, 28.09it/s]
Loading 0:  80%|████████  | 292/363 [00:13<00:02, 35.23it/s]
Loading 0:  82%|████████▏ | 297/363 [00:14<00:01, 36.49it/s]
Loading 0:  83%|████████▎ | 302/363 [00:14<00:01, 37.90it/s]
Loading 0:  85%|████████▍ | 307/363 [00:14<00:01, 38.52it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:01, 32.65it/s]
Loading 0:  88%|████████▊ | 319/363 [00:14<00:01, 40.25it/s]
Loading 0:  89%|████████▉ | 324/363 [00:14<00:00, 40.27it/s]
Loading 0:  91%|█████████ | 329/363 [00:14<00:00, 40.60it/s]
Loading 0:  92%|█████████▏| 334/363 [00:14<00:00, 41.56it/s]
Loading 0:  93%|█████████▎| 339/363 [00:15<00:00, 34.77it/s]
Loading 0:  95%|█████████▌| 346/363 [00:15<00:00, 41.59it/s]
Loading 0:  97%|█████████▋| 351/363 [00:15<00:00, 40.92it/s]
Loading 0:  98%|█████████▊| 356/363 [00:15<00:00, 40.74it/s]
Loading 0:  99%|█████████▉| 361/363 [00:15<00:00, 41.77it/s]
                                                            
Job chaiml-nemo-20241010-tie-9990-v2-mkmlizer completed after 113.09s with status: succeeded
Stopping job with name chaiml-nemo-20241010-tie-9990-v2-mkmlizer
Pipeline stage MKMLizer completed in 113.58s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-nemo-20241010-tie-9990-v2
Waiting for inference service chaiml-nemo-20241010-tie-9990-v2 to be ready
Inference service chaiml-nemo-20241010-tie-9990-v2 ready after 140.54667162895203s
Pipeline stage MKMLDeployer completed in 141.01s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.0965685844421387s
Received healthy response to inference request in 1.9755277633666992s
Received healthy response to inference request in 1.5845105648040771s
Received healthy response to inference request in 1.9583861827850342s
5 requests
1 failed requests
5th percentile: 1.6592856884002685
10th percentile: 1.73406081199646
20th percentile: 1.8836110591888429
30th percentile: 1.9618144989013673
40th percentile: 1.9686711311340332
50th percentile: 1.9755277633666992
60th percentile: 2.023944091796875
70th percentile: 2.0723604202270507
80th percentile: 5.707575225830081
90th percentile: 12.92958850860596
95th percentile: 16.540595149993894
99th percentile: 19.429400463104248
mean time: 5.553318977355957
%s, retrying in %s seconds...
Received healthy response to inference request in 1.668454885482788s
Received healthy response to inference request in 1.4847710132598877s
Received healthy response to inference request in 1.631758451461792s
Received healthy response to inference request in 1.6028995513916016s
Received healthy response to inference request in 1.4547991752624512s
5 requests
0 failed requests
5th percentile: 1.4607935428619385
10th percentile: 1.4667879104614259
20th percentile: 1.4787766456604003
30th percentile: 1.5083967208862306
40th percentile: 1.555648136138916
50th percentile: 1.6028995513916016
60th percentile: 1.6144431114196778
70th percentile: 1.625986671447754
80th percentile: 1.6390977382659913
90th percentile: 1.6537763118743896
95th percentile: 1.6611155986785888
99th percentile: 1.6669870281219483
mean time: 1.5685366153717042
Pipeline stage StressChecker completed in 39.06s
Shutdown handler de-registered
chaiml-nemo-20241010-tie_9990_v2 status is now deployed due to DeploymentManager action
chaiml-nemo-20241010-tie_9990_v2 status is now inactive due to auto deactivation removed underperforming models
chaiml-nemo-20241010-tie_9990_v2 status is now torndown due to DeploymentManager action