chaiml-nemo-20241010-t_5991

submission_id: chaiml-nemo-20241010-t_5991_v155

developer_uid: chai_backend_admin

best_of: 4

celo_rating: 1250.23

display_name: chaiml-nemo-20241010-t_5991_v155

family_friendly_score: 0.5798172757475083

family_friendly_standard_error: 0.0063376819328883945

formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '### Response:\n{bot_name}:', 'truncate_by_message': False}

generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|eot_id|>', '<|end_of_text|>', 'You:'], 'max_input_tokens': 1024, 'best_of': 4, 'max_output_tokens': 64}

is_internal_developer: True

language_model: ChaiML/nemo-20241010_tier_merge_v4-albert

max_input_tokens: 1024

max_output_tokens: 64

model_architecture: MistralForCausalLM

model_group: ChaiML/nemo-20241010_tie

model_name: chaiml-nemo-20241010-t_5991_v155

model_num_parameters: 12772070400.0

model_repo: ChaiML/nemo-20241010_tier_merge_v4-albert

model_size: 13B

num_battles: 6338

num_wins: 3463

ranking_group: single

status: torndown

submission_type: basic

timestamp: 2024-10-15T12:00:33+00:00

us_pacific_date: 2024-10-15

win_ratio: 0.5463868728305459

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-nemo-20241010-t-5991-v155-mkmlizer
Waiting for job on chaiml-nemo-20241010-t-5991-v155-mkmlizer to finish
chaiml-nemo-20241010-t-5991-v155-mkmlizer: Downloaded to shared memory in 28.936s
chaiml-nemo-20241010-t-5991-v155-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpnhmym73o, device:0
chaiml-nemo-20241010-t-5991-v155-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission nousresearch-llama-2-13_8998_v10: ('http://chaiml-llama-8b-pairwis-8189-v27-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:40488->127.0.0.1:8080: read: connection reset by peer\n')
chaiml-nemo-20241010-t-5991-v155-mkmlizer: quantized model in 37.675s
chaiml-nemo-20241010-t-5991-v155-mkmlizer: Processed model ChaiML/nemo-20241010_tier_merge_v4-albert in 66.611s
chaiml-nemo-20241010-t-5991-v155-mkmlizer: creating bucket guanaco-mkml-models
chaiml-nemo-20241010-t-5991-v155-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-nemo-20241010-t-5991-v155-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-nemo-20241010-t-5991-v155
chaiml-nemo-20241010-t-5991-v155-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-nemo-20241010-t-5991-v155/special_tokens_map.json
chaiml-nemo-20241010-t-5991-v155-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-nemo-20241010-t-5991-v155/config.json
chaiml-nemo-20241010-t-5991-v155-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-nemo-20241010-t-5991-v155/tokenizer_config.json
chaiml-nemo-20241010-t-5991-v155-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-nemo-20241010-t-5991-v155/tokenizer.json
chaiml-nemo-20241010-t-5991-v155-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-nemo-20241010-t-5991-v155/flywheel_model.0.safetensors
chaiml-nemo-20241010-t-5991-v155-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/363 [00:06<18:13,  3.03s/it]
Loading 0:   2%|▏         | 6/363 [00:06<04:52,  1.22it/s]
Loading 0:   3%|▎         | 11/363 [00:06<02:08,  2.74it/s]
Loading 0:   4%|▍         | 15/363 [00:06<01:21,  4.25it/s]
Loading 0:   6%|▌         | 22/363 [00:06<00:43,  7.90it/s]
Loading 0:   7%|▋         | 26/363 [00:06<00:33, 10.07it/s]
Loading 0:   9%|▊         | 31/363 [00:06<00:24, 13.66it/s]
Loading 0:  10%|▉         | 35/363 [00:06<00:20, 16.31it/s]
Loading 0:  11%|█         | 40/363 [00:07<00:19, 16.16it/s]
Loading 0:  12%|█▏        | 44/363 [00:07<00:17, 18.67it/s]
Loading 0:  13%|█▎        | 49/363 [00:07<00:13, 23.05it/s]
Loading 0:  15%|█▍        | 53/363 [00:07<00:12, 24.93it/s]
Loading 0:  16%|█▌        | 57/363 [00:07<00:11, 27.79it/s]
Loading 0:  17%|█▋        | 61/363 [00:07<00:11, 26.29it/s]
Loading 0:  18%|█▊        | 67/363 [00:08<00:09, 32.16it/s]
Loading 0:  20%|█▉        | 71/363 [00:08<00:09, 31.98it/s]
Loading 0:  21%|██        | 76/363 [00:08<00:08, 34.74it/s]
Loading 0:  22%|██▏       | 80/363 [00:08<00:08, 33.41it/s]
Loading 0:  23%|██▎       | 85/363 [00:08<00:07, 36.33it/s]
Loading 0:  25%|██▍       | 89/363 [00:08<00:07, 34.54it/s]
Loading 0:  26%|██▌       | 94/363 [00:08<00:07, 36.50it/s]
Loading 0:  27%|██▋       | 98/363 [00:08<00:07, 35.12it/s]
Loading 0:  28%|██▊       | 103/363 [00:09<00:06, 38.47it/s]
Loading 0:  29%|██▉       | 107/363 [00:09<00:06, 36.61it/s]
Loading 0:  31%|███       | 112/363 [00:09<00:06, 39.27it/s]
Loading 0:  32%|███▏      | 116/363 [00:09<00:06, 37.13it/s]
Loading 0:  33%|███▎      | 121/363 [00:09<00:09, 26.12it/s]
Loading 0:  34%|███▍      | 125/363 [00:09<00:08, 27.37it/s]
Loading 0:  36%|███▌      | 130/363 [00:09<00:07, 31.51it/s]
Loading 0:  37%|███▋      | 134/363 [00:10<00:07, 31.45it/s]
Loading 0:  38%|███▊      | 139/363 [00:10<00:06, 34.05it/s]
Loading 0:  39%|███▉      | 143/363 [00:10<00:06, 33.38it/s]
Loading 0:  41%|████      | 148/363 [00:10<00:05, 36.56it/s]
Loading 0:  42%|████▏     | 152/363 [00:10<00:06, 34.81it/s]
Loading 0:  43%|████▎     | 157/363 [00:10<00:05, 37.24it/s]
Loading 0:  44%|████▍     | 161/363 [00:10<00:05, 34.97it/s]
Loading 0:  46%|████▌     | 166/363 [00:10<00:05, 37.55it/s]
Loading 0:  47%|████▋     | 170/363 [00:11<00:05, 34.65it/s]
Loading 0:  48%|████▊     | 174/363 [00:11<00:05, 35.98it/s]
Loading 0:  49%|████▉     | 178/363 [00:11<00:05, 32.55it/s]
Loading 0:  51%|█████     | 184/363 [00:11<00:04, 36.84it/s]
Loading 0:  52%|█████▏    | 188/363 [00:11<00:05, 34.78it/s]
Loading 0:  53%|█████▎    | 193/363 [00:11<00:04, 37.82it/s]
Loading 0:  54%|█████▍    | 197/363 [00:11<00:04, 36.25it/s]
Loading 0:  56%|█████▌    | 202/363 [00:12<00:06, 25.71it/s]
Loading 0:  57%|█████▋    | 206/363 [00:12<00:05, 27.07it/s]
Loading 0:  58%|█████▊    | 211/363 [00:12<00:04, 31.01it/s]
Loading 0:  59%|█████▉    | 215/363 [00:12<00:04, 31.20it/s]
Loading 0:  61%|██████    | 220/363 [00:12<00:04, 34.59it/s]
Loading 0:  62%|██████▏   | 224/363 [00:12<00:04, 34.07it/s]
Loading 0:  63%|██████▎   | 229/363 [00:12<00:03, 37.54it/s]
Loading 0:  64%|██████▍   | 233/363 [00:12<00:03, 36.07it/s]
Loading 0:  66%|██████▌   | 238/363 [00:13<00:03, 38.05it/s]
Loading 0:  67%|██████▋   | 242/363 [00:13<00:03, 36.73it/s]
Loading 0:  68%|██████▊   | 248/363 [00:13<00:02, 40.63it/s]
Loading 0:  70%|██████▉   | 253/363 [00:13<00:02, 42.19it/s]
Loading 0:  71%|███████   | 258/363 [00:13<00:02, 35.14it/s]
Loading 0:  73%|███████▎  | 265/363 [00:13<00:02, 41.13it/s]
Loading 0:  74%|███████▍  | 270/363 [00:13<00:02, 39.75it/s]
Loading 0:  76%|███████▌  | 275/363 [00:13<00:02, 38.59it/s]
Loading 0:  77%|███████▋  | 279/363 [00:14<00:02, 38.59it/s]
Loading 0:  78%|███████▊  | 283/363 [00:14<00:03, 25.72it/s]
Loading 0:  79%|███████▉  | 287/363 [00:14<00:02, 27.05it/s]
Loading 0:  80%|████████  | 292/363 [00:14<00:02, 31.20it/s]
Loading 0:  82%|████████▏ | 296/363 [00:14<00:02, 31.99it/s]
Loading 0:  83%|████████▎ | 301/363 [00:14<00:01, 34.82it/s]
Loading 0:  84%|████████▍ | 305/363 [00:14<00:01, 34.05it/s]
Loading 0:  85%|████████▌ | 310/363 [00:15<00:01, 37.12it/s]
Loading 0:  87%|████████▋ | 314/363 [00:15<00:01, 35.49it/s]
Loading 0:  88%|████████▊ | 319/363 [00:15<00:01, 38.22it/s]
Loading 0:  89%|████████▉ | 323/363 [00:15<00:01, 36.70it/s]
Loading 0:  90%|█████████ | 328/363 [00:15<00:00, 39.97it/s]
Loading 0:  92%|█████████▏| 333/363 [00:15<00:00, 40.36it/s]
Loading 0:  93%|█████████▎| 338/363 [00:15<00:00, 41.51it/s]
Loading 0:  95%|█████████▍| 344/363 [00:15<00:00, 40.99it/s]
Loading 0:  96%|█████████▌| 349/363 [00:16<00:00, 41.20it/s]
Loading 0:  98%|█████████▊| 356/363 [00:16<00:00, 46.84it/s]
Loading 0: 100%|█████████▉| 362/363 [00:16<00:00, 44.09it/s]
                                                            
Job chaiml-nemo-20241010-t-5991-v155-mkmlizer completed after 92.9s with status: succeeded
Stopping job with name chaiml-nemo-20241010-t-5991-v155-mkmlizer
Pipeline stage MKMLizer completed in 93.42s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.20s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-nemo-20241010-t-5991-v155
Waiting for inference service chaiml-nemo-20241010-t-5991-v155 to be ready
Inference service chaiml-nemo-20241010-t-5991-v155 ready after 160.59121561050415s
Pipeline stage MKMLDeployer completed in 161.08s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.8215296268463135s
Received healthy response to inference request in 1.474519968032837s
Received healthy response to inference request in 1.327406883239746s
Received healthy response to inference request in 1.3586127758026123s
5 requests
1 failed requests
5th percentile: 1.3336480617523194
10th percentile: 1.3398892402648925
20th percentile: 1.352371597290039
30th percentile: 1.3817942142486572
40th percentile: 1.428157091140747
50th percentile: 1.474519968032837
60th percentile: 1.6133238315582275
70th percentile: 1.752127695083618
80th percentile: 5.489253139495853
90th percentile: 12.824700164794923
95th percentile: 16.492423677444457
99th percentile: 19.426602487564086
mean time: 5.228443288803101
%s, retrying in %s seconds...
Received healthy response to inference request in 1.4998016357421875s
Received healthy response to inference request in 1.5789616107940674s
Received healthy response to inference request in 1.391026496887207s
Received healthy response to inference request in 1.6186928749084473s
Received healthy response to inference request in 1.4011521339416504s
5 requests
0 failed requests
5th percentile: 1.3930516242980957
10th percentile: 1.3950767517089844
20th percentile: 1.3991270065307617
30th percentile: 1.4208820343017579
40th percentile: 1.4603418350219726
50th percentile: 1.4998016357421875
60th percentile: 1.5314656257629395
70th percentile: 1.5631296157836914
80th percentile: 1.5869078636169434
90th percentile: 1.6028003692626953
95th percentile: 1.6107466220855713
99th percentile: 1.6171036243438721
mean time: 1.497926950454712
Pipeline stage StressChecker completed in 37.00s
Shutdown handler de-registered
chaiml-nemo-20241010-t_5991_v155 status is now deployed due to DeploymentManager action
chaiml-nemo-20241010-t_5991_v155 status is now inactive due to auto deactivation removed underperforming models
chaiml-nemo-20241010-t_5991_v155 status is now torndown due to DeploymentManager action