zonemercy-vingt-deux-v0-1e5

submission_id: zonemercy-vingt-deux-v0-1e5_v12

developer_uid: chai_backend_admin

best_of: 8

celo_rating: 1245.66

display_name: temp-2

family_friendly_score: 0.0

formatter: {'memory_template': 'Bot Name: {bot_name}\n####\n', 'prompt_template': '', 'bot_template': 'Bot: {message}\n', 'user_template': 'User: {message}\n', 'response_template': 'Bot:', 'truncate_by_message': False}

generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '####', 'Bot:', 'User:', 'You:', '<|im_end|>', '<|eot_id|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}

gpu_counts: {'NVIDIA RTX A6000': 1}

is_internal_developer: True

language_model: zonemercy/Vingt-Deux-v0-1e5

latencies: [{'batch_size': 1, 'throughput': 0.3811018963356682, 'latency_mean': 2.6239052414894104, 'latency_p50': 2.6380767822265625, 'latency_p90': 2.8945515155792236}, {'batch_size': 2, 'throughput': 0.604846737425986, 'latency_mean': 3.3019170081615448, 'latency_p50': 3.281363368034363, 'latency_p90': 3.6509014844894407}, {'batch_size': 3, 'throughput': 0.7583591020983999, 'latency_mean': 3.937962874174118, 'latency_p50': 3.931504011154175, 'latency_p90': 4.315424704551697}, {'batch_size': 4, 'throughput': 0.8848746088326525, 'latency_mean': 4.492236130237579, 'latency_p50': 4.52222216129303, 'latency_p90': 5.007832765579224}, {'batch_size': 5, 'throughput': 0.9733559105678249, 'latency_mean': 5.100026845932007, 'latency_p50': 5.100275874137878, 'latency_p90': 5.7895451307296755}]

max_input_tokens: 1024

max_output_tokens: 64

model_architecture: MistralForCausalLM

model_group: zonemercy/Vingt-Deux-v0-

model_name: temp-2

model_num_parameters: 22247282688.0

model_repo: zonemercy/Vingt-Deux-v0-1e5

model_size: 22B

num_battles: 53642

num_wins: 26862

ranking_group: single

status: torndown

submission_type: basic

throughput_3p7s: 0.71

timestamp: 2024-09-23T09:38:47+00:00

us_pacific_date: 2024-09-23

win_ratio: 0.5007643264606092

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name zonemercy-vingt-deux-v0-1e5-v12-mkmlizer
Waiting for job on zonemercy-vingt-deux-v0-1e5-v12-mkmlizer to finish
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║     _____            __           __                                ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║       /___/                                                         ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║  Version: 0.10.1                                                    ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║  https://mk1.ai                                                     ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║  The license key for the current software has been verified as      ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║  belonging to:                                                      ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║  Chai Research Corp.                                                ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: Downloaded to shared memory in 48.962s
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmppdmhg7xg, device:0
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: Saving flywheel model at /dev/shm/model_cache
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: quantized model in 45.875s
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: Processed model zonemercy/Vingt-Deux-v0-1e5 in 94.837s
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-vingt-deux-v0-1e5-v12
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v0-1e5-v12/tokenizer_config.json
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v0-1e5-v12/tokenizer.json
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v0-1e5-v12/flywheel_model.1.safetensors
zonemercy-vingt-deux-v0-1e5-v12-mkmlizer: 
Loading 0:   0%|          | 0/507 [00:00<?, ?it/s]
Loading 0:   1%|          | 5/507 [00:00<00:21, 23.74it/s]
Loading 0:   2%|▏         | 12/507 [00:00<00:12, 39.11it/s]
Loading 0:   3%|▎         | 17/507 [00:00<00:13, 36.21it/s]
Loading 0:   4%|▍         | 22/507 [00:00<00:12, 37.38it/s]
Loading 0:   5%|▌         | 27/507 [00:00<00:12, 39.59it/s]
Loading 0:   6%|▋         | 32/507 [00:00<00:14, 33.58it/s]
Loading 0:   8%|▊         | 39/507 [00:01<00:11, 40.98it/s]
Loading 0:   9%|▊         | 44/507 [00:01<00:11, 40.77it/s]
Loading 0:  10%|▉         | 49/507 [00:01<00:12, 36.00it/s]
Loading 0:  10%|█         | 53/507 [00:01<00:17, 25.26it/s]
Loading 0:  11%|█         | 57/507 [00:01<00:17, 25.28it/s]
Loading 0:  12%|█▏        | 63/507 [00:01<00:14, 30.71it/s]
Loading 0:  13%|█▎        | 67/507 [00:02<00:14, 30.78it/s]
Loading 0:  14%|█▍        | 71/507 [00:02<00:13, 32.23it/s]
Loading 0:  15%|█▍        | 75/507 [00:02<00:13, 31.42it/s]
Loading 0:  16%|█▌        | 80/507 [00:02<00:14, 29.19it/s]
Loading 0:  17%|█▋        | 85/507 [00:02<00:12, 33.27it/s]
Loading 0:  18%|█▊        | 89/507 [00:02<00:13, 29.88it/s]
Loading 0:  19%|█▊        | 94/507 [00:02<00:12, 33.97it/s]
Loading 0:  19%|█▉        | 98/507 [00:03<00:13, 29.39it/s]
Loading 0:  20%|██        | 103/507 [00:03<00:12, 32.68it/s]
Loading 0:  21%|██        | 107/507 [00:03<00:14, 27.79it/s]
Loading 0:  22%|██▏       | 112/507 [00:03<00:12, 31.38it/s]
Loading 0:  23%|██▎       | 116/507 [00:03<00:19, 19.89it/s]
Loading 0:  24%|██▍       | 122/507 [00:04<00:17, 22.57it/s]
Loading 0:  25%|██▌       | 127/507 [00:04<00:14, 26.43it/s]
Loading 0:  26%|██▌       | 131/507 [00:04<00:15, 24.73it/s]
Loading 0:  27%|██▋       | 136/507 [00:04<00:12, 28.73it/s]
Loading 0:  28%|██▊       | 140/507 [00:04<00:13, 26.66it/s]
Loading 0:  29%|██▊       | 145/507 [00:04<00:11, 30.59it/s]
Loading 0:  29%|██▉       | 149/507 [00:04<00:12, 27.66it/s]
Loading 0:  30%|███       | 154/507 [00:05<00:11, 31.36it/s]
Loading 0:  31%|███       | 158/507 [00:05<00:12, 28.65it/s]
Loading 0:  32%|███▏      | 164/507 [00:05<00:10, 33.35it/s]
Loading 0:  33%|███▎      | 169/507 [00:05<00:12, 26.92it/s]
Loading 0:  34%|███▍      | 173/507 [00:05<00:11, 28.32it/s]
Loading 0:  35%|███▍      | 177/507 [00:05<00:11, 28.39it/s]
Loading 0:  36%|███▌      | 183/507 [00:06<00:09, 33.91it/s]
Loading 0:  37%|███▋      | 187/507 [00:06<00:09, 32.53it/s]
Loading 0:  38%|███▊      | 192/507 [00:06<00:09, 34.75it/s]
Loading 0:  39%|███▊      | 196/507 [00:06<00:09, 33.10it/s]
Loading 0:  40%|███▉      | 201/507 [00:06<00:08, 34.77it/s]
Loading 0:  40%|████      | 205/507 [00:06<00:08, 33.93it/s]
Loading 0:  42%|████▏     | 211/507 [00:06<00:07, 37.63it/s]
Loading 0:  43%|████▎     | 217/507 [00:06<00:07, 38.68it/s]
Loading 0:  44%|████▎     | 221/507 [00:07<00:07, 36.76it/s]
Loading 0:  44%|████▍     | 225/507 [00:07<00:10, 26.62it/s]
Loading 0:  45%|████▌     | 230/507 [00:07<00:10, 26.60it/s]
Loading 0:  46%|████▋     | 235/507 [00:07<00:08, 30.58it/s]
Loading 0:  47%|████▋     | 239/507 [00:07<00:09, 28.22it/s]
Loading 0:  48%|████▊     | 244/507 [00:07<00:08, 32.61it/s]
Loading 0:  49%|████▉     | 248/507 [00:08<00:08, 30.16it/s]
Loading 0:  50%|█████     | 254/507 [00:08<00:06, 36.82it/s]
Loading 0:  51%|█████     | 259/507 [00:08<00:07, 32.93it/s]
Loading 0:  52%|█████▏    | 263/507 [00:08<00:07, 33.48it/s]
Loading 0:  53%|█████▎    | 267/507 [00:08<00:08, 29.68it/s]
Loading 0:  53%|█████▎    | 271/507 [00:08<00:07, 30.75it/s]
Loading 0:  54%|█████▍    | 275/507 [00:08<00:08, 28.72it/s]
Loading 0:  55%|█████▌    | 280/507 [00:09<00:07, 32.28it/s]
Loading 0:  56%|█████▌    | 284/507 [00:09<00:09, 23.55it/s]
Loading 0:  57%|█████▋    | 287/507 [00:09<00:10, 21.46it/s]
Loading 0:  58%|█████▊    | 293/507 [00:09<00:08, 24.45it/s]
Loading 0:  59%|█████▉    | 298/507 [00:09<00:07, 28.56it/s]
Loading 0:  59%|█████▉    | 299/507 [00:19<00:07, 28.56it/s]
Loading 0:  59%|█████▉    | 300/507 [00:24<04:08,  1.20s/it]
Loading 0:  60%|█████▉    | 302/507 [00:25<03:24,  1.00it/s]
Loading 0:  61%|██████    | 307/507 [00:25<02:04,  1.61it/s]
Loading 0:  61%|██████    | 310/507 [00:25<01:33,  2.10it/s]
Loading 0:  62%|██████▏   | 313/507 [00:25<01:09,  2.78it/s]
Loading 0:  62%|██████▏   | 316/507 [00:25<00:51,  3.69it/s]
Loading 0:  63%|██████▎   | 320/507 [00:25<00:36,  5.18it/s]
Loading 0:  64%|██████▍   | 325/507 [00:25<00:23,  7.82it/s]
Loading 0:  65%|██████▍   | 329/507 [00:26<00:18,  9.75it/s]
Loading 0:  66%|██████▌   | 334/507 [00:26<00:13, 13.30it/s]
Loading 0:  67%|██████▋   | 339/507 [00:26<00:09, 16.87it/s]
Loading 0:  68%|██████▊   | 343/507 [00:26<00:10, 15.33it/s]
Loading 0:  68%|██████▊   | 347/507 [00:26<00:09, 17.44it/s]
Loading 0:  70%|██████▉   | 354/507 [00:26<00:06, 24.77it/s]
Loading 0:  71%|███████   | 358/507 [00:27<00:05, 26.45it/s]
Loading 0:  72%|███████▏  | 363/507 [00:27<00:04, 30.80it/s]
Loading 0:  72%|███████▏  | 367/507 [00:27<00:04, 31.34it/s]
Loading 0:  73%|███████▎  | 372/507 [00:27<00:03, 35.15it/s]
Loading 0:  74%|███████▍  | 377/507 [00:27<00:03, 35.87it/s]
Loading 0:  75%|███████▌  | 382/507 [00:27<00:03, 37.66it/s]
Loading 0:  77%|███████▋  | 388/507 [00:27<00:03, 38.97it/s]
Loading 0:  78%|███████▊  | 393/507 [00:27<00:02, 38.38it/s]
Loading 0:  78%|███████▊  | 397/507 [00:28<00:03, 28.88it/s]
Loading 0:  79%|███████▉  | 401/507 [00:28<00:03, 28.29it/s]
Loading 0:  80%|████████  | 408/507 [00:28<00:02, 35.74it/s]
Loading 0:  81%|████████▏ | 412/507 [00:28<00:02, 35.64it/s]
Loading 0:  82%|████████▏ | 417/507 [00:28<00:02, 37.67it/s]
Loading 0:  83%|████████▎ | 421/507 [00:28<00:02, 34.00it/s]
Loading 0:  84%|████████▍ | 425/507 [00:28<00:02, 33.74it/s]
Loading 0:  85%|████████▍ | 429/507 [00:29<00:02, 29.27it/s]
Loading 0:  86%|████████▌ | 435/507 [00:29<00:02, 33.89it/s]
Loading 0:  87%|████████▋ | 439/507 [00:29<00:02, 32.88it/s]
Loading 0:  88%|████████▊ | 444/507 [00:29<00:01, 34.87it/s]
Loading 0:  88%|████████▊ | 448/507 [00:29<00:01, 33.17it/s]
Loading 0:  89%|████████▉ | 452/507 [00:29<00:01, 33.82it/s]
Loading 0:  90%|████████▉ | 456/507 [00:32<00:09,  5.28it/s]
Loading 0:  91%|█████████ | 459/507 [00:32<00:07,  6.37it/s]
Loading 0:  92%|█████████▏| 465/507 [00:32<00:04,  9.53it/s]
Loading 0:  93%|█████████▎| 472/507 [00:32<00:02, 14.37it/s]
Loading 0:  94%|█████████▍| 476/507 [00:32<00:01, 16.84it/s]
Loading 0:  95%|█████████▍| 481/507 [00:32<00:01, 20.94it/s]
Loading 0:  96%|█████████▌| 485/507 [00:32<00:00, 23.36it/s]
Loading 0:  97%|█████████▋| 490/507 [00:33<00:00, 27.70it/s]
Loading 0:  98%|█████████▊| 495/507 [00:33<00:00, 29.98it/s]
Loading 0:  99%|█████████▊| 500/507 [00:33<00:00, 32.34it/s]
Loading 0: 100%|█████████▉| 505/507 [00:33<00:00, 35.46it/s]
                                                            
Job zonemercy-vingt-deux-v0-1e5-v12-mkmlizer completed after 129.02s with status: succeeded
Stopping job with name zonemercy-vingt-deux-v0-1e5-v12-mkmlizer
Pipeline stage MKMLizer completed in 129.69s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.22s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service zonemercy-vingt-deux-v0-1e5-v12
Waiting for inference service zonemercy-vingt-deux-v0-1e5-v12 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service zonemercy-vingt-deux-v0-1e5-v12 ready after 192.05274057388306s
Pipeline stage MKMLDeployer completed in 194.38s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.665044546127319s
Received healthy response to inference request in 4.785362005233765s
Received healthy response to inference request in 4.6944849491119385s
Received healthy response to inference request in 3.067976951599121s
Received healthy response to inference request in 3.8613460063934326s
5 requests
0 failed requests
5th percentile: 3.226650762557983
10th percentile: 3.385324573516846
20th percentile: 3.7026721954345705
30th percentile: 4.02208571434021
40th percentile: 4.343565130233765
50th percentile: 4.665044546127319
60th percentile: 4.676820707321167
70th percentile: 4.688596868515015
80th percentile: 4.7126603603363035
90th percentile: 4.7490111827850345
95th percentile: 4.7671865940094
99th percentile: 4.7817269229888915
mean time: 4.214842891693115
%s, retrying in %s seconds...
Received healthy response to inference request in 2.890087842941284s
Received healthy response to inference request in 3.160526752471924s
Received healthy response to inference request in 2.5684261322021484s
Received healthy response to inference request in 3.491121292114258s
Received healthy response to inference request in 3.6394827365875244s
5 requests
0 failed requests
5th percentile: 2.6327584743499757
10th percentile: 2.697090816497803
20th percentile: 2.825755500793457
30th percentile: 2.944175624847412
40th percentile: 3.052351188659668
50th percentile: 3.160526752471924
60th percentile: 3.2927645683288573
70th percentile: 3.425002384185791
80th percentile: 3.520793581008911
90th percentile: 3.580138158798218
95th percentile: 3.609810447692871
99th percentile: 3.6335482788085938
mean time: 3.1499289512634276
Pipeline stage StressChecker completed in 40.62s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 6.95s
Shutdown handler de-registered
zonemercy-vingt-deux-v0-1e5_v12 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.13s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service zonemercy-vingt-deux-v0-1e5-v12-profiler
Waiting for inference service zonemercy-vingt-deux-v0-1e5-v12-profiler to be ready
Inference service zonemercy-vingt-deux-v0-1e5-v12-profiler ready after 190.43655228614807s
Pipeline stage MKMLProfilerDeployer completed in 190.81s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/zonemercy-vingt-deux09e839cfba211b74805ec70097c7ce18-deplolm4l4:/code/chaiverse_profiler_1727084936 --namespace tenant-chaiml-guanaco
kubectl exec -it zonemercy-vingt-deux09e839cfba211b74805ec70097c7ce18-deplolm4l4 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1727084936 && python profiles.py profile --best_of_n 8 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 64 --summary /code/chaiverse_profiler_1727084936/summary.json'
kubectl exec -it zonemercy-vingt-deux09e839cfba211b74805ec70097c7ce18-deplolm4l4 --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1727084936/summary.json'
Pipeline stage MKMLProfilerRunner completed in 1556.46s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service zonemercy-vingt-deux-v0-1e5-v12-profiler is running
Tearing down inference service zonemercy-vingt-deux-v0-1e5-v12-profiler
Service zonemercy-vingt-deux-v0-1e5-v12-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.31s
Shutdown handler de-registered
zonemercy-vingt-deux-v0-1e5_v12 status is now inactive due to auto deactivation removed underperforming models
run pipeline %s
admin requested tearing down of zonemercy-vingt-deux-v0-1e5_v12
run pipeline stage %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Running pipeline stage MKMLDeleter
run pipeline %s
admin requested tearing down of zonemercy-vingt-deux-v0-1e5_v13
run pipeline stage %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Running pipeline stage MKMLDeleter
run pipeline %s
%s, retrying in %s seconds...
run pipeline stage %s
%s, retrying in %s seconds...
Running pipeline stage MKMLDeleter
clean up pipeline due to error=TeardownError("module 'kubernetes.config' has no attribute 'load_kube_config'")
%s, retrying in %s seconds...
Shutdown handler de-registered
%s, retrying in %s seconds...
zonemercy-vingt-deux-v0-1e5_v12 status is now torndown due to DeploymentManager action