developer_uid: sao10k
submission_id: sao10k-l3-1-70b-euryale-v2-2_v2
model_name: Euryale-2-2
model_group: Sao10K/L3.1-70B-Euryale-
status: torndown
timestamp: 2024-08-28T00:45:39+00:00
num_battles: 11310
num_wins: 5274
celo_rating: 1213.88
family_friendly_score: 0.0
submission_type: basic
model_repo: Sao10K/L3.1-70B-Euryale-v2.2
model_architecture: LlamaForCausalLM
model_num_parameters: 70553706496.0
best_of: 4
max_input_tokens: 512
max_output_tokens: 64
latencies: [{'batch_size': 1, 'throughput': 0.255097682797627, 'latency_mean': 3.9200111079216002, 'latency_p50': 3.932070732116699, 'latency_p90': 4.372938466072083}, {'batch_size': 2, 'throughput': 0.42418591134794753, 'latency_mean': 4.698371912240982, 'latency_p50': 4.690353870391846, 'latency_p90': 5.163253760337829}, {'batch_size': 3, 'throughput': 0.5794970069010901, 'latency_mean': 5.165480179786682, 'latency_p50': 5.18385922908783, 'latency_p90': 5.700179886817932}, {'batch_size': 5, 'throughput': 0.8131430483092956, 'latency_mean': 6.115543599128723, 'latency_p50': 6.117327928543091, 'latency_p90': 6.885808634757995}]
gpu_counts: {'NVIDIA A100-SXM4-80GB': 1}
display_name: Euryale-2-2
is_internal_developer: False
language_model: Sao10K/L3.1-70B-Euryale-v2.2
model_size: 71B
ranking_group: single
us_pacific_date: 2024-08-27
win_ratio: 0.4663129973474801
generation_params: {'temperature': 1.2, 'top_p': 1.0, 'min_p': 0.2, 'top_k': 50, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '\n\n', '<|eot_id|>', '\n\n{user_name}', '\nYou:', '<|end_header_id|>'], 'max_input_tokens': 512, 'best_of': 4, 'max_output_tokens': 64}
formatter: {'memory_template': "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{bot_name}'s Persona: {memory}\n\n", 'prompt_template': '{prompt}<|eot_id|>', 'bot_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}: {message}<|eot_id|>', 'user_template': '<|start_header_id|>user<|end_header_id|>\n\n{user_name}: {message}<|eot_id|>', 'response_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Running pipeline stage MKMLizer
Starting job with name sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer
Waiting for job on sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer to finish
Stopping job with name sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer
%s, retrying in %s seconds...
Starting job with name sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer
Waiting for job on sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer to finish
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║     _____            __           __                                ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║       /___/                                                         ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║                                                                     ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║  Version: 0.10.1                                                    ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║  https://mk1.ai                                                     ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║                                                                     ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║  belonging to:                                                      ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║                                                                     ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║  Chai Research Corp.                                                ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ║                                                                     ║
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: Downloaded to shared memory in 203.804s
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpmy2pga5l, device:0
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: quantized model in 134.111s
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: Processed model Sao10K/L3.1-70B-Euryale-v2.2 in 337.915s
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: creating bucket guanaco-mkml-models
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/sao10k-l3-1-70b-euryale-v2-2-v2
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/sao10k-l3-1-70b-euryale-v2-2-v2/config.json
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/sao10k-l3-1-70b-euryale-v2-2-v2/special_tokens_map.json
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/sao10k-l3-1-70b-euryale-v2-2-v2/tokenizer_config.json
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/sao10k-l3-1-70b-euryale-v2-2-v2/tokenizer.json
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.4.safetensors s3://guanaco-mkml-models/sao10k-l3-1-70b-euryale-v2-2-v2/flywheel_model.4.safetensors
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.3.safetensors s3://guanaco-mkml-models/sao10k-l3-1-70b-euryale-v2-2-v2/flywheel_model.3.safetensors
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/sao10k-l3-1-70b-euryale-v2-2-v2/flywheel_model.1.safetensors
sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer: 
Loading 0:   0%|          | 0/723 [00:00<?, ?it/s]
Loading 0:   1%|          | 4/723 [00:00<00:19, 37.06it/s]
Loading 0:   2%|▏         | 13/723 [00:00<00:11, 62.11it/s]
Loading 0:   3%|▎         | 20/723 [00:01<00:52, 13.43it/s]
Loading 0:   3%|▎         | 25/723 [00:01<00:40, 17.37it/s]
Loading 0:   5%|▍         | 34/723 [00:01<00:25, 27.24it/s]
Loading 0:   6%|▌         | 42/723 [00:02<00:47, 14.46it/s]
Loading 0:   7%|▋         | 49/723 [00:02<00:35, 18.93it/s]
Loading 0:   8%|▊         | 58/723 [00:02<00:25, 25.92it/s]
Loading 0:   9%|▉         | 67/723 [00:02<00:20, 32.06it/s]
Loading 0:  10%|█         | 73/723 [00:03<00:42, 15.42it/s]
Loading 0:  11%|█         | 79/723 [00:03<00:34, 18.92it/s]
Loading 0:  12%|█▏        | 88/723 [00:04<00:24, 26.13it/s]
Loading 0:  13%|█▎        | 94/723 [00:05<00:43, 14.37it/s]
Loading 0:  14%|█▍        | 103/723 [00:05<00:30, 20.31it/s]
Loading 0:  15%|█▌        | 112/723 [00:05<00:22, 26.98it/s]
Loading 0:  16%|█▋        | 119/723 [00:06<00:38, 15.79it/s]
Loading 0:  17%|█▋        | 124/723 [00:06<00:32, 18.15it/s]
Loading 0:  17%|█▋        | 124/723 [00:21<00:32, 18.15it/s]
Loading 0:  17%|█▋        | 125/723 [00:21<09:04,  1.10it/s]
Loading 0:  18%|█▊        | 133/723 [00:21<05:26,  1.81it/s]
Loading 0:  20%|█▉        | 142/723 [00:22<03:43,  2.60it/s]
Loading 0:  21%|██        | 151/723 [00:22<02:24,  3.97it/s]
Loading 0:  22%|██▏       | 160/723 [00:22<01:36,  5.86it/s]
Loading 0:  23%|██▎       | 168/723 [00:23<01:25,  6.49it/s]
Loading 0:  24%|██▍       | 175/723 [00:23<01:04,  8.55it/s]
Loading 0:  25%|██▌       | 184/723 [00:23<00:43, 12.25it/s]
Loading 0:  27%|██▋       | 193/723 [00:23<00:31, 16.93it/s]
Loading 0:  28%|██▊       | 200/723 [00:24<00:41, 12.48it/s]
Loading 0:  28%|██▊       | 205/723 [00:24<00:35, 14.77it/s]
Loading 0:  30%|██▉       | 215/723 [00:24<00:23, 21.76it/s]
Loading 0:  31%|███       | 222/723 [00:25<00:34, 14.45it/s]
Loading 0:  32%|███▏      | 232/723 [00:26<00:24, 20.12it/s]
Loading 0:  33%|███▎      | 241/723 [00:26<00:18, 26.34it/s]
Loading 0:  34%|███▍      | 248/723 [00:27<00:28, 16.57it/s]
Loading 0:  35%|███▌      | 256/723 [00:27<00:22, 21.14it/s]
Loading 0:  37%|███▋      | 265/723 [00:27<00:16, 28.09it/s]
Loading 0:  37%|███▋      | 265/723 [00:41<00:16, 28.09it/s]
Loading 0:  37%|███▋      | 266/723 [00:41<05:37,  1.35it/s]
Loading 0:  37%|███▋      | 268/723 [00:42<05:29,  1.38it/s]
Loading 0:  38%|███▊      | 277/723 [00:43<03:02,  2.44it/s]
Loading 0:  40%|███▉      | 286/723 [00:43<01:52,  3.89it/s]
Loading 0:  41%|████      | 294/723 [00:44<01:30,  4.73it/s]
Loading 0:  42%|████▏     | 301/723 [00:44<01:05,  6.48it/s]
Loading 0:  43%|████▎     | 310/723 [00:44<00:43,  9.57it/s]
Loading 0:  44%|████▍     | 319/723 [00:44<00:29, 13.59it/s]
Loading 0:  45%|████▌     | 326/723 [00:45<00:36, 10.96it/s]
Loading 0:  46%|████▌     | 332/723 [00:45<00:28, 13.70it/s]
Loading 0:  47%|████▋     | 340/723 [00:45<00:20, 18.56it/s]
Loading 0:  48%|████▊     | 346/723 [00:46<00:30, 12.39it/s]
Loading 0:  49%|████▉     | 357/723 [00:46<00:19, 19.24it/s]
Loading 0:  50%|█████     | 364/723 [00:46<00:15, 23.83it/s]
Loading 0:  51%|█████▏    | 371/723 [00:47<00:22, 15.38it/s]
Loading 0:  52%|█████▏    | 376/723 [00:47<00:19, 17.88it/s]
Loading 0:  53%|█████▎    | 385/723 [00:47<00:13, 25.08it/s]
Loading 0:  54%|█████▍    | 394/723 [00:48<00:20, 15.87it/s]
Loading 0:  56%|█████▌    | 402/723 [01:03<00:20, 15.87it/s]
Loading 0:  56%|█████▌    | 403/723 [01:03<03:00,  1.77it/s]
Loading 0:  57%|█████▋    | 412/723 [01:03<02:00,  2.58it/s]
Loading 0:  58%|█████▊    | 420/723 [01:04<01:34,  3.20it/s]
Loading 0:  59%|█████▉    | 427/723 [01:04<01:09,  4.27it/s]
Loading 0:  61%|██████    | 439/723 [01:04<00:41,  6.80it/s]
Loading 0:  62%|██████▏   | 446/723 [01:05<00:39,  6.98it/s]
Loading 0:  63%|██████▎   | 454/723 [01:05<00:28,  9.41it/s]
Loading 0:  64%|██████▍   | 463/723 [01:05<00:19, 13.08it/s]
Loading 0:  65%|██████▌   | 470/723 [01:06<00:22, 11.23it/s]
Loading 0:  66%|██████▌   | 475/723 [01:06<00:18, 13.23it/s]
Loading 0:  67%|██████▋   | 484/723 [01:06<00:12, 18.75it/s]
Loading 0:  68%|██████▊   | 493/723 [01:06<00:09, 24.76it/s]
Loading 0:  69%|██████▉   | 499/723 [01:07<00:14, 15.23it/s]
Loading 0:  70%|███████   | 508/723 [01:07<00:10, 20.87it/s]
Loading 0:  72%|███████▏  | 517/723 [01:08<00:07, 27.28it/s]
Loading 0:  72%|███████▏  | 523/723 [01:08<00:12, 15.91it/s]
Loading 0:  73%|███████▎  | 529/723 [01:09<00:10, 19.30it/s]
Loading 0:  74%|███████▍  | 537/723 [01:23<00:09, 19.30it/s]
Loading 0:  74%|███████▍  | 538/723 [01:23<01:49,  1.69it/s]
Loading 0:  76%|███████▌  | 546/723 [01:24<01:19,  2.22it/s]
Loading 0:  76%|███████▋  | 553/723 [01:24<00:56,  3.03it/s]
Loading 0:  78%|███████▊  | 562/723 [01:24<00:35,  4.48it/s]
Loading 0:  79%|███████▉  | 571/723 [01:24<00:23,  6.43it/s]
Loading 0:  80%|███████▉  | 577/723 [01:25<00:22,  6.44it/s]
Loading 0:  81%|████████  | 583/723 [01:25<00:16,  8.31it/s]
Loading 0:  82%|████████▏ | 592/723 [01:25<00:10, 12.20it/s]
Loading 0:  83%|████████▎ | 598/723 [01:26<00:12, 10.13it/s]
Loading 0:  84%|████████▍ | 607/723 [01:26<00:07, 14.66it/s]
Loading 0:  85%|████████▌ | 616/723 [01:27<00:05, 19.98it/s]
Loading 0:  86%|████████▌ | 623/723 [01:27<00:06, 14.45it/s]
Loading 0:  87%|████████▋ | 628/723 [01:27<00:05, 16.72it/s]
Loading 0:  88%|████████▊ | 637/723 [01:28<00:03, 23.50it/s]
Loading 0:  89%|████████▉ | 646/723 [01:29<00:04, 16.03it/s]
Loading 0:  91%|█████████ | 655/723 [01:29<00:03, 21.83it/s]
Loading 0:  92%|█████████▏| 664/723 [01:29<00:02, 28.02it/s]
Loading 0:  93%|█████████▎| 672/723 [01:30<00:03, 17.00it/s]
Loading 0:  93%|█████████▎| 672/723 [01:44<00:03, 17.00it/s]
Loading 0:  93%|█████████▎| 673/723 [01:44<00:37,  1.34it/s]
Loading 0:  94%|█████████▍| 682/723 [01:44<00:19,  2.15it/s]
Loading 0:  96%|█████████▌| 691/723 [01:44<00:09,  3.27it/s]
Loading 0:  97%|█████████▋| 698/723 [01:45<00:06,  3.87it/s]
Loading 0:  98%|█████████▊| 706/723 [01:45<00:03,  5.49it/s]
Loading 0:  99%|█████████▉| 715/723 [01:45<00:01,  8.00it/s]
Loading 0: 100%|█████████▉| 722/723 [01:56<00:00,  8.00it/s]
Loading 0: 100%|██████████| 723/723 [01:56<00:00,  2.10it/s]
                                                            

Job sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer completed after 386.67s with status: succeeded
Stopping job with name sao10k-l3-1-70b-euryale-v2-2-v2-mkmlizer
Pipeline stage MKMLizer completed in 388.04s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.08s
Running pipeline stage ISVCDeployer
Creating inference service sao10k-l3-1-70b-euryale-v2-2-v2
Waiting for inference service sao10k-l3-1-70b-euryale-v2-2-v2 to be ready
Failed to get response for submission blend_koran_2024-08-16: ('http://neversleep-noromaid-v0-8068-v149-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:41684->127.0.0.1:8080: read: connection reset by peer\n')
Inference service sao10k-l3-1-70b-euryale-v2-2-v2 ready after 201.2774202823639s
Pipeline stage ISVCDeployer completed in 201.87s
Running pipeline stage StressChecker
Received healthy response to inference request in 5.022865533828735s
Received healthy response to inference request in 4.231633901596069s
Received healthy response to inference request in 4.844794034957886s
Received healthy response to inference request in 4.169628381729126s
Received healthy response to inference request in 4.2215800285339355s
5 requests
0 failed requests
5th percentile: 4.180018711090088
10th percentile: 4.1904090404510494
20th percentile: 4.211189699172974
30th percentile: 4.2235908031463625
40th percentile: 4.2276123523712155
50th percentile: 4.231633901596069
60th percentile: 4.476897954940796
70th percentile: 4.7221620082855225
80th percentile: 4.8804083347320555
90th percentile: 4.951636934280396
95th percentile: 4.987251234054566
99th percentile: 5.015742673873901
mean time: 4.49810037612915
Connection pool is full, discarding connection: %s. Connection pool size: %s
Pipeline stage StressChecker completed in 23.33s
sao10k-l3-1-70b-euryale-v2-2_v2 status is now deployed due to DeploymentManager action
sao10k-l3-1-70b-euryale-v2-2_v2 status is now inactive due to auto deactivation removed underperforming models
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service sao10k-l3-1-70b-euryale-v2-2-v2-profiler
Waiting for inference service sao10k-l3-1-70b-euryale-v2-2-v2-profiler to be ready
Inference service sao10k-l3-1-70b-euryale-v2-2-v2-profiler ready after 270.6090099811554s
Pipeline stage MKMLProfilerDeployer completed in 270.99s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/sao10k-l3-1-70b-eury3451daba09184c205960c7a6d52499f3-deplofbgc6:/code/chaiverse_profiler_1725514944 --namespace tenant-chaiml-guanaco
kubectl exec -it sao10k-l3-1-70b-eury3451daba09184c205960c7a6d52499f3-deplofbgc6 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1725514944 && python profiles.py profile --best_of_n 4 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725514944/summary.json'
kubectl exec -it sao10k-l3-1-70b-eury3451daba09184c205960c7a6d52499f3-deplofbgc6 --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1725514944/summary.json'
Pipeline stage MKMLProfilerRunner completed in 1854.63s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service sao10k-l3-1-70b-euryale-v2-2-v2-profiler is running
Tearing down inference service sao10k-l3-1-70b-euryale-v2-2-v2-profiler
Service sao10k-l3-1-70b-euryale-v2-2-v2-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.74s
Shutdown handler de-registered
sao10k-l3-1-70b-euryale-v2-2_v2 status is now torndown due to DeploymentManager action