cycy233-l3-ba-se-v5-c1

developer_uid: shiroe40
submission_id: cycy233-l3-ba-se-v5-c1_v1
model_name: auto
model_group: cycy233/L3-ba-se-v5-c1
status: torndown
timestamp: 2024-08-22T02:01:53+00:00
num_battles: 10314
num_wins: 5354
celo_rating: 1251.61
family_friendly_score: 0.0
submission_type: basic
model_repo: cycy233/L3-ba-se-v5-c1
model_architecture: LlamaForCausalLM
model_num_parameters: 8030261248.0
best_of: 16
max_input_tokens: 512
max_output_tokens: 64
display_name: auto
is_internal_developer: False
language_model: cycy233/L3-ba-se-v5-c1
model_size: 8B
ranking_group: single
us_pacific_date: 2024-08-21
win_ratio: 0.5191002520845452
generation_params: {'temperature': 1.0, 'top_p': 0.9, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|end_header_id|>', '<|eot_id|>'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64}
formatter: {'memory_template': "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{bot_name}'s Persona: {memory}\n\n", 'prompt_template': '{prompt}<|eot_id|>', 'bot_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}: {message}<|eot_id|>', 'user_template': '<|start_header_id|>user<|end_header_id|>\n\n{user_name}: {message}<|eot_id|>', 'response_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Running pipeline stage MKMLizer
Starting job with name cycy233-l3-ba-se-v5-c1-v1-mkmlizer
Waiting for job on cycy233-l3-ba-se-v5-c1-v1-mkmlizer to finish
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║     _____            __           __                                ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║       /___/                                                         ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║                                                                     ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║  Version: 0.10.1                                                    ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║  https://mk1.ai                                                     ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║                                                                     ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║  belonging to:                                                      ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║                                                                     ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║  Chai Research Corp.                                                ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ║                                                                     ║
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Running pipeline stage MKMLizer
Starting job with name cycy233-l3-ba-se-v5-c2-v1-mkmlizer
Waiting for job on cycy233-l3-ba-se-v5-c2-v1-mkmlizer to finish
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║     _____            __           __                                ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║       /___/                                                         ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║                                                                     ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║  Version: 0.10.1                                                    ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║  https://mk1.ai                                                     ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║                                                                     ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║  belonging to:                                                      ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║                                                                     ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║  Chai Research Corp.                                                ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ║                                                                     ║
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: Downloaded to shared memory in 42.052s
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpw361eg1c, device:0
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission blend_sehof_2024-08-22: ('http://chaiml-elo-alignment-run-3-v34-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"ValueError : [TypeError(\\"\'numpy.int64\' object is not iterable\\"), TypeError(\'vars() argument must have __dict__ attribute\')]"}')
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: Downloaded to shared memory in 38.776s
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpzirhm4vb, device:0
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: quantized model in 26.958s
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: Processed model cycy233/L3-ba-se-v5-c1 in 69.011s
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: creating bucket guanaco-mkml-models
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/cycy233-l3-ba-se-v5-c1-v1
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/cycy233-l3-ba-se-v5-c1-v1/config.json
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/cycy233-l3-ba-se-v5-c1-v1/special_tokens_map.json
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/cycy233-l3-ba-se-v5-c1-v1/tokenizer_config.json
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/cycy233-l3-ba-se-v5-c1-v1/tokenizer.json
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/cycy233-l3-ba-se-v5-c1-v1/flywheel_model.0.safetensors
cycy233-l3-ba-se-v5-c1-v1-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/291 [00:00<00:08, 34.22it/s]
Loading 0:   4%|▍         | 13/291 [00:00<00:05, 53.43it/s]
Loading 0:   7%|▋         | 19/291 [00:00<00:05, 46.12it/s]
Loading 0:   8%|▊         | 24/291 [00:00<00:05, 44.67it/s]
Loading 0:  11%|█         | 31/291 [00:00<00:05, 50.35it/s]
Loading 0:  13%|█▎        | 37/291 [00:00<00:05, 45.34it/s]
Loading 0:  14%|█▍        | 42/291 [00:00<00:05, 44.23it/s]
Loading 0:  17%|█▋        | 49/291 [00:01<00:04, 49.61it/s]
Loading 0:  19%|█▉        | 55/291 [00:01<00:05, 44.94it/s]
Loading 0:  21%|██        | 60/291 [00:01<00:05, 43.96it/s]
Loading 0:  23%|██▎       | 67/291 [00:01<00:04, 48.45it/s]
Loading 0:  25%|██▌       | 73/291 [00:01<00:04, 44.12it/s]
Loading 0:  27%|██▋       | 78/291 [00:01<00:04, 43.71it/s]
Loading 0:  29%|██▊       | 83/291 [00:02<00:07, 29.71it/s]
Loading 0:  30%|██▉       | 87/291 [00:02<00:06, 30.28it/s]
Loading 0:  32%|███▏      | 93/291 [00:02<00:05, 35.91it/s]
Loading 0:  34%|███▎      | 98/291 [00:02<00:05, 38.50it/s]
Loading 0:  35%|███▌      | 103/291 [00:02<00:04, 40.92it/s]
Loading 0:  37%|███▋      | 108/291 [00:02<00:04, 42.92it/s]
Loading 0:  39%|███▉      | 113/291 [00:02<00:04, 36.95it/s]
Loading 0:  42%|████▏     | 121/291 [00:02<00:03, 45.78it/s]
Loading 0:  44%|████▎     | 127/291 [00:03<00:03, 42.99it/s]
Loading 0:  45%|████▌     | 132/291 [00:03<00:03, 42.96it/s]
Loading 0:  48%|████▊     | 139/291 [00:03<00:03, 47.91it/s]
Loading 0:  49%|████▉     | 144/291 [00:03<00:03, 48.00it/s]
Loading 0:  51%|█████     | 149/291 [00:03<00:03, 40.15it/s]
Loading 0:  54%|█████▍    | 157/291 [00:03<00:02, 47.83it/s]
Loading 0:  56%|█████▌    | 163/291 [00:03<00:02, 44.68it/s]
Loading 0:  58%|█████▊    | 168/291 [00:03<00:02, 44.19it/s]
Loading 0:  59%|█████▉    | 173/291 [00:04<00:02, 44.87it/s]
Loading 0:  62%|██████▏   | 180/291 [00:04<00:02, 49.66it/s]
Loading 0:  64%|██████▍   | 186/291 [00:04<00:02, 46.50it/s]
Loading 0:  66%|██████▌   | 191/291 [00:04<00:03, 31.80it/s]
Loading 0:  67%|██████▋   | 195/291 [00:04<00:02, 32.38it/s]
Loading 0:  69%|██████▉   | 202/291 [00:04<00:02, 38.98it/s]
Loading 0:  71%|███████▏  | 208/291 [00:04<00:02, 38.46it/s]
Loading 0:  73%|███████▎  | 213/291 [00:05<00:01, 39.12it/s]
Loading 0:  75%|███████▌  | 219/291 [00:05<00:01, 43.44it/s]
Loading 0:  77%|███████▋  | 224/291 [00:05<00:01, 43.65it/s]
Loading 0:  79%|███████▊  | 229/291 [00:05<00:01, 44.40it/s]
Loading 0:  81%|████████  | 235/291 [00:05<00:01, 42.64it/s]
Loading 0:  82%|████████▏ | 240/291 [00:05<00:01, 42.70it/s]
Loading 0:  85%|████████▍ | 247/291 [00:05<00:00, 47.70it/s]
Loading 0:  87%|████████▋ | 253/291 [00:05<00:00, 45.14it/s]
Loading 0:  89%|████████▊ | 258/291 [00:06<00:00, 44.92it/s]
Loading 0:  91%|█████████ | 264/291 [00:06<00:00, 48.72it/s]
Loading 0:  92%|█████████▏| 269/291 [00:06<00:00, 48.44it/s]
Loading 0:  94%|█████████▍| 274/291 [00:06<00:00, 48.66it/s]
Loading 0:  96%|█████████▌| 280/291 [00:06<00:00, 45.21it/s]
Loading 0:  98%|█████████▊| 285/291 [00:06<00:00, 44.19it/s]
Loading 0: 100%|█████████▉| 290/291 [00:12<00:00,  3.05it/s]
                                                            
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: quantized model in 26.246s
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: Processed model cycy233/L3-ba-se-v5-c2 in 65.023s
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: creating bucket guanaco-mkml-models
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/cycy233-l3-ba-se-v5-c2-v1
Job cycy233-l3-ba-se-v5-c1-v1-mkmlizer completed after 96.17s with status: succeeded
Stopping job with name cycy233-l3-ba-se-v5-c1-v1-mkmlizer
Pipeline stage MKMLizer completed in 97.16s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.10s
Running pipeline stage ISVCDeployer
Creating inference service cycy233-l3-ba-se-v5-c1-v1
Waiting for inference service cycy233-l3-ba-se-v5-c1-v1 to be ready
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/cycy233-l3-ba-se-v5-c2-v1/flywheel_model.0.safetensors
cycy233-l3-ba-se-v5-c2-v1-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/291 [00:00<00:08, 34.80it/s]
Loading 0:   4%|▍         | 13/291 [00:00<00:05, 53.67it/s]
Loading 0:   7%|▋         | 19/291 [00:00<00:05, 46.03it/s]
Loading 0:   8%|▊         | 24/291 [00:00<00:05, 45.49it/s]
Loading 0:  11%|█         | 31/291 [00:00<00:05, 51.35it/s]
Loading 0:  13%|█▎        | 37/291 [00:00<00:05, 45.79it/s]
Loading 0:  14%|█▍        | 42/291 [00:00<00:05, 44.95it/s]
Loading 0:  17%|█▋        | 49/291 [00:01<00:04, 49.95it/s]
Loading 0:  19%|█▉        | 55/291 [00:01<00:05, 46.18it/s]
Loading 0:  21%|██        | 60/291 [00:01<00:05, 46.14it/s]
Loading 0:  23%|██▎       | 67/291 [00:01<00:04, 51.11it/s]
Loading 0:  25%|██▌       | 73/291 [00:01<00:04, 48.18it/s]
Loading 0:  27%|██▋       | 78/291 [00:01<00:04, 47.49it/s]
Loading 0:  29%|██▊       | 83/291 [00:01<00:05, 34.75it/s]
Loading 0:  30%|██▉       | 87/291 [00:02<00:05, 35.07it/s]
Loading 0:  32%|███▏      | 94/291 [00:02<00:04, 42.64it/s]
Loading 0:  34%|███▍      | 100/291 [00:02<00:04, 42.43it/s]
Loading 0:  36%|███▌      | 105/291 [00:02<00:04, 43.09it/s]
Loading 0:  38%|███▊      | 112/291 [00:02<00:03, 49.23it/s]
Loading 0:  41%|████      | 118/291 [00:02<00:03, 45.43it/s]
Loading 0:  42%|████▏     | 123/291 [00:02<00:03, 43.88it/s]
Loading 0:  45%|████▍     | 130/291 [00:02<00:03, 49.31it/s]
Loading 0:  47%|████▋     | 136/291 [00:03<00:03, 46.29it/s]
Loading 0:  48%|████▊     | 141/291 [00:03<00:03, 45.54it/s]
Loading 0:  51%|█████     | 148/291 [00:03<00:02, 50.57it/s]
Loading 0:  53%|█████▎    | 154/291 [00:03<00:02, 46.13it/s]
Loading 0:  55%|█████▍    | 159/291 [00:03<00:02, 44.76it/s]
Loading 0:  57%|█████▋    | 165/291 [00:03<00:02, 47.01it/s]
Loading 0:  58%|█████▊    | 170/291 [00:03<00:02, 46.49it/s]
Loading 0:  60%|██████    | 176/291 [00:03<00:02, 49.60it/s]
Loading 0:  63%|██████▎   | 182/291 [00:04<00:02, 41.89it/s]
Loading 0:  64%|██████▍   | 187/291 [00:04<00:03, 32.39it/s]
Loading 0:  66%|██████▌   | 191/291 [00:04<00:02, 33.75it/s]
Loading 0:  67%|██████▋   | 195/291 [00:04<00:02, 33.60it/s]
Loading 0:  69%|██████▉   | 201/291 [00:04<00:02, 39.53it/s]
Loading 0:  71%|███████   | 206/291 [00:04<00:02, 41.22it/s]
Loading 0:  73%|███████▎  | 211/291 [00:04<00:01, 40.63it/s]
Loading 0:  75%|███████▍  | 217/291 [00:04<00:01, 40.17it/s]
Loading 0:  76%|███████▋  | 222/291 [00:05<00:01, 40.69it/s]
Loading 0:  79%|███████▊  | 229/291 [00:05<00:01, 46.12it/s]
Loading 0:  81%|████████  | 235/291 [00:05<00:01, 42.52it/s]
Loading 0:  82%|████████▏ | 240/291 [00:05<00:01, 41.05it/s]
Loading 0:  85%|████████▍ | 246/291 [00:05<00:01, 44.81it/s]
Loading 0:  86%|████████▋ | 251/291 [00:05<00:00, 45.72it/s]
Loading 0:  88%|████████▊ | 256/291 [00:05<00:00, 46.58it/s]
Loading 0:  90%|█████████ | 262/291 [00:05<00:00, 44.21it/s]
Loading 0:  92%|█████████▏| 267/291 [00:06<00:00, 44.24it/s]
Loading 0:  94%|█████████▍| 274/291 [00:06<00:00, 49.75it/s]
Loading 0:  96%|█████████▌| 280/291 [00:06<00:00, 46.34it/s]
Loading 0:  98%|█████████▊| 285/291 [00:06<00:00, 45.96it/s]
Loading 0: 100%|█████████▉| 290/291 [00:11<00:00,  3.22it/s]
                                                            
Job cycy233-l3-ba-se-v5-c2-v1-mkmlizer completed after 88.11s with status: succeeded
Stopping job with name cycy233-l3-ba-se-v5-c2-v1-mkmlizer
Pipeline stage MKMLizer completed in 88.80s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.11s
Running pipeline stage ISVCDeployer
Creating inference service cycy233-l3-ba-se-v5-c2-v1
Waiting for inference service cycy233-l3-ba-se-v5-c2-v1 to be ready
Inference service cycy233-l3-ba-se-v5-c2-v1 ready after 252.87273955345154s
Pipeline stage ISVCDeployer completed in 253.70s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.3621363639831543s
Inference service cycy233-l3-ba-se-v5-c1-v1 ready after 262.87149143218994s
Pipeline stage ISVCDeployer completed in 264.49s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.484961748123169s
Received healthy response to inference request in 2.085844039916992s
Received healthy response to inference request in 1.847987413406372s
Received healthy response to inference request in 1.8501067161560059s
Received healthy response to inference request in 1.9990546703338623s
Received healthy response to inference request in 1.8650550842285156s
Received healthy response to inference request in 1.9121332168579102s
5 requests
0 failed requests
5th percentile: 1.8608165740966798
10th percentile: 1.8736457347869873
20th percentile: 1.8993040561676025
30th percentile: 1.9295175075531006
40th percentile: 1.9642860889434814
50th percentile: 1.9990546703338623
60th percentile: 2.144287347793579
70th percentile: 2.289520025253296
80th percentile: 2.3867014408111573
90th percentile: 2.435831594467163
95th percentile: 2.460396671295166
99th percentile: 2.4800487327575684
mean time: 2.1212546825408936
Pipeline stage StressChecker completed in 12.31s
Received healthy response to inference request in 1.8858907222747803s
cycy233-l3-ba-se-v5-c2_v1 status is now deployed due to DeploymentManager action
Received healthy response to inference request in 2.1375443935394287s
5 requests
0 failed requests
5th percentile: 1.8530963897705077
10th percentile: 1.8560860633850098
20th percentile: 1.8620654106140138
30th percentile: 1.8692222118377686
40th percentile: 1.8775564670562743
50th percentile: 1.8858907222747803
60th percentile: 1.965872049331665
70th percentile: 2.04585337638855
80th percentile: 2.0961841106414796
90th percentile: 2.116864252090454
95th percentile: 2.1272043228149413
99th percentile: 2.135476379394531
mean time: 1.9648881912231446
Pipeline stage StressChecker completed in 11.09s
cycy233-l3-ba-se-v5-c1_v1 status is now deployed due to DeploymentManager action
cycy233-l3-ba-se-v5-c1_v1 status is now inactive due to auto deactivation removed underperforming models
cycy233-l3-ba-se-v5-c1_v1 status is now torndown due to DeploymentManager action