mistralai-mixtral-8x7b_3473

developer_uid: zonemercy
submission_id: mistralai-mixtral-8x7b_3473_v139
model_name: mistralai-mixtral-8x7b_3473_v139
model_group: mistralai/Mixtral-8x7B-I
status: torndown
timestamp: 2024-09-16T14:42:07+00:00
num_battles: 11620
num_wins: 4621
celo_rating: 1181.51
family_friendly_score: 0.0
submission_type: basic
model_repo: mistralai/Mixtral-8x7B-Instruct-v0.1
model_architecture: MixtralForCausalLM
model_num_parameters: 46702792704.0
best_of: 2
max_input_tokens: 512
max_output_tokens: 64
latencies: [{'batch_size': 1, 'throughput': 0.5503516168826115, 'latency_mean': 1.816965115070343, 'latency_p50': 1.8129088878631592, 'latency_p90': 2.055765151977539}, {'batch_size': 5, 'throughput': 1.5315271657516762, 'latency_mean': 3.2479244756698606, 'latency_p50': 3.2482649087905884, 'latency_p90': 3.635074281692505}, {'batch_size': 10, 'throughput': 2.5868020575809636, 'latency_mean': 3.8058273649215697, 'latency_p50': 3.7814775705337524, 'latency_p90': 4.31525228023529}, {'batch_size': 15, 'throughput': 3.373140245371733, 'latency_mean': 4.3250349688529965, 'latency_p50': 4.329188346862793, 'latency_p90': 4.944095921516419}, {'batch_size': 20, 'throughput': 4.028155650770792, 'latency_mean': 4.841404349803924, 'latency_p50': 4.8882046937942505, 'latency_p90': 5.572861123085022}, {'batch_size': 25, 'throughput': 4.561470216903015, 'latency_mean': 5.283430631160736, 'latency_p50': 5.2662113904953, 'latency_p90': 6.153449273109436}]
gpu_counts: {'NVIDIA A100-SXM4-80GB': 1}
display_name: mistralai-mixtral-8x7b_3473_v139
is_internal_developer: True
language_model: mistralai/Mixtral-8x7B-Instruct-v0.1
model_size: 47B
ranking_group: single
throughput_3p7s: 2.45
us_pacific_date: 2024-09-16
win_ratio: 0.39767641996557657
generation_params: {'temperature': 1.1235, 'top_p': 1.0, 'min_p': 0.1, 'top_k': 100, 'presence_penalty': 0.5, 'frequency_penalty': 0.5, 'stopping_words': ['\n', '</s>', '<|user|>', '###'], 'max_input_tokens': 512, 'best_of': 2, 'max_output_tokens': 64}
formatter: {'memory_template': '{memory}\n####\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}</s>', 'user_template': '[INST] {user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Failed to get response for submission blend_hokok_2024-09-09: ('http://neversleep-noromaid-v0-8068-v150-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Starting job with name mistralai-mixtral-8x7b-3473-v139-mkmlizer
Waiting for job on mistralai-mixtral-8x7b-3473-v139-mkmlizer to finish
Connection pool is full, discarding connection: %s. Connection pool size: %s
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║     _____            __           __                                ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║       /___/                                                         ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║                                                                     ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║  Version: 0.10.1                                                    ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║  https://mk1.ai                                                     ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║                                                                     ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║  The license key for the current software has been verified as      ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║  belonging to:                                                      ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║                                                                     ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║  Chai Research Corp.                                                ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ║                                                                     ║
mistralai-mixtral-8x7b-3473-v139-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission blend_garut_2024-09-14: ('http://zonemercy-virgo-edit-v1-1e5-v3-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"ValueError : [TypeError(\\"\'numpy.int64\' object is not iterable\\"), TypeError(\'vars() argument must have __dict__ attribute\')]"}')
Failed to get response for submission blend_hokok_2024-09-09: ('http://neversleep-noromaid-v0-8068-v150-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
mistralai-mixtral-8x7b-3473-v139-mkmlizer: Downloaded to shared memory in 183.226s
mistralai-mixtral-8x7b-3473-v139-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpd6an_qn3, device:0
mistralai-mixtral-8x7b-3473-v139-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
mistralai-mixtral-8x7b-3473-v139-mkmlizer: quantized model in 91.274s
mistralai-mixtral-8x7b-3473-v139-mkmlizer: Processed model mistralai/Mixtral-8x7B-Instruct-v0.1 in 274.501s
mistralai-mixtral-8x7b-3473-v139-mkmlizer: creating bucket guanaco-mkml-models
mistralai-mixtral-8x7b-3473-v139-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
Connection pool is full, discarding connection: %s. Connection pool size: %s
mistralai-mixtral-8x7b-3473-v139-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/mistralai-mixtral-8x7b-3473-v139
mistralai-mixtral-8x7b-3473-v139-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/mistralai-mixtral-8x7b-3473-v139/config.json
mistralai-mixtral-8x7b-3473-v139-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/mistralai-mixtral-8x7b-3473-v139/tokenizer_config.json
mistralai-mixtral-8x7b-3473-v139-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/mistralai-mixtral-8x7b-3473-v139/special_tokens_map.json
mistralai-mixtral-8x7b-3473-v139-mkmlizer: cp /dev/shm/model_cache/tokenizer.model s3://guanaco-mkml-models/mistralai-mixtral-8x7b-3473-v139/tokenizer.model
mistralai-mixtral-8x7b-3473-v139-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/mistralai-mixtral-8x7b-3473-v139/tokenizer.json
mistralai-mixtral-8x7b-3473-v139-mkmlizer: cp /dev/shm/model_cache/flywheel_model.3.safetensors s3://guanaco-mkml-models/mistralai-mixtral-8x7b-3473-v139/flywheel_model.3.safetensors
mistralai-mixtral-8x7b-3473-v139-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/mistralai-mixtral-8x7b-3473-v139/flywheel_model.0.safetensors
mistralai-mixtral-8x7b-3473-v139-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/mistralai-mixtral-8x7b-3473-v139/flywheel_model.1.safetensors
mistralai-mixtral-8x7b-3473-v139-mkmlizer: cp /dev/shm/model_cache/flywheel_model.2.safetensors s3://guanaco-mkml-models/mistralai-mixtral-8x7b-3473-v139/flywheel_model.2.safetensors
mistralai-mixtral-8x7b-3473-v139-mkmlizer: 
Loading 0:   0%|          | 0/995 [00:00<?, ?it/s]
Loading 0:   0%|          | 4/995 [00:00<00:28, 34.37it/s]
Loading 0:   1%|          | 8/995 [00:00<00:32, 30.76it/s]
Loading 0:   1%|          | 12/995 [00:00<00:31, 30.87it/s]
Loading 0:   2%|▏         | 16/995 [00:00<00:37, 26.19it/s]
Loading 0:   2%|▏         | 19/995 [00:00<00:37, 26.15it/s]
Loading 0:   2%|▏         | 22/995 [00:00<00:37, 25.74it/s]
Loading 0:   3%|▎         | 25/995 [00:00<00:38, 24.95it/s]
Loading 0:   3%|▎         | 33/995 [00:01<00:24, 39.15it/s]
Loading 0:   4%|▍         | 38/995 [00:01<00:29, 32.32it/s]
Loading 0:   4%|▍         | 42/995 [00:01<00:30, 31.15it/s]
Loading 0:   5%|▍         | 46/995 [00:01<00:30, 30.94it/s]
Loading 0:   5%|▌         | 52/995 [00:01<00:38, 24.27it/s]
Loading 0:   6%|▌         | 55/995 [00:01<00:39, 23.94it/s]
Loading 0:   6%|▌         | 58/995 [00:02<00:38, 24.43it/s]
Loading 0:   6%|▌         | 61/995 [00:02<00:37, 24.95it/s]
Loading 0:   7%|▋         | 66/995 [00:02<00:31, 29.22it/s]
Loading 0:   7%|▋         | 70/995 [00:02<00:31, 29.02it/s]
Loading 0:   7%|▋         | 74/995 [00:02<00:31, 29.50it/s]
Loading 0:   8%|▊         | 78/995 [00:02<00:35, 26.06it/s]
Loading 0:   8%|▊         | 81/995 [00:02<00:35, 25.94it/s]
Loading 0:   8%|▊         | 84/995 [00:03<00:37, 24.62it/s]
Loading 0:   9%|▊         | 87/995 [00:03<00:36, 24.97it/s]
Loading 0:  10%|▉         | 95/995 [00:03<00:23, 38.12it/s]
Loading 0:  10%|█         | 100/995 [00:03<00:28, 31.80it/s]
Loading 0:  11%|█         | 107/995 [00:03<00:31, 28.54it/s]
Loading 0:  11%|█         | 111/995 [00:03<00:33, 26.45it/s]
Loading 0:  11%|█▏        | 114/995 [00:04<00:33, 26.41it/s]
Loading 0:  12%|█▏        | 117/995 [00:04<00:33, 26.19it/s]
Loading 0:  12%|█▏        | 120/995 [00:04<00:33, 26.20it/s]
Loading 0:  12%|█▏        | 123/995 [00:04<00:33, 26.29it/s]
Loading 0:  13%|█▎        | 128/995 [00:04<00:28, 30.83it/s]
Loading 0:  13%|█▎        | 132/995 [00:04<00:28, 30.06it/s]
Loading 0:  14%|█▎        | 136/995 [00:04<00:28, 30.06it/s]
Loading 0:  14%|█▍        | 140/995 [00:05<00:33, 25.86it/s]
Loading 0:  14%|█▍        | 143/995 [00:05<00:33, 25.61it/s]
Loading 0:  15%|█▍        | 146/995 [00:05<00:33, 25.40it/s]
Loading 0:  15%|█▍        | 149/995 [00:05<00:33, 25.55it/s]
Loading 0:  16%|█▌        | 160/995 [00:05<00:18, 46.27it/s]
Loading 0:  17%|█▋        | 166/995 [00:05<00:29, 28.35it/s]
Loading 0:  17%|█▋        | 171/995 [00:06<00:31, 26.39it/s]
Loading 0:  18%|█▊        | 175/995 [00:06<00:30, 27.31it/s]
Loading 0:  18%|█▊        | 179/995 [00:06<00:32, 25.27it/s]
Loading 0:  18%|█▊        | 182/995 [00:06<00:31, 25.49it/s]
Loading 0:  19%|█▊        | 185/995 [00:06<00:31, 25.73it/s]
Loading 0:  19%|█▉        | 190/995 [00:06<00:26, 29.94it/s]
Loading 0:  19%|█▉        | 194/995 [00:06<00:27, 29.63it/s]
Loading 0:  20%|█▉        | 198/995 [00:07<00:26, 29.64it/s]
Loading 0:  20%|██        | 202/995 [00:07<00:30, 25.61it/s]
Loading 0:  21%|██        | 209/995 [00:07<00:23, 33.31it/s]
Loading 0:  21%|██▏       | 213/995 [00:07<00:36, 21.41it/s]
Loading 0:  22%|██▏       | 216/995 [00:07<00:35, 22.08it/s]
Loading 0:  22%|██▏       | 221/995 [00:07<00:29, 26.31it/s]
Loading 0:  23%|██▎       | 225/995 [00:08<00:28, 26.75it/s]
Loading 0:  23%|██▎       | 229/995 [00:08<00:27, 27.94it/s]
Loading 0:  23%|██▎       | 233/995 [00:08<00:30, 24.79it/s]
Loading 0:  24%|██▎       | 236/995 [00:08<00:30, 24.70it/s]
Loading 0:  24%|██▍       | 239/995 [00:08<00:30, 24.91it/s]
Loading 0:  24%|██▍       | 242/995 [00:08<00:29, 25.20it/s]
Loading 0:  25%|██▌       | 250/995 [00:08<00:19, 38.40it/s]
Loading 0:  26%|██▌       | 255/995 [00:09<00:23, 31.94it/s]
Loading 0:  26%|██▌       | 259/995 [00:09<00:24, 30.54it/s]
Loading 0:  27%|██▋       | 265/995 [00:09<00:26, 27.24it/s]
Loading 0:  27%|██▋       | 270/995 [00:09<00:23, 31.23it/s]
Loading 0:  28%|██▊       | 274/995 [00:09<00:24, 29.17it/s]
Loading 0:  28%|██▊       | 278/995 [00:09<00:24, 29.20it/s]
Loading 0:  28%|██▊       | 282/995 [00:10<00:24, 29.26it/s]
Loading 0:  29%|██▊       | 286/995 [00:24<12:44,  1.08s/it]
Loading 0:  29%|██▉       | 291/995 [00:25<08:30,  1.38it/s]
Loading 0:  30%|██▉       | 295/995 [00:25<06:13,  1.87it/s]
Loading 0:  30%|███       | 299/995 [00:25<04:32,  2.55it/s]
Loading 0:  30%|███       | 303/995 [00:25<03:22,  3.41it/s]
Loading 0:  31%|███       | 306/995 [00:25<02:40,  4.30it/s]
Loading 0:  31%|███       | 309/995 [00:25<02:06,  5.42it/s]
Loading 0:  31%|███▏      | 312/995 [00:25<01:39,  6.85it/s]
Loading 0:  32%|███▏      | 320/995 [00:26<01:00, 11.07it/s]
Loading 0:  32%|███▏      | 323/995 [00:26<00:53, 12.51it/s]
Loading 0:  33%|███▎      | 326/995 [00:26<00:47, 14.21it/s]
Loading 0:  33%|███▎      | 329/995 [00:26<00:41, 16.10it/s]
Loading 0:  33%|███▎      | 332/995 [00:26<00:37, 17.86it/s]
Loading 0:  34%|███▎      | 335/995 [00:26<00:34, 19.27it/s]
Loading 0:  34%|███▍      | 338/995 [00:26<00:31, 20.62it/s]
Loading 0:  34%|███▍      | 343/995 [00:27<00:24, 26.11it/s]
Loading 0:  35%|███▍      | 347/995 [00:27<00:23, 27.37it/s]
Loading 0:  35%|███▌      | 351/995 [00:27<00:25, 24.81it/s]
Loading 0:  36%|███▌      | 354/995 [00:27<00:25, 25.24it/s]
Loading 0:  36%|███▌      | 357/995 [00:27<00:25, 25.35it/s]
Loading 0:  36%|███▌      | 360/995 [00:27<00:25, 24.58it/s]
Loading 0:  37%|███▋      | 367/995 [00:27<00:18, 33.61it/s]
Loading 0:  37%|███▋      | 371/995 [00:28<00:29, 21.25it/s]
Loading 0:  38%|███▊      | 376/995 [00:28<00:24, 25.35it/s]
Loading 0:  38%|███▊      | 380/995 [00:28<00:23, 26.13it/s]
Loading 0:  39%|███▊      | 384/995 [00:28<00:22, 27.12it/s]
Loading 0:  39%|███▉      | 388/995 [00:28<00:24, 25.20it/s]
Loading 0:  39%|███▉      | 391/995 [00:28<00:24, 25.12it/s]
Loading 0:  40%|███▉      | 394/995 [00:29<00:23, 25.38it/s]
Loading 0:  40%|███▉      | 397/995 [00:29<00:23, 25.69it/s]
Loading 0:  41%|████      | 406/995 [00:29<00:14, 40.61it/s]
Loading 0:  41%|████▏     | 411/995 [00:29<00:17, 33.35it/s]
Loading 0:  42%|████▏     | 415/995 [00:29<00:17, 32.38it/s]
Loading 0:  42%|████▏     | 420/995 [00:29<00:15, 36.00it/s]
Loading 0:  43%|████▎     | 424/995 [00:30<00:23, 23.80it/s]
Loading 0:  43%|████▎     | 428/995 [00:30<00:22, 24.90it/s]
Loading 0:  43%|████▎     | 432/995 [00:30<00:21, 26.39it/s]
Loading 0:  44%|████▍     | 437/995 [00:30<00:18, 30.64it/s]
Loading 0:  44%|████▍     | 441/995 [00:30<00:20, 27.65it/s]
Loading 0:  45%|████▍     | 445/995 [00:30<00:19, 28.17it/s]
Loading 0:  45%|████▌     | 449/995 [00:30<00:18, 29.09it/s]
Loading 0:  46%|████▌     | 453/995 [00:31<00:20, 26.17it/s]
Loading 0:  46%|████▌     | 456/995 [00:31<00:20, 25.96it/s]
Loading 0:  46%|████▌     | 459/995 [00:31<00:20, 26.01it/s]
Loading 0:  47%|████▋     | 467/995 [00:31<00:13, 37.94it/s]
Loading 0:  47%|████▋     | 472/995 [00:31<00:16, 30.85it/s]
Loading 0:  48%|████▊     | 478/995 [00:31<00:18, 27.70it/s]
Loading 0:  48%|████▊     | 482/995 [00:32<00:18, 28.05it/s]
Loading 0:  49%|████▉     | 486/995 [00:32<00:19, 25.60it/s]
Loading 0:  49%|████▉     | 489/995 [00:32<00:19, 25.44it/s]
Loading 0:  49%|████▉     | 492/995 [00:32<00:19, 25.64it/s]
Loading 0:  50%|████▉     | 495/995 [00:32<00:19, 25.69it/s]
Loading 0:  50%|█████     | 500/995 [00:32<00:16, 30.59it/s]
Loading 0:  51%|█████     | 504/995 [00:32<00:16, 30.21it/s]
Loading 0:  51%|█████     | 508/995 [00:32<00:16, 30.11it/s]
Loading 0:  51%|█████▏    | 512/995 [00:33<00:18, 26.05it/s]
Loading 0:  52%|█████▏    | 515/995 [00:33<00:18, 25.33it/s]
Loading 0:  52%|█████▏    | 518/995 [00:33<00:18, 25.17it/s]
Loading 0:  53%|█████▎    | 525/995 [00:33<00:13, 34.42it/s]
Loading 0:  53%|█████▎    | 529/995 [00:33<00:19, 24.50it/s]
Loading 0:  53%|█████▎    | 532/995 [00:33<00:18, 24.43it/s]
Loading 0:  54%|█████▍    | 535/995 [00:34<00:18, 24.44it/s]
Loading 0:  54%|█████▍    | 538/995 [00:34<00:18, 24.54it/s]
Loading 0:  54%|█████▍    | 541/995 [00:34<00:18, 24.85it/s]
Loading 0:  55%|█████▍    | 544/995 [00:34<00:17, 25.10it/s]
Loading 0:  55%|█████▍    | 547/995 [00:34<00:17, 25.25it/s]
Loading 0:  55%|█████▌    | 550/995 [00:34<00:18, 24.54it/s]
Loading 0:  56%|█████▌    | 556/995 [00:34<00:13, 33.32it/s]
Loading 0:  56%|█████▋    | 561/995 [00:34<00:11, 37.07it/s]
Loading 0:  56%|█████▋    | 561/995 [00:49<00:11, 37.07it/s]
Loading 0:  56%|█████▋    | 562/995 [00:49<09:51,  1.37s/it]
Loading 0:  57%|█████▋    | 564/995 [00:49<07:53,  1.10s/it]
Loading 0:  57%|█████▋    | 568/995 [00:50<05:06,  1.39it/s]
Loading 0:  57%|█████▋    | 571/995 [00:50<03:42,  1.91it/s]
Loading 0:  58%|█████▊    | 574/995 [00:50<02:41,  2.60it/s]
Loading 0:  58%|█████▊    | 580/995 [00:50<01:31,  4.56it/s]
Loading 0:  59%|█████▊    | 584/995 [00:50<01:12,  5.64it/s]
Loading 0:  59%|█████▉    | 587/995 [00:50<00:58,  6.97it/s]
Loading 0:  59%|█████▉    | 591/995 [00:50<00:42,  9.42it/s]
Loading 0:  60%|█████▉    | 595/995 [00:51<00:33, 11.98it/s]
Loading 0:  60%|██████    | 599/995 [00:51<00:28, 13.81it/s]
Loading 0:  61%|██████    | 602/995 [00:51<00:25, 15.53it/s]
Loading 0:  61%|██████    | 605/995 [00:51<00:22, 17.42it/s]
Loading 0:  61%|██████    | 608/995 [00:51<00:20, 18.68it/s]
Loading 0:  61%|██████▏   | 611/995 [00:51<00:19, 20.13it/s]
Loading 0:  62%|██████▏   | 614/995 [00:51<00:18, 20.63it/s]
Loading 0:  63%|██████▎   | 622/995 [00:51<00:11, 33.32it/s]
Loading 0:  63%|██████▎   | 627/995 [00:52<00:12, 28.46it/s]
Loading 0:  63%|██████▎   | 631/995 [00:52<00:12, 29.05it/s]
Loading 0:  64%|██████▍   | 636/995 [00:52<00:14, 25.10it/s]
Loading 0:  64%|██████▍   | 639/995 [00:52<00:14, 25.22it/s]
Loading 0:  65%|██████▍   | 642/995 [00:52<00:13, 25.53it/s]
Loading 0:  65%|██████▍   | 645/995 [00:52<00:13, 25.62it/s]
Loading 0:  65%|██████▌   | 648/995 [00:53<00:13, 25.57it/s]
Loading 0:  66%|██████▌   | 653/995 [00:53<00:11, 30.48it/s]
Loading 0:  66%|██████▌   | 657/995 [00:53<00:11, 29.94it/s]
Loading 0:  66%|██████▋   | 661/995 [00:53<00:12, 26.03it/s]
Loading 0:  67%|██████▋   | 664/995 [00:53<00:12, 25.83it/s]
Loading 0:  67%|██████▋   | 667/995 [00:53<00:12, 25.87it/s]
Loading 0:  67%|██████▋   | 670/995 [00:53<00:12, 26.04it/s]
Loading 0:  68%|██████▊   | 673/995 [00:53<00:12, 26.06it/s]
Loading 0:  68%|██████▊   | 676/995 [00:54<00:12, 25.55it/s]
Loading 0:  69%|██████▊   | 684/995 [00:54<00:07, 39.25it/s]
Loading 0:  69%|██████▉   | 691/995 [00:54<00:10, 29.11it/s]
Loading 0:  70%|██████▉   | 695/995 [00:54<00:10, 29.13it/s]
Loading 0:  70%|███████   | 699/995 [00:54<00:10, 29.46it/s]
Loading 0:  71%|███████   | 703/995 [00:54<00:10, 26.83it/s]
Loading 0:  71%|███████   | 706/995 [00:55<00:11, 26.11it/s]
Loading 0:  71%|███████▏  | 709/995 [00:55<00:11, 25.96it/s]
Loading 0:  72%|███████▏  | 712/995 [00:55<00:10, 26.15it/s]
Loading 0:  72%|███████▏  | 717/995 [00:55<00:08, 31.02it/s]
Loading 0:  72%|███████▏  | 721/995 [00:55<00:09, 29.95it/s]
Loading 0:  73%|███████▎  | 725/995 [00:55<00:09, 29.88it/s]
Loading 0:  73%|███████▎  | 729/995 [00:55<00:10, 25.79it/s]
Loading 0:  74%|███████▎  | 732/995 [00:56<00:10, 25.88it/s]
Loading 0:  74%|███████▍  | 739/995 [00:56<00:09, 25.76it/s]
Loading 0:  75%|███████▍  | 742/995 [00:56<00:09, 25.38it/s]
Loading 0:  75%|███████▍  | 746/995 [00:56<00:08, 28.29it/s]
Loading 0:  75%|███████▌  | 749/995 [00:56<00:08, 27.98it/s]
Loading 0:  76%|███████▌  | 752/995 [00:56<00:08, 27.51it/s]
Loading 0:  76%|███████▌  | 755/995 [00:56<00:08, 27.26it/s]
Loading 0:  76%|███████▌  | 758/995 [00:57<00:08, 26.80it/s]
Loading 0:  76%|███████▋  | 761/995 [00:57<00:08, 26.53it/s]
Loading 0:  77%|███████▋  | 764/995 [00:57<00:08, 25.81it/s]
Loading 0:  77%|███████▋  | 767/995 [00:57<00:08, 25.74it/s]
Loading 0:  78%|███████▊  | 774/995 [00:57<00:05, 37.06it/s]
Loading 0:  78%|███████▊  | 779/995 [00:57<00:06, 33.59it/s]
Loading 0:  79%|███████▊  | 783/995 [00:57<00:06, 31.84it/s]
Loading 0:  79%|███████▉  | 787/995 [00:57<00:06, 30.58it/s]
Loading 0:  80%|███████▉  | 793/995 [00:58<00:05, 33.67it/s]
Loading 0:  80%|████████  | 797/995 [00:58<00:08, 23.60it/s]
Loading 0:  80%|████████  | 800/995 [00:58<00:08, 24.18it/s]
Loading 0:  81%|████████  | 803/995 [00:58<00:07, 24.50it/s]
Loading 0:  81%|████████  | 806/995 [00:58<00:07, 25.44it/s]
Loading 0:  81%|████████▏ | 810/995 [00:58<00:06, 27.94it/s]
Loading 0:  82%|████████▏ | 813/995 [00:58<00:06, 27.57it/s]
Loading 0:  82%|████████▏ | 816/995 [00:59<00:06, 27.29it/s]
Loading 0:  82%|████████▏ | 819/995 [00:59<00:06, 26.63it/s]
Loading 0:  83%|████████▎ | 822/995 [00:59<00:06, 25.68it/s]
Loading 0:  83%|████████▎ | 825/995 [00:59<00:06, 25.01it/s]
Loading 0:  83%|████████▎ | 828/995 [00:59<00:06, 25.24it/s]
Loading 0:  84%|████████▎ | 831/995 [00:59<00:06, 25.35it/s]
Loading 0:  84%|████████▍ | 840/995 [00:59<00:03, 41.24it/s]
Loading 0:  85%|████████▍ | 842/995 [01:10<00:03, 41.24it/s]
Loading 0:  85%|████████▍ | 843/995 [01:14<02:40,  1.05s/it]
Loading 0:  85%|████████▌ | 848/995 [01:14<01:44,  1.41it/s]
Loading 0:  86%|████████▌ | 853/995 [01:15<01:12,  1.96it/s]
Loading 0:  86%|████████▌ | 857/995 [01:15<00:52,  2.62it/s]
Loading 0:  87%|████████▋ | 861/995 [01:15<00:38,  3.45it/s]
Loading 0:  87%|████████▋ | 864/995 [01:15<00:30,  4.31it/s]
Loading 0:  87%|████████▋ | 867/995 [01:15<00:23,  5.37it/s]
Loading 0:  88%|████████▊ | 872/995 [01:15<00:15,  7.96it/s]
Loading 0:  88%|████████▊ | 876/995 [01:15<00:11, 10.18it/s]
Loading 0:  88%|████████▊ | 880/995 [01:16<00:09, 12.52it/s]
Loading 0:  89%|████████▊ | 883/995 [01:16<00:07, 14.27it/s]
Loading 0:  89%|████████▉ | 886/995 [01:16<00:06, 15.88it/s]
Loading 0:  89%|████████▉ | 889/995 [01:16<00:05, 17.70it/s]
Loading 0:  90%|████████▉ | 892/995 [01:16<00:05, 19.70it/s]
Loading 0:  90%|█████████ | 897/995 [01:16<00:04, 19.85it/s]
Loading 0:  91%|█████████ | 902/995 [01:16<00:03, 24.68it/s]
Loading 0:  91%|█████████ | 906/995 [01:17<00:03, 22.72it/s]
Loading 0:  91%|█████████▏| 909/995 [01:17<00:03, 23.48it/s]
Loading 0:  92%|█████████▏| 912/995 [01:17<00:03, 23.84it/s]
Loading 0:  92%|█████████▏| 915/995 [01:17<00:03, 24.27it/s]
Loading 0:  92%|█████████▏| 918/995 [01:17<00:03, 24.53it/s]
Loading 0:  93%|█████████▎| 921/995 [01:17<00:03, 24.64it/s]
Loading 0:  93%|█████████▎| 924/995 [01:17<00:02, 24.40it/s]
Loading 0:  94%|█████████▎| 932/995 [01:17<00:01, 38.14it/s]
Loading 0:  94%|█████████▍| 937/995 [01:18<00:01, 32.34it/s]
Loading 0:  95%|█████████▍| 941/995 [01:18<00:01, 30.67it/s]
Loading 0:  95%|█████████▍| 945/995 [01:18<00:01, 30.50it/s]
Loading 0:  96%|█████████▌| 951/995 [01:18<00:01, 33.78it/s]
Loading 0:  96%|█████████▌| 955/995 [01:20<00:04,  8.02it/s]
Loading 0:  96%|█████████▋| 958/995 [01:20<00:03,  9.49it/s]
Loading 0:  97%|█████████▋| 961/995 [01:20<00:03, 11.20it/s]
Loading 0:  97%|█████████▋| 966/995 [01:20<00:01, 15.37it/s]
Loading 0:  97%|█████████▋| 970/995 [01:20<00:01, 17.95it/s]
Loading 0:  98%|█████████▊| 974/995 [01:20<00:01, 20.17it/s]
Loading 0:  98%|█████████▊| 978/995 [01:20<00:00, 20.09it/s]
Loading 0:  99%|█████████▊| 981/995 [01:21<00:00, 21.31it/s]
Loading 0:  99%|█████████▉| 984/995 [01:21<00:00, 22.38it/s]
Loading 0:  99%|█████████▉| 987/995 [01:21<00:00, 22.59it/s]
                                                            
Job mistralai-mixtral-8x7b-3473-v139-mkmlizer completed after 310.04s with status: succeeded
Connection pool is full, discarding connection: %s. Connection pool size: %s
Stopping job with name mistralai-mixtral-8x7b-3473-v139-mkmlizer
Pipeline stage MKMLizer completed in 343.34s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Connection pool is full, discarding connection: %s. Connection pool size: %s
Pipeline stage MKMLTemplater completed in 10.12s
run pipeline stage %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission blend_hokok_2024-09-09: ('http://neversleep-noromaid-v0-8068-v150-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Running pipeline stage MKMLDeployer
Creating inference service mistralai-mixtral-8x7b-3473-v139
Waiting for inference service mistralai-mixtral-8x7b-3473-v139 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission blend_hokok_2024-09-09: ('http://neversleep-noromaid-v0-8068-v150-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_hokok_2024-09-09: ('http://neversleep-noromaid-v0-8068-v150-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission blend_hokok_2024-09-09: ('http://neversleep-noromaid-v0-8068-v150-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Inference service mistralai-mixtral-8x7b-3473-v139 ready after 161.74526190757751s
Pipeline stage MKMLDeployer completed in 164.88s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.3537325859069824s
Received healthy response to inference request in 1.2735700607299805s
Received healthy response to inference request in 1.129814624786377s
Received healthy response to inference request in 2.229365587234497s
Received healthy response to inference request in 1.891204833984375s
5 requests
0 failed requests
5th percentile: 1.1585657119750976
10th percentile: 1.1873167991638183
20th percentile: 1.2448189735412598
30th percentile: 1.3970970153808593
40th percentile: 1.6441509246826174
50th percentile: 1.891204833984375
60th percentile: 2.0264691352844237
70th percentile: 2.1617334365844725
80th percentile: 2.254238986968994
90th percentile: 2.3039857864379885
95th percentile: 2.3288591861724854
99th percentile: 2.348757905960083
mean time: 1.7755375385284424
Pipeline stage StressChecker completed in 11.85s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 16.56s
Shutdown handler de-registered
Connection pool is full, discarding connection: %s. Connection pool size: %s
mistralai-mixtral-8x7b_3473_v139 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service mistralai-mixtral-8x7b-3473-v139-profiler
Waiting for inference service mistralai-mixtral-8x7b-3473-v139-profiler to be ready
Inference service mistralai-mixtral-8x7b-3473-v139-profiler ready after 170.3892743587494s
Pipeline stage MKMLProfilerDeployer completed in 170.89s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/mistralai-mixtral-8x23cefe0c52d44b296990a42a5b60fbd7-deploks84m:/code/chaiverse_profiler_1726498509 --namespace tenant-chaiml-guanaco
kubectl exec -it mistralai-mixtral-8x23cefe0c52d44b296990a42a5b60fbd7-deploks84m --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1726498509 && python profiles.py profile --best_of_n 2 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1726498509/summary.json'
kubectl exec -it mistralai-mixtral-8x23cefe0c52d44b296990a42a5b60fbd7-deploks84m --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1726498509/summary.json'
Pipeline stage MKMLProfilerRunner completed in 740.73s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service mistralai-mixtral-8x7b-3473-v139-profiler is running
Tearing down inference service mistralai-mixtral-8x7b-3473-v139-profiler
Service mistralai-mixtral-8x7b-3473-v139-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 22.84s
Shutdown handler de-registered
mistralai-mixtral-8x7b_3473_v139 status is now inactive due to auto deactivation removed underperforming models
mistralai-mixtral-8x7b_3473_v139 status is now torndown due to DeploymentManager action