alexdaoud-trainer-bagir-_6353

developer_uid: chai_backend_admin
submission_id: alexdaoud-trainer-bagir-_6353_v1
model_name: alexdaoud-trainer-bagir-_6353_v1
model_group: alexdaoud/trainer_bagir_
status: inactive
timestamp: 2024-12-19T09:03:20+00:00
num_battles: 19192
num_wins: 9855
celo_rating: 1271.59
family_friendly_score: 0.5778
family_friendly_standard_error: 0.006984943235273999
submission_type: basic
model_repo: alexdaoud/trainer_bagir_2024-12-11-checkpoint-106
model_architecture: LlamaForSequenceClassification
model_num_parameters: 8030261248.0
best_of: 1
max_input_tokens: 256
max_output_tokens: 1
display_name: alexdaoud-trainer-bagir-_6353_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: alexdaoud/trainer_bagir_2024-12-11-checkpoint-106
model_size: 8B
ranking_group: single
us_pacific_date: 2024-12-19
win_ratio: 0.5134952063359733
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 256, 'best_of': 1, 'max_output_tokens': 1}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name alexdaoud-trainer-bagir-6353-v1-mkmlizer
Waiting for job on alexdaoud-trainer-bagir-6353-v1-mkmlizer to finish
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║     _____            __           __                                ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║       /___/                                                         ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║                                                                     ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║  https://mk1.ai                                                     ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║                                                                     ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║  belonging to:                                                      ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║                                                                     ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║  Chai Research Corp.                                                ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ║                                                                     ║
alexdaoud-trainer-bagir-6353-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
alexdaoud-trainer-bagir-6353-v1-mkmlizer: Downloaded to shared memory in 30.547s
alexdaoud-trainer-bagir-6353-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmp737co5uz, device:0
alexdaoud-trainer-bagir-6353-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
alexdaoud-trainer-bagir-6353-v1-mkmlizer: quantized model in 85.195s
alexdaoud-trainer-bagir-6353-v1-mkmlizer: Processed model alexdaoud/trainer_bagir_2024-12-11-checkpoint-106 in 115.742s
alexdaoud-trainer-bagir-6353-v1-mkmlizer: creating bucket guanaco-mkml-models
alexdaoud-trainer-bagir-6353-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
alexdaoud-trainer-bagir-6353-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/alexdaoud-trainer-bagir-6353-v1
alexdaoud-trainer-bagir-6353-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-6353-v1/config.json
alexdaoud-trainer-bagir-6353-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-6353-v1/special_tokens_map.json
alexdaoud-trainer-bagir-6353-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-6353-v1/tokenizer_config.json
alexdaoud-trainer-bagir-6353-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-6353-v1/tokenizer.json
alexdaoud-trainer-bagir-6353-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/alexdaoud-trainer-bagir-6353-v1/flywheel_model.0.safetensors
alexdaoud-trainer-bagir-6353-v1-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/291 [00:00<00:57,  5.05it/s]
Loading 0:   1%|▏         | 4/291 [00:01<01:32,  3.11it/s]
Loading 0:   2%|▏         | 5/291 [00:01<02:03,  2.32it/s]
Loading 0:   3%|▎         | 8/291 [00:02<01:03,  4.43it/s]
Loading 0:   3%|▎         | 9/291 [00:02<01:02,  4.50it/s]
Loading 0:   3%|▎         | 10/291 [00:02<00:54,  5.13it/s]
Loading 0:   4%|▍         | 12/291 [00:02<01:05,  4.28it/s]
Loading 0:   4%|▍         | 13/291 [00:03<01:26,  3.21it/s]
Loading 0:   5%|▍         | 14/291 [00:04<01:50,  2.52it/s]
Loading 0:   6%|▌         | 17/291 [00:04<01:03,  4.34it/s]
Loading 0:   6%|▌         | 18/291 [00:04<00:59,  4.57it/s]
Loading 0:   7%|▋         | 19/291 [00:04<00:53,  5.09it/s]
Loading 0:   7%|▋         | 21/291 [00:05<01:02,  4.31it/s]
Loading 0:   8%|▊         | 22/291 [00:05<01:22,  3.26it/s]
Loading 0:   8%|▊         | 23/291 [00:06<01:44,  2.57it/s]
Loading 0:   9%|▉         | 26/291 [00:06<01:00,  4.38it/s]
Loading 0:   9%|▉         | 27/291 [00:06<00:57,  4.62it/s]
Loading 0:  10%|█         | 30/291 [00:07<00:57,  4.53it/s]
Loading 0:  11%|█         | 31/291 [00:08<01:13,  3.53it/s]
Loading 0:  11%|█         | 32/291 [00:08<01:33,  2.77it/s]
Loading 0:  12%|█▏        | 35/291 [00:09<00:57,  4.45it/s]
Loading 0:  12%|█▏        | 36/291 [00:09<00:54,  4.67it/s]
Loading 0:  13%|█▎        | 37/291 [00:09<00:48,  5.24it/s]
Loading 0:  13%|█▎        | 39/291 [00:09<00:57,  4.41it/s]
Loading 0:  14%|█▎        | 40/291 [00:10<01:15,  3.33it/s]
Loading 0:  14%|█▍        | 41/291 [00:11<01:35,  2.61it/s]
Loading 0:  15%|█▌        | 44/291 [00:11<00:56,  4.40it/s]
Loading 0:  15%|█▌        | 45/291 [00:11<00:53,  4.63it/s]
Loading 0:  16%|█▌        | 46/291 [00:11<00:46,  5.22it/s]
Loading 0:  16%|█▋        | 48/291 [00:12<00:55,  4.38it/s]
Loading 0:  17%|█▋        | 49/291 [00:12<01:13,  3.31it/s]
Loading 0:  17%|█▋        | 50/291 [00:13<01:33,  2.59it/s]
Loading 0:  18%|█▊        | 53/291 [00:13<00:53,  4.44it/s]
Loading 0:  19%|█▊        | 54/291 [00:13<00:50,  4.68it/s]
Loading 0:  19%|█▉        | 55/291 [00:13<00:44,  5.28it/s]
Loading 0:  20%|█▉        | 57/291 [00:14<00:53,  4.39it/s]
Loading 0:  20%|█▉        | 58/291 [00:15<01:10,  3.31it/s]
Loading 0:  20%|██        | 59/291 [00:15<01:28,  2.62it/s]
Loading 0:  21%|██▏       | 62/291 [00:15<00:51,  4.43it/s]
Loading 0:  22%|██▏       | 63/291 [00:16<00:48,  4.68it/s]
Loading 0:  22%|██▏       | 64/291 [00:16<00:43,  5.27it/s]
Loading 0:  23%|██▎       | 66/291 [00:16<00:51,  4.39it/s]
Loading 0:  23%|██▎       | 67/291 [00:17<01:07,  3.31it/s]
Loading 0:  23%|██▎       | 68/291 [00:17<01:25,  2.61it/s]
Loading 0:  24%|██▍       | 71/291 [00:18<00:49,  4.48it/s]
Loading 0:  25%|██▍       | 72/291 [00:18<00:46,  4.73it/s]
Loading 0:  26%|██▌       | 75/291 [00:18<00:47,  4.59it/s]
Loading 0:  26%|██▌       | 76/291 [00:19<01:00,  3.55it/s]
Loading 0:  26%|██▋       | 77/291 [00:20<01:16,  2.81it/s]
Loading 0:  27%|██▋       | 80/291 [00:20<00:46,  4.49it/s]
Loading 0:  28%|██▊       | 81/291 [00:20<00:44,  4.72it/s]
Loading 0:  28%|██▊       | 82/291 [00:20<00:39,  5.25it/s]
Loading 0:  29%|██▊       | 83/291 [00:20<00:39,  5.23it/s]
Loading 0:  29%|██▉       | 84/291 [00:21<00:59,  3.51it/s]
Loading 0:  29%|██▉       | 85/291 [00:22<01:14,  2.77it/s]
Loading 0:  30%|██▉       | 86/291 [00:22<01:29,  2.29it/s]
Loading 0:  31%|███       | 89/291 [00:22<00:48,  4.20it/s]
Loading 0:  31%|███       | 90/291 [00:23<00:44,  4.48it/s]
Loading 0:  31%|███▏      | 91/291 [00:23<00:39,  5.11it/s]
Loading 0:  32%|███▏      | 93/291 [00:23<00:45,  4.31it/s]
Loading 0:  32%|███▏      | 94/291 [00:24<01:02,  3.17it/s]
Loading 0:  33%|███▎      | 95/291 [00:24<01:18,  2.51it/s]
Loading 0:  34%|███▎      | 98/291 [00:25<00:44,  4.36it/s]
Loading 0:  34%|███▍      | 99/291 [00:25<00:41,  4.61it/s]
Loading 0:  35%|███▍      | 101/291 [00:25<00:29,  6.36it/s]
Loading 0:  35%|███▌      | 103/291 [00:26<00:55,  3.36it/s]
Loading 0:  36%|███▌      | 104/291 [00:27<01:07,  2.75it/s]
Loading 0:  37%|███▋      | 107/291 [00:27<00:41,  4.41it/s]
Loading 0:  37%|███▋      | 108/291 [00:27<00:39,  4.64it/s]
Loading 0:  37%|███▋      | 109/291 [00:27<00:34,  5.20it/s]
Loading 0:  38%|███▊      | 111/291 [00:28<00:40,  4.39it/s]
Loading 0:  38%|███▊      | 112/291 [00:28<00:53,  3.34it/s]
Loading 0:  39%|███▉      | 113/291 [00:29<01:08,  2.61it/s]
Loading 0:  40%|███▉      | 116/291 [00:29<00:39,  4.45it/s]
Loading 0:  40%|████      | 117/291 [00:29<00:37,  4.69it/s]
Loading 0:  41%|████      | 118/291 [00:29<00:32,  5.29it/s]
Loading 0:  41%|████      | 120/291 [00:30<00:38,  4.39it/s]
Loading 0:  42%|████▏     | 121/291 [00:31<00:51,  3.30it/s]
Loading 0:  42%|████▏     | 122/291 [00:31<01:05,  2.59it/s]
Loading 0:  43%|████▎     | 125/291 [00:31<00:37,  4.45it/s]
Loading 0:  43%|████▎     | 126/291 [00:32<00:35,  4.69it/s]
Loading 0:  44%|████▎     | 127/291 [00:32<00:31,  5.27it/s]
Loading 0:  44%|████▍     | 129/291 [00:32<00:36,  4.39it/s]
Loading 0:  45%|████▍     | 130/291 [00:33<00:48,  3.32it/s]
Loading 0:  45%|████▌     | 131/291 [00:34<01:01,  2.62it/s]
Loading 0:  46%|████▌     | 134/291 [00:34<00:34,  4.51it/s]
Loading 0:  46%|████▋     | 135/291 [00:34<00:32,  4.75it/s]
Loading 0:  47%|████▋     | 136/291 [00:34<00:29,  5.32it/s]
Loading 0:  47%|████▋     | 138/291 [00:35<00:34,  4.43it/s]
Loading 0:  48%|████▊     | 139/291 [00:35<00:46,  3.30it/s]
Loading 0:  48%|████▊     | 140/291 [00:36<00:57,  2.63it/s]
Loading 0:  49%|████▉     | 143/291 [00:36<00:33,  4.48it/s]
Loading 0:  49%|████▉     | 144/291 [00:36<00:31,  4.72it/s]
Loading 0:  50%|████▉     | 145/291 [00:36<00:27,  5.32it/s]
Loading 0:  51%|█████     | 147/291 [00:37<00:32,  4.42it/s]
Loading 0:  51%|█████     | 148/291 [00:37<00:42,  3.34it/s]
Loading 0:  51%|█████     | 149/291 [00:38<00:54,  2.62it/s]
Loading 0:  52%|█████▏    | 152/291 [00:38<00:31,  4.44it/s]
Loading 0:  53%|█████▎    | 153/291 [00:38<00:29,  4.68it/s]
Loading 0:  53%|█████▎    | 154/291 [00:38<00:26,  5.25it/s]
Loading 0:  54%|█████▎    | 156/291 [00:39<00:30,  4.41it/s]
Loading 0:  54%|█████▍    | 157/291 [00:40<00:40,  3.34it/s]
Loading 0:  54%|█████▍    | 158/291 [00:40<00:50,  2.62it/s]
Loading 0:  55%|█████▌    | 161/291 [00:40<00:29,  4.43it/s]
Loading 0:  56%|█████▌    | 162/291 [00:41<00:27,  4.67it/s]
Loading 0:  56%|█████▌    | 163/291 [00:41<00:24,  5.24it/s]
Loading 0:  57%|█████▋    | 165/291 [00:41<00:28,  4.40it/s]
Loading 0:  57%|█████▋    | 166/291 [00:42<00:37,  3.32it/s]
Loading 0:  57%|█████▋    | 167/291 [00:43<00:46,  2.64it/s]
Loading 0:  58%|█████▊    | 170/291 [00:43<00:27,  4.47it/s]
Loading 0:  59%|█████▉    | 171/291 [00:43<00:25,  4.71it/s]
Loading 0:  59%|█████▉    | 172/291 [00:43<00:22,  5.31it/s]
Loading 0:  59%|█████▉    | 173/291 [00:44<00:32,  3.58it/s]
Loading 0:  60%|██████    | 175/291 [00:44<00:23,  4.89it/s]
Loading 0:  60%|██████    | 176/291 [00:44<00:22,  5.12it/s]
Loading 0:  61%|██████    | 177/291 [00:44<00:19,  5.77it/s]
Loading 0:  62%|██████▏   | 179/291 [00:45<00:24,  4.56it/s]
Loading 0:  62%|██████▏   | 180/291 [00:45<00:33,  3.35it/s]
Loading 0:  62%|██████▏   | 181/291 [00:46<00:42,  2.60it/s]
Loading 0:  63%|██████▎   | 184/291 [00:46<00:23,  4.49it/s]
Loading 0:  64%|██████▎   | 185/291 [00:46<00:22,  4.73it/s]
Loading 0:  64%|██████▍   | 186/291 [00:46<00:19,  5.34it/s]
Loading 0:  64%|██████▍   | 187/291 [00:46<00:19,  5.45it/s]
Loading 0:  65%|██████▍   | 188/291 [00:47<00:29,  3.54it/s]
Loading 0:  65%|██████▍   | 189/291 [00:48<00:38,  2.65it/s]
Loading 0:  66%|██████▌   | 192/291 [00:48<00:27,  3.59it/s]
Loading 0:  66%|██████▋   | 193/291 [00:49<00:33,  2.96it/s]
Loading 0:  67%|██████▋   | 194/291 [00:49<00:39,  2.47it/s]
Loading 0:  68%|██████▊   | 197/291 [00:50<00:22,  4.17it/s]
Loading 0:  68%|██████▊   | 198/291 [00:50<00:20,  4.43it/s]
Loading 0:  68%|██████▊   | 199/291 [00:50<00:18,  4.97it/s]
Loading 0:  69%|██████▉   | 201/291 [00:51<00:20,  4.29it/s]
Loading 0:  69%|██████▉   | 202/291 [00:51<00:27,  3.27it/s]
Loading 0:  70%|██████▉   | 203/291 [00:52<00:33,  2.61it/s]
Loading 0:  71%|███████   | 206/291 [00:52<00:19,  4.42it/s]
Loading 0:  71%|███████   | 207/291 [00:52<00:18,  4.66it/s]
Loading 0:  71%|███████▏  | 208/291 [00:52<00:15,  5.24it/s]
Loading 0:  72%|███████▏  | 210/291 [00:53<00:18,  4.39it/s]
Loading 0:  73%|███████▎  | 211/291 [00:53<00:24,  3.31it/s]
Loading 0:  73%|███████▎  | 212/291 [00:54<00:30,  2.63it/s]
Loading 0:  74%|███████▍  | 215/291 [00:54<00:17,  4.45it/s]
Loading 0:  74%|███████▍  | 216/291 [00:54<00:15,  4.70it/s]
Loading 0:  75%|███████▍  | 217/291 [00:54<00:14,  5.28it/s]
Loading 0:  75%|███████▌  | 219/291 [00:55<00:16,  4.41it/s]
Loading 0:  76%|███████▌  | 220/291 [00:56<00:21,  3.33it/s]
Loading 0:  76%|███████▌  | 221/291 [00:56<00:26,  2.62it/s]
Loading 0:  77%|███████▋  | 224/291 [00:56<00:15,  4.43it/s]
Loading 0:  77%|███████▋  | 225/291 [00:57<00:14,  4.66it/s]
Loading 0:  78%|███████▊  | 226/291 [00:57<00:12,  5.24it/s]
Loading 0:  78%|███████▊  | 228/291 [00:57<00:14,  4.39it/s]
Loading 0:  79%|███████▊  | 229/291 [00:58<00:18,  3.32it/s]
Loading 0:  79%|███████▉  | 230/291 [00:59<00:23,  2.63it/s]
Loading 0:  80%|████████  | 233/291 [00:59<00:13,  4.46it/s]
Loading 0:  80%|████████  | 234/291 [00:59<00:12,  4.70it/s]
Loading 0:  81%|████████  | 235/291 [00:59<00:10,  5.30it/s]
Loading 0:  81%|████████▏ | 237/291 [01:00<00:12,  4.44it/s]
Loading 0:  82%|████████▏ | 238/291 [01:00<00:15,  3.35it/s]
Loading 0:  82%|████████▏ | 239/291 [01:01<00:19,  2.66it/s]
Loading 0:  83%|████████▎ | 242/291 [01:01<00:10,  4.57it/s]
Loading 0:  84%|████████▎ | 243/291 [01:01<00:10,  4.80it/s]
Loading 0:  84%|████████▍ | 244/291 [01:01<00:08,  5.27it/s]
Loading 0:  85%|████████▍ | 246/291 [01:02<00:10,  4.40it/s]
Loading 0:  85%|████████▍ | 247/291 [01:02<00:13,  3.32it/s]
Loading 0:  85%|████████▌ | 248/291 [01:03<00:16,  2.63it/s]
Loading 0:  86%|████████▋ | 251/291 [01:03<00:08,  4.47it/s]
Loading 0:  87%|████████▋ | 252/291 [01:03<00:08,  4.71it/s]
Loading 0:  87%|████████▋ | 253/291 [01:04<00:07,  5.31it/s]
Loading 0:  88%|████████▊ | 255/291 [01:04<00:08,  4.43it/s]
Loading 0:  88%|████████▊ | 256/291 [01:05<00:10,  3.34it/s]
Loading 0:  88%|████████▊ | 257/291 [01:05<00:12,  2.65it/s]
Loading 0:  89%|████████▉ | 260/291 [01:05<00:06,  4.53it/s]
Loading 0:  90%|████████▉ | 261/291 [01:06<00:06,  4.76it/s]
Loading 0:  90%|█████████ | 262/291 [01:06<00:05,  5.35it/s]
Loading 0:  91%|█████████ | 264/291 [01:06<00:06,  4.43it/s]
Loading 0:  91%|█████████ | 265/291 [01:07<00:07,  3.36it/s]
Loading 0:  91%|█████████▏| 266/291 [01:08<00:09,  2.65it/s]
Loading 0:  92%|█████████▏| 269/291 [01:08<00:04,  4.54it/s]
Loading 0:  93%|█████████▎| 270/291 [01:08<00:04,  4.78it/s]
Loading 0:  93%|█████████▎| 271/291 [01:08<00:03,  5.35it/s]
Loading 0:  94%|█████████▍| 273/291 [01:09<00:04,  4.44it/s]
Loading 0:  94%|█████████▍| 274/291 [01:09<00:05,  3.35it/s]
Loading 0:  95%|█████████▍| 275/291 [01:10<00:06,  2.65it/s]
Loading 0:  96%|█████████▌| 278/291 [01:10<00:02,  4.50it/s]
Loading 0:  96%|█████████▌| 279/291 [01:10<00:02,  4.73it/s]
Loading 0:  96%|█████████▌| 280/291 [01:10<00:02,  5.32it/s]
Loading 0:  97%|█████████▋| 281/291 [01:11<00:02,  3.59it/s]
Loading 0:  97%|█████████▋| 282/291 [01:11<00:03,  2.71it/s]
Loading 0:  98%|█████████▊| 284/291 [01:12<00:01,  3.94it/s]
Loading 0:  98%|█████████▊| 285/291 [01:12<00:01,  4.30it/s]
Loading 0:  98%|█████████▊| 286/291 [01:12<00:01,  4.99it/s]
Loading 0:  99%|█████████▊| 287/291 [01:12<00:00,  5.19it/s]
Loading 0:  99%|█████████▉| 288/291 [01:13<00:00,  3.39it/s]
                                                            
Job alexdaoud-trainer-bagir-6353-v1-mkmlizer completed after 145.72s with status: succeeded
Stopping job with name alexdaoud-trainer-bagir-6353-v1-mkmlizer
Pipeline stage MKMLizer completed in 146.28s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service alexdaoud-trainer-bagir-6353-v1
Waiting for inference service alexdaoud-trainer-bagir-6353-v1 to be ready
Inference service alexdaoud-trainer-bagir-6353-v1 ready after 250.8622853755951s
Pipeline stage MKMLDeployer completed in 251.40s
run pipeline stage %s
Running pipeline stage StressChecker
Connection pool is full, discarding connection: %s. Connection pool size: %s
Received healthy response to inference request in 3.158071279525757s
Received healthy response to inference request in 5.302440166473389s
Received healthy response to inference request in 3.7494747638702393s
Received healthy response to inference request in 3.5901684761047363s
Received healthy response to inference request in 1.982496738433838s
5 requests
0 failed requests
5th percentile: 2.2176116466522218
10th percentile: 2.4527265548706056
20th percentile: 2.922956371307373
30th percentile: 3.244490718841553
40th percentile: 3.4173295974731444
50th percentile: 3.5901684761047363
60th percentile: 3.6538909912109374
70th percentile: 3.7176135063171385
80th percentile: 4.060067844390869
90th percentile: 4.6812540054321286
95th percentile: 4.991847085952759
99th percentile: 5.240321550369263
mean time: 3.5565302848815916
%s, retrying in %s seconds...
read tcp 127.0.0.1:46822->127.0.0.1:8080: read: connection reset by peer
Received unhealthy response to inference request!
Received healthy response to inference request in 3.1877779960632324s
Received healthy response to inference request in 4.129073143005371s
Received healthy response to inference request in 2.821587085723877s
Received healthy response to inference request in 2.5261104106903076s
5 requests
1 failed requests
5th percentile: 1.11130633354187
10th percentile: 1.4650073528289795
20th percentile: 2.1724093914031983
30th percentile: 2.5852057456970217
40th percentile: 2.7033964157104493
50th percentile: 2.821587085723877
60th percentile: 2.9680634498596192
70th percentile: 3.114539813995361
80th percentile: 3.3760370254516605
90th percentile: 3.752555084228516
95th percentile: 3.940814113616943
99th percentile: 4.091421337127685
mean time: 2.68443078994751
%s, retrying in %s seconds...
Received healthy response to inference request in 3.5239086151123047s
Received healthy response to inference request in 3.2445404529571533s
Received healthy response to inference request in 2.915687084197998s
Received healthy response to inference request in 3.4429614543914795s
Received healthy response to inference request in 3.896029233932495s
5 requests
0 failed requests
5th percentile: 2.981457757949829
10th percentile: 3.0472284317016602
20th percentile: 3.1787697792053224
30th percentile: 3.2842246532440185
40th percentile: 3.363593053817749
50th percentile: 3.4429614543914795
60th percentile: 3.4753403186798097
70th percentile: 3.5077191829681396
80th percentile: 3.598332738876343
90th percentile: 3.7471809864044188
95th percentile: 3.821605110168457
99th percentile: 3.8811444091796874
mean time: 3.4046253681182863
Pipeline stage StressChecker completed in 52.36s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.21s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.20s
Shutdown handler de-registered
alexdaoud-trainer-bagir-_6353_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.13s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service alexdaoud-trainer-bagir-6353-v1-profiler
Waiting for inference service alexdaoud-trainer-bagir-6353-v1-profiler to be ready
Inference service alexdaoud-trainer-bagir-6353-v1-profiler ready after 250.549063205719s
Pipeline stage MKMLProfilerDeployer completed in 250.92s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba8a55b33d5aec584d3ad1b89cf30525af-deploqjhpp:/code/chaiverse_profiler_1734599737 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba8a55b33d5aec584d3ad1b89cf30525af-deploqjhpp --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734599737 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734599737/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba8a55b33d5aec584d3ad1b89cf30525af-deploqjhpp:/code/chaiverse_profiler_1734602513 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba8a55b33d5aec584d3ad1b89cf30525af-deploqjhpp --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734602513 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734602513/summary.json'
Received signal 2, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-6353-v1-profiler is running
Tearing down inference service alexdaoud-trainer-bagir-6353-v1-profiler
Service alexdaoud-trainer-bagir-6353-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.64s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-6353-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 2.80s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service alexdaoud-trainer-bagir-6353-v1-profiler
Waiting for inference service alexdaoud-trainer-bagir-6353-v1-profiler to be ready
Inference service alexdaoud-trainer-bagir-6353-v1-profiler ready after 40.11465620994568s
Pipeline stage MKMLProfilerDeployer completed in 40.50s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba8a55b33d5aec584d3ad1b89cf30525af-deplonvkgr:/code/chaiverse_profiler_1734603152 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba8a55b33d5aec584d3ad1b89cf30525af-deplonvkgr --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734603152 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734603152/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba8a55b33d5aec584d3ad1b89cf30525af-deplonvkgr:/code/chaiverse_profiler_1734605987 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba8a55b33d5aec584d3ad1b89cf30525af-deplonvkgr --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734605987 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734605987/summary.json'
Received signal 2, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-6353-v1-profiler is running
Tearing down inference service alexdaoud-trainer-bagir-6353-v1-profiler
Service alexdaoud-trainer-bagir-6353-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.48s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-6353-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 2.81s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service alexdaoud-trainer-bagir-6353-v1-profiler
Waiting for inference service alexdaoud-trainer-bagir-6353-v1-profiler to be ready
Inference service alexdaoud-trainer-bagir-6353-v1-profiler ready after 50.13836121559143s
Pipeline stage MKMLProfilerDeployer completed in 50.49s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba8a55b33d5aec584d3ad1b89cf30525af-deploxb5tv:/code/chaiverse_profiler_1734606789 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba8a55b33d5aec584d3ad1b89cf30525af-deploxb5tv --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734606789 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734606789/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba8a55b33d5aec584d3ad1b89cf30525af-deploxb5tv:/code/chaiverse_profiler_1734609583 --namespace tenant-chaiml-guanaco
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba8a55b33d5aec584d3ad1b89cf30525af-deploxb5tv:/code/chaiverse_profiler_1734609583 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba8a55b33d5aec584d3ad1b89cf30525af-deploxb5tv --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734609583 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734609583/summary.json'
Received signal 2, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-6353-v1-profiler is running
Tearing down inference service alexdaoud-trainer-bagir-6353-v1-profiler
Service alexdaoud-trainer-bagir-6353-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.80s
Shutdown handler de-registered
alexdaoud-trainer-bagir-_6353_v1 status is now inactive due to auto deactivation removed underperforming models