alexdaoud-trainer-bagir-_8564

submission_id: alexdaoud-trainer-bagir-_8564_v1
developer_uid: chai_backend_admin
status: failed
model_repo: alexdaoud/trainer_bagir_2024-12-11-checkpoint-107
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 256, 'best_of': 1, 'max_output_tokens': 1}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '', 'truncate_by_message': True}
timestamp: 2024-12-19T10:30:23+00:00
model_name: alexdaoud-trainer-bagir-_8564_v1
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name alexdaoud-trainer-bagir-8564-v1-mkmlizer
Waiting for job on alexdaoud-trainer-bagir-8564-v1-mkmlizer to finish
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║     _____            __           __                                ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║       /___/                                                         ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║                                                                     ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║  https://mk1.ai                                                     ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║                                                                     ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║  belonging to:                                                      ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║                                                                     ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║  Chai Research Corp.                                                ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ║                                                                     ║
alexdaoud-trainer-bagir-8564-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
alexdaoud-trainer-bagir-8564-v1-mkmlizer: Downloaded to shared memory in 31.610s
alexdaoud-trainer-bagir-8564-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpiw8qryfn, device:0
alexdaoud-trainer-bagir-8564-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
alexdaoud-trainer-bagir-8564-v1-mkmlizer: quantized model in 85.814s
alexdaoud-trainer-bagir-8564-v1-mkmlizer: Processed model alexdaoud/trainer_bagir_2024-12-11-checkpoint-107 in 117.424s
alexdaoud-trainer-bagir-8564-v1-mkmlizer: creating bucket guanaco-mkml-models
alexdaoud-trainer-bagir-8564-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
alexdaoud-trainer-bagir-8564-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8564-v1
alexdaoud-trainer-bagir-8564-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8564-v1/config.json
alexdaoud-trainer-bagir-8564-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8564-v1/special_tokens_map.json
alexdaoud-trainer-bagir-8564-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8564-v1/tokenizer_config.json
alexdaoud-trainer-bagir-8564-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8564-v1/tokenizer.json
alexdaoud-trainer-bagir-8564-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8564-v1/flywheel_model.0.safetensors
alexdaoud-trainer-bagir-8564-v1-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/291 [00:00<00:56,  5.08it/s]
Loading 0:   1%|▏         | 4/291 [00:01<01:32,  3.11it/s]
Loading 0:   2%|▏         | 5/291 [00:01<02:02,  2.34it/s]
Loading 0:   3%|▎         | 8/291 [00:02<01:03,  4.48it/s]
Loading 0:   3%|▎         | 9/291 [00:02<01:02,  4.55it/s]
Loading 0:   3%|▎         | 10/291 [00:02<00:54,  5.18it/s]
Loading 0:   4%|▍         | 12/291 [00:02<01:05,  4.29it/s]
Loading 0:   4%|▍         | 13/291 [00:03<01:26,  3.22it/s]
Loading 0:   5%|▍         | 14/291 [00:04<01:49,  2.53it/s]
Loading 0:   6%|▌         | 17/291 [00:04<01:02,  4.36it/s]
Loading 0:   6%|▌         | 18/291 [00:04<00:59,  4.58it/s]
Loading 0:   7%|▋         | 19/291 [00:04<00:54,  5.03it/s]
Loading 0:   7%|▋         | 21/291 [00:05<01:03,  4.26it/s]
Loading 0:   8%|▊         | 22/291 [00:05<01:23,  3.23it/s]
Loading 0:   8%|▊         | 23/291 [00:06<01:45,  2.53it/s]
Loading 0:   9%|▉         | 26/291 [00:06<01:01,  4.30it/s]
Loading 0:   9%|▉         | 27/291 [00:06<00:58,  4.54it/s]
Loading 0:  10%|▉         | 28/291 [00:07<00:51,  5.09it/s]
Loading 0:  10%|█         | 30/291 [00:07<01:00,  4.29it/s]
Loading 0:  11%|█         | 31/291 [00:08<01:19,  3.26it/s]
Loading 0:  11%|█         | 32/291 [00:08<01:41,  2.55it/s]
Loading 0:  12%|█▏        | 35/291 [00:09<00:58,  4.34it/s]
Loading 0:  12%|█▏        | 36/291 [00:09<00:55,  4.59it/s]
Loading 0:  13%|█▎        | 37/291 [00:09<00:49,  5.17it/s]
Loading 0:  13%|█▎        | 39/291 [00:09<00:57,  4.35it/s]
Loading 0:  14%|█▎        | 40/291 [00:10<01:15,  3.30it/s]
Loading 0:  14%|█▍        | 41/291 [00:11<01:36,  2.59it/s]
Loading 0:  15%|█▌        | 44/291 [00:11<00:55,  4.45it/s]
Loading 0:  15%|█▌        | 45/291 [00:11<00:52,  4.68it/s]
Loading 0:  16%|█▌        | 46/291 [00:11<00:46,  5.27it/s]
Loading 0:  16%|█▋        | 48/291 [00:12<00:55,  4.41it/s]
Loading 0:  17%|█▋        | 49/291 [00:12<01:13,  3.31it/s]
Loading 0:  17%|█▋        | 50/291 [00:13<01:33,  2.58it/s]
Loading 0:  18%|█▊        | 53/291 [00:13<00:54,  4.38it/s]
Loading 0:  19%|█▊        | 54/291 [00:13<00:51,  4.62it/s]
Loading 0:  19%|█▉        | 55/291 [00:13<00:45,  5.21it/s]
Loading 0:  20%|█▉        | 57/291 [00:14<00:53,  4.37it/s]
Loading 0:  20%|█▉        | 58/291 [00:15<01:10,  3.29it/s]
Loading 0:  20%|██        | 59/291 [00:15<01:29,  2.59it/s]
Loading 0:  21%|██▏       | 62/291 [00:15<00:51,  4.45it/s]
Loading 0:  22%|██▏       | 63/291 [00:16<00:48,  4.69it/s]
Loading 0:  22%|██▏       | 64/291 [00:16<00:42,  5.29it/s]
Loading 0:  23%|██▎       | 66/291 [00:16<00:51,  4.41it/s]
Loading 0:  23%|██▎       | 67/291 [00:17<01:07,  3.30it/s]
Loading 0:  23%|██▎       | 68/291 [00:17<01:26,  2.59it/s]
Loading 0:  24%|██▍       | 71/291 [00:18<00:49,  4.41it/s]
Loading 0:  25%|██▍       | 72/291 [00:18<00:47,  4.66it/s]
Loading 0:  25%|██▌       | 73/291 [00:18<00:41,  5.26it/s]
Loading 0:  26%|██▌       | 75/291 [00:19<00:49,  4.38it/s]
Loading 0:  26%|██▌       | 76/291 [00:19<01:05,  3.30it/s]
Loading 0:  26%|██▋       | 77/291 [00:20<01:22,  2.60it/s]
Loading 0:  27%|██▋       | 80/291 [00:20<00:47,  4.43it/s]
Loading 0:  28%|██▊       | 81/291 [00:20<00:44,  4.67it/s]
Loading 0:  28%|██▊       | 82/291 [00:20<00:39,  5.28it/s]
Loading 0:  29%|██▊       | 83/291 [00:20<00:39,  5.27it/s]
Loading 0:  29%|██▉       | 84/291 [00:21<00:59,  3.49it/s]
Loading 0:  29%|██▉       | 85/291 [00:22<01:14,  2.76it/s]
Loading 0:  30%|██▉       | 86/291 [00:22<01:30,  2.27it/s]
Loading 0:  31%|███       | 89/291 [00:22<00:48,  4.18it/s]
Loading 0:  31%|███       | 90/291 [00:23<00:44,  4.47it/s]
Loading 0:  31%|███▏      | 91/291 [00:23<00:39,  5.10it/s]
Loading 0:  32%|███▏      | 93/291 [00:23<00:45,  4.31it/s]
Loading 0:  32%|███▏      | 94/291 [00:24<01:00,  3.26it/s]
Loading 0:  33%|███▎      | 95/291 [00:24<01:16,  2.57it/s]
Loading 0:  34%|███▎      | 98/291 [00:25<00:43,  4.44it/s]
Loading 0:  34%|███▍      | 99/291 [00:25<00:40,  4.69it/s]
Loading 0:  35%|███▍      | 101/291 [00:25<00:29,  6.45it/s]
Loading 0:  35%|███▌      | 103/291 [00:26<00:55,  3.37it/s]
Loading 0:  36%|███▌      | 104/291 [00:27<01:07,  2.75it/s]
Loading 0:  37%|███▋      | 107/291 [00:27<00:41,  4.41it/s]
Loading 0:  37%|███▋      | 108/291 [00:27<00:39,  4.62it/s]
Loading 0:  37%|███▋      | 109/291 [00:27<00:35,  5.08it/s]
Loading 0:  38%|███▊      | 111/291 [00:28<00:41,  4.32it/s]
Loading 0:  38%|███▊      | 112/291 [00:28<00:54,  3.30it/s]
Loading 0:  39%|███▉      | 113/291 [00:29<01:07,  2.62it/s]
Loading 0:  40%|███▉      | 116/291 [00:29<00:39,  4.48it/s]
Loading 0:  40%|████      | 117/291 [00:29<00:36,  4.71it/s]
Loading 0:  41%|████      | 118/291 [00:29<00:32,  5.31it/s]
Loading 0:  41%|████      | 120/291 [00:30<00:38,  4.43it/s]
Loading 0:  42%|████▏     | 121/291 [00:31<00:51,  3.33it/s]
Loading 0:  42%|████▏     | 122/291 [00:31<01:04,  2.62it/s]
Loading 0:  43%|████▎     | 125/291 [00:31<00:36,  4.49it/s]
Loading 0:  43%|████▎     | 126/291 [00:32<00:34,  4.73it/s]
Loading 0:  44%|████▎     | 127/291 [00:32<00:30,  5.29it/s]
Loading 0:  44%|████▍     | 129/291 [00:32<00:36,  4.40it/s]
Loading 0:  45%|████▍     | 130/291 [00:33<00:48,  3.31it/s]
Loading 0:  45%|████▌     | 131/291 [00:34<01:01,  2.61it/s]
Loading 0:  46%|████▌     | 134/291 [00:34<00:35,  4.49it/s]
Loading 0:  46%|████▋     | 135/291 [00:34<00:33,  4.71it/s]
Loading 0:  47%|████▋     | 136/291 [00:34<00:29,  5.27it/s]
Loading 0:  47%|████▋     | 138/291 [00:35<00:34,  4.38it/s]
Loading 0:  48%|████▊     | 139/291 [00:35<00:46,  3.29it/s]
Loading 0:  48%|████▊     | 140/291 [00:36<00:58,  2.59it/s]
Loading 0:  49%|████▉     | 143/291 [00:36<00:33,  4.44it/s]
Loading 0:  49%|████▉     | 144/291 [00:36<00:31,  4.67it/s]
Loading 0:  50%|████▉     | 145/291 [00:36<00:27,  5.23it/s]
Loading 0:  51%|█████     | 147/291 [00:37<00:32,  4.37it/s]
Loading 0:  51%|█████     | 148/291 [00:37<00:43,  3.29it/s]
Loading 0:  51%|█████     | 149/291 [00:38<00:55,  2.56it/s]
Loading 0:  52%|█████▏    | 152/291 [00:38<00:31,  4.41it/s]
Loading 0:  53%|█████▎    | 153/291 [00:38<00:29,  4.63it/s]
Loading 0:  53%|█████▎    | 154/291 [00:39<00:26,  5.23it/s]
Loading 0:  54%|█████▎    | 156/291 [00:39<00:31,  4.35it/s]
Loading 0:  54%|█████▍    | 157/291 [00:40<00:40,  3.27it/s]
Loading 0:  54%|█████▍    | 158/291 [00:40<00:51,  2.59it/s]
Loading 0:  55%|█████▌    | 161/291 [00:41<00:29,  4.41it/s]
Loading 0:  56%|█████▌    | 162/291 [00:41<00:27,  4.64it/s]
Loading 0:  56%|█████▌    | 163/291 [00:41<00:24,  5.23it/s]
Loading 0:  57%|█████▋    | 165/291 [00:41<00:28,  4.37it/s]
Loading 0:  57%|█████▋    | 166/291 [00:42<00:37,  3.29it/s]
Loading 0:  57%|█████▋    | 167/291 [00:43<00:47,  2.59it/s]
Loading 0:  58%|█████▊    | 170/291 [00:43<00:27,  4.40it/s]
Loading 0:  59%|█████▉    | 171/291 [00:43<00:25,  4.62it/s]
Loading 0:  59%|█████▉    | 172/291 [00:43<00:23,  5.14it/s]
Loading 0:  59%|█████▉    | 173/291 [00:44<00:33,  3.49it/s]
Loading 0:  60%|██████    | 175/291 [00:44<00:24,  4.80it/s]
Loading 0:  60%|██████    | 176/291 [00:44<00:22,  5.02it/s]
Loading 0:  61%|██████    | 177/291 [00:44<00:20,  5.67it/s]
Loading 0:  62%|██████▏   | 179/291 [00:45<00:25,  4.46it/s]
Loading 0:  62%|██████▏   | 180/291 [00:45<00:33,  3.28it/s]
Loading 0:  62%|██████▏   | 181/291 [00:46<00:42,  2.58it/s]
Loading 0:  63%|██████▎   | 184/291 [00:46<00:23,  4.53it/s]
Loading 0:  64%|██████▎   | 185/291 [00:46<00:22,  4.76it/s]
Loading 0:  64%|██████▍   | 186/291 [00:46<00:19,  5.33it/s]
Loading 0:  64%|██████▍   | 187/291 [00:47<00:19,  5.40it/s]
Loading 0:  65%|██████▍   | 188/291 [00:47<00:29,  3.50it/s]
Loading 0:  65%|██████▍   | 189/291 [00:48<00:38,  2.63it/s]
Loading 0:  66%|██████▌   | 192/291 [00:48<00:27,  3.54it/s]
Loading 0:  66%|██████▋   | 193/291 [00:49<00:33,  2.92it/s]
Loading 0:  67%|██████▋   | 194/291 [00:50<00:39,  2.43it/s]
Loading 0:  68%|██████▊   | 197/291 [00:50<00:22,  4.09it/s]
Loading 0:  68%|██████▊   | 198/291 [00:50<00:21,  4.34it/s]
Loading 0:  68%|██████▊   | 199/291 [00:50<00:18,  4.90it/s]
Loading 0:  69%|██████▉   | 201/291 [00:51<00:21,  4.22it/s]
Loading 0:  69%|██████▉   | 202/291 [00:51<00:27,  3.22it/s]
Loading 0:  70%|██████▉   | 203/291 [00:52<00:33,  2.59it/s]
Loading 0:  71%|███████   | 206/291 [00:52<00:19,  4.42it/s]
Loading 0:  71%|███████   | 207/291 [00:52<00:18,  4.64it/s]
Loading 0:  71%|███████▏  | 208/291 [00:52<00:15,  5.22it/s]
Loading 0:  72%|███████▏  | 210/291 [00:53<00:18,  4.36it/s]
Loading 0:  73%|███████▎  | 211/291 [00:54<00:24,  3.28it/s]
Loading 0:  73%|███████▎  | 212/291 [00:54<00:30,  2.58it/s]
Loading 0:  74%|███████▍  | 215/291 [00:54<00:17,  4.45it/s]
Loading 0:  74%|███████▍  | 216/291 [00:55<00:16,  4.67it/s]
Loading 0:  75%|███████▍  | 217/291 [00:55<00:14,  5.23it/s]
Loading 0:  75%|███████▌  | 219/291 [00:55<00:16,  4.37it/s]
Loading 0:  76%|███████▌  | 220/291 [00:56<00:21,  3.29it/s]
Loading 0:  76%|███████▌  | 221/291 [00:57<00:26,  2.61it/s]
Loading 0:  77%|███████▋  | 224/291 [00:57<00:15,  4.47it/s]
Loading 0:  77%|███████▋  | 225/291 [00:57<00:14,  4.69it/s]
Loading 0:  78%|███████▊  | 226/291 [00:57<00:12,  5.28it/s]
Loading 0:  78%|███████▊  | 228/291 [00:58<00:14,  4.39it/s]
Loading 0:  79%|███████▊  | 229/291 [00:58<00:18,  3.31it/s]
Loading 0:  79%|███████▉  | 230/291 [00:59<00:23,  2.60it/s]
Loading 0:  80%|████████  | 233/291 [00:59<00:13,  4.39it/s]
Loading 0:  80%|████████  | 234/291 [00:59<00:12,  4.63it/s]
Loading 0:  81%|████████  | 235/291 [00:59<00:10,  5.17it/s]
Loading 0:  81%|████████▏ | 237/291 [01:00<00:12,  4.33it/s]
Loading 0:  82%|████████▏ | 238/291 [01:01<00:16,  3.25it/s]
Loading 0:  82%|████████▏ | 239/291 [01:01<00:20,  2.56it/s]
Loading 0:  83%|████████▎ | 242/291 [01:01<00:11,  4.36it/s]
Loading 0:  84%|████████▎ | 243/291 [01:02<00:10,  4.59it/s]
Loading 0:  84%|████████▍ | 244/291 [01:02<00:09,  5.16it/s]
Loading 0:  85%|████████▍ | 246/291 [01:02<00:10,  4.33it/s]
Loading 0:  85%|████████▍ | 247/291 [01:03<00:13,  3.28it/s]
Loading 0:  85%|████████▌ | 248/291 [01:03<00:16,  2.61it/s]
Loading 0:  86%|████████▋ | 251/291 [01:04<00:08,  4.48it/s]
Loading 0:  87%|████████▋ | 252/291 [01:04<00:08,  4.72it/s]
Loading 0:  87%|████████▋ | 253/291 [01:04<00:07,  5.30it/s]
Loading 0:  88%|████████▊ | 255/291 [01:04<00:08,  4.40it/s]
Loading 0:  88%|████████▊ | 256/291 [01:05<00:10,  3.31it/s]
Loading 0:  88%|████████▊ | 257/291 [01:06<00:13,  2.60it/s]
Loading 0:  89%|████████▉ | 260/291 [01:06<00:06,  4.43it/s]
Loading 0:  90%|████████▉ | 261/291 [01:06<00:06,  4.67it/s]
Loading 0:  90%|█████████ | 262/291 [01:06<00:05,  5.28it/s]
Loading 0:  91%|█████████ | 264/291 [01:07<00:06,  4.39it/s]
Loading 0:  91%|█████████ | 265/291 [01:07<00:07,  3.31it/s]
Loading 0:  91%|█████████▏| 266/291 [01:08<00:09,  2.62it/s]
Loading 0:  92%|█████████▏| 269/291 [01:08<00:04,  4.50it/s]
Loading 0:  93%|█████████▎| 270/291 [01:08<00:04,  4.74it/s]
Loading 0:  93%|█████████▎| 271/291 [01:08<00:03,  5.30it/s]
Loading 0:  94%|█████████▍| 273/291 [01:09<00:04,  4.41it/s]
Loading 0:  94%|█████████▍| 274/291 [01:10<00:05,  3.30it/s]
Loading 0:  95%|█████████▍| 275/291 [01:10<00:06,  2.59it/s]
Loading 0:  96%|█████████▌| 278/291 [01:10<00:02,  4.45it/s]
Loading 0:  96%|█████████▌| 279/291 [01:11<00:02,  4.69it/s]
Loading 0:  96%|█████████▌| 280/291 [01:11<00:02,  5.28it/s]
Loading 0:  97%|█████████▋| 281/291 [01:11<00:02,  3.53it/s]
Loading 0:  97%|█████████▋| 282/291 [01:12<00:03,  2.67it/s]
Loading 0:  98%|█████████▊| 284/291 [01:12<00:01,  3.83it/s]
Loading 0:  98%|█████████▊| 285/291 [01:12<00:01,  4.18it/s]
Loading 0:  98%|█████████▊| 286/291 [01:12<00:01,  4.85it/s]
Loading 0:  99%|█████████▊| 287/291 [01:13<00:00,  5.07it/s]
Loading 0:  99%|█████████▉| 288/291 [01:13<00:00,  3.29it/s]
                                                            
Job alexdaoud-trainer-bagir-8564-v1-mkmlizer completed after 144.96s with status: succeeded
Stopping job with name alexdaoud-trainer-bagir-8564-v1-mkmlizer
Pipeline stage MKMLizer completed in 145.45s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service alexdaoud-trainer-bagir-8564-v1
Waiting for inference service alexdaoud-trainer-bagir-8564-v1 to be ready
Retrying (%r) after connection broken by '%r': %s
Inference service alexdaoud-trainer-bagir-8564-v1 ready after 251.53011536598206s
Pipeline stage MKMLDeployer completed in 252.09s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.307368993759155s
Received healthy response to inference request in 4.470407962799072s
Received healthy response to inference request in 2.817237138748169s
Received healthy response to inference request in 2.8876612186431885s
Received healthy response to inference request in 3.1805636882781982s
5 requests
0 failed requests
5th percentile: 2.831321954727173
10th percentile: 2.8454067707061768
20th percentile: 2.8735764026641846
30th percentile: 2.9462417125701905
40th percentile: 3.063402700424194
50th percentile: 3.1805636882781982
60th percentile: 3.631285810470581
70th percentile: 4.082007932662964
80th percentile: 4.339976787567139
90th percentile: 4.4051923751831055
95th percentile: 4.437800168991089
99th percentile: 4.463886404037476
mean time: 3.532647800445557
%s, retrying in %s seconds...
Received healthy response to inference request in 5.332309722900391s
Received healthy response to inference request in 3.662525177001953s
Received healthy response to inference request in 3.293253183364868s
Received healthy response to inference request in 5.410370349884033s
Received healthy response to inference request in 2.7933361530303955s
5 requests
0 failed requests
5th percentile: 2.89331955909729
10th percentile: 2.9933029651641845
20th percentile: 3.1932697772979735
30th percentile: 3.3671075820922853
40th percentile: 3.5148163795471192
50th percentile: 3.662525177001953
60th percentile: 4.330438995361328
70th percentile: 4.998352813720703
80th percentile: 5.347921848297119
90th percentile: 5.379146099090576
95th percentile: 5.394758224487305
99th percentile: 5.407247924804688
mean time: 4.098358917236328
%s, retrying in %s seconds...
{"detail":"('http://chaiml-elo-alignment-run-3-v44-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:40742->127.0.0.1:8080: read: connection reset by peer\\n')"}
Received unhealthy response to inference request!
Received healthy response to inference request in 4.530014276504517s
Received healthy response to inference request in 4.926225423812866s
Received healthy response to inference request in 3.2053210735321045s
Received healthy response to inference request in 3.131324529647827s
5 requests
1 failed requests
5th percentile: 1.107928228378296
10th percentile: 1.6137773036956788
20th percentile: 2.6254754543304446
30th percentile: 3.146123838424683
40th percentile: 3.1757224559783936
50th percentile: 3.2053210735321045
60th percentile: 3.735198354721069
70th percentile: 4.265075635910034
80th percentile: 4.6092565059661865
90th percentile: 4.767740964889526
95th percentile: 4.846983194351196
99th percentile: 4.910376977920532
mean time: 3.2789928913116455
clean up pipeline due to error=DeploymentChecksError('Unacceptable number of predict errors: 20.0%')
Shutdown handler de-registered
alexdaoud-trainer-bagir-_8564_v1 status is now failed due to DeploymentManager action