alexdaoud-trainer-bagir-_2925

submission_id: alexdaoud-trainer-bagir-_2925_v1
developer_uid: chai_backend_admin
status: failed
model_repo: alexdaoud/trainer_bagir_2024-12-11-checkpoint-105
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 256, 'best_of': 1, 'max_output_tokens': 1}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '', 'truncate_by_message': True}
timestamp: 2024-12-19T07:51:55+00:00
model_name: alexdaoud-trainer-bagir-_2925_v1
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name alexdaoud-trainer-bagir-2925-v1-mkmlizer
Waiting for job on alexdaoud-trainer-bagir-2925-v1-mkmlizer to finish
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║     _____            __           __                                ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║       /___/                                                         ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║                                                                     ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║  https://mk1.ai                                                     ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║                                                                     ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║  belonging to:                                                      ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║                                                                     ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║  Chai Research Corp.                                                ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ║                                                                     ║
alexdaoud-trainer-bagir-2925-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
alexdaoud-trainer-bagir-2925-v1-mkmlizer: Downloaded to shared memory in 30.647s
alexdaoud-trainer-bagir-2925-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmp9acavoun, device:0
alexdaoud-trainer-bagir-2925-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
alexdaoud-trainer-bagir-2925-v1-mkmlizer: quantized model in 84.691s
alexdaoud-trainer-bagir-2925-v1-mkmlizer: Processed model alexdaoud/trainer_bagir_2024-12-11-checkpoint-105 in 115.338s
alexdaoud-trainer-bagir-2925-v1-mkmlizer: creating bucket guanaco-mkml-models
alexdaoud-trainer-bagir-2925-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
alexdaoud-trainer-bagir-2925-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/alexdaoud-trainer-bagir-2925-v1
alexdaoud-trainer-bagir-2925-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-2925-v1/special_tokens_map.json
alexdaoud-trainer-bagir-2925-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-2925-v1/config.json
alexdaoud-trainer-bagir-2925-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-2925-v1/tokenizer_config.json
alexdaoud-trainer-bagir-2925-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-2925-v1/tokenizer.json
alexdaoud-trainer-bagir-2925-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/alexdaoud-trainer-bagir-2925-v1/flywheel_model.0.safetensors
alexdaoud-trainer-bagir-2925-v1-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/291 [00:00<00:56,  5.11it/s]
Loading 0:   1%|▏         | 4/291 [00:01<01:30,  3.15it/s]
Loading 0:   2%|▏         | 5/291 [00:01<02:01,  2.36it/s]
Loading 0:   3%|▎         | 8/291 [00:02<01:02,  4.50it/s]
Loading 0:   3%|▎         | 9/291 [00:02<01:01,  4.58it/s]
Loading 0:   3%|▎         | 10/291 [00:02<00:53,  5.22it/s]
Loading 0:   4%|▍         | 12/291 [00:02<01:04,  4.34it/s]
Loading 0:   4%|▍         | 13/291 [00:03<01:25,  3.26it/s]
Loading 0:   5%|▍         | 14/291 [00:04<01:47,  2.57it/s]
Loading 0:   6%|▌         | 17/291 [00:04<01:02,  4.41it/s]
Loading 0:   6%|▌         | 18/291 [00:04<00:58,  4.66it/s]
Loading 0:   7%|▋         | 19/291 [00:04<00:51,  5.25it/s]
Loading 0:   7%|▋         | 21/291 [00:05<01:01,  4.41it/s]
Loading 0:   8%|▊         | 22/291 [00:05<01:20,  3.33it/s]
Loading 0:   8%|▊         | 23/291 [00:06<01:43,  2.60it/s]
Loading 0:   9%|▉         | 26/291 [00:06<01:00,  4.42it/s]
Loading 0:   9%|▉         | 27/291 [00:06<00:56,  4.66it/s]
Loading 0:  10%|▉         | 28/291 [00:06<00:49,  5.26it/s]
Loading 0:  10%|█         | 30/291 [00:07<00:59,  4.40it/s]
Loading 0:  11%|█         | 31/291 [00:08<01:18,  3.32it/s]
Loading 0:  11%|█         | 32/291 [00:08<01:39,  2.61it/s]
Loading 0:  12%|█▏        | 35/291 [00:08<00:57,  4.44it/s]
Loading 0:  12%|█▏        | 36/291 [00:09<00:54,  4.68it/s]
Loading 0:  13%|█▎        | 37/291 [00:09<00:48,  5.18it/s]
Loading 0:  13%|█▎        | 39/291 [00:09<00:57,  4.36it/s]
Loading 0:  14%|█▎        | 40/291 [00:10<01:15,  3.31it/s]
Loading 0:  14%|█▍        | 41/291 [00:10<01:35,  2.61it/s]
Loading 0:  15%|█▌        | 44/291 [00:11<00:54,  4.49it/s]
Loading 0:  15%|█▌        | 45/291 [00:11<00:51,  4.74it/s]
Loading 0:  16%|█▌        | 46/291 [00:11<00:46,  5.30it/s]
Loading 0:  16%|█▋        | 48/291 [00:12<00:54,  4.43it/s]
Loading 0:  17%|█▋        | 49/291 [00:12<01:12,  3.34it/s]
Loading 0:  17%|█▋        | 50/291 [00:13<01:32,  2.59it/s]
Loading 0:  18%|█▊        | 53/291 [00:13<00:53,  4.47it/s]
Loading 0:  19%|█▊        | 54/291 [00:13<00:50,  4.71it/s]
Loading 0:  19%|█▉        | 55/291 [00:13<00:44,  5.31it/s]
Loading 0:  20%|█▉        | 57/291 [00:14<00:53,  4.40it/s]
Loading 0:  20%|█▉        | 58/291 [00:14<01:10,  3.33it/s]
Loading 0:  20%|██        | 59/291 [00:15<01:27,  2.65it/s]
Loading 0:  21%|██▏       | 62/291 [00:15<00:50,  4.57it/s]
Loading 0:  22%|██▏       | 63/291 [00:15<00:47,  4.80it/s]
Loading 0:  23%|██▎       | 66/291 [00:16<00:48,  4.63it/s]
Loading 0:  23%|██▎       | 67/291 [00:17<01:02,  3.58it/s]
Loading 0:  23%|██▎       | 68/291 [00:17<01:18,  2.83it/s]
Loading 0:  24%|██▍       | 71/291 [00:17<00:48,  4.54it/s]
Loading 0:  25%|██▍       | 72/291 [00:18<00:46,  4.75it/s]
Loading 0:  25%|██▌       | 73/291 [00:18<00:41,  5.25it/s]
Loading 0:  26%|██▌       | 75/291 [00:18<00:48,  4.42it/s]
Loading 0:  26%|██▌       | 76/291 [00:19<01:04,  3.35it/s]
Loading 0:  26%|██▋       | 77/291 [00:19<01:21,  2.63it/s]
Loading 0:  27%|██▋       | 80/291 [00:20<00:47,  4.43it/s]
Loading 0:  28%|██▊       | 81/291 [00:20<00:44,  4.67it/s]
Loading 0:  28%|██▊       | 82/291 [00:20<00:39,  5.26it/s]
Loading 0:  29%|██▊       | 83/291 [00:20<00:38,  5.37it/s]
Loading 0:  29%|██▉       | 84/291 [00:21<00:58,  3.53it/s]
Loading 0:  29%|██▉       | 85/291 [00:21<01:13,  2.78it/s]
Loading 0:  30%|██▉       | 86/291 [00:22<01:29,  2.28it/s]
Loading 0:  31%|███       | 89/291 [00:22<00:47,  4.24it/s]
Loading 0:  31%|███       | 90/291 [00:22<00:44,  4.51it/s]
Loading 0:  31%|███▏      | 91/291 [00:22<00:39,  5.11it/s]
Loading 0:  32%|███▏      | 93/291 [00:23<00:45,  4.34it/s]
Loading 0:  32%|███▏      | 94/291 [00:24<00:59,  3.29it/s]
Loading 0:  33%|███▎      | 95/291 [00:24<01:15,  2.59it/s]
Loading 0:  34%|███▎      | 98/291 [00:24<00:43,  4.44it/s]
Loading 0:  34%|███▍      | 99/291 [00:25<00:40,  4.69it/s]
Loading 0:  34%|███▍      | 100/291 [00:25<00:36,  5.28it/s]
Loading 0:  35%|███▌      | 102/291 [00:25<00:42,  4.43it/s]
Loading 0:  35%|███▌      | 103/291 [00:26<00:56,  3.34it/s]
Loading 0:  36%|███▌      | 104/291 [00:26<01:10,  2.64it/s]
Loading 0:  37%|███▋      | 107/291 [00:27<00:40,  4.53it/s]
Loading 0:  37%|███▋      | 108/291 [00:27<00:38,  4.78it/s]
Loading 0:  37%|███▋      | 109/291 [00:27<00:33,  5.36it/s]
Loading 0:  38%|███▊      | 111/291 [00:27<00:40,  4.48it/s]
Loading 0:  38%|███▊      | 112/291 [00:28<00:53,  3.38it/s]
Loading 0:  39%|███▉      | 113/291 [00:29<01:06,  2.66it/s]
Loading 0:  40%|███▉      | 116/291 [00:29<00:38,  4.56it/s]
Loading 0:  40%|████      | 117/291 [00:29<00:36,  4.79it/s]
Loading 0:  41%|████      | 120/291 [00:30<00:36,  4.62it/s]
Loading 0:  42%|████▏     | 121/291 [00:30<00:47,  3.60it/s]
Loading 0:  42%|████▏     | 122/291 [00:31<00:59,  2.85it/s]
Loading 0:  43%|████▎     | 125/291 [00:31<00:36,  4.61it/s]
Loading 0:  43%|████▎     | 126/291 [00:31<00:34,  4.83it/s]
Loading 0:  44%|████▎     | 127/291 [00:31<00:30,  5.40it/s]
Loading 0:  44%|████▍     | 129/291 [00:32<00:35,  4.53it/s]
Loading 0:  45%|████▍     | 130/291 [00:32<00:47,  3.42it/s]
Loading 0:  45%|████▌     | 131/291 [00:33<01:00,  2.64it/s]
Loading 0:  46%|████▌     | 134/291 [00:33<00:34,  4.51it/s]
Loading 0:  46%|████▋     | 135/291 [00:33<00:32,  4.76it/s]
Loading 0:  47%|████▋     | 136/291 [00:34<00:28,  5.36it/s]
Loading 0:  47%|████▋     | 138/291 [00:34<00:34,  4.49it/s]
Loading 0:  48%|████▊     | 139/291 [00:35<00:44,  3.40it/s]
Loading 0:  48%|████▊     | 140/291 [00:35<00:56,  2.69it/s]
Loading 0:  49%|████▉     | 143/291 [00:36<00:32,  4.55it/s]
Loading 0:  49%|████▉     | 144/291 [00:36<00:30,  4.80it/s]
Loading 0:  50%|████▉     | 145/291 [00:36<00:27,  5.32it/s]
Loading 0:  51%|█████     | 147/291 [00:36<00:32,  4.49it/s]
Loading 0:  51%|█████     | 148/291 [00:37<00:42,  3.40it/s]
Loading 0:  51%|█████     | 149/291 [00:38<00:53,  2.67it/s]
Loading 0:  52%|█████▏    | 152/291 [00:38<00:30,  4.53it/s]
Loading 0:  53%|█████▎    | 153/291 [00:38<00:28,  4.77it/s]
Loading 0:  53%|█████▎    | 154/291 [00:38<00:25,  5.31it/s]
Loading 0:  54%|█████▎    | 156/291 [00:39<00:30,  4.46it/s]
Loading 0:  54%|█████▍    | 157/291 [00:39<00:39,  3.37it/s]
Loading 0:  54%|█████▍    | 158/291 [00:40<00:50,  2.65it/s]
Loading 0:  55%|█████▌    | 161/291 [00:40<00:28,  4.50it/s]
Loading 0:  56%|█████▌    | 162/291 [00:40<00:27,  4.75it/s]
Loading 0:  56%|█████▌    | 163/291 [00:40<00:23,  5.34it/s]
Loading 0:  57%|█████▋    | 165/291 [00:41<00:28,  4.48it/s]
Loading 0:  57%|█████▋    | 166/291 [00:41<00:36,  3.38it/s]
Loading 0:  57%|█████▋    | 167/291 [00:42<00:46,  2.65it/s]
Loading 0:  58%|█████▊    | 170/291 [00:42<00:27,  4.46it/s]
Loading 0:  59%|█████▉    | 171/291 [00:42<00:25,  4.69it/s]
Loading 0:  59%|█████▉    | 172/291 [00:43<00:22,  5.27it/s]
Loading 0:  59%|█████▉    | 173/291 [00:43<00:33,  3.55it/s]
Loading 0:  60%|██████    | 175/291 [00:43<00:23,  4.84it/s]
Loading 0:  60%|██████    | 176/291 [00:43<00:22,  5.05it/s]
Loading 0:  61%|██████    | 178/291 [00:44<00:15,  7.12it/s]
Loading 0:  62%|██████▏   | 180/291 [00:45<00:32,  3.37it/s]
Loading 0:  62%|██████▏   | 181/291 [00:45<00:40,  2.71it/s]
Loading 0:  63%|██████▎   | 184/291 [00:46<00:24,  4.41it/s]
Loading 0:  64%|██████▎   | 185/291 [00:46<00:22,  4.64it/s]
Loading 0:  64%|██████▍   | 186/291 [00:46<00:20,  5.20it/s]
Loading 0:  64%|██████▍   | 187/291 [00:46<00:19,  5.37it/s]
Loading 0:  65%|██████▍   | 188/291 [00:47<00:28,  3.56it/s]
Loading 0:  65%|██████▍   | 189/291 [00:47<00:38,  2.68it/s]
Loading 0:  66%|██████▌   | 192/291 [00:48<00:27,  3.59it/s]
Loading 0:  66%|██████▋   | 193/291 [00:48<00:33,  2.96it/s]
Loading 0:  67%|██████▋   | 194/291 [00:49<00:39,  2.46it/s]
Loading 0:  68%|██████▊   | 197/291 [00:49<00:22,  4.20it/s]
Loading 0:  68%|██████▊   | 198/291 [00:49<00:21,  4.43it/s]
Loading 0:  68%|██████▊   | 199/291 [00:49<00:18,  4.99it/s]
Loading 0:  69%|██████▉   | 201/291 [00:50<00:21,  4.24it/s]
Loading 0:  69%|██████▉   | 202/291 [00:51<00:27,  3.25it/s]
Loading 0:  70%|██████▉   | 203/291 [00:51<00:33,  2.60it/s]
Loading 0:  71%|███████   | 206/291 [00:51<00:19,  4.41it/s]
Loading 0:  71%|███████   | 207/291 [00:52<00:18,  4.64it/s]
Loading 0:  71%|███████▏  | 208/291 [00:52<00:15,  5.20it/s]
Loading 0:  72%|███████▏  | 210/291 [00:52<00:18,  4.37it/s]
Loading 0:  73%|███████▎  | 211/291 [00:53<00:24,  3.30it/s]
Loading 0:  73%|███████▎  | 212/291 [00:54<00:30,  2.61it/s]
Loading 0:  74%|███████▍  | 215/291 [00:54<00:17,  4.42it/s]
Loading 0:  74%|███████▍  | 216/291 [00:54<00:16,  4.66it/s]
Loading 0:  75%|███████▍  | 217/291 [00:54<00:14,  5.23it/s]
Loading 0:  75%|███████▌  | 219/291 [00:55<00:16,  4.39it/s]
Loading 0:  76%|███████▌  | 220/291 [00:55<00:21,  3.32it/s]
Loading 0:  76%|███████▌  | 221/291 [00:56<00:26,  2.61it/s]
Loading 0:  77%|███████▋  | 224/291 [00:56<00:14,  4.48it/s]
Loading 0:  77%|███████▋  | 225/291 [00:56<00:14,  4.71it/s]
Loading 0:  78%|███████▊  | 226/291 [00:56<00:12,  5.28it/s]
Loading 0:  78%|███████▊  | 228/291 [00:57<00:14,  4.41it/s]
Loading 0:  79%|███████▊  | 229/291 [00:57<00:18,  3.33it/s]
Loading 0:  79%|███████▉  | 230/291 [00:58<00:23,  2.64it/s]
Loading 0:  80%|████████  | 233/291 [00:58<00:12,  4.51it/s]
Loading 0:  80%|████████  | 234/291 [00:58<00:12,  4.74it/s]
Loading 0:  81%|████████▏ | 237/291 [00:59<00:11,  4.59it/s]
Loading 0:  82%|████████▏ | 238/291 [01:00<00:14,  3.54it/s]
Loading 0:  82%|████████▏ | 239/291 [01:00<00:18,  2.83it/s]
Loading 0:  83%|████████▎ | 242/291 [01:01<00:10,  4.56it/s]
Loading 0:  84%|████████▎ | 243/291 [01:01<00:10,  4.77it/s]
Loading 0:  84%|████████▍ | 244/291 [01:01<00:08,  5.33it/s]
Loading 0:  85%|████████▍ | 246/291 [01:01<00:10,  4.44it/s]
Loading 0:  85%|████████▍ | 247/291 [01:02<00:13,  3.36it/s]
Loading 0:  85%|████████▌ | 248/291 [01:03<00:16,  2.66it/s]
Loading 0:  86%|████████▋ | 251/291 [01:03<00:08,  4.52it/s]
Loading 0:  87%|████████▋ | 252/291 [01:03<00:08,  4.75it/s]
Loading 0:  88%|████████▊ | 255/291 [01:04<00:07,  4.61it/s]
Loading 0:  88%|████████▊ | 256/291 [01:04<00:09,  3.59it/s]
Loading 0:  88%|████████▊ | 257/291 [01:05<00:11,  2.87it/s]
Loading 0:  89%|████████▉ | 260/291 [01:05<00:06,  4.63it/s]
Loading 0:  90%|████████▉ | 261/291 [01:05<00:06,  4.91it/s]
Loading 0:  90%|█████████ | 263/291 [01:05<00:04,  6.61it/s]
Loading 0:  91%|█████████ | 265/291 [01:06<00:07,  3.47it/s]
Loading 0:  91%|█████████▏| 266/291 [01:07<00:08,  2.81it/s]
Loading 0:  92%|█████████▏| 269/291 [01:07<00:04,  4.45it/s]
Loading 0:  93%|█████████▎| 270/291 [01:07<00:04,  4.66it/s]
Loading 0:  93%|█████████▎| 271/291 [01:07<00:03,  5.20it/s]
Loading 0:  94%|█████████▍| 273/291 [01:08<00:04,  4.41it/s]
Loading 0:  94%|█████████▍| 274/291 [01:09<00:05,  3.37it/s]
Loading 0:  95%|█████████▍| 275/291 [01:09<00:05,  2.68it/s]
Loading 0:  96%|█████████▌| 278/291 [01:09<00:02,  4.50it/s]
Loading 0:  96%|█████████▌| 279/291 [01:10<00:02,  4.73it/s]
Loading 0:  96%|█████████▌| 280/291 [01:10<00:02,  5.31it/s]
Loading 0:  97%|█████████▋| 281/291 [01:10<00:02,  3.59it/s]
Loading 0:  97%|█████████▋| 282/291 [01:11<00:03,  2.69it/s]
Loading 0:  98%|█████████▊| 284/291 [01:11<00:01,  3.85it/s]
Loading 0:  98%|█████████▊| 285/291 [01:11<00:01,  4.20it/s]
Loading 0:  98%|█████████▊| 286/291 [01:11<00:01,  4.85it/s]
Loading 0:  99%|█████████▊| 287/291 [01:12<00:00,  5.05it/s]
Loading 0:  99%|█████████▉| 288/291 [01:12<00:00,  3.32it/s]
                                                            
Job alexdaoud-trainer-bagir-2925-v1-mkmlizer completed after 145.5s with status: succeeded
Stopping job with name alexdaoud-trainer-bagir-2925-v1-mkmlizer
Pipeline stage MKMLizer completed in 146.19s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service alexdaoud-trainer-bagir-2925-v1
Waiting for inference service alexdaoud-trainer-bagir-2925-v1 to be ready
Inference service alexdaoud-trainer-bagir-2925-v1 ready after 230.8324739933014s
Pipeline stage MKMLDeployer completed in 231.40s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.356414556503296s
Received healthy response to inference request in 4.0242085456848145s
Received healthy response to inference request in 2.801297903060913s
Received healthy response to inference request in 4.157505035400391s
Received healthy response to inference request in 2.948928117752075s
5 requests
0 failed requests
5th percentile: 2.8308239459991453
10th percentile: 2.860349988937378
20th percentile: 2.919402074813843
30th percentile: 3.163984203338623
40th percentile: 3.594096374511719
50th percentile: 4.0242085456848145
60th percentile: 4.077527141571045
70th percentile: 4.130845737457276
80th percentile: 4.197286939620971
90th percentile: 4.276850748062134
95th percentile: 4.316632652282715
99th percentile: 4.34845817565918
mean time: 3.657670831680298
%s, retrying in %s seconds...
Received healthy response to inference request in 3.100484609603882s
Received healthy response to inference request in 3.9845995903015137s
Received healthy response to inference request in 2.6489686965942383s
Received healthy response to inference request in 3.5723888874053955s
Received healthy response to inference request in 3.775789976119995s
5 requests
0 failed requests
5th percentile: 2.739271879196167
10th percentile: 2.8295750617980957
20th percentile: 3.010181427001953
30th percentile: 3.1948654651641846
40th percentile: 3.38362717628479
50th percentile: 3.5723888874053955
60th percentile: 3.6537493228912354
70th percentile: 3.7351097583770754
80th percentile: 3.8175518989562987
90th percentile: 3.901075744628906
95th percentile: 3.94283766746521
99th percentile: 3.976247205734253
mean time: 3.416446352005005
%s, retrying in %s seconds...
Received healthy response to inference request in 1.9644451141357422s
Received healthy response to inference request in 4.818966865539551s
Received healthy response to inference request in 3.1059153079986572s
Received healthy response to inference request in 4.280075788497925s
Received healthy response to inference request in 3.4731316566467285s
5 requests
0 failed requests
5th percentile: 2.192739152908325
10th percentile: 2.421033191680908
20th percentile: 2.8776212692260743
30th percentile: 3.1793585777282716
40th percentile: 3.3262451171875
50th percentile: 3.4731316566467285
60th percentile: 3.795909309387207
70th percentile: 4.118686962127685
80th percentile: 4.38785400390625
90th percentile: 4.603410434722901
95th percentile: 4.711188650131225
99th percentile: 4.797411222457885
mean time: 3.528506946563721
clean up pipeline due to error=DeploymentChecksError('Unacceptable 70th percentile latency 4.118686962127685s')
Shutdown handler de-registered
alexdaoud-trainer-bagir-_2925_v1 status is now failed due to DeploymentManager action