nischaydnk-exp14-exp9d_94555

developer_uid: chai_backend_admin
submission_id: nischaydnk-exp14-exp9d_94555_v14
model_name: nischaydnk-exp14-exp9d_94555_v14
model_group: NischayDnk/exp14-exp9DPO
status: torndown
timestamp: 2025-06-24T19:24:30+00:00
num_battles: 8631
num_wins: 4333
celo_rating: 1290.05
family_friendly_score: 0.5252
family_friendly_standard_error: 0.007062081279622884
submission_type: basic
model_repo: NischayDnk/exp14-exp9DPOft-Chaidatav1dpo6k-Mistral24B
model_architecture: MistralForCausalLM
model_num_parameters: 24096691200.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
reward_model: default
latencies: [{'batch_size': 1, 'throughput': 0.40720333814537746, 'latency_mean': 2.455649870634079, 'latency_p50': 2.413457751274109, 'latency_p90': 2.692944550514221}, {'batch_size': 2, 'throughput': 0.6007493462590632, 'latency_mean': 3.314390625953674, 'latency_p50': 3.303214907646179, 'latency_p90': 3.6797152519226075}, {'batch_size': 3, 'throughput': 0.715218464352699, 'latency_mean': 4.174315402507782, 'latency_p50': 4.16531229019165, 'latency_p90': 4.64607264995575}, {'batch_size': 4, 'throughput': 0.8384324490539405, 'latency_mean': 4.7480187833309175, 'latency_p50': 4.910488605499268, 'latency_p90': 5.5362214803695675}, {'batch_size': 5, 'throughput': 0.8583629012531158, 'latency_mean': 5.793382252454758, 'latency_p50': 5.852248430252075, 'latency_p90': 6.604658913612366}]
gpu_counts: {'NVIDIA A100-SXM4-80GB': 1}
display_name: nischaydnk-exp14-exp9d_94555_v14
is_internal_developer: True
language_model: NischayDnk/exp14-exp9DPOft-Chaidatav1dpo6k-Mistral24B
model_size: 24B
ranking_group: single
throughput_3p7s: 0.66
us_pacific_date: 2025-06-24
win_ratio: 0.5020275750202757
generation_params: {'temperature': 0.45, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.45, 'frequency_penalty': 0.45, 'stopping_words': ['###', '</s>', '\n', 'You:'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '[INST]', 'prompt_template': "''", 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{user_name}: {message}<|im_end|>\n', 'response_template': '[/INST]{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name nischaydnk-exp14-exp9d-94555-v14-mkmlizer
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name nischaydnk-exp14-exp9d-94555-v14-mkmlizer
Stopping job with name nischaydnk-exp14-exp9d-94555-v14-mkmlizer
%s, retrying in %s seconds...
Starting job with name nischaydnk-exp14-exp9d-94555-v14-mkmlizer
Waiting for job on nischaydnk-exp14-exp9d-94555-v14-mkmlizer to finish
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: Downloaded to shared memory in 116.122s
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: Checking if NischayDnk/exp14-exp9DPOft-Chaidatav1dpo6k-Mistral24B already exists in ChaiML
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpchansgm6, device:0
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: Saving flywheel model at /dev/shm/model_cache
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: quantized model in 48.430s
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: Processed model NischayDnk/exp14-exp9DPOft-Chaidatav1dpo6k-Mistral24B in 164.634s
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: creating bucket guanaco-mkml-models
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v14
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v14/config.json
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v14/special_tokens_map.json
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v14/tokenizer_config.json
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v14/tokenizer.json
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v14/flywheel_model.1.safetensors
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v14/flywheel_model.0.safetensors
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 4/363 [00:00<00:09, 35.94it/s]
Loading 0:   2%|▏         | 8/363 [00:00<00:12, 28.59it/s]
Loading 0:   3%|▎         | 12/363 [00:00<00:11, 29.89it/s]
Loading 0:   4%|▍         | 16/363 [00:00<00:13, 26.44it/s]
Loading 0:   6%|▌         | 21/363 [00:00<00:11, 30.15it/s]
Loading 0:   7%|▋         | 25/363 [00:00<00:12, 27.11it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:09, 34.64it/s]
Loading 0:  10%|▉         | 35/363 [00:01<00:16, 20.32it/s]
Loading 0:  10%|█         | 38/363 [00:01<00:15, 21.01it/s]
Loading 0:  11%|█▏        | 41/363 [00:01<00:18, 17.60it/s]
Loading 0:  13%|█▎        | 46/363 [00:01<00:13, 22.76it/s]
Loading 0:  13%|█▎        | 49/363 [00:01<00:13, 24.03it/s]
Loading 0:  14%|█▍        | 52/363 [00:02<00:13, 23.78it/s]
Loading 0:  16%|█▌        | 57/363 [00:02<00:11, 26.62it/s]
Loading 0:  17%|█▋        | 60/363 [00:02<00:12, 23.40it/s]
Loading 0:  18%|█▊        | 65/363 [00:02<00:11, 26.17it/s]
Loading 0:  19%|█▉        | 70/363 [00:02<00:13, 21.23it/s]
Loading 0:  20%|██        | 73/363 [00:03<00:15, 18.78it/s]
Loading 0:  21%|██▏       | 78/363 [00:03<00:11, 23.95it/s]
Loading 0:  22%|██▏       | 81/363 [00:03<00:13, 21.57it/s]
Loading 0:  23%|██▎       | 84/363 [00:03<00:12, 23.15it/s]
Loading 0:  24%|██▍       | 87/363 [00:03<00:13, 20.80it/s]
Loading 0:  25%|██▌       | 91/363 [00:03<00:11, 23.99it/s]
Loading 0:  26%|██▌       | 94/363 [00:03<00:11, 22.72it/s]
Loading 0:  27%|██▋       | 97/363 [00:04<00:11, 24.09it/s]
Loading 0:  28%|██▊       | 100/363 [00:04<00:10, 24.96it/s]
Loading 0:  28%|██▊       | 103/363 [00:04<00:10, 24.78it/s]
Loading 0:  29%|██▉       | 107/363 [00:04<00:13, 19.01it/s]
Loading 0:  31%|███       | 112/363 [00:04<00:10, 22.83it/s]
Loading 0:  32%|███▏      | 115/363 [00:04<00:10, 22.90it/s]
Loading 0:  33%|███▎      | 120/363 [00:05<00:09, 26.09it/s]
Loading 0:  34%|███▍      | 123/363 [00:05<00:10, 23.99it/s]
Loading 0:  36%|███▌      | 129/363 [00:05<00:08, 28.30it/s]
Loading 0:  36%|███▋      | 132/363 [00:05<00:09, 25.43it/s]
Loading 0:  38%|███▊      | 138/363 [00:05<00:07, 29.96it/s]
Loading 0:  39%|███▉      | 142/363 [00:05<00:07, 28.34it/s]
Loading 0:  41%|████      | 148/363 [00:05<00:06, 34.75it/s]
Loading 0:  42%|████▏     | 152/363 [00:06<00:09, 23.11it/s]
Loading 0:  43%|████▎     | 155/363 [00:06<00:08, 23.28it/s]
Loading 0:  44%|████▎     | 158/363 [00:06<00:10, 19.93it/s]
Loading 0:  45%|████▌     | 164/363 [00:06<00:07, 27.28it/s]
Loading 0:  46%|████▋     | 168/363 [00:06<00:08, 23.95it/s]
Loading 0:  48%|████▊     | 174/363 [00:07<00:06, 28.19it/s]
Loading 0:  49%|████▉     | 178/363 [00:07<00:06, 27.17it/s]
Loading 0:  50%|█████     | 182/363 [00:07<00:06, 27.42it/s]
Loading 0:  52%|█████▏    | 187/363 [00:07<00:07, 23.61it/s]
Loading 0:  52%|█████▏    | 190/363 [00:07<00:07, 22.37it/s]
Loading 0:  53%|█████▎    | 193/363 [00:07<00:07, 23.60it/s]
Loading 0:  54%|█████▍    | 196/363 [00:08<00:06, 24.23it/s]
Loading 0:  55%|█████▌    | 200/363 [00:22<00:06, 24.23it/s]
Loading 0:  55%|█████▌    | 201/363 [00:22<02:57,  1.09s/it]
Loading 0:  56%|█████▌    | 203/363 [00:22<02:27,  1.09it/s]
Loading 0:  57%|█████▋    | 208/363 [00:22<01:29,  1.74it/s]
Loading 0:  58%|█████▊    | 211/363 [00:22<01:08,  2.23it/s]
Loading 0:  59%|█████▉    | 214/363 [00:22<00:50,  2.93it/s]
Loading 0:  60%|██████    | 218/363 [00:23<00:34,  4.15it/s]
Loading 0:  61%|██████    | 221/363 [00:23<00:26,  5.28it/s]
Loading 0:  62%|██████▏   | 224/363 [00:23<00:22,  6.17it/s]
Loading 0:  63%|██████▎   | 228/363 [00:23<00:15,  8.56it/s]
Loading 0:  64%|██████▎   | 231/363 [00:23<00:13, 10.00it/s]
Loading 0:  65%|██████▍   | 235/363 [00:23<00:09, 13.31it/s]
Loading 0:  66%|██████▌   | 238/363 [00:23<00:08, 15.27it/s]
Loading 0:  66%|██████▋   | 241/363 [00:24<00:07, 17.21it/s]
Loading 0:  68%|██████▊   | 246/363 [00:24<00:05, 21.49it/s]
Loading 0:  69%|██████▊   | 249/363 [00:24<00:05, 20.73it/s]
Loading 0:  70%|███████   | 255/363 [00:24<00:04, 26.03it/s]
Loading 0:  71%|███████▏  | 259/363 [00:24<00:04, 25.47it/s]
Loading 0:  73%|███████▎  | 264/363 [00:24<00:03, 30.26it/s]
Loading 0:  74%|███████▍  | 268/363 [00:25<00:04, 22.51it/s]
Loading 0:  75%|███████▍  | 271/363 [00:25<00:04, 21.03it/s]
Loading 0:  75%|███████▌  | 274/363 [00:25<00:04, 22.23it/s]
Loading 0:  76%|███████▋  | 277/363 [00:25<00:03, 22.85it/s]
Loading 0:  78%|███████▊  | 282/363 [00:25<00:03, 25.94it/s]
Loading 0:  79%|███████▊  | 285/363 [00:25<00:03, 23.86it/s]
Loading 0:  80%|████████  | 291/363 [00:25<00:02, 28.96it/s]
Loading 0:  81%|████████▏ | 295/363 [00:26<00:02, 27.73it/s]
Loading 0:  82%|████████▏ | 299/363 [00:26<00:02, 28.02it/s]
Loading 0:  84%|████████▎ | 304/363 [00:26<00:02, 24.55it/s]
Loading 0:  85%|████████▍ | 307/363 [00:26<00:02, 23.15it/s]
Loading 0:  85%|████████▌ | 310/363 [00:26<00:02, 24.32it/s]
Loading 0:  86%|████████▌ | 313/363 [00:26<00:02, 24.81it/s]
Loading 0:  88%|████████▊ | 318/363 [00:27<00:01, 27.66it/s]
Loading 0:  88%|████████▊ | 321/363 [00:27<00:01, 25.16it/s]
Loading 0:  90%|█████████ | 327/363 [00:27<00:01, 30.15it/s]
Loading 0:  91%|█████████ | 331/363 [00:27<00:01, 28.50it/s]
Loading 0:  92%|█████████▏| 335/363 [00:27<00:00, 28.70it/s]
Loading 0:  93%|█████████▎| 338/363 [00:27<00:00, 26.83it/s]
Loading 0:  94%|█████████▍| 341/363 [00:28<00:01, 16.08it/s]
Loading 0:  96%|█████████▌| 347/363 [00:28<00:00, 21.90it/s]
Loading 0:  96%|█████████▋| 350/363 [00:28<00:00, 22.72it/s]
Loading 0:  97%|█████████▋| 353/363 [00:28<00:00, 23.87it/s]
Loading 0:  98%|█████████▊| 357/363 [00:28<00:00, 21.72it/s]
Loading 0: 100%|█████████▉| 362/363 [00:28<00:00, 27.27it/s]
                                                            
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║                                                                     ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║                                                                     ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║  Version: 0.29.3                                                    ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║  https://mk1.ai                                                     ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║                                                                     ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║  The license key for the current software has been verified as      ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║  belonging to:                                                      ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║                                                                     ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║  Chai Research Corp.                                                ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ║                                                                     ║
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission chaiml-giyu-disabled-hu_13897_v2: ('http://chaiml-giyu-disabled-hu-13897-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: Downloaded to shared memory in 56.966s
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: Checking if NischayDnk/exp14-exp9DPOft-Chaidatav1dpo6k-Mistral24B already exists in ChaiML
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp6_bgzf4c, device:0
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: Saving flywheel model at /dev/shm/model_cache
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: quantized model in 50.060s
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: Processed model NischayDnk/exp14-exp9DPOft-Chaidatav1dpo6k-Mistral24B in 107.124s
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: creating bucket guanaco-mkml-models
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v14
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v14/config.json
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v14/special_tokens_map.json
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v14/tokenizer_config.json
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v14/tokenizer.json
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v14/flywheel_model.1.safetensors
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v14/flywheel_model.0.safetensors
nischaydnk-exp14-exp9d-94555-v14-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:17, 20.82it/s]
Loading 0:   3%|▎         | 12/363 [00:00<00:10, 34.47it/s]
Loading 0:   4%|▍         | 16/363 [00:00<00:11, 29.32it/s]
Loading 0:   6%|▌         | 21/363 [00:00<00:10, 32.09it/s]
Loading 0:   7%|▋         | 25/363 [00:00<00:11, 28.74it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:09, 35.93it/s]
Loading 0:  10%|▉         | 35/363 [00:01<00:14, 23.08it/s]
Loading 0:  11%|█         | 39/363 [00:01<00:14, 22.86it/s]
Loading 0:  12%|█▏        | 42/363 [00:01<00:15, 21.23it/s]
Loading 0:  13%|█▎        | 46/363 [00:01<00:12, 24.65it/s]
Loading 0:  14%|█▍        | 50/363 [00:01<00:14, 22.24it/s]
Loading 0:  15%|█▌        | 55/363 [00:02<00:11, 27.39it/s]
Loading 0:  16%|█▋        | 59/363 [00:02<00:12, 24.08it/s]
Loading 0:  18%|█▊        | 65/363 [00:02<00:10, 28.28it/s]
Loading 0:  19%|█▉        | 70/363 [00:02<00:12, 23.92it/s]
Loading 0:  20%|██        | 73/363 [00:02<00:14, 19.98it/s]
Loading 0:  22%|██▏       | 79/363 [00:03<00:11, 24.71it/s]
Loading 0:  23%|██▎       | 82/363 [00:03<00:11, 24.19it/s]
Loading 0:  24%|██▎       | 86/363 [00:03<00:10, 25.78it/s]
Loading 0:  25%|██▍       | 89/363 [00:03<00:10, 26.23it/s]
Loading 0:  25%|██▌       | 92/363 [00:03<00:13, 20.84it/s]
Loading 0:  27%|██▋       | 99/363 [00:03<00:09, 28.13it/s]
Loading 0:  28%|██▊       | 103/363 [00:04<00:09, 26.86it/s]
Loading 0:  29%|██▉       | 107/363 [00:04<00:11, 22.88it/s]
Loading 0:  31%|███       | 112/363 [00:04<00:09, 25.78it/s]
Loading 0:  32%|███▏      | 115/363 [00:04<00:09, 25.28it/s]
Loading 0:  33%|███▎      | 120/363 [00:04<00:08, 27.75it/s]
Loading 0:  34%|███▍      | 123/363 [00:04<00:09, 25.18it/s]
Loading 0:  36%|███▌      | 129/363 [00:04<00:07, 29.65it/s]
Loading 0:  37%|███▋      | 133/363 [00:05<00:08, 27.77it/s]
Loading 0:  38%|███▊      | 138/363 [00:05<00:07, 29.75it/s]
Loading 0:  39%|███▉      | 142/363 [00:05<00:07, 27.96it/s]
Loading 0:  40%|████      | 147/363 [00:05<00:06, 32.10it/s]
Loading 0:  42%|████▏     | 151/363 [00:05<00:09, 23.51it/s]
Loading 0:  42%|████▏     | 154/363 [00:06<00:09, 21.83it/s]
Loading 0:  43%|████▎     | 157/363 [00:06<00:08, 23.18it/s]
Loading 0:  44%|████▍     | 160/363 [00:06<00:08, 22.73it/s]
Loading 0:  45%|████▍     | 163/363 [00:06<00:08, 23.75it/s]
Loading 0:  46%|████▌     | 166/363 [00:06<00:07, 24.92it/s]
Loading 0:  47%|████▋     | 169/363 [00:06<00:08, 24.22it/s]
Loading 0:  48%|████▊     | 174/363 [00:06<00:07, 26.75it/s]
Loading 0:  49%|████▉     | 177/363 [00:06<00:07, 23.86it/s]
Loading 0:  50%|█████     | 182/363 [00:07<00:06, 26.70it/s]
Loading 0:  52%|█████▏    | 187/363 [00:07<00:07, 23.65it/s]
Loading 0:  52%|█████▏    | 190/363 [00:07<00:07, 22.03it/s]
Loading 0:  53%|█████▎    | 193/363 [00:07<00:07, 23.28it/s]
Loading 0:  54%|█████▍    | 196/363 [00:07<00:07, 23.28it/s]
Loading 0:  55%|█████▌    | 200/363 [00:23<00:07, 23.28it/s]
Loading 0:  55%|█████▌    | 201/363 [00:23<03:15,  1.21s/it]
Loading 0:  56%|█████▌    | 203/363 [00:23<02:41,  1.01s/it]
Loading 0:  57%|█████▋    | 208/363 [00:23<01:36,  1.60it/s]
Loading 0:  58%|█████▊    | 211/363 [00:23<01:13,  2.06it/s]
Loading 0:  59%|█████▉    | 214/363 [00:23<00:54,  2.71it/s]
Loading 0:  60%|██████    | 218/363 [00:23<00:37,  3.88it/s]
Loading 0:  61%|██████    | 221/363 [00:24<00:28,  4.97it/s]
Loading 0:  62%|██████▏   | 224/363 [00:24<00:23,  5.95it/s]
Loading 0:  63%|██████▎   | 228/363 [00:24<00:16,  8.39it/s]
Loading 0:  64%|██████▎   | 231/363 [00:24<00:13,  9.75it/s]
Loading 0:  65%|██████▌   | 237/363 [00:24<00:08, 14.48it/s]
Loading 0:  66%|██████▌   | 240/363 [00:24<00:08, 15.37it/s]
Loading 0:  68%|██████▊   | 246/363 [00:25<00:05, 20.57it/s]
Loading 0:  69%|██████▉   | 250/363 [00:25<00:05, 21.45it/s]
Loading 0:  70%|███████   | 255/363 [00:25<00:04, 24.51it/s]
Loading 0:  71%|███████▏  | 259/363 [00:25<00:04, 24.33it/s]
Loading 0:  73%|███████▎  | 264/363 [00:25<00:03, 29.17it/s]
Loading 0:  74%|███████▍  | 268/363 [00:25<00:04, 22.96it/s]
Loading 0:  75%|███████▍  | 271/363 [00:26<00:04, 21.94it/s]
Loading 0:  75%|███████▌  | 274/363 [00:26<00:03, 23.20it/s]
Loading 0:  76%|███████▋  | 277/363 [00:26<00:03, 24.02it/s]
Loading 0:  78%|███████▊  | 282/363 [00:26<00:02, 27.23it/s]
Loading 0:  79%|███████▊  | 285/363 [00:26<00:03, 24.69it/s]
Loading 0:  80%|████████  | 291/363 [00:26<00:02, 29.43it/s]
Loading 0:  81%|████████▏ | 295/363 [00:26<00:02, 27.55it/s]
Loading 0:  82%|████████▏ | 299/363 [00:27<00:02, 27.76it/s]
Loading 0:  84%|████████▎ | 304/363 [00:27<00:02, 24.70it/s]
Loading 0:  85%|████████▍ | 307/363 [00:27<00:02, 23.20it/s]
Loading 0:  85%|████████▌ | 310/363 [00:27<00:02, 24.04it/s]
Loading 0:  86%|████████▌ | 313/363 [00:27<00:02, 24.51it/s]
Loading 0:  88%|████████▊ | 318/363 [00:27<00:01, 27.18it/s]
Loading 0:  88%|████████▊ | 321/363 [00:27<00:01, 24.27it/s]
Loading 0:  90%|█████████ | 327/363 [00:28<00:01, 28.55it/s]
Loading 0:  91%|█████████ | 330/363 [00:28<00:01, 25.23it/s]
Loading 0:  92%|█████████▏| 335/363 [00:28<00:01, 27.36it/s]
Loading 0:  93%|█████████▎| 338/363 [00:28<00:00, 25.68it/s]
Loading 0:  94%|█████████▍| 341/363 [00:29<00:01, 14.37it/s]
Loading 0:  95%|█████████▌| 346/363 [00:29<00:00, 19.40it/s]
Loading 0:  96%|█████████▌| 349/363 [00:29<00:00, 18.83it/s]
Loading 0:  97%|█████████▋| 353/363 [00:29<00:00, 22.44it/s]
Loading 0:  98%|█████████▊| 356/363 [00:29<00:00, 23.94it/s]
Loading 0:  99%|█████████▉| 359/363 [00:29<00:00, 23.66it/s]
                                                            
Job nischaydnk-exp14-exp9d-94555-v14-mkmlizer completed after 342.94s with status: succeeded
Stopping job with name nischaydnk-exp14-exp9d-94555-v14-mkmlizer
Pipeline stage MKMLizer completed in 343.71s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service nischaydnk-exp14-exp9d-94555-v14
Ignoring service nischaydnk-exp14-exp9d-94555-v14 already deployed
Waiting for inference service nischaydnk-exp14-exp9d-94555-v14 to be ready
Inference service nischaydnk-exp14-exp9d-94555-v14 ready after 10.084222555160522s
Pipeline stage MKMLDeployer completed in 10.83s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.910032033920288s
Received healthy response to inference request in 2.120682954788208s
Received healthy response to inference request in 1.8121836185455322s
Received healthy response to inference request in 2.0092785358428955s
5 requests
1 failed requests
5th percentile: 1.8317533016204834
10th percentile: 1.8513229846954347
20th percentile: 1.8904623508453369
30th percentile: 1.9298813343048096
40th percentile: 1.9695799350738525
50th percentile: 2.0092785358428955
60th percentile: 2.0538403034210204
70th percentile: 2.0984020709991453
80th percentile: 5.726174116134647
90th percentile: 12.937156438827516
95th percentile: 16.542647600173947
99th percentile: 19.427040529251098
mean time: 5.600063180923462
%s, retrying in %s seconds...
Received healthy response to inference request in 1.7962541580200195s
Received healthy response to inference request in 1.8430492877960205s
Received healthy response to inference request in 1.9580707550048828s
Received healthy response to inference request in 1.8087255954742432s
Received healthy response to inference request in 2.0586061477661133s
5 requests
0 failed requests
5th percentile: 1.7987484455108642
10th percentile: 1.801242733001709
20th percentile: 1.8062313079833985
30th percentile: 1.8155903339385986
40th percentile: 1.8293198108673097
50th percentile: 1.8430492877960205
60th percentile: 1.8890578746795654
70th percentile: 1.9350664615631104
80th percentile: 1.978177833557129
90th percentile: 2.018391990661621
95th percentile: 2.0384990692138674
99th percentile: 2.0545847320556643
mean time: 1.8929411888122558
Pipeline stage StressChecker completed in 40.65s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.70s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.72s
Shutdown handler de-registered
nischaydnk-exp14-exp9d_94555_v14 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.18s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service nischaydnk-exp14-exp9d-94555-v14-profiler
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3552.71s
Shutdown handler de-registered
nischaydnk-exp14-exp9d_94555_v14 status is now inactive due to auto deactivation removed underperforming models
nischaydnk-exp14-exp9d_94555_v14 status is now torndown due to DeploymentManager action