arushimgupta-final-check_3580

submission_id: arushimgupta-final-check_3580_v2
developer_uid: immaculate_possum_03470
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 0.95, 'top_p': 1.0, 'min_p': 0.075, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|eot_id|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
model_name: nemo_base_1
model_repo: arushimgupta/final_checkpoint_dpo2
status: torndown
timestamp: 2024-09-27T23:16:14+00:00
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name arushimgupta-final-check-3580-v2-mkmlizer
Waiting for job on arushimgupta-final-check-3580-v2-mkmlizer to finish
arushimgupta-final-check-3580-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
arushimgupta-final-check-3580-v2-mkmlizer: ║     _____            __           __                                ║
arushimgupta-final-check-3580-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
arushimgupta-final-check-3580-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
arushimgupta-final-check-3580-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
arushimgupta-final-check-3580-v2-mkmlizer: ║       /___/                                                         ║
arushimgupta-final-check-3580-v2-mkmlizer: ║                                                                     ║
arushimgupta-final-check-3580-v2-mkmlizer: ║  Version: 0.11.12                                                   ║
arushimgupta-final-check-3580-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
arushimgupta-final-check-3580-v2-mkmlizer: ║  https://mk1.ai                                                     ║
arushimgupta-final-check-3580-v2-mkmlizer: ║                                                                     ║
arushimgupta-final-check-3580-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
arushimgupta-final-check-3580-v2-mkmlizer: ║  belonging to:                                                      ║
arushimgupta-final-check-3580-v2-mkmlizer: ║                                                                     ║
arushimgupta-final-check-3580-v2-mkmlizer: ║  Chai Research Corp.                                                ║
arushimgupta-final-check-3580-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
arushimgupta-final-check-3580-v2-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
arushimgupta-final-check-3580-v2-mkmlizer: ║                                                                     ║
arushimgupta-final-check-3580-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
arushimgupta-final-check-3580-v2-mkmlizer: Downloaded to shared memory in 32.625s
arushimgupta-final-check-3580-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpnsu62xnq, device:0
arushimgupta-final-check-3580-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
arushimgupta-final-check-3580-v2-mkmlizer: quantized model in 42.982s
arushimgupta-final-check-3580-v2-mkmlizer: Processed model arushimgupta/final_checkpoint_dpo2 in 75.607s
arushimgupta-final-check-3580-v2-mkmlizer: creating bucket guanaco-mkml-models
arushimgupta-final-check-3580-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
arushimgupta-final-check-3580-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/arushimgupta-final-check-3580-v2
arushimgupta-final-check-3580-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/arushimgupta-final-check-3580-v2/config.json
arushimgupta-final-check-3580-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/arushimgupta-final-check-3580-v2/special_tokens_map.json
arushimgupta-final-check-3580-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/arushimgupta-final-check-3580-v2/tokenizer_config.json
arushimgupta-final-check-3580-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/arushimgupta-final-check-3580-v2/tokenizer.json
arushimgupta-final-check-3580-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/arushimgupta-final-check-3580-v2/flywheel_model.0.safetensors
arushimgupta-final-check-3580-v2-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:12, 29.20it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:07, 48.08it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 43.69it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:07, 42.66it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 48.89it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:07, 45.34it/s]
Loading 0:  12%|█▏        | 42/363 [00:00<00:07, 43.44it/s]
Loading 0:  13%|█▎        | 48/363 [00:01<00:06, 47.01it/s]
Loading 0:  15%|█▍        | 53/363 [00:01<00:06, 45.38it/s]
Loading 0:  16%|█▋        | 59/363 [00:01<00:06, 48.58it/s]
Loading 0:  18%|█▊        | 64/363 [00:01<00:10, 27.81it/s]
Loading 0:  20%|█▉        | 71/363 [00:01<00:08, 34.68it/s]
Loading 0:  21%|██        | 76/363 [00:01<00:07, 36.87it/s]
Loading 0:  22%|██▏       | 81/363 [00:02<00:07, 39.14it/s]
Loading 0:  24%|██▍       | 87/363 [00:02<00:07, 38.72it/s]
Loading 0:  25%|██▌       | 92/363 [00:02<00:06, 39.49it/s]
Loading 0:  27%|██▋       | 98/363 [00:02<00:06, 44.09it/s]
Loading 0:  28%|██▊       | 103/363 [00:02<00:05, 43.39it/s]
Loading 0:  30%|███       | 109/363 [00:02<00:05, 47.16it/s]
Loading 0:  31%|███▏      | 114/363 [00:02<00:06, 39.92it/s]
Loading 0:  33%|███▎      | 119/363 [00:02<00:06, 39.10it/s]
Loading 0:  35%|███▍      | 126/363 [00:03<00:05, 44.74it/s]
Loading 0:  36%|███▋      | 132/363 [00:03<00:05, 42.49it/s]
Loading 0:  38%|███▊      | 137/363 [00:03<00:05, 40.35it/s]
Loading 0:  39%|███▉      | 142/363 [00:03<00:07, 30.90it/s]
Loading 0:  40%|████      | 146/363 [00:03<00:06, 31.74it/s]
Loading 0:  41%|████▏     | 150/363 [00:03<00:07, 30.36it/s]
Loading 0:  43%|████▎     | 156/363 [00:03<00:05, 36.44it/s]
Loading 0:  44%|████▍     | 160/363 [00:04<00:05, 35.66it/s]
Loading 0:  45%|████▌     | 165/363 [00:04<00:05, 37.99it/s]
Loading 0:  47%|████▋     | 169/363 [00:04<00:05, 36.65it/s]
Loading 0:  48%|████▊     | 174/363 [00:04<00:04, 39.86it/s]
Loading 0:  49%|████▉     | 179/363 [00:04<00:04, 39.98it/s]
Loading 0:  51%|█████     | 184/363 [00:04<00:04, 40.12it/s]
Loading 0:  52%|█████▏    | 189/363 [00:04<00:04, 41.58it/s]
Loading 0:  53%|█████▎    | 194/363 [00:04<00:04, 34.78it/s]
Loading 0:  55%|█████▌    | 201/363 [00:05<00:03, 41.61it/s]
Loading 0:  57%|█████▋    | 206/363 [00:05<00:03, 41.21it/s]
Loading 0:  58%|█████▊    | 211/363 [00:05<00:03, 40.07it/s]
Loading 0:  60%|█████▉    | 216/363 [00:05<00:03, 41.77it/s]
Loading 0:  61%|██████    | 221/363 [00:05<00:03, 42.82it/s]
Loading 0:  62%|██████▏   | 226/363 [00:05<00:05, 25.44it/s]
Loading 0:  63%|██████▎   | 230/363 [00:06<00:05, 25.95it/s]
Loading 0:  65%|██████▌   | 237/363 [00:06<00:03, 33.79it/s]
Loading 0:  67%|██████▋   | 242/363 [00:06<00:03, 35.29it/s]
Loading 0:  68%|██████▊   | 247/363 [00:06<00:03, 36.42it/s]
Loading 0:  69%|██████▉   | 252/363 [00:06<00:02, 39.26it/s]
Loading 0:  71%|███████   | 257/363 [00:06<00:03, 33.78it/s]
Loading 0:  73%|███████▎  | 264/363 [00:06<00:02, 41.35it/s]
Loading 0:  74%|███████▍  | 269/363 [00:06<00:02, 41.67it/s]
Loading 0:  75%|███████▌  | 274/363 [00:07<00:02, 41.42it/s]
Loading 0:  77%|███████▋  | 279/363 [00:07<00:01, 42.99it/s]
Loading 0:  78%|███████▊  | 284/363 [00:07<00:02, 36.18it/s]
Loading 0:  80%|████████  | 291/363 [00:07<00:01, 42.42it/s]
Loading 0:  82%|████████▏ | 296/363 [00:07<00:01, 41.48it/s]
Loading 0:  83%|████████▎ | 301/363 [00:07<00:01, 43.41it/s]
Loading 0:  84%|████████▍ | 306/363 [00:15<00:24,  2.30it/s]
Loading 0:  85%|████████▌ | 310/363 [00:15<00:17,  2.98it/s]
Loading 0:  87%|████████▋ | 314/363 [00:15<00:12,  3.92it/s]
Loading 0:  88%|████████▊ | 320/363 [00:15<00:07,  5.84it/s]
Loading 0:  90%|████████▉ | 325/363 [00:15<00:04,  7.93it/s]
Loading 0:  91%|█████████ | 330/363 [00:15<00:03,  9.95it/s]
Loading 0:  93%|█████████▎| 338/363 [00:15<00:01, 15.18it/s]
Loading 0:  95%|█████████▍| 344/363 [00:15<00:01, 18.58it/s]
Loading 0:  96%|█████████▌| 349/363 [00:16<00:00, 21.55it/s]
Loading 0:  98%|█████████▊| 355/363 [00:16<00:00, 26.85it/s]
Loading 0:  99%|█████████▉| 360/363 [00:16<00:00, 29.86it/s]
                                                            
Job arushimgupta-final-check-3580-v2-mkmlizer completed after 102.46s with status: succeeded
Stopping job with name arushimgupta-final-check-3580-v2-mkmlizer
Pipeline stage MKMLizer completed in 102.77s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.07s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service arushimgupta-final-check-3580-v2
Waiting for inference service arushimgupta-final-check-3580-v2 to be ready
admin requested tearing down of chaiml-llama-8b-pairwis_8189_v28
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-llama-8b-pairwis-8189-v28 is running
Tearing down inference service chaiml-llama-8b-pairwis-8189-v28
Service chaiml-llama-8b-pairwis-8189-v28 has been torndown
Pipeline stage MKMLDeleter completed in 2.37s
run pipeline stage %s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key chaiml-llama-8b-pairwis-8189-v28/config.json from bucket guanaco-mkml-models
Deleting key chaiml-llama-8b-pairwis-8189-v28/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key chaiml-llama-8b-pairwis-8189-v28/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key chaiml-llama-8b-pairwis-8189-v28/tokenizer.json from bucket guanaco-mkml-models
Deleting key chaiml-llama-8b-pairwis-8189-v28/tokenizer_config.json from bucket guanaco-mkml-models
Pipeline stage MKMLModelDeleter completed in 1.43s
Shutdown handler de-registered
chaiml-llama-8b-pairwis_8189_v28 status is now torndown due to DeploymentManager action
Inference service arushimgupta-final-check-3580-v2 ready after 220.49695801734924s
Pipeline stage MKMLDeployer completed in 220.74s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.963096857070923s
Received healthy response to inference request in 2.1939280033111572s
HTTPSConnectionPool(host='guanaco-submitter.chai-research.com', port=443): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.7269487380981445s
Received healthy response to inference request in 2.4336016178131104s
5 requests
1 failed requests
5th percentile: 2.241862726211548
10th percentile: 2.2897974491119384
20th percentile: 2.3856668949127195
30th percentile: 2.4922710418701173
40th percentile: 2.6096098899841307
50th percentile: 2.7269487380981445
60th percentile: 2.821407985687256
70th percentile: 2.915867233276367
80th percentile: 6.3879447460174585
90th percentile: 13.237640523910525
95th percentile: 16.66248841285705
99th percentile: 19.402366724014282
mean time: 6.080982303619384
%s, retrying in %s seconds...
Received healthy response to inference request in 15.379749536514282s
Received healthy response to inference request in 2.582622766494751s
Received healthy response to inference request in 2.2022945880889893s
HTTPSConnectionPool(host='guanaco-submitter.chai-research.com', port=443): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.464167833328247s
5 requests
1 failed requests
5th percentile: 2.2546692371368406
10th percentile: 2.3070438861846925
20th percentile: 2.4117931842803957
30th percentile: 2.4878588199615477
40th percentile: 2.5352407932281493
50th percentile: 2.582622766494751
60th percentile: 7.701473474502563
70th percentile: 12.820324182510374
80th percentile: 16.316614055633547
90th percentile: 18.19034309387207
95th percentile: 19.12720761299133
99th percentile: 19.876699228286743
mean time: 8.538581371307373
%s, retrying in %s seconds...
HTTPSConnectionPool(host='guanaco-submitter.chai-research.com', port=443): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.7885534763336182s
Received healthy response to inference request in 2.1746981143951416s
Received healthy response to inference request in 2.382127523422241s
Received healthy response to inference request in 2.887272834777832s
5 requests
1 failed requests
5th percentile: 1.8657824039459228
10th percentile: 1.9430113315582276
20th percentile: 2.097469186782837
30th percentile: 2.2161839962005616
40th percentile: 2.299155759811401
50th percentile: 2.382127523422241
60th percentile: 2.5841856479644774
70th percentile: 2.786243772506714
80th percentile: 6.325284814834598
90th percentile: 13.20130877494812
95th percentile: 16.63932075500488
99th percentile: 19.389730339050292
mean time: 5.8619969367980955
clean up pipeline due to error=DeploymentChecksError('Unacceptable number of predict errors: 20.0%')
Shutdown handler de-registered
arushimgupta-final-check_3580_v2 status is now failed due to DeploymentManager action
arushimgupta-final-check_3580_v2 status is now torndown due to DeploymentManager action