pawankrd-cosmosrp-12b

developer_uid: PawanOsman

submission_id: pawankrd-cosmosrp-12b_v2

model_name: pawankrd-cosmosrp-12b_v2

model_group: PawanKrd/CosmosRP-12b

status: torndown

timestamp: 2024-09-24T18:13:52+00:00

num_battles: 4711

num_wins: 2449

celo_rating: 1266.62

family_friendly_score: 0.0

submission_type: basic

model_repo: PawanKrd/CosmosRP-12b

model_architecture: MistralForCausalLM

model_num_parameters: 12772070400.0

best_of: 8

max_input_tokens: 512

max_output_tokens: 64

latencies: [{'batch_size': 1, 'throughput': 0.6980635059768854, 'latency_mean': 1.432439887523651, 'latency_p50': 1.431013822555542, 'latency_p90': 1.6030401706695556}, {'batch_size': 3, 'throughput': 1.3496556202979684, 'latency_mean': 2.215686005353928, 'latency_p50': 2.211166262626648, 'latency_p90': 2.4196743249893187}, {'batch_size': 5, 'throughput': 1.5749849357451406, 'latency_mean': 3.162906334400177, 'latency_p50': 3.1640595197677612, 'latency_p90': 3.5771309614181517}, {'batch_size': 6, 'throughput': 1.635778170275851, 'latency_mean': 3.6397255384922027, 'latency_p50': 3.5948961973190308, 'latency_p90': 4.171315741539002}, {'batch_size': 8, 'throughput': 1.6106562664982258, 'latency_mean': 4.94255651473999, 'latency_p50': 4.945273041725159, 'latency_p90': 5.54102258682251}, {'batch_size': 10, 'throughput': 1.5583437310127777, 'latency_mean': 6.37179181933403, 'latency_p50': 6.47674834728241, 'latency_p90': 7.217730450630188}]

gpu_counts: {'NVIDIA RTX A5000': 1}

display_name: pawankrd-cosmosrp-12b_v2

ineligible_reason: num_battles<5000

is_internal_developer: False

language_model: PawanKrd/CosmosRP-12b

model_size: 13B

ranking_group: single

throughput_3p7s: 1.65

us_pacific_date: 2024-09-24

win_ratio: 0.5198471662067502

generation_params: {'temperature': 0.7, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<', '>'], 'max_input_tokens': 512, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name pawankrd-cosmosrp-12b-v2-mkmlizer
Waiting for job on pawankrd-cosmosrp-12b-v2-mkmlizer to finish
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v11: ('http://zonemercy-vingt-deux-v2-1e5-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v11: ('http://zonemercy-vingt-deux-v2-1e5-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
pawankrd-cosmosrp-12b-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
pawankrd-cosmosrp-12b-v2-mkmlizer: ║     _____            __           __                                ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ║       /___/                                                         ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ║                                                                     ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ║  Version: 0.10.1                                                    ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ║  https://mk1.ai                                                     ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ║                                                                     ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ║  belonging to:                                                      ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ║                                                                     ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ║  Chai Research Corp.                                                ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ║                                                                     ║
pawankrd-cosmosrp-12b-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission chaiml-chaiml-sft-horror_v2: ('http://chaiml-chaiml-sft-horror-v2-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:60742->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission chaiml-elo-alignment-run-3_v44: ('http://chaiml-elo-alignment-run-3-v44-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:60436->127.0.0.1:8080: read: connection reset by peer\n')
pawankrd-cosmosrp-12b-v2-mkmlizer: Downloaded to shared memory in 49.362s
pawankrd-cosmosrp-12b-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp5yygktur, device:0
pawankrd-cosmosrp-12b-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v11: ('http://zonemercy-vingt-deux-v2-1e5-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v11: ('http://zonemercy-vingt-deux-v2-1e5-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v2: ('http://zonemercy-vingt-deux-v2-1e5-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
pawankrd-cosmosrp-12b-v2-mkmlizer: quantized model in 35.300s
pawankrd-cosmosrp-12b-v2-mkmlizer: Processed model PawanKrd/CosmosRP-12b in 84.662s
pawankrd-cosmosrp-12b-v2-mkmlizer: creating bucket guanaco-mkml-models
pawankrd-cosmosrp-12b-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
pawankrd-cosmosrp-12b-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v2
pawankrd-cosmosrp-12b-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v2/config.json
pawankrd-cosmosrp-12b-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v2/special_tokens_map.json
pawankrd-cosmosrp-12b-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v2/tokenizer_config.json
pawankrd-cosmosrp-12b-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v2/tokenizer.json
pawankrd-cosmosrp-12b-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v2/flywheel_model.0.safetensors
pawankrd-cosmosrp-12b-v2-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:10, 33.02it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 55.16it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:06, 52.52it/s]
Loading 0:   7%|▋         | 25/363 [00:00<00:06, 53.21it/s]
Loading 0:   9%|▉         | 32/363 [00:00<00:06, 48.27it/s]
Loading 0:  11%|█         | 40/363 [00:00<00:05, 56.72it/s]
Loading 0:  13%|█▎        | 46/363 [00:00<00:05, 54.16it/s]
Loading 0:  14%|█▍        | 52/363 [00:00<00:05, 53.92it/s]
Loading 0:  16%|█▌        | 58/363 [00:01<00:05, 54.98it/s]
Loading 0:  18%|█▊        | 64/363 [00:01<00:09, 32.95it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:07, 41.44it/s]
Loading 0:  21%|██▏       | 78/363 [00:01<00:06, 42.19it/s]
Loading 0:  23%|██▎       | 84/363 [00:01<00:06, 39.95it/s]
Loading 0:  25%|██▍       | 90/363 [00:01<00:06, 43.48it/s]
Loading 0:  26%|██▌       | 95/363 [00:02<00:06, 44.51it/s]
Loading 0:  28%|██▊       | 100/363 [00:02<00:06, 37.69it/s]
Loading 0:  30%|██▉       | 108/363 [00:02<00:05, 47.00it/s]
Loading 0:  31%|███▏      | 114/363 [00:02<00:05, 42.47it/s]
Loading 0:  33%|███▎      | 119/363 [00:02<00:05, 41.51it/s]
Loading 0:  35%|███▍      | 126/363 [00:02<00:05, 45.71it/s]
Loading 0:  36%|███▋      | 132/363 [00:02<00:05, 44.75it/s]
Loading 0:  38%|███▊      | 137/363 [00:03<00:05, 43.44it/s]
Loading 0:  39%|███▉      | 142/363 [00:03<00:06, 33.09it/s]
Loading 0:  40%|████      | 146/363 [00:03<00:06, 34.16it/s]
Loading 0:  41%|████▏     | 150/363 [00:03<00:06, 32.75it/s]
Loading 0:  43%|████▎     | 156/363 [00:03<00:05, 37.82it/s]
Loading 0:  44%|████▍     | 161/363 [00:03<00:05, 40.00it/s]
Loading 0:  46%|████▌     | 166/363 [00:03<00:04, 42.45it/s]
Loading 0:  47%|████▋     | 172/363 [00:04<00:04, 42.67it/s]
Loading 0:  49%|████▉     | 177/363 [00:04<00:04, 42.65it/s]
Loading 0:  50%|█████     | 183/363 [00:04<00:03, 46.81it/s]
Loading 0:  52%|█████▏    | 188/363 [00:04<00:03, 44.60it/s]
Loading 0:  53%|█████▎    | 193/363 [00:04<00:03, 43.55it/s]
Loading 0:  55%|█████▍    | 198/363 [00:04<00:03, 44.30it/s]
Loading 0:  56%|█████▌    | 203/363 [00:04<00:04, 35.74it/s]
Loading 0:  58%|█████▊    | 211/363 [00:04<00:03, 44.35it/s]
Loading 0:  60%|█████▉    | 217/363 [00:05<00:03, 44.10it/s]
Loading 0:  61%|██████▏   | 223/363 [00:05<00:04, 34.96it/s]
Loading 0:  63%|██████▎   | 227/363 [00:05<00:03, 35.72it/s]
Loading 0:  64%|██████▎   | 231/363 [00:05<00:03, 35.21it/s]
Loading 0:  65%|██████▌   | 237/363 [00:05<00:03, 40.80it/s]
Loading 0:  67%|██████▋   | 242/363 [00:05<00:02, 42.30it/s]
Loading 0:  68%|██████▊   | 248/363 [00:05<00:02, 40.13it/s]
Loading 0:  71%|███████   | 256/363 [00:06<00:02, 48.03it/s]
Loading 0:  72%|███████▏  | 262/363 [00:06<00:02, 41.90it/s]
Loading 0:  74%|███████▎  | 267/363 [00:06<00:02, 42.04it/s]
Loading 0:  75%|███████▌  | 274/363 [00:06<00:01, 46.50it/s]
Loading 0:  77%|███████▋  | 280/363 [00:06<00:01, 46.49it/s]
Loading 0:  79%|███████▊  | 285/363 [00:06<00:01, 45.00it/s]
Loading 0:  80%|████████  | 292/363 [00:06<00:01, 49.33it/s]
Loading 0:  82%|████████▏ | 298/363 [00:06<00:01, 48.56it/s]
Loading 0:  84%|████████▎ | 304/363 [00:13<00:20,  2.85it/s]
Loading 0:  85%|████████▍ | 308/363 [00:13<00:15,  3.58it/s]
Loading 0:  86%|████████▌ | 312/363 [00:13<00:11,  4.54it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:05,  7.33it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:03,  9.79it/s]
Loading 0:  91%|█████████ | 331/363 [00:14<00:02, 12.39it/s]
Loading 0:  93%|█████████▎| 339/363 [00:14<00:01, 17.26it/s]
Loading 0:  96%|█████████▌| 347/363 [00:14<00:00, 23.79it/s]
Loading 0:  97%|█████████▋| 353/363 [00:14<00:00, 27.61it/s]
Loading 0:  99%|█████████▉| 359/363 [00:14<00:00, 31.52it/s]
                                                            
Job pawankrd-cosmosrp-12b-v2-mkmlizer completed after 115.74s with status: succeeded
Stopping job with name pawankrd-cosmosrp-12b-v2-mkmlizer
Pipeline stage MKMLizer completed in 116.95s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.52s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service pawankrd-cosmosrp-12b-v2
Waiting for inference service pawankrd-cosmosrp-12b-v2 to be ready
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v2: ('http://zonemercy-vingt-deux-v2-1e5-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v11: ('http://zonemercy-vingt-deux-v2-1e5-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v2: ('http://zonemercy-vingt-deux-v2-1e5-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v11: ('http://zonemercy-vingt-deux-v2-1e5-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v11: ('http://zonemercy-vingt-deux-v2-1e5-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v2: ('http://zonemercy-vingt-deux-v2-1e5-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v2: ('http://zonemercy-vingt-deux-v2-1e5-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v11: ('http://zonemercy-vingt-deux-v2-1e5-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v2: ('http://zonemercy-vingt-deux-v2-1e5-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v11: ('http://zonemercy-vingt-deux-v2-1e5-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v2: ('http://zonemercy-vingt-deux-v2-1e5-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission blend_pudib_2024-09-20: ('http://mistralai-mistral-small-5341-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:58644->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v2: ('http://zonemercy-vingt-deux-v2-1e5-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Inference service pawankrd-cosmosrp-12b-v2 ready after 212.49027585983276s
Pipeline stage MKMLDeployer completed in 213.48s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.334132671356201s
Received healthy response to inference request in 1.6902799606323242s
Received healthy response to inference request in 1.758122205734253s
Received healthy response to inference request in 1.6735506057739258s
Received healthy response to inference request in 1.6448566913604736s
5 requests
0 failed requests
5th percentile: 1.650595474243164
10th percentile: 1.6563342571258546
20th percentile: 1.6678118228912353
30th percentile: 1.6768964767456054
40th percentile: 1.6835882186889648
50th percentile: 1.6902799606323242
60th percentile: 1.7174168586730958
70th percentile: 1.7445537567138671
80th percentile: 1.8733242988586427
90th percentile: 2.103728485107422
95th percentile: 2.2189305782318116
99th percentile: 2.3110922527313233
mean time: 1.8201884269714355
Pipeline stage StressChecker completed in 9.84s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Pipeline stage TriggerMKMLProfilingPipeline completed in 4.95s
Shutdown handler de-registered
pawankrd-cosmosrp-12b_v2 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.13s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service pawankrd-cosmosrp-12b-v2-profiler
Waiting for inference service pawankrd-cosmosrp-12b-v2-profiler to be ready
Inference service pawankrd-cosmosrp-12b-v2-profiler ready after 210.47888326644897s
Pipeline stage MKMLProfilerDeployer completed in 210.88s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/pawankrd-cosmosrp-12b-v2-profiler-predictor-00001-deploymebls9c:/code/chaiverse_profiler_1727202234 --namespace tenant-chaiml-guanaco
kubectl exec -it pawankrd-cosmosrp-12b-v2-profiler-predictor-00001-deploymebls9c --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1727202234 && python profiles.py profile --best_of_n 8 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1727202234/summary.json'
kubectl exec -it pawankrd-cosmosrp-12b-v2-profiler-predictor-00001-deploymebls9c --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1727202234/summary.json'
Pipeline stage MKMLProfilerRunner completed in 940.65s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service pawankrd-cosmosrp-12b-v2-profiler is running
Tearing down inference service pawankrd-cosmosrp-12b-v2-profiler
Service pawankrd-cosmosrp-12b-v2-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.51s
Shutdown handler de-registered
pawankrd-cosmosrp-12b_v2 status is now inactive due to auto deactivation removed underperforming models
run pipeline %s
admin requested tearing down of pawankrd-cosmosrp-12b_v2
run pipeline stage %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Running pipeline stage MKMLDeleter
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError("module 'kubernetes.config' has no attribute 'load_kube_config'")
Shutdown handler de-registered
pawankrd-cosmosrp-12b_v2 status is now torndown due to DeploymentManager action