marinaraspaghetti-nemomi_1739

developer_uid: azuruce
submission_id: marinaraspaghetti-nemomi_1739_v8
model_name: marinaraspaghetti-nemomi_1739_v8
model_group: MarinaraSpaghetti/NemoMi
status: torndown
timestamp: 2024-09-26T07:36:31+00:00
num_battles: 3148
num_wins: 1615
celo_rating: 1262.26
family_friendly_score: 0.5589561855670103
family_friendly_standard_error: 0.008884463431116714
submission_type: basic
model_repo: MarinaraSpaghetti/NemoMix-Unleashed-12B
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
latencies: [{'batch_size': 1, 'throughput': 0.6141517122271819, 'latency_mean': 1.628201938867569, 'latency_p50': 1.6192891597747803, 'latency_p90': 1.789552593231201}, {'batch_size': 3, 'throughput': 1.0825436821819343, 'latency_mean': 2.7623359155654907, 'latency_p50': 2.7704029083251953, 'latency_p90': 3.0660599946975706}, {'batch_size': 5, 'throughput': 1.2292520826626705, 'latency_mean': 4.047072145938873, 'latency_p50': 4.053423523902893, 'latency_p90': 4.554346299171447}, {'batch_size': 6, 'throughput': 1.261689391941661, 'latency_mean': 4.733972043991089, 'latency_p50': 4.695183157920837, 'latency_p90': 5.478676342964173}, {'batch_size': 8, 'throughput': 1.2340282305423682, 'latency_mean': 6.4461485338211055, 'latency_p50': 6.486296057701111, 'latency_p90': 7.424980497360229}, {'batch_size': 10, 'throughput': 1.2015365449993836, 'latency_mean': 8.273986248970031, 'latency_p50': 8.321347713470459, 'latency_p90': 9.259292173385619}]
gpu_counts: {'NVIDIA RTX A5000': 1}
display_name: marinaraspaghetti-nemomi_1739_v8
ineligible_reason: num_battles<5000
is_internal_developer: True
language_model: MarinaraSpaghetti/NemoMix-Unleashed-12B
model_size: 13B
ranking_group: single
throughput_3p7s: 1.21
us_pacific_date: 2024-09-26
win_ratio: 0.5130241423125794
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|eot_id|>', '<|end_of_text|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '### Response:\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name marinaraspaghetti-nemomi-1739-v8-mkmlizer
Waiting for job on marinaraspaghetti-nemomi-1739-v8-mkmlizer to finish
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║     _____            __           __                                ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║       /___/                                                         ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║                                                                     ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║  Version: 0.11.12                                                   ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║  https://mk1.ai                                                     ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║                                                                     ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║  The license key for the current software has been verified as      ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║  belonging to:                                                      ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║                                                                     ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║  Chai Research Corp.                                                ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ║                                                                     ║
marinaraspaghetti-nemomi-1739-v8-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
marinaraspaghetti-nemomi-1739-v8-mkmlizer: Downloaded to shared memory in 43.370s
marinaraspaghetti-nemomi-1739-v8-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpe1c9wmys, device:0
marinaraspaghetti-nemomi-1739-v8-mkmlizer: Saving flywheel model at /dev/shm/model_cache
marinaraspaghetti-nemomi-1739-v8-mkmlizer: quantized model in 35.260s
marinaraspaghetti-nemomi-1739-v8-mkmlizer: Processed model MarinaraSpaghetti/NemoMix-Unleashed-12B in 78.631s
marinaraspaghetti-nemomi-1739-v8-mkmlizer: creating bucket guanaco-mkml-models
marinaraspaghetti-nemomi-1739-v8-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
marinaraspaghetti-nemomi-1739-v8-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/marinaraspaghetti-nemomi-1739-v8
marinaraspaghetti-nemomi-1739-v8-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/marinaraspaghetti-nemomi-1739-v8/config.json
marinaraspaghetti-nemomi-1739-v8-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/marinaraspaghetti-nemomi-1739-v8/special_tokens_map.json
marinaraspaghetti-nemomi-1739-v8-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/marinaraspaghetti-nemomi-1739-v8/tokenizer_config.json
marinaraspaghetti-nemomi-1739-v8-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/marinaraspaghetti-nemomi-1739-v8/tokenizer.json
marinaraspaghetti-nemomi-1739-v8-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/marinaraspaghetti-nemomi-1739-v8/flywheel_model.0.safetensors
marinaraspaghetti-nemomi-1739-v8-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/363 [00:06<18:10,  3.02s/it]
Loading 0:   2%|▏         | 6/363 [00:06<04:49,  1.23it/s]
Loading 0:   4%|▎         | 13/363 [00:06<01:43,  3.39it/s]
Loading 0:   5%|▍         | 18/363 [00:06<01:03,  5.39it/s]
Loading 0:   6%|▋         | 23/363 [00:06<00:42,  7.96it/s]
Loading 0:   8%|▊         | 29/363 [00:06<00:28, 11.57it/s]
Loading 0:   9%|▉         | 34/363 [00:06<00:21, 14.99it/s]
Loading 0:  11%|█         | 40/363 [00:07<00:19, 16.73it/s]
Loading 0:  12%|█▏        | 44/363 [00:07<00:16, 19.45it/s]
Loading 0:  14%|█▍        | 50/363 [00:07<00:12, 24.91it/s]
Loading 0:  15%|█▌        | 56/363 [00:07<00:10, 28.42it/s]
Loading 0:  17%|█▋        | 61/363 [00:07<00:09, 31.23it/s]
Loading 0:  19%|█▊        | 68/363 [00:07<00:07, 37.68it/s]
Loading 0:  20%|██        | 74/363 [00:07<00:07, 38.38it/s]
Loading 0:  22%|██▏       | 79/363 [00:07<00:07, 39.68it/s]
Loading 0:  24%|██▎       | 86/363 [00:08<00:06, 45.01it/s]
Loading 0:  25%|██▌       | 92/363 [00:08<00:06, 43.51it/s]
Loading 0:  27%|██▋       | 97/363 [00:08<00:06, 43.18it/s]
Loading 0:  29%|██▊       | 104/363 [00:08<00:05, 47.60it/s]
Loading 0:  30%|███       | 110/363 [00:08<00:05, 44.90it/s]
Loading 0:  32%|███▏      | 115/363 [00:08<00:05, 44.13it/s]
Loading 0:  33%|███▎      | 121/363 [00:08<00:06, 34.70it/s]
Loading 0:  34%|███▍      | 125/363 [00:09<00:06, 34.76it/s]
Loading 0:  36%|███▌      | 130/363 [00:09<00:06, 37.95it/s]
Loading 0:  37%|███▋      | 135/363 [00:09<00:05, 39.76it/s]
Loading 0:  39%|███▊      | 140/363 [00:09<00:05, 41.18it/s]
Loading 0:  40%|████      | 146/363 [00:09<00:05, 40.26it/s]
Loading 0:  42%|████▏     | 151/363 [00:09<00:05, 40.65it/s]
Loading 0:  43%|████▎     | 157/363 [00:09<00:04, 45.16it/s]
Loading 0:  45%|████▍     | 162/363 [00:09<00:04, 45.48it/s]
Loading 0:  46%|████▌     | 167/363 [00:09<00:04, 45.77it/s]
Loading 0:  48%|████▊     | 173/363 [00:10<00:04, 43.97it/s]
Loading 0:  49%|████▉     | 178/363 [00:10<00:04, 42.91it/s]
Loading 0:  51%|█████     | 184/363 [00:10<00:03, 46.88it/s]
Loading 0:  52%|█████▏    | 189/363 [00:10<00:03, 46.70it/s]
Loading 0:  53%|█████▎    | 194/363 [00:10<00:03, 46.63it/s]
Loading 0:  55%|█████▌    | 200/363 [00:10<00:03, 43.63it/s]
Loading 0:  56%|█████▋    | 205/363 [00:10<00:04, 31.73it/s]
Loading 0:  58%|█████▊    | 211/363 [00:11<00:04, 36.66it/s]
Loading 0:  60%|█████▉    | 216/363 [00:11<00:03, 38.65it/s]
Loading 0:  61%|██████    | 221/363 [00:11<00:03, 40.20it/s]
Loading 0:  63%|██████▎   | 227/363 [00:11<00:03, 40.01it/s]
Loading 0:  64%|██████▍   | 232/363 [00:11<00:03, 40.51it/s]
Loading 0:  66%|██████▌   | 238/363 [00:11<00:02, 44.96it/s]
Loading 0:  67%|██████▋   | 243/363 [00:11<00:02, 45.55it/s]
Loading 0:  68%|██████▊   | 248/363 [00:11<00:02, 45.93it/s]
Loading 0:  70%|██████▉   | 254/363 [00:12<00:02, 43.73it/s]
Loading 0:  71%|███████▏  | 259/363 [00:12<00:02, 42.76it/s]
Loading 0:  73%|███████▎  | 265/363 [00:12<00:02, 47.13it/s]
Loading 0:  74%|███████▍  | 270/363 [00:12<00:02, 46.34it/s]
Loading 0:  76%|███████▌  | 275/363 [00:12<00:01, 46.16it/s]
Loading 0:  77%|███████▋  | 281/363 [00:12<00:01, 44.04it/s]
Loading 0:  79%|███████▉  | 286/363 [00:12<00:02, 32.01it/s]
Loading 0:  80%|████████  | 292/363 [00:13<00:01, 37.69it/s]
Loading 0:  82%|████████▏ | 297/363 [00:13<00:01, 39.56it/s]
Loading 0:  83%|████████▎ | 302/363 [00:13<00:01, 40.93it/s]
Loading 0:  85%|████████▍ | 307/363 [00:13<00:01, 42.86it/s]
Loading 0:  86%|████████▌ | 312/363 [00:13<00:01, 37.33it/s]
Loading 0:  88%|████████▊ | 319/363 [00:13<00:00, 44.57it/s]
Loading 0:  89%|████████▉ | 324/363 [00:13<00:00, 44.55it/s]
Loading 0:  91%|█████████ | 329/363 [00:13<00:00, 44.98it/s]
Loading 0:  92%|█████████▏| 335/363 [00:13<00:00, 43.52it/s]
Loading 0:  94%|█████████▎| 340/363 [00:14<00:00, 42.52it/s]
Loading 0:  95%|█████████▌| 346/363 [00:14<00:00, 46.15it/s]
Loading 0:  97%|█████████▋| 351/363 [00:14<00:00, 46.19it/s]
Loading 0:  98%|█████████▊| 356/363 [00:14<00:00, 46.25it/s]
Loading 0: 100%|█████████▉| 362/363 [00:14<00:00, 44.01it/s]
                                                            
Job marinaraspaghetti-nemomi-1739-v8-mkmlizer completed after 102.42s with status: succeeded
Stopping job with name marinaraspaghetti-nemomi-1739-v8-mkmlizer
Pipeline stage MKMLizer completed in 103.15s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service marinaraspaghetti-nemomi-1739-v8
Waiting for inference service marinaraspaghetti-nemomi-1739-v8 to be ready
Inference service marinaraspaghetti-nemomi-1739-v8 ready after 220.4927999973297s
Pipeline stage MKMLDeployer completed in 220.85s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.187873363494873s
Received healthy response to inference request in 1.5907845497131348s
Received healthy response to inference request in 1.622178077697754s
Received healthy response to inference request in 1.6452200412750244s
Received healthy response to inference request in 1.5009701251983643s
5 requests
0 failed requests
5th percentile: 1.5189330101013183
10th percentile: 1.5368958950042724
20th percentile: 1.5728216648101807
30th percentile: 1.5970632553100585
40th percentile: 1.6096206665039063
50th percentile: 1.622178077697754
60th percentile: 1.631394863128662
70th percentile: 1.6406116485595703
80th percentile: 1.7537507057189943
90th percentile: 1.9708120346069335
95th percentile: 2.0793426990509034
99th percentile: 2.1661672306060793
mean time: 1.7094052314758301
Pipeline stage StressChecker completed in 9.15s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 5.10s
Shutdown handler de-registered
marinaraspaghetti-nemomi_1739_v8 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service marinaraspaghetti-nemomi-1739-v8-profiler
Waiting for inference service marinaraspaghetti-nemomi-1739-v8-profiler to be ready
Inference service marinaraspaghetti-nemomi-1739-v8-profiler ready after 210.52306699752808s
Pipeline stage MKMLProfilerDeployer completed in 210.92s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/marinaraspaghetti-ne4786a2bcd0e1f29a82a5938ec7b1b350-deplon99mz:/code/chaiverse_profiler_1727336794 --namespace tenant-chaiml-guanaco
kubectl exec -it marinaraspaghetti-ne4786a2bcd0e1f29a82a5938ec7b1b350-deplon99mz --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1727336794 && python profiles.py profile --best_of_n 8 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 64 --summary /code/chaiverse_profiler_1727336794/summary.json'
kubectl exec -it marinaraspaghetti-ne4786a2bcd0e1f29a82a5938ec7b1b350-deplon99mz --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1727336794/summary.json'
Pipeline stage MKMLProfilerRunner completed in 1164.80s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service marinaraspaghetti-nemomi-1739-v8-profiler is running
Tearing down inference service marinaraspaghetti-nemomi-1739-v8-profiler
Service marinaraspaghetti-nemomi-1739-v8-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.36s
Shutdown handler de-registered
marinaraspaghetti-nemomi_1739_v8 status is now inactive due to auto deactivation removed underperforming models
run pipeline %s
Shutdown handler not registered because Python interpreter is not running in the main thread
admin requested tearing down of chaiml-nemo-ties-esao-albert_v2
admin requested tearing down of marinaraspaghetti-nemomi_1739_v8
run pipeline stage %s
run pipeline %s
Shutdown handler not registered because Python interpreter is not running in the main thread
admin requested tearing down of chaiml-nemo-ties-lexical_2876_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
Running pipeline stage ProductionBlendMKMLTemplater
run pipeline stage %s
run pipeline %s
admin requested tearing down of blend_rofur_2024-10-03
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
admin requested tearing down of chaiml-virgo-edit-v1-1e5_v7
Pipeline stage %s skipped, reason=%s
Running pipeline stage MKMLDeleter
run pipeline stage %s
run pipeline %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Shutdown handler not registered because Python interpreter is not running in the main thread
admin requested tearing down of chaiml-nemo-slerp-lexica_9631_v2
Pipeline stage %s skipped, reason=%s
run pipeline stage %s
Pipeline stage ProductionBlendMKMLTemplater completed in 47.13s
Running pipeline stage MKMLDeleter
run pipeline stage %s
run pipeline %s
run pipeline %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Pipeline stage MKMLDeleter completed in 41.17s
Running pipeline stage MKMLDeleter
run pipeline stage %s
Pipeline stage %s skipped, reason=%s
Running pipeline stage MKMLDeleter
run pipeline stage %s
run pipeline stage %s
run pipeline %s
run pipeline stage %s
Checking if service marinaraspaghetti-nemomi-1739-v8 is running
Running pipeline stage MKMLDeployer
Pipeline stage MKMLDeleter completed in 53.25s
Skipping teardown as no inference service was successfully deployed
Running pipeline stage ProductionBlendMKMLTemplater
Running pipeline stage MKMLDeleter
run pipeline stage %s
Running pipeline stage MKMLModelDeleter
Tearing down inference service marinaraspaghetti-nemomi-1739-v8
Creating inference service blend-rofur-2024-10-03
run pipeline stage %s
Pipeline stage MKMLDeleter completed in 51.17s
Pipeline stage %s skipped, reason=%s
Skipping teardown as no inference service was successfully deployed
Running pipeline stage MKMLDeleter
Cleaning model data from S3
Service marinaraspaghetti-nemomi-1739-v8 has been torndown
Running pipeline stage MKMLModelDeleter
Ignoring service blend-rofur-2024-10-03 already deployed
run pipeline stage %s
Pipeline stage ProductionBlendMKMLTemplater completed in 49.79s
Pipeline stage MKMLDeleter completed in 51.37s
Checking if service chaiml-nemo-slerp-lexica-9631-v2 is running
Cleaning model data from model cache
Pipeline stage MKMLDeleter completed in 104.40s
Cleaning model data from S3
Waiting for inference service blend-rofur-2024-10-03 to be ready
Running pipeline stage MKMLModelDeleter
run pipeline stage %s
run pipeline stage %s
Deleting key chaiml-nemo-slerp-esao-b-7186-v1/special_tokens_map.json from bucket guanaco-mkml-models
Tearing down inference service chaiml-nemo-slerp-lexica-9631-v2
run pipeline stage %s
Cleaning model data from model cache
Cleaning model data from S3
Running pipeline stage MKMLDeployer
Running pipeline stage MKMLModelDeleter
Deleting key chaiml-nemo-slerp-esao-b-7186-v1/tokenizer.json from bucket guanaco-mkml-models
Service chaiml-nemo-slerp-lexica-9631-v2 has been torndown
Running pipeline stage MKMLModelDeleter
Deleting key chaiml-nemo-ties-esao-albert-v2/config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Creating inference service blend-rofur-2024-10-03
Skipping deletion as no model was successfully uploaded
Deleting key chaiml-nemo-slerp-esao-b-7186-v1/tokenizer_config.json from bucket guanaco-mkml-models
Pipeline stage MKMLDeleter completed in 108.99s
Cleaning model data from S3
Deleting key chaiml-nemo-ties-esao-albert-v2/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key chaiml-nemo-ties-lexical-2876-v1/config.json from bucket guanaco-mkml-models
Ignoring service blend-rofur-2024-10-03 already deployed
Pipeline stage MKMLModelDeleter completed in 45.67s
admin requested tearing down of blend_rofur_2024-10-03
Pipeline stage MKMLModelDeleter completed in 153.37s
run pipeline stage %s
Cleaning model data from model cache
admin requested tearing down of marinaraspaghetti-nemomi_1739_v8
Deleting key chaiml-nemo-ties-esao-albert-v2/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key chaiml-nemo-ties-lexical-2876-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Waiting for inference service blend-rofur-2024-10-03 to be ready
Shutdown handler de-registered
Shutdown handler not registered because Python interpreter is not running in the main thread
Deleting key chaiml-nemo-ties-lexical-2876-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Waiting for inference service blend-rofur-2024-10-03 to be ready
Shutdown handler de-registered
Shutdown handler not registered because Python interpreter is not running in the main thread
Shutdown handler de-registered
Deleting key marinaraspaghetti-nemomi-1739-v8/config.json from bucket guanaco-mkml-models
Deleting key chaiml-nemo-ties-esao-albert-v2/tokenizer.json from bucket guanaco-mkml-models
Shutdown handler not registered because Python interpreter is not running in the main thread
admin requested tearing down of blend_rofur_2024-10-03
Deleting key chaiml-nemo-ties-lexical-2876-v1/special_tokens_map.json from bucket guanaco-mkml-models
chaiml-virgo-edit-v1-1e5_v7 status is now torndown due to DeploymentManager action
run pipeline %s
chaiml-nemo-slerp-esao-b_7186_v1 status is now torndown due to DeploymentManager action
Tearing down inference service blend-rofur-2024-10-03
Cleaning model data from S3
Deleting key chaiml-nemo-ties-esao-albert-v2/tokenizer_config.json from bucket guanaco-mkml-models
Deleting key marinaraspaghetti-nemomi-1739-v8/flywheel_model.0.safetensors from bucket guanaco-mkml-models
run pipeline %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Deleting key chaiml-nemo-ties-lexical-2876-v1/tokenizer.json from bucket guanaco-mkml-models
Tearing down inference service blend-rofur-2024-10-03
run pipeline stage %s
%s, retrying in %s seconds...
Tearing down inference service blend-rofur-2024-10-03
Tearing down inference service blend-rofur-2024-10-03
Cleaning model data from model cache
Tearing down inference service blend-rofur-2024-10-03
Pipeline stage MKMLModelDeleter completed in 254.02s
Deleting key marinaraspaghetti-nemomi-1739-v8/special_tokens_map.json from bucket guanaco-mkml-models
run pipeline stage %s
run pipeline %s
Deleting key chaiml-nemo-ties-lexical-2876-v1/tokenizer_config.json from bucket guanaco-mkml-models
%s, retrying in %s seconds...
Running pipeline stage ProductionBlendMKMLTemplater
Creating inference service blend-rofur-2024-10-03
%s, retrying in %s seconds...
%s, retrying in %s seconds...
Deleting key chaiml-nemo-slerp-lexica-9631-v2/config.json from bucket guanaco-mkml-models
%s, retrying in %s seconds...
Shutdown handler de-registered
Deleting key marinaraspaghetti-nemomi-1739-v8/tokenizer.json from bucket guanaco-mkml-models
run pipeline stage %s
Pipeline stage MKMLModelDeleter completed in 294.96s
Creating inference service blend-rofur-2024-10-03
Pipeline stage %s skipped, reason=%s
Waiting for inference service blend-rofur-2024-10-03 to be ready
Creating inference service blend-rofur-2024-10-03
Creating inference service blend-rofur-2024-10-03
Deleting key chaiml-nemo-slerp-lexica-9631-v2/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Creating inference service blend-rofur-2024-10-03
chaiml-nemo-ties-esao-albert_v2 status is now torndown due to DeploymentManager action
Deleting key marinaraspaghetti-nemomi-1739-v8/tokenizer_config.json from bucket guanaco-mkml-models
Pipeline stage %s skipped, reason=%s
Running pipeline stage ProductionBlendMKMLTemplater
Pipeline stage ProductionBlendMKMLTemplater completed in 108.45s
Pipeline stage ProductionBlendMKMLTemplater completed in 112.31s
marinaraspaghetti-nemomi_1739_v8 status is now torndown due to DeploymentManager action