submission_id: mistralai-mistral-nem_93303_v320
developer_uid: RandomForest1024
status: failed
model_repo: mistralai/Mistral-Nemo-Instruct-2407
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': "'<Context>Previous conversation shows your personality as described above.</Context> {prompt}\n<START>\n'", 'bot_template': "'{bot_name}: {message}\n [Maintain teenage perspective while being sophisticated. Use occasional gaming or sports references. Keep political topics neutral.]'", 'user_template': "'{user_name}: {message}\n [Focus on building genuine connections while maintaining appropriate boundaries.]'", 'response_template': "'{bot_name}: [Think as a Gen Z teenager with unique life experiences, then respond]', 'truncate_by_message': False}", 'truncate_by_message': False}
timestamp: 2025-02-10T18:39:02+00:00
model_name: mistralai-mistral-nem_93303_v320
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name mistralai-mistral-nem-93303-v320-mkmlizer
Waiting for job on mistralai-mistral-nem-93303-v320-mkmlizer to finish
mistralai-mistral-nem-93303-v320-mkmlizer: Downloaded to shared memory in 52.890s
mistralai-mistral-nem-93303-v320-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmphun4wh98, device:0
mistralai-mistral-nem-93303-v320-mkmlizer: Saving flywheel model at /dev/shm/model_cache
mistralai-mistral-nem-93303-v320-mkmlizer: quantized model in 37.718s
mistralai-mistral-nem-93303-v320-mkmlizer: Processed model mistralai/Mistral-Nemo-Instruct-2407 in 90.608s
mistralai-mistral-nem-93303-v320-mkmlizer: creating bucket guanaco-mkml-models
mistralai-mistral-nem-93303-v320-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
mistralai-mistral-nem-93303-v320-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/mistralai-mistral-nem-93303-v320
mistralai-mistral-nem-93303-v320-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/mistralai-mistral-nem-93303-v320/config.json
mistralai-mistral-nem-93303-v320-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/mistralai-mistral-nem-93303-v320/special_tokens_map.json
mistralai-mistral-nem-93303-v320-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/mistralai-mistral-nem-93303-v320/tokenizer_config.json
mistralai-mistral-nem-93303-v320-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/mistralai-mistral-nem-93303-v320/flywheel_model.0.safetensors
mistralai-mistral-nem-93303-v320-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 31.43it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 51.51it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 46.72it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:07, 45.83it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 52.12it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:06, 48.26it/s]
Loading 0:  12%|█▏        | 42/363 [00:00<00:06, 47.17it/s]
Loading 0:  13%|█▎        | 49/363 [00:00<00:05, 52.51it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 48.08it/s]
Loading 0:  17%|█▋        | 60/363 [00:01<00:06, 48.41it/s]
Loading 0:  18%|█▊        | 65/363 [00:01<00:10, 29.22it/s]
Loading 0:  20%|█▉        | 71/363 [00:01<00:08, 34.92it/s]
Loading 0:  21%|██        | 76/363 [00:01<00:07, 37.12it/s]
Loading 0:  22%|██▏       | 81/363 [00:01<00:07, 39.11it/s]
Loading 0:  24%|██▍       | 87/363 [00:02<00:07, 39.23it/s]
Loading 0:  25%|██▌       | 92/363 [00:02<00:06, 39.25it/s]
Loading 0:  27%|██▋       | 99/363 [00:02<00:05, 44.88it/s]
Loading 0:  29%|██▉       | 105/363 [00:02<00:05, 43.30it/s]
Loading 0:  31%|███       | 112/363 [00:02<00:05, 46.48it/s]
Loading 0:  32%|███▏      | 117/363 [00:02<00:05, 43.95it/s]
Loading 0:  34%|███▍      | 123/363 [00:02<00:05, 41.64it/s]
Loading 0:  35%|███▌      | 128/363 [00:03<00:05, 40.11it/s]
Loading 0:  37%|███▋      | 134/363 [00:03<00:05, 43.57it/s]
Loading 0:  38%|███▊      | 139/363 [00:03<00:05, 43.14it/s]
Loading 0:  40%|███▉      | 144/363 [00:03<00:08, 24.76it/s]
Loading 0:  41%|████      | 149/363 [00:03<00:07, 27.51it/s]
Loading 0:  43%|████▎     | 156/363 [00:03<00:06, 34.19it/s]
Loading 0:  44%|████▍     | 161/363 [00:04<00:05, 36.33it/s]
Loading 0:  46%|████▌     | 166/363 [00:04<00:05, 38.60it/s]
Loading 0:  47%|████▋     | 172/363 [00:04<00:05, 38.13it/s]
Loading 0:  49%|████▉     | 177/363 [00:04<00:04, 38.59it/s]
Loading 0:  50%|█████     | 183/363 [00:04<00:04, 43.53it/s]
Loading 0:  52%|█████▏    | 188/363 [00:04<00:03, 44.42it/s]
Loading 0:  53%|█████▎    | 193/363 [00:04<00:03, 45.06it/s]
Loading 0:  55%|█████▍    | 199/363 [00:04<00:03, 42.53it/s]
Loading 0:  56%|█████▌    | 204/363 [00:05<00:04, 38.28it/s]
Loading 0:  58%|█████▊    | 210/363 [00:05<00:03, 43.31it/s]
Loading 0:  59%|█████▉    | 215/363 [00:05<00:03, 43.26it/s]
Loading 0:  61%|██████    | 220/363 [00:05<00:03, 43.97it/s]
Loading 0:  62%|██████▏   | 225/363 [00:05<00:05, 24.52it/s]
Loading 0:  63%|██████▎   | 230/363 [00:05<00:04, 27.21it/s]
Loading 0:  65%|██████▌   | 237/363 [00:06<00:03, 34.60it/s]
Loading 0:  67%|██████▋   | 242/363 [00:06<00:03, 37.12it/s]
Loading 0:  68%|██████▊   | 247/363 [00:06<00:02, 39.40it/s]
Loading 0:  69%|██████▉   | 252/363 [00:06<00:02, 41.77it/s]
Loading 0:  71%|███████   | 257/363 [00:06<00:02, 35.59it/s]
Loading 0:  73%|███████▎  | 264/363 [00:06<00:02, 43.29it/s]
Loading 0:  74%|███████▍  | 269/363 [00:06<00:02, 43.95it/s]
Loading 0:  75%|███████▌  | 274/363 [00:06<00:02, 44.37it/s]
Loading 0:  77%|███████▋  | 280/363 [00:07<00:02, 40.95it/s]
Loading 0:  79%|███████▊  | 285/363 [00:07<00:01, 40.21it/s]
Loading 0:  80%|████████  | 291/363 [00:07<00:01, 44.84it/s]
Loading 0:  82%|████████▏ | 296/363 [00:07<00:01, 45.28it/s]
Loading 0:  83%|████████▎ | 301/363 [00:07<00:01, 46.35it/s]
Loading 0:  84%|████████▍ | 306/363 [00:14<00:24,  2.37it/s]
Loading 0:  85%|████████▌ | 310/363 [00:14<00:17,  3.07it/s]
Loading 0:  87%|████████▋ | 314/363 [00:14<00:12,  4.05it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:07,  6.06it/s]
Loading 0:  90%|████████▉ | 326/363 [00:15<00:04,  8.53it/s]
Loading 0:  91%|█████████ | 331/363 [00:15<00:02, 11.08it/s]
Loading 0:  93%|█████████▎| 338/363 [00:15<00:01, 15.83it/s]
Loading 0:  95%|█████████▍| 344/363 [00:15<00:00, 19.38it/s]
Loading 0:  96%|█████████▌| 349/363 [00:15<00:00, 22.48it/s]
Loading 0:  98%|█████████▊| 355/363 [00:15<00:00, 27.90it/s]
Loading 0:  99%|█████████▉| 360/363 [00:15<00:00, 31.37it/s]
                                                            

Job mistralai-mistral-nem-93303-v320-mkmlizer completed after 122.78s with status: succeeded
Stopping job with name mistralai-mistral-nem-93303-v320-mkmlizer
Pipeline stage MKMLizer completed in 123.26s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service mistralai-mistral-nem-93303-v320
Waiting for inference service mistralai-mistral-nem-93303-v320 to be ready
Failed to get response for submission randomforest1024-v25_v8: HTTPConnectionPool(host='randomforest1024-v25-v8-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission randomforest1024-v25_v8: HTTPConnectionPool(host='randomforest1024-v25-v8-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service mistralai-mistral-nem-93303-v320 ready after 200.93559741973877s
Pipeline stage MKMLDeployer completed in 201.42s
run pipeline stage %s
Running pipeline stage StressChecker
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 0.15201306343078613
10th percentile: 0.15299487113952637
20th percentile: 0.15495848655700684
30th percentile: 0.15609159469604492
40th percentile: 0.15639419555664064
50th percentile: 0.15669679641723633
60th percentile: 0.15831480026245118
70th percentile: 0.15993280410766603
80th percentile: 0.1621126651763916
90th percentile: 0.16485438346862794
95th percentile: 0.16622524261474608
99th percentile: 0.16732192993164063
mean time: 0.1584012508392334
%s, retrying in %s seconds...
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 0.16489953994750978
10th percentile: 0.16585793495178222
20th percentile: 0.16777472496032714
30th percentile: 0.18418416976928711
40th percentile: 0.21508626937866213
50th percentile: 0.2459883689880371
60th percentile: 0.2544851303100586
70th percentile: 0.2629818916320801
80th percentile: 0.26892571449279784
90th percentile: 0.27231659889221194
95th percentile: 0.27401204109191896
99th percentile: 0.2753683948516846
mean time: 0.22432007789611816
%s, retrying in %s seconds...
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 0.1774519443511963
10th percentile: 0.1796574115753174
20th percentile: 0.18406834602355956
30th percentile: 0.18937721252441406
40th percentile: 0.19558401107788087
50th percentile: 0.20179080963134766
60th percentile: 0.21128368377685547
70th percentile: 0.22077655792236328
80th percentile: 0.25846652984619145
90th percentile: 0.32435359954833987
95th percentile: 0.357297134399414
99th percentile: 0.38365196228027343
mean time: 0.2358149528503418
clean up pipeline due to error=DeploymentChecksError('Unacceptable number of predict errors: 100.0%')
Shutdown handler de-registered
mistralai-mistral-nem_93303_v320 status is now failed due to DeploymentManager action