natong19-mistral-nemo-i_86670

developer_uid: zmeeks

submission_id: natong19-mistral-nemo-i_86670_v2

model_name: ab-1000

model_group: natong19/Mistral-Nemo-In

status: torndown

timestamp: 2025-06-28T18:52:35+00:00

num_battles: 7128

num_wins: 3108

celo_rating: 1243.36

family_friendly_score: 0.606

family_friendly_standard_error: 0.00691034007846213

submission_type: basic

model_repo: natong19/Mistral-Nemo-Instruct-2407-abliterated

model_architecture: MistralForCausalLM

model_num_parameters: 12772070400.0

best_of: 8

max_input_tokens: 1024

max_output_tokens: 64

reward_model: default

latencies: [{'batch_size': 1, 'throughput': 0.5898934070887052, 'latency_mean': 1.695109511613846, 'latency_p50': 1.7081902027130127, 'latency_p90': 1.8420788049697876}, {'batch_size': 3, 'throughput': 1.072792655507296, 'latency_mean': 2.784257332086563, 'latency_p50': 2.7795578241348267, 'latency_p90': 3.0907755851745606}, {'batch_size': 5, 'throughput': 1.2760966564028298, 'latency_mean': 3.8901285409927366, 'latency_p50': 3.89261531829834, 'latency_p90': 4.350487399101257}, {'batch_size': 6, 'throughput': 1.3520828226419388, 'latency_mean': 4.414051740169525, 'latency_p50': 4.410121440887451, 'latency_p90': 4.901059627532959}, {'batch_size': 8, 'throughput': 1.4127962819109599, 'latency_mean': 5.621511626243591, 'latency_p50': 5.628851532936096, 'latency_p90': 6.400016045570373}, {'batch_size': 10, 'throughput': 1.43918458431408, 'latency_mean': 6.882994691133499, 'latency_p50': 6.8607417345047, 'latency_p90': 7.803118968009949}]

gpu_counts: {'NVIDIA RTX A5000': 1}

display_name: ab-1000

is_internal_developer: False

language_model: natong19/Mistral-Nemo-Instruct-2407-abliterated

model_size: 13B

ranking_group: single

throughput_3p7s: 1.26

us_pacific_date: 2025-06-28

win_ratio: 0.43602693602693604

generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name natong19-mistral-nemo-i-86670-v2-mkmlizer
Waiting for job on natong19-mistral-nemo-i-86670-v2-mkmlizer to finish
natong19-mistral-nemo-i-86670-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║                                                                     ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║                                                                     ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║  Version: 0.29.3                                                    ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║  https://mk1.ai                                                     ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║                                                                     ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║  belonging to:                                                      ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║                                                                     ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║  Chai Research Corp.                                                ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ║                                                                     ║
natong19-mistral-nemo-i-86670-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
natong19-mistral-nemo-i-86670-v2-mkmlizer: Downloaded to shared memory in 45.064s
natong19-mistral-nemo-i-86670-v2-mkmlizer: Checking if natong19/Mistral-Nemo-Instruct-2407-abliterated already exists in ChaiML
natong19-mistral-nemo-i-86670-v2-mkmlizer: Creating repo ChaiML/Mistral-Nemo-Instruct-2407-abliterated and uploading /tmp/tmpee5mnqr3 to it
natong19-mistral-nemo-i-86670-v2-mkmlizer: 
  0%|          | 0/5 [00:00<?, ?it/s]
 20%|██        | 1/5 [00:03<00:14,  3.75s/it]
 40%|████      | 2/5 [00:07<00:11,  3.68s/it]
 60%|██████    | 3/5 [00:12<00:08,  4.41s/it]
 80%|████████  | 4/5 [00:16<00:04,  4.18s/it]
100%|██████████| 5/5 [00:20<00:00,  4.19s/it]
100%|██████████| 5/5 [00:20<00:00,  4.14s/it]
natong19-mistral-nemo-i-86670-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpee5mnqr3, device:0
natong19-mistral-nemo-i-86670-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
natong19-mistral-nemo-i-86670-v2-mkmlizer: quantized model in 29.737s
natong19-mistral-nemo-i-86670-v2-mkmlizer: Processed model natong19/Mistral-Nemo-Instruct-2407-abliterated in 122.751s
natong19-mistral-nemo-i-86670-v2-mkmlizer: creating bucket guanaco-mkml-models
natong19-mistral-nemo-i-86670-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
natong19-mistral-nemo-i-86670-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/natong19-mistral-nemo-i-86670-v2
natong19-mistral-nemo-i-86670-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/natong19-mistral-nemo-i-86670-v2/config.json
natong19-mistral-nemo-i-86670-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/natong19-mistral-nemo-i-86670-v2/special_tokens_map.json
natong19-mistral-nemo-i-86670-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/natong19-mistral-nemo-i-86670-v2/tokenizer_config.json
natong19-mistral-nemo-i-86670-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/natong19-mistral-nemo-i-86670-v2/tokenizer.json
natong19-mistral-nemo-i-86670-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/natong19-mistral-nemo-i-86670-v2/flywheel_model.0.safetensors
natong19-mistral-nemo-i-86670-v2-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:10, 32.80it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 52.07it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 45.46it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:07, 42.79it/s]
Loading 0:   8%|▊         | 30/363 [00:00<00:07, 47.52it/s]
Loading 0:  10%|▉         | 35/363 [00:00<00:06, 47.17it/s]
Loading 0:  11%|█         | 40/363 [00:00<00:06, 46.97it/s]
Loading 0:  13%|█▎        | 46/363 [00:01<00:07, 44.02it/s]
Loading 0:  14%|█▍        | 51/363 [00:01<00:07, 43.40it/s]
Loading 0:  16%|█▌        | 58/363 [00:01<00:06, 50.16it/s]
Loading 0:  18%|█▊        | 64/363 [00:01<00:09, 31.71it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:07, 39.44it/s]
Loading 0:  21%|██▏       | 78/363 [00:01<00:07, 39.00it/s]
Loading 0:  23%|██▎       | 83/363 [00:01<00:07, 39.58it/s]
Loading 0:  25%|██▍       | 89/363 [00:02<00:06, 44.03it/s]
Loading 0:  26%|██▌       | 94/363 [00:02<00:06, 44.01it/s]
Loading 0:  27%|██▋       | 99/363 [00:02<00:05, 44.56it/s]
Loading 0:  29%|██▊       | 104/363 [00:02<00:05, 45.92it/s]
Loading 0:  30%|███       | 110/363 [00:02<00:05, 43.32it/s]
Loading 0:  32%|███▏      | 115/363 [00:02<00:05, 44.04it/s]
Loading 0:  33%|███▎      | 120/363 [00:02<00:05, 41.86it/s]
Loading 0:  34%|███▍      | 125/363 [00:02<00:05, 43.73it/s]
Loading 0:  36%|███▌      | 130/363 [00:03<00:05, 43.83it/s]
Loading 0:  37%|███▋      | 135/363 [00:03<00:05, 43.91it/s]
Loading 0:  39%|███▊      | 140/363 [00:03<00:04, 45.34it/s]
Loading 0:  40%|███▉      | 145/363 [00:03<00:07, 28.88it/s]
Loading 0:  41%|████      | 149/363 [00:03<00:07, 29.07it/s]
Loading 0:  43%|████▎     | 156/363 [00:03<00:05, 36.92it/s]
Loading 0:  44%|████▍     | 161/363 [00:03<00:05, 38.67it/s]
Loading 0:  46%|████▌     | 166/363 [00:04<00:05, 39.11it/s]
Loading 0:  47%|████▋     | 171/363 [00:04<00:04, 41.05it/s]
Loading 0:  48%|████▊     | 176/363 [00:04<00:05, 34.59it/s]
Loading 0:  50%|█████     | 183/363 [00:04<00:04, 41.88it/s]
Loading 0:  52%|█████▏    | 188/363 [00:04<00:04, 41.42it/s]
Loading 0:  53%|█████▎    | 193/363 [00:04<00:04, 41.97it/s]
Loading 0:  55%|█████▍    | 198/363 [00:04<00:03, 43.64it/s]
Loading 0:  56%|█████▌    | 203/363 [00:04<00:04, 36.43it/s]
Loading 0:  58%|█████▊    | 210/363 [00:05<00:03, 42.88it/s]
Loading 0:  59%|█████▉    | 215/363 [00:05<00:03, 42.67it/s]
Loading 0:  61%|██████    | 220/363 [00:05<00:03, 43.93it/s]
Loading 0:  62%|██████▏   | 225/363 [00:05<00:04, 28.42it/s]
Loading 0:  63%|██████▎   | 230/363 [00:05<00:04, 30.43it/s]
Loading 0:  65%|██████▌   | 237/363 [00:05<00:03, 37.47it/s]
Loading 0:  67%|██████▋   | 242/363 [00:06<00:03, 39.17it/s]
Loading 0:  68%|██████▊   | 247/363 [00:06<00:02, 40.52it/s]
Loading 0:  69%|██████▉   | 252/363 [00:06<00:02, 42.33it/s]
Loading 0:  71%|███████   | 257/363 [00:06<00:02, 35.40it/s]
Loading 0:  73%|███████▎  | 264/363 [00:06<00:02, 42.38it/s]
Loading 0:  74%|███████▍  | 269/363 [00:06<00:02, 42.41it/s]
Loading 0:  75%|███████▌  | 274/363 [00:06<00:02, 42.35it/s]
Loading 0:  77%|███████▋  | 279/363 [00:06<00:01, 44.20it/s]
Loading 0:  78%|███████▊  | 284/363 [00:07<00:02, 37.51it/s]
Loading 0:  80%|████████  | 291/363 [00:07<00:01, 43.85it/s]
Loading 0:  82%|████████▏ | 296/363 [00:07<00:01, 43.52it/s]
Loading 0:  83%|████████▎ | 302/363 [00:07<00:01, 47.01it/s]
Loading 0:  85%|████████▍ | 307/363 [00:07<00:02, 23.01it/s]
Loading 0:  86%|████████▌ | 312/363 [00:08<00:02, 25.27it/s]
Loading 0:  88%|████████▊ | 320/363 [00:08<00:01, 33.71it/s]
Loading 0:  90%|████████▉ | 326/363 [00:08<00:01, 34.93it/s]
Loading 0:  91%|█████████ | 331/363 [00:08<00:00, 36.49it/s]
Loading 0:  93%|█████████▎| 338/363 [00:08<00:00, 42.02it/s]
Loading 0:  95%|█████████▍| 344/363 [00:08<00:00, 40.35it/s]
Loading 0:  96%|█████████▌| 349/363 [00:08<00:00, 40.35it/s]
Loading 0:  98%|█████████▊| 355/363 [00:08<00:00, 44.78it/s]
Loading 0:  99%|█████████▉| 360/363 [00:09<00:00, 44.40it/s]
                                                            
Job natong19-mistral-nemo-i-86670-v2-mkmlizer completed after 155.86s with status: succeeded
Stopping job with name natong19-mistral-nemo-i-86670-v2-mkmlizer
Pipeline stage MKMLizer completed in 156.34s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service natong19-mistral-nemo-i-86670-v2
Waiting for inference service natong19-mistral-nemo-i-86670-v2 to be ready
Inference service natong19-mistral-nemo-i-86670-v2 ready after 170.56384897232056s
Pipeline stage MKMLDeployer completed in 171.10s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.3640425205230713s
Received healthy response to inference request in 1.7119255065917969s
Received healthy response to inference request in 2.037034511566162s
Received healthy response to inference request in 2.0569448471069336s
5 requests
1 failed requests
5th percentile: 1.77694730758667
10th percentile: 1.841969108581543
20th percentile: 1.972012710571289
30th percentile: 2.0410165786743164
40th percentile: 2.048980712890625
50th percentile: 2.0569448471069336
60th percentile: 2.1797839164733888
70th percentile: 2.3026229858398435
80th percentile: 5.921020984649662
90th percentile: 13.034977912902834
95th percentile: 16.591956377029415
99th percentile: 19.43753914833069
mean time: 5.663776445388794
%s, retrying in %s seconds...
Received healthy response to inference request in 1.384185552597046s
Received healthy response to inference request in 1.1303999423980713s
Received healthy response to inference request in 1.7940797805786133s
Received healthy response to inference request in 1.6163628101348877s
Received healthy response to inference request in 3.288674831390381s
5 requests
0 failed requests
5th percentile: 1.1811570644378662
10th percentile: 1.231914186477661
20th percentile: 1.333428430557251
30th percentile: 1.4306210041046143
40th percentile: 1.523491907119751
50th percentile: 1.6163628101348877
60th percentile: 1.6874495983123778
70th percentile: 1.7585363864898682
80th percentile: 2.092998790740967
90th percentile: 2.690836811065674
95th percentile: 2.989755821228027
99th percentile: 3.2288910293579103
mean time: 1.8427405834197998
Pipeline stage StressChecker completed in 40.23s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.73s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 1.00s
Shutdown handler de-registered
natong19-mistral-nemo-i_86670_v2 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.13s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service natong19-mistral-nemo-i-86670-v2-profiler
Waiting for inference service natong19-mistral-nemo-i-86670-v2-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3254.58s
Shutdown handler de-registered
natong19-mistral-nemo-i_86670_v2 status is now inactive due to auto deactivation removed underperforming models
natong19-mistral-nemo-i_86670_v2 status is now torndown due to DeploymentManager action