developer_uid: azuruce
submission_id: chaiml-mistral-nemo-simp_1866_v2
model_name: chaiml-mistral-nemo-simp_1866_v2
model_group: ChaiML/mistral_nemo_simp
status: inactive
timestamp: 2024-12-18T19:08:11+00:00
num_battles: 12961
num_wins: 6057
celo_rating: 1234.54
family_friendly_score: 0.5893999999999999
family_friendly_standard_error: 0.006957120668782452
submission_type: basic
model_repo: ChaiML/mistral_nemo_simpo_baseline_albert_20241217_v1-checkpoint-125
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 4
max_input_tokens: 1024
max_output_tokens: 64
latencies: [{'batch_size': 1, 'throughput': 0.6377123868305431, 'latency_mean': 1.5680364418029784, 'latency_p50': 1.5761327743530273, 'latency_p90': 1.7386414766311646}, {'batch_size': 3, 'throughput': 1.264894325316662, 'latency_mean': 2.358267904520035, 'latency_p50': 2.365749955177307, 'latency_p90': 2.583300733566284}, {'batch_size': 5, 'throughput': 1.5954662190591211, 'latency_mean': 3.1110729002952575, 'latency_p50': 3.107487916946411, 'latency_p90': 3.529320311546326}, {'batch_size': 6, 'throughput': 1.7113682498311442, 'latency_mean': 3.4868385100364687, 'latency_p50': 3.4950801134109497, 'latency_p90': 3.857900214195251}, {'batch_size': 8, 'throughput': 1.887586706473468, 'latency_mean': 4.210103975534439, 'latency_p50': 4.20859158039093, 'latency_p90': 4.756027388572693}, {'batch_size': 10, 'throughput': 1.9530839353586318, 'latency_mean': 5.089314419031143, 'latency_p50': 5.095231533050537, 'latency_p90': 5.791120839118958}]
gpu_counts: {'NVIDIA RTX A5000': 1}
display_name: chaiml-mistral-nemo-simp_1866_v2
is_internal_developer: True
language_model: ChaiML/mistral_nemo_simpo_baseline_albert_20241217_v1-checkpoint-125
model_size: 13B
ranking_group: single
throughput_3p7s: 1.78
us_pacific_date: 2024-12-18
win_ratio: 0.46732505207931485
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 100, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '####', 'Bot:', 'User:', 'You:', '<|im_end|>', '<|eot_id|>'], 'max_input_tokens': 1024, 'best_of': 4, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-mistral-nemo-simp-1866-v2-mkmlizer
Waiting for job on chaiml-mistral-nemo-simp-1866-v2-mkmlizer to finish
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║     _____            __           __                                ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║       /___/                                                         ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║                                                                     ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║  Version: 0.11.12                                                   ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║                                                                     ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║  belonging to:                                                      ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║                                                                     ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ║                                                                     ║
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: Downloaded to shared memory in 33.099s
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp1hry8yrq, device:0
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: quantized model in 36.262s
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: Processed model ChaiML/mistral_nemo_simpo_baseline_albert_20241217_v1-checkpoint-125 in 69.362s
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: creating bucket guanaco-mkml-models
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-1866-v2
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-1866-v2/config.json
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-1866-v2/special_tokens_map.json
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-1866-v2/tokenizer_config.json
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-1866-v2/tokenizer.json
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-1866-v2/flywheel_model.0.safetensors
chaiml-mistral-nemo-simp-1866-v2-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:12, 29.00it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 50.47it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 43.97it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:08, 42.30it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 48.55it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:07, 43.36it/s]
Loading 0:  12%|█▏        | 42/363 [00:00<00:07, 42.68it/s]
Loading 0:  13%|█▎        | 49/363 [00:01<00:06, 47.68it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 44.16it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:08, 34.16it/s]
Loading 0:  18%|█▊        | 65/363 [00:01<00:09, 33.08it/s]
Loading 0:  20%|█▉        | 71/363 [00:01<00:07, 38.24it/s]
Loading 0:  21%|██        | 76/363 [00:01<00:07, 39.08it/s]
Loading 0:  22%|██▏       | 81/363 [00:01<00:06, 40.37it/s]
Loading 0:  24%|██▍       | 87/363 [00:02<00:06, 41.03it/s]
Loading 0:  25%|██▌       | 92/363 [00:02<00:06, 41.44it/s]
Loading 0:  27%|██▋       | 98/363 [00:02<00:05, 45.36it/s]
Loading 0:  28%|██▊       | 103/363 [00:02<00:05, 45.40it/s]
Loading 0:  30%|███       | 110/363 [00:02<00:05, 45.36it/s]
Loading 0:  32%|███▏      | 115/363 [00:02<00:05, 44.96it/s]
Loading 0:  33%|███▎      | 120/363 [00:02<00:05, 41.72it/s]
Loading 0:  35%|███▍      | 126/363 [00:02<00:05, 45.29it/s]
Loading 0:  36%|███▋      | 132/363 [00:03<00:05, 43.55it/s]
Loading 0:  38%|███▊      | 137/363 [00:03<00:05, 43.02it/s]
Loading 0:  39%|███▉      | 142/363 [00:03<00:06, 32.93it/s]
Loading 0:  40%|████      | 146/363 [00:03<00:06, 33.67it/s]
Loading 0:  41%|████▏     | 150/363 [00:03<00:06, 31.72it/s]
Loading 0:  43%|████▎     | 157/363 [00:03<00:05, 39.19it/s]
Loading 0:  45%|████▍     | 163/363 [00:04<00:05, 39.95it/s]
Loading 0:  46%|████▋     | 168/363 [00:04<00:04, 40.70it/s]
Loading 0:  48%|████▊     | 175/363 [00:04<00:04, 46.08it/s]
Loading 0:  50%|████▉     | 181/363 [00:04<00:04, 42.18it/s]
Loading 0:  51%|█████     | 186/363 [00:04<00:04, 41.03it/s]
Loading 0:  53%|█████▎    | 193/363 [00:04<00:03, 45.79it/s]
Loading 0:  55%|█████▍    | 198/363 [00:04<00:03, 46.71it/s]
Loading 0:  56%|█████▌    | 203/363 [00:04<00:04, 38.94it/s]
Loading 0:  58%|█████▊    | 211/363 [00:05<00:03, 46.68it/s]
Loading 0:  60%|█████▉    | 216/363 [00:05<00:03, 47.46it/s]
Loading 0:  61%|██████    | 222/363 [00:05<00:03, 44.15it/s]
Loading 0:  63%|██████▎   | 227/363 [00:05<00:04, 31.52it/s]
Loading 0:  64%|██████▎   | 231/363 [00:05<00:04, 29.89it/s]
Loading 0:  65%|██████▌   | 237/363 [00:05<00:03, 35.88it/s]
Loading 0:  67%|██████▋   | 242/363 [00:05<00:03, 37.66it/s]
Loading 0:  68%|██████▊   | 247/363 [00:06<00:02, 39.58it/s]
Loading 0:  69%|██████▉   | 252/363 [00:06<00:02, 41.96it/s]
Loading 0:  71%|███████   | 257/363 [00:06<00:03, 34.47it/s]
Loading 0:  73%|███████▎  | 264/363 [00:06<00:02, 41.43it/s]
Loading 0:  74%|███████▍  | 269/363 [00:06<00:02, 41.34it/s]
Loading 0:  75%|███████▌  | 274/363 [00:06<00:02, 42.06it/s]
Loading 0:  77%|███████▋  | 279/363 [00:06<00:01, 43.16it/s]
Loading 0:  78%|███████▊  | 284/363 [00:07<00:02, 36.59it/s]
Loading 0:  80%|████████  | 291/363 [00:07<00:01, 42.62it/s]
Loading 0:  82%|████████▏ | 296/363 [00:07<00:01, 42.64it/s]
Loading 0:  83%|████████▎ | 302/363 [00:07<00:01, 46.51it/s]
Loading 0:  85%|████████▍ | 307/363 [00:14<00:22,  2.53it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:14,  3.43it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:07,  5.49it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:04,  7.41it/s]
Loading 0:  91%|█████████ | 331/363 [00:14<00:03,  9.46it/s]
Loading 0:  93%|█████████▎| 338/363 [00:14<00:01, 13.39it/s]
Loading 0:  95%|█████████▍| 344/363 [00:15<00:01, 16.17it/s]
Loading 0:  96%|█████████▌| 349/363 [00:15<00:00, 19.16it/s]
Loading 0:  98%|█████████▊| 356/363 [00:15<00:00, 24.96it/s]
Loading 0:  99%|█████████▉| 361/363 [00:15<00:00, 28.40it/s]
                                                            

Job chaiml-mistral-nemo-simp-1866-v2-mkmlizer completed after 94.14s with status: succeeded
Stopping job with name chaiml-mistral-nemo-simp-1866-v2-mkmlizer
Pipeline stage MKMLizer completed in 94.62s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.22s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-mistral-nemo-simp-1866-v2
Waiting for inference service chaiml-mistral-nemo-simp-1866-v2 to be ready
Inference service chaiml-mistral-nemo-simp-1866-v2 ready after 241.1892740726471s
Pipeline stage MKMLDeployer completed in 241.67s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.7384350299835205s
Received healthy response to inference request in 0.9783823490142822s
Received healthy response to inference request in 1.4686100482940674s
Received healthy response to inference request in 1.551957368850708s
5 requests
1 failed requests
5th percentile: 1.0764278888702392
10th percentile: 1.1744734287261962
20th percentile: 1.3705645084381104
30th percentile: 1.4852795124053955
40th percentile: 1.5186184406280518
50th percentile: 1.551957368850708
60th percentile: 1.626548433303833
70th percentile: 1.701139497756958
80th percentile: 5.421317958831791
90th percentile: 12.787083816528323
95th percentile: 16.469966745376585
99th percentile: 19.4162730884552
mean time: 5.1780468940734865
%s, retrying in %s seconds...
Received healthy response to inference request in 1.1341872215270996s
Received healthy response to inference request in 1.4505274295806885s
Received healthy response to inference request in 1.1958000659942627s
Received healthy response to inference request in 0.8403282165527344s
Received healthy response to inference request in 0.7262506484985352s
5 requests
0 failed requests
5th percentile: 0.749066162109375
10th percentile: 0.7718816757202148
20th percentile: 0.8175127029418945
30th percentile: 0.8991000175476074
40th percentile: 1.0166436195373536
50th percentile: 1.1341872215270996
60th percentile: 1.1588323593139649
70th percentile: 1.1834774971008302
80th percentile: 1.2467455387115478
90th percentile: 1.3486364841461183
95th percentile: 1.3995819568634034
99th percentile: 1.4403383350372314
mean time: 1.069418716430664
Pipeline stage StressChecker completed in 33.76s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.95s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.35s
Shutdown handler de-registered
chaiml-mistral-nemo-simp_1866_v2 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2941.92s
Shutdown handler de-registered
chaiml-mistral-nemo-simp_1866_v2 status is now inactive due to auto deactivation removed underperforming models