chaiml-mistral-nemo-simp_1866

developer_uid: azuruce

submission_id: chaiml-mistral-nemo-simp_1866_v4

model_name: chaiml-mistral-nemo-simp_1866_v4

model_group: ChaiML/mistral_nemo_simp

status: inactive

timestamp: 2024-12-18T19:22:34+00:00

num_battles: 16526

num_wins: 7731

celo_rating: 1235.09

family_friendly_score: 0.5886

family_friendly_standard_error: 0.006959167191553886

submission_type: basic

model_repo: ChaiML/mistral_nemo_simpo_baseline_albert_20241217_v1-checkpoint-125

model_architecture: MistralForCausalLM

model_num_parameters: 12772070400.0

best_of: 4

max_input_tokens: 1024

max_output_tokens: 64

latencies: [{'batch_size': 1, 'throughput': 0.6355346784228202, 'latency_mean': 1.5733938014507294, 'latency_p50': 1.5697928667068481, 'latency_p90': 1.7260351181030273}, {'batch_size': 3, 'throughput': 1.2579768762368075, 'latency_mean': 2.3781853330135347, 'latency_p50': 2.390535354614258, 'latency_p90': 2.5984230756759645}, {'batch_size': 5, 'throughput': 1.5939541755427815, 'latency_mean': 3.1198113775253296, 'latency_p50': 3.109305500984192, 'latency_p90': 3.542547178268433}, {'batch_size': 6, 'throughput': 1.6914800923048876, 'latency_mean': 3.524151768684387, 'latency_p50': 3.5082350969314575, 'latency_p90': 3.956793713569641}, {'batch_size': 8, 'throughput': 1.8498877280246433, 'latency_mean': 4.2940918779373165, 'latency_p50': 4.287263751029968, 'latency_p90': 4.883940482139588}, {'batch_size': 10, 'throughput': 1.9421333216554477, 'latency_mean': 5.104416782855988, 'latency_p50': 5.068885445594788, 'latency_p90': 5.778931355476379}]

gpu_counts: {'NVIDIA RTX A5000': 1}

display_name: chaiml-mistral-nemo-simp_1866_v4

is_internal_developer: True

language_model: ChaiML/mistral_nemo_simpo_baseline_albert_20241217_v1-checkpoint-125

model_size: 13B

ranking_group: single

throughput_3p7s: 1.75

us_pacific_date: 2024-12-18

win_ratio: 0.4678083020694663

generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 100, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '####', 'Bot:', 'User:', 'You:', '<|im_end|>', '<|eot_id|>'], 'max_input_tokens': 1024, 'best_of': 4, 'max_output_tokens': 64}

formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-mistral-nemo-simp-1866-v4-mkmlizer
Waiting for job on chaiml-mistral-nemo-simp-1866-v4-mkmlizer to finish
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║     _____            __           __                                ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║       /___/                                                         ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║                                                                     ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║  Version: 0.11.12                                                   ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║                                                                     ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║  belonging to:                                                      ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║                                                                     ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ║                                                                     ║
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: Downloaded to shared memory in 30.657s
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpartxt650, device:0
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: quantized model in 35.797s
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: Processed model ChaiML/mistral_nemo_simpo_baseline_albert_20241217_v1-checkpoint-125 in 66.454s
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: creating bucket guanaco-mkml-models
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-1866-v4
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-1866-v4/config.json
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-1866-v4/special_tokens_map.json
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-1866-v4/tokenizer_config.json
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-1866-v4/tokenizer.json
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-1866-v4/flywheel_model.0.safetensors
chaiml-mistral-nemo-simp-1866-v4-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:12, 28.14it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:07, 49.88it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 48.56it/s]
Loading 0:   7%|▋         | 25/363 [00:00<00:06, 49.74it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 52.49it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:06, 50.76it/s]
Loading 0:  12%|█▏        | 43/363 [00:00<00:06, 51.65it/s]
Loading 0:  13%|█▎        | 49/363 [00:00<00:06, 52.30it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 45.14it/s]
Loading 0:  17%|█▋        | 60/363 [00:01<00:06, 46.08it/s]
Loading 0:  18%|█▊        | 65/363 [00:01<00:10, 29.78it/s]
Loading 0:  20%|█▉        | 71/363 [00:01<00:08, 34.96it/s]
Loading 0:  21%|██        | 76/363 [00:01<00:07, 36.32it/s]
Loading 0:  22%|██▏       | 81/363 [00:01<00:07, 37.71it/s]
Loading 0:  24%|██▎       | 86/363 [00:02<00:07, 39.33it/s]
Loading 0:  25%|██▌       | 91/363 [00:02<00:08, 33.10it/s]
Loading 0:  27%|██▋       | 99/363 [00:02<00:06, 41.38it/s]
Loading 0:  29%|██▉       | 105/363 [00:02<00:06, 41.47it/s]
Loading 0:  31%|███       | 112/363 [00:02<00:05, 45.87it/s]
Loading 0:  32%|███▏      | 117/363 [00:02<00:05, 43.95it/s]
Loading 0:  34%|███▎      | 122/363 [00:02<00:05, 44.75it/s]
Loading 0:  35%|███▍      | 127/363 [00:03<00:06, 38.76it/s]
Loading 0:  37%|███▋      | 135/363 [00:03<00:04, 46.66it/s]
Loading 0:  39%|███▉      | 141/363 [00:03<00:04, 45.26it/s]
Loading 0:  40%|████      | 146/363 [00:03<00:06, 31.32it/s]
Loading 0:  41%|████▏     | 150/363 [00:03<00:06, 31.55it/s]
Loading 0:  43%|████▎     | 157/363 [00:03<00:05, 38.47it/s]
Loading 0:  45%|████▍     | 163/363 [00:03<00:05, 38.96it/s]
Loading 0:  46%|████▋     | 168/363 [00:04<00:05, 38.64it/s]
Loading 0:  48%|████▊     | 174/363 [00:04<00:04, 42.48it/s]
Loading 0:  49%|████▉     | 179/363 [00:04<00:04, 41.85it/s]
Loading 0:  51%|█████     | 184/363 [00:04<00:04, 42.00it/s]
Loading 0:  52%|█████▏    | 190/363 [00:04<00:04, 41.66it/s]
Loading 0:  54%|█████▎    | 195/363 [00:04<00:04, 41.76it/s]
Loading 0:  56%|█████▌    | 202/363 [00:04<00:03, 46.46it/s]
Loading 0:  57%|█████▋    | 208/363 [00:05<00:03, 43.95it/s]
Loading 0:  59%|█████▊    | 213/363 [00:05<00:03, 42.98it/s]
Loading 0:  60%|██████    | 219/363 [00:05<00:03, 47.12it/s]
Loading 0:  62%|██████▏   | 224/363 [00:05<00:04, 33.49it/s]
Loading 0:  63%|██████▎   | 228/363 [00:05<00:04, 33.56it/s]
Loading 0:  64%|██████▍   | 232/363 [00:05<00:03, 33.42it/s]
Loading 0:  66%|██████▌   | 238/363 [00:05<00:03, 39.24it/s]
Loading 0:  67%|██████▋   | 244/363 [00:05<00:03, 39.65it/s]
Loading 0:  69%|██████▊   | 249/363 [00:06<00:02, 40.62it/s]
Loading 0:  71%|███████   | 256/363 [00:06<00:02, 46.29it/s]
Loading 0:  72%|███████▏  | 262/363 [00:06<00:02, 45.71it/s]
Loading 0:  74%|███████▎  | 267/363 [00:06<00:02, 45.22it/s]
Loading 0:  75%|███████▌  | 274/363 [00:06<00:01, 49.47it/s]
Loading 0:  77%|███████▋  | 280/363 [00:06<00:01, 48.04it/s]
Loading 0:  79%|███████▊  | 285/363 [00:06<00:01, 47.35it/s]
Loading 0:  80%|████████  | 292/363 [00:06<00:01, 51.42it/s]
Loading 0:  82%|████████▏ | 298/363 [00:07<00:01, 48.37it/s]
Loading 0:  84%|████████▎ | 304/363 [00:13<00:20,  2.83it/s]
Loading 0:  85%|████████▍ | 308/363 [00:14<00:15,  3.55it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:11,  4.52it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:05,  7.30it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:03,  9.68it/s]
Loading 0:  91%|█████████ | 331/363 [00:14<00:02, 12.09it/s]
Loading 0:  93%|█████████▎| 338/363 [00:14<00:01, 16.78it/s]
Loading 0:  95%|█████████▍| 344/363 [00:14<00:00, 20.32it/s]
Loading 0:  96%|█████████▌| 349/363 [00:14<00:00, 23.59it/s]
Loading 0:  98%|█████████▊| 356/363 [00:15<00:00, 30.16it/s]
Loading 0: 100%|█████████▉| 362/363 [00:15<00:00, 32.34it/s]
                                                            
Job chaiml-mistral-nemo-simp-1866-v4-mkmlizer completed after 93.95s with status: succeeded
Stopping job with name chaiml-mistral-nemo-simp-1866-v4-mkmlizer
Pipeline stage MKMLizer completed in 94.46s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-mistral-nemo-simp-1866-v4
Waiting for inference service chaiml-mistral-nemo-simp-1866-v4 to be ready
Inference service chaiml-mistral-nemo-simp-1866-v4 ready after 251.06308841705322s
Pipeline stage MKMLDeployer completed in 251.65s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.1137621402740479s
Received healthy response to inference request in 1.2799293994903564s
Received healthy response to inference request in 0.6065690517425537s
Received healthy response to inference request in 1.0021030902862549s
{"detail":"('http://chaiml-llama-8b-multihea-7878-v5-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:42464->127.0.0.1:8080: read: connection reset by peer\\n')"}
Received unhealthy response to inference request!
5 requests
1 failed requests
5th percentile: 0.5824927806854248
10th percentile: 0.588511848449707
20th percentile: 0.6005499839782715
30th percentile: 0.685675859451294
40th percentile: 0.8438894748687744
50th percentile: 1.0021030902862549
60th percentile: 1.046766710281372
70th percentile: 1.0914303302764892
80th percentile: 1.1469955921173096
90th percentile: 1.213462495803833
95th percentile: 1.2466959476470947
99th percentile: 1.273282709121704
mean time: 0.9157674789428711
%s, retrying in %s seconds...
Received healthy response to inference request in 1.8280649185180664s
Received healthy response to inference request in 2.656248092651367s
Received healthy response to inference request in 1.6010761260986328s
Received healthy response to inference request in 0.6924588680267334s
Received healthy response to inference request in 0.767561674118042s
5 requests
0 failed requests
5th percentile: 0.7074794292449951
10th percentile: 0.7224999904632569
20th percentile: 0.7525411128997803
30th percentile: 0.9342645645141601
40th percentile: 1.2676703453063967
50th percentile: 1.6010761260986328
60th percentile: 1.6918716430664062
70th percentile: 1.7826671600341797
80th percentile: 1.9937015533447267
90th percentile: 2.324974822998047
95th percentile: 2.490611457824707
99th percentile: 2.623120765686035
mean time: 1.5090819358825684
Pipeline stage StressChecker completed in 15.01s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.27s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.15s
Shutdown handler de-registered
chaiml-mistral-nemo-simp_1866_v4 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2936.87s
Shutdown handler de-registered
chaiml-mistral-nemo-simp_1866_v4 status is now inactive due to auto deactivation removed underperforming models