developer_uid: azuruce
submission_id: chaiml-mistral-nemo-simp_8423_v1
model_name: chaiml-mistral-nemo-simp_8423_v1
model_group: ChaiML/mistral_nemo_simp
status: torndown
timestamp: 2024-12-15T23:14:41+00:00
num_battles: 9198
num_wins: 4339
celo_rating: 1209.0
family_friendly_score: 0.5731999999999999
family_friendly_standard_error: 0.00699488041355962
submission_type: basic
model_repo: ChaiML/mistral_nemo_simpo_baseline_albert_20241213_v2_mad-checkpoint-936
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 4
max_input_tokens: 1024
max_output_tokens: 64
latencies: [{'batch_size': 1, 'throughput': 0.6363769516006841, 'latency_mean': 1.5713356220722199, 'latency_p50': 1.5704753398895264, 'latency_p90': 1.732342576980591}, {'batch_size': 3, 'throughput': 1.2705995601990263, 'latency_mean': 2.3496360790729525, 'latency_p50': 2.3502039909362793, 'latency_p90': 2.657196307182312}, {'batch_size': 5, 'throughput': 1.6041549666070332, 'latency_mean': 3.0952789723873138, 'latency_p50': 3.1010504961013794, 'latency_p90': 3.532555103302002}, {'batch_size': 6, 'throughput': 1.728767513053969, 'latency_mean': 3.456528433561325, 'latency_p50': 3.463939666748047, 'latency_p90': 3.858552074432373}, {'batch_size': 8, 'throughput': 1.884183379146091, 'latency_mean': 4.216424978971482, 'latency_p50': 4.219954013824463, 'latency_p90': 4.8924195051193236}, {'batch_size': 10, 'throughput': 1.9601597387293646, 'latency_mean': 5.05728222489357, 'latency_p50': 5.0908684730529785, 'latency_p90': 5.69502956867218}]
gpu_counts: {'NVIDIA RTX A5000': 1}
display_name: chaiml-mistral-nemo-simp_8423_v1
is_internal_developer: True
language_model: ChaiML/mistral_nemo_simpo_baseline_albert_20241213_v2_mad-checkpoint-936
model_size: 13B
ranking_group: single
throughput_3p7s: 1.79
us_pacific_date: 2024-12-15
win_ratio: 0.47173298543161557
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 100, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '####', 'Bot:', 'User:', 'You:', '<|im_end|>', '<|eot_id|>'], 'max_input_tokens': 1024, 'best_of': 4, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-mistral-nemo-simp-8423-v1-mkmlizer
Waiting for job on chaiml-mistral-nemo-simp-8423-v1-mkmlizer to finish
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║     _____            __           __                                ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║       /___/                                                         ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║                                                                     ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║                                                                     ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║  belonging to:                                                      ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║                                                                     ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ║                                                                     ║
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: Downloaded to shared memory in 50.219s
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpcv26k6pi, device:0
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: quantized model in 38.157s
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: Processed model ChaiML/mistral_nemo_simpo_baseline_albert_20241213_v2_mad-checkpoint-936 in 88.377s
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: creating bucket guanaco-mkml-models
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-8423-v1
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-8423-v1/config.json
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-8423-v1/special_tokens_map.json
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-8423-v1/tokenizer_config.json
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-mistral-nemo-simp-8423-v1/tokenizer.json
chaiml-mistral-nemo-simp-8423-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:13, 26.25it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:07, 45.32it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:08, 42.38it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:08, 40.76it/s]
Loading 0:   8%|▊         | 30/363 [00:00<00:07, 45.84it/s]
Loading 0:  10%|▉         | 35/363 [00:00<00:07, 45.19it/s]
Loading 0:  11%|█         | 40/363 [00:00<00:07, 45.55it/s]
Loading 0:  13%|█▎        | 46/363 [00:01<00:07, 43.28it/s]
Loading 0:  14%|█▍        | 51/363 [00:01<00:07, 41.03it/s]
Loading 0:  15%|█▌        | 56/363 [00:01<00:07, 42.43it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:10, 30.04it/s]
Loading 0:  18%|█▊        | 65/363 [00:01<00:10, 29.06it/s]
Loading 0:  20%|█▉        | 71/363 [00:01<00:08, 34.52it/s]
Loading 0:  21%|██        | 75/363 [00:01<00:08, 34.07it/s]
Loading 0:  22%|██▏       | 81/363 [00:02<00:07, 38.15it/s]
Loading 0:  24%|██▎       | 86/363 [00:02<00:06, 39.98it/s]
Loading 0:  25%|██▌       | 91/363 [00:02<00:08, 33.35it/s]
Loading 0:  27%|██▋       | 98/363 [00:02<00:06, 39.97it/s]
Loading 0:  28%|██▊       | 103/363 [00:02<00:06, 39.73it/s]
Loading 0:  30%|██▉       | 108/363 [00:02<00:06, 41.78it/s]
Loading 0:  31%|███       | 113/363 [00:02<00:07, 34.52it/s]
Loading 0:  33%|███▎      | 118/363 [00:03<00:07, 34.03it/s]
Loading 0:  34%|███▍      | 125/363 [00:03<00:05, 40.86it/s]
Loading 0:  36%|███▌      | 130/363 [00:03<00:05, 39.90it/s]
Loading 0:  37%|███▋      | 135/363 [00:03<00:05, 39.69it/s]
Loading 0:  39%|███▊      | 140/363 [00:03<00:05, 41.29it/s]
Loading 0:  40%|███▉      | 145/363 [00:04<00:08, 24.40it/s]
Loading 0:  41%|████      | 149/363 [00:04<00:08, 24.65it/s]
Loading 0:  43%|████▎     | 156/363 [00:04<00:06, 31.62it/s]
Loading 0:  44%|████▍     | 160/363 [00:04<00:06, 31.49it/s]
Loading 0:  45%|████▌     | 165/363 [00:04<00:05, 34.08it/s]
Loading 0:  47%|████▋     | 169/363 [00:04<00:05, 33.02it/s]
Loading 0:  48%|████▊     | 174/363 [00:04<00:05, 35.44it/s]
Loading 0:  49%|████▉     | 178/363 [00:04<00:05, 34.01it/s]
Loading 0:  50%|█████     | 183/363 [00:05<00:04, 36.48it/s]
Loading 0:  52%|█████▏    | 187/363 [00:05<00:05, 34.60it/s]
Loading 0:  53%|█████▎    | 192/363 [00:05<00:04, 37.30it/s]
Loading 0:  54%|█████▍    | 196/363 [00:05<00:04, 35.37it/s]
Loading 0:  55%|█████▌    | 201/363 [00:05<00:04, 37.51it/s]
Loading 0:  56%|█████▋    | 205/363 [00:05<00:04, 35.55it/s]
Loading 0:  58%|█████▊    | 210/363 [00:05<00:04, 37.67it/s]
Loading 0:  59%|█████▉    | 214/363 [00:05<00:04, 34.07it/s]
Loading 0:  60%|██████    | 218/363 [00:06<00:04, 33.98it/s]
Loading 0:  61%|██████▏   | 223/363 [00:06<00:05, 24.97it/s]
Loading 0:  62%|██████▏   | 226/363 [00:06<00:05, 24.27it/s]
Loading 0:  63%|██████▎   | 230/363 [00:06<00:05, 24.78it/s]
Loading 0:  65%|██████▌   | 237/363 [00:06<00:03, 33.70it/s]
Loading 0:  66%|██████▋   | 241/363 [00:06<00:03, 33.46it/s]
Loading 0:  68%|██████▊   | 246/363 [00:06<00:03, 36.58it/s]
Loading 0:  69%|██████▉   | 250/363 [00:07<00:03, 35.40it/s]
Loading 0:  70%|███████   | 255/363 [00:07<00:02, 37.86it/s]
Loading 0:  71%|███████▏  | 259/363 [00:07<00:02, 36.49it/s]
Loading 0:  73%|███████▎  | 264/363 [00:07<00:02, 38.76it/s]
Loading 0:  74%|███████▍  | 268/363 [00:07<00:02, 36.68it/s]
Loading 0:  75%|███████▌  | 273/363 [00:07<00:02, 39.09it/s]
Loading 0:  76%|███████▋  | 277/363 [00:07<00:02, 37.78it/s]
Loading 0:  78%|███████▊  | 282/363 [00:07<00:01, 40.84it/s]
Loading 0:  79%|███████▉  | 287/363 [00:08<00:01, 41.09it/s]
Loading 0:  80%|████████  | 292/363 [00:08<00:01, 39.91it/s]
Loading 0:  82%|████████▏ | 297/363 [00:08<00:01, 39.22it/s]
Loading 0:  83%|████████▎ | 302/363 [00:08<00:01, 41.82it/s]
Loading 0:  85%|████████▍ | 307/363 [00:15<00:25,  2.23it/s]
Loading 0:  86%|████████▌ | 312/363 [00:15<00:16,  3.12it/s]
Loading 0:  88%|████████▊ | 320/363 [00:15<00:08,  5.13it/s]
Loading 0:  90%|████████▉ | 326/363 [00:15<00:05,  6.99it/s]
Loading 0:  91%|█████████ | 331/363 [00:15<00:03,  9.00it/s]
Loading 0:  93%|█████████▎| 338/363 [00:16<00:01, 12.81it/s]
Loading 0:  94%|█████████▍| 343/363 [00:16<00:01, 15.82it/s]
Loading 0:  96%|█████████▌| 348/363 [00:16<00:00, 17.94it/s]
Loading 0:  98%|█████████▊| 356/363 [00:16<00:00, 24.93it/s]
Loading 0: 100%|█████████▉| 362/363 [00:16<00:00, 28.02it/s]
                                                            

Job chaiml-mistral-nemo-simp-8423-v1-mkmlizer completed after 114.77s with status: succeeded
Stopping job with name chaiml-mistral-nemo-simp-8423-v1-mkmlizer
Pipeline stage MKMLizer completed in 115.25s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-mistral-nemo-simp-8423-v1
Waiting for inference service chaiml-mistral-nemo-simp-8423-v1 to be ready
Inference service chaiml-mistral-nemo-simp-8423-v1 ready after 211.10233092308044s
Pipeline stage MKMLDeployer completed in 211.54s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.068664789199829s
Received healthy response to inference request in 1.6635117530822754s
Received healthy response to inference request in 1.5267918109893799s
Received healthy response to inference request in 1.3532383441925049s
5 requests
1 failed requests
5th percentile: 1.38794903755188
10th percentile: 1.422659730911255
20th percentile: 1.4920811176300048
30th percentile: 1.554135799407959
40th percentile: 1.6088237762451172
50th percentile: 1.6635117530822754
60th percentile: 1.8255729675292969
70th percentile: 1.9876341819763184
80th percentile: 5.681430625915531
90th percentile: 12.906962299346926
95th percentile: 16.519728136062618
99th percentile: 19.40994080543518
mean time: 5.348940134048462
%s, retrying in %s seconds...
Received healthy response to inference request in 1.3405492305755615s
Received healthy response to inference request in 1.7945797443389893s
Received healthy response to inference request in 1.3736083507537842s
Received healthy response to inference request in 1.3185782432556152s
Received healthy response to inference request in 1.6112608909606934s
5 requests
0 failed requests
5th percentile: 1.3229724407196044
10th percentile: 1.3273666381835938
20th percentile: 1.3361550331115724
30th percentile: 1.347161054611206
40th percentile: 1.3603847026824951
50th percentile: 1.3736083507537842
60th percentile: 1.4686693668365478
70th percentile: 1.5637303829193114
80th percentile: 1.6479246616363525
90th percentile: 1.721252202987671
95th percentile: 1.75791597366333
99th percentile: 1.7872469902038575
mean time: 1.4877152919769288
Pipeline stage StressChecker completed in 36.55s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.01s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.03s
Shutdown handler de-registered
chaiml-mistral-nemo-simp_8423_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2442.34s
Shutdown handler de-registered
chaiml-mistral-nemo-simp_8423_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-mistral-nemo-simp_8423_v1 status is now torndown due to DeploymentManager action
chaiml-mistral-nemo-simp_8423_v1 status is now torndown due to DeploymentManager action
chaiml-mistral-nemo-simp_8423_v1 status is now torndown due to DeploymentManager action
ChatRequest
Generation Params
Prompt Formatter
Chat History
ChatMessage 1