jic062-orpo-v0-5-nemo

submission_id: jic062-orpo-v0-5-nemo_v1
developer_uid: chace9580
best_of: 8
celo_rating: 1275.51
display_name: jic062-orpo-v0-5-nemo_v1
family_friendly_score: 0.5684
family_friendly_standard_error: 0.007004590494811242
formatter: {'memory_template': '[INST]system\n{memory}[/INST]\n', 'prompt_template': '[INST]user\n{prompt}[/INST]\n', 'bot_template': '[INST]assistant\n{bot_name}: {message}[/INST]\n', 'user_template': '[INST]user\n{user_name}: {message}[/INST]\n', 'response_template': '[INST]assistant\n{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 0.9, 'min_p': 0.05, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
is_internal_developer: False
language_model: jic062/orpo-v0.5-Nemo
max_input_tokens: 1024
max_output_tokens: 64
model_architecture: MistralForCausalLM
model_group: jic062/orpo-v0.5-Nemo
model_name: jic062-orpo-v0-5-nemo_v1
model_num_parameters: 12772070400.0
model_repo: jic062/orpo-v0.5-Nemo
model_size: 13B
num_battles: 13857
num_wins: 7708
ranking_group: single
status: inactive
submission_type: basic
timestamp: 2024-10-25T15:38:27+00:00
us_pacific_date: 2024-10-25
win_ratio: 0.5562531572490438
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name jic062-orpo-v0-5-nemo-v1-mkmlizer
Waiting for job on jic062-orpo-v0-5-nemo-v1-mkmlizer to finish
jic062-orpo-v0-5-nemo-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║     _____            __           __                                ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║       /___/                                                         ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║                                                                     ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║  https://mk1.ai                                                     ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║                                                                     ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║  belonging to:                                                      ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║                                                                     ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║  Chai Research Corp.                                                ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ║                                                                     ║
jic062-orpo-v0-5-nemo-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
jic062-orpo-v0-5-nemo-v1-mkmlizer: Downloaded to shared memory in 55.567s
jic062-orpo-v0-5-nemo-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpp3u0j6y_, device:0
jic062-orpo-v0-5-nemo-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
jic062-orpo-v0-5-nemo-v1-mkmlizer: quantized model in 37.426s
jic062-orpo-v0-5-nemo-v1-mkmlizer: Processed model jic062/orpo-v0.5-Nemo in 92.993s
jic062-orpo-v0-5-nemo-v1-mkmlizer: creating bucket guanaco-mkml-models
jic062-orpo-v0-5-nemo-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
jic062-orpo-v0-5-nemo-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/jic062-orpo-v0-5-nemo-v1
jic062-orpo-v0-5-nemo-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/jic062-orpo-v0-5-nemo-v1/config.json
jic062-orpo-v0-5-nemo-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/jic062-orpo-v0-5-nemo-v1/special_tokens_map.json
jic062-orpo-v0-5-nemo-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/jic062-orpo-v0-5-nemo-v1/tokenizer_config.json
jic062-orpo-v0-5-nemo-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/jic062-orpo-v0-5-nemo-v1/tokenizer.json
jic062-orpo-v0-5-nemo-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/jic062-orpo-v0-5-nemo-v1/flywheel_model.0.safetensors
jic062-orpo-v0-5-nemo-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:12, 29.67it/s]
Loading 0:   3%|▎         | 12/363 [00:00<00:07, 47.38it/s]
Loading 0:   5%|▍         | 18/363 [00:00<00:07, 48.03it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:09, 37.16it/s]
Loading 0:   8%|▊         | 30/363 [00:00<00:07, 41.92it/s]
Loading 0:  10%|▉         | 35/363 [00:00<00:07, 41.17it/s]
Loading 0:  11%|█         | 40/363 [00:00<00:07, 41.05it/s]
Loading 0:  12%|█▏        | 45/363 [00:01<00:07, 42.27it/s]
Loading 0:  14%|█▍        | 50/363 [00:01<00:09, 34.63it/s]
Loading 0:  15%|█▌        | 56/363 [00:01<00:07, 39.74it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:09, 30.87it/s]
Loading 0:  18%|█▊        | 65/363 [00:01<00:09, 30.33it/s]
Loading 0:  20%|█▉        | 71/363 [00:01<00:08, 35.84it/s]
Loading 0:  21%|██        | 76/363 [00:02<00:07, 36.82it/s]
Loading 0:  22%|██▏       | 81/363 [00:02<00:07, 37.71it/s]
Loading 0:  23%|██▎       | 85/363 [00:02<00:07, 37.68it/s]
Loading 0:  25%|██▍       | 89/363 [00:02<00:07, 37.73it/s]
Loading 0:  26%|██▌       | 93/363 [00:02<00:07, 35.80it/s]
Loading 0:  27%|██▋       | 98/363 [00:02<00:07, 37.73it/s]
Loading 0:  28%|██▊       | 102/363 [00:02<00:07, 35.70it/s]
Loading 0:  29%|██▉       | 106/363 [00:02<00:07, 34.79it/s]
Loading 0:  30%|███       | 110/363 [00:02<00:07, 35.82it/s]
Loading 0:  31%|███▏      | 114/363 [00:03<00:07, 34.40it/s]
Loading 0:  33%|███▎      | 118/363 [00:03<00:07, 32.01it/s]
Loading 0:  34%|███▍      | 125/363 [00:03<00:06, 39.00it/s]
Loading 0:  36%|███▌      | 129/363 [00:03<00:06, 37.08it/s]
Loading 0:  37%|███▋      | 134/363 [00:03<00:05, 38.53it/s]
Loading 0:  38%|███▊      | 138/363 [00:03<00:06, 36.72it/s]
Loading 0:  39%|███▉      | 142/363 [00:04<00:08, 25.52it/s]
Loading 0:  40%|████      | 146/363 [00:04<00:07, 27.46it/s]
Loading 0:  41%|████▏     | 150/363 [00:04<00:07, 27.18it/s]
Loading 0:  42%|████▏     | 154/363 [00:04<00:06, 29.96it/s]
Loading 0:  44%|████▎     | 158/363 [00:04<00:07, 29.08it/s]
Loading 0:  45%|████▍     | 163/363 [00:04<00:05, 33.60it/s]
Loading 0:  46%|████▌     | 167/363 [00:04<00:06, 31.16it/s]
Loading 0:  48%|████▊     | 174/363 [00:04<00:04, 38.20it/s]
Loading 0:  49%|████▉     | 179/363 [00:05<00:04, 38.15it/s]
Loading 0:  50%|█████     | 183/363 [00:05<00:04, 36.94it/s]
Loading 0:  52%|█████▏    | 187/363 [00:05<00:04, 35.36it/s]
Loading 0:  53%|█████▎    | 192/363 [00:05<00:04, 37.31it/s]
Loading 0:  54%|█████▍    | 196/363 [00:05<00:04, 34.99it/s]
Loading 0:  55%|█████▌    | 201/363 [00:05<00:04, 36.20it/s]
Loading 0:  56%|█████▋    | 205/363 [00:05<00:04, 34.38it/s]
Loading 0:  58%|█████▊    | 210/363 [00:05<00:04, 36.40it/s]
Loading 0:  59%|█████▉    | 214/363 [00:06<00:04, 36.46it/s]
Loading 0:  60%|██████    | 218/363 [00:06<00:03, 36.46it/s]
Loading 0:  61%|██████▏   | 223/363 [00:06<00:04, 29.88it/s]
Loading 0:  63%|██████▎   | 227/363 [00:06<00:04, 31.33it/s]
Loading 0:  64%|██████▎   | 231/363 [00:06<00:04, 30.83it/s]
Loading 0:  65%|██████▌   | 237/363 [00:06<00:03, 36.98it/s]
Loading 0:  66%|██████▋   | 241/363 [00:06<00:03, 37.24it/s]
Loading 0:  68%|██████▊   | 247/363 [00:06<00:02, 41.24it/s]
Loading 0:  69%|██████▉   | 252/363 [00:07<00:02, 43.27it/s]
Loading 0:  71%|███████   | 257/363 [00:07<00:03, 35.12it/s]
Loading 0:  73%|███████▎  | 264/363 [00:07<00:02, 42.55it/s]
Loading 0:  74%|███████▍  | 269/363 [00:07<00:02, 42.75it/s]
Loading 0:  75%|███████▌  | 274/363 [00:07<00:02, 42.54it/s]
Loading 0:  77%|███████▋  | 279/363 [00:07<00:01, 43.29it/s]
Loading 0:  78%|███████▊  | 284/363 [00:07<00:02, 34.96it/s]
Loading 0:  80%|████████  | 291/363 [00:08<00:01, 40.40it/s]
Loading 0:  82%|████████▏ | 296/363 [00:08<00:01, 40.15it/s]
Loading 0:  83%|████████▎ | 301/363 [00:08<00:01, 41.50it/s]
Loading 0:  84%|████████▍ | 306/363 [00:15<00:23,  2.46it/s]
Loading 0:  85%|████████▌ | 310/363 [00:15<00:16,  3.18it/s]
Loading 0:  87%|████████▋ | 314/363 [00:15<00:11,  4.17it/s]
Loading 0:  88%|████████▊ | 319/363 [00:15<00:07,  5.88it/s]
Loading 0:  89%|████████▉ | 323/363 [00:15<00:05,  7.56it/s]
Loading 0:  91%|█████████ | 329/363 [00:15<00:03, 10.95it/s]
Loading 0:  92%|█████████▏| 335/363 [00:15<00:01, 14.44it/s]
Loading 0:  94%|█████████▎| 340/363 [00:15<00:01, 17.76it/s]
Loading 0:  96%|█████████▌| 347/363 [00:16<00:00, 24.01it/s]
Loading 0:  97%|█████████▋| 353/363 [00:16<00:00, 27.27it/s]
Loading 0:  99%|█████████▊| 358/363 [00:16<00:00, 29.80it/s]
                                                            
Job jic062-orpo-v0-5-nemo-v1-mkmlizer completed after 114.33s with status: succeeded
Stopping job with name jic062-orpo-v0-5-nemo-v1-mkmlizer
Pipeline stage MKMLizer completed in 114.86s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.22s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service jic062-orpo-v0-5-nemo-v1
Waiting for inference service jic062-orpo-v0-5-nemo-v1 to be ready
Inference service jic062-orpo-v0-5-nemo-v1 ready after 130.92505884170532s
Pipeline stage MKMLDeployer completed in 131.63s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.3437986373901367s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Received healthy response to inference request in 2.1376030445098877s
Received healthy response to inference request in 1.83058500289917s
Received healthy response to inference request in 1.9067418575286865s
5 requests
1 failed requests
5th percentile: 1.8458163738250732
10th percentile: 1.8610477447509766
20th percentile: 1.8915104866027832
30th percentile: 1.9529140949249268
40th percentile: 2.045258569717407
50th percentile: 2.1376030445098877
60th percentile: 2.2200812816619875
70th percentile: 2.302559518814087
80th percentile: 5.909163904190066
90th percentile: 13.039894437789918
95th percentile: 16.60525970458984
99th percentile: 19.457551918029786
mean time: 5.6778707027435305
%s, retrying in %s seconds...
Received healthy response to inference request in 1.7307369709014893s
Received healthy response to inference request in 1.768824815750122s
Received healthy response to inference request in 2.230334520339966s
Received healthy response to inference request in 2.1028153896331787s
Received healthy response to inference request in 1.8275184631347656s
5 requests
0 failed requests
5th percentile: 1.738354539871216
10th percentile: 1.7459721088409423
20th percentile: 1.7612072467803954
30th percentile: 1.7805635452270507
40th percentile: 1.8040410041809083
50th percentile: 1.8275184631347656
60th percentile: 1.9376372337341308
70th percentile: 2.047756004333496
80th percentile: 2.1283192157745363
90th percentile: 2.179326868057251
95th percentile: 2.204830694198608
99th percentile: 2.2252337551116943
mean time: 1.9320460319519044
Pipeline stage StressChecker completed in 40.86s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 5.44s
Shutdown handler de-registered
jic062-orpo-v0-5-nemo_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2610.97s
Shutdown handler de-registered
jic062-orpo-v0-5-nemo_v1 status is now inactive due to auto deactivation removed underperforming models