developer_uid: Riverise
submission_id: rica40325-dpo-1008-5162s_v5
model_name: rica40325-dpo-1008-5162s_v4
model_group: rica40325/dpo_1008_5162s
status: torndown
timestamp: 2024-10-08T13:10:30+00:00
num_battles: 6301
num_wins: 3460
celo_rating: 1289.84
family_friendly_score: 0.5708670075320248
family_friendly_standard_error: 0.006333523599278311
submission_type: basic
model_repo: rica40325/dpo_1008_5162s
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
display_name: rica40325-dpo-1008-5162s_v4
is_internal_developer: False
language_model: rica40325/dpo_1008_5162s
model_size: 13B
ranking_group: single
us_pacific_date: 2024-10-08
win_ratio: 0.5491191874305665
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rica40325-dpo-1008-5162s-v5-mkmlizer
Waiting for job on rica40325-dpo-1008-5162s-v5-mkmlizer to finish
Failed to get response for submission chaiml-elo-alignment-run-3_v34: ('http://chaiml-elo-alignment-run-3-v34-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'readfrom tcp 127.0.0.1:58214->127.0.0.1:8080: write tcp 127.0.0.1:58214->127.0.0.1:8080: write: broken pipe\n')
rica40325-dpo-1008-5162s-v5-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rica40325-dpo-1008-5162s-v5-mkmlizer: ║     _____            __           __                                ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ║       /___/                                                         ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ║                                                                     ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ║  Version: 0.11.12                                                   ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ║  https://mk1.ai                                                     ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ║                                                                     ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ║  The license key for the current software has been verified as      ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ║  belonging to:                                                      ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ║                                                                     ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ║  Chai Research Corp.                                                ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ║                                                                     ║
rica40325-dpo-1008-5162s-v5-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rica40325-dpo-1008-5162s-v5-mkmlizer: Downloaded to shared memory in 53.924s
rica40325-dpo-1008-5162s-v5-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpw9fzcdm7, device:0
rica40325-dpo-1008-5162s-v5-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rica40325-dpo-1008-5162s-v5-mkmlizer: quantized model in 41.866s
rica40325-dpo-1008-5162s-v5-mkmlizer: Processed model rica40325/dpo_1008_5162s in 95.790s
rica40325-dpo-1008-5162s-v5-mkmlizer: creating bucket guanaco-mkml-models
rica40325-dpo-1008-5162s-v5-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rica40325-dpo-1008-5162s-v5-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rica40325-dpo-1008-5162s-v5
rica40325-dpo-1008-5162s-v5-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rica40325-dpo-1008-5162s-v5/config.json
rica40325-dpo-1008-5162s-v5-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rica40325-dpo-1008-5162s-v5/special_tokens_map.json
rica40325-dpo-1008-5162s-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rica40325-dpo-1008-5162s-v5/tokenizer_config.json
rica40325-dpo-1008-5162s-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rica40325-dpo-1008-5162s-v5/tokenizer.json
rica40325-dpo-1008-5162s-v5-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rica40325-dpo-1008-5162s-v5/flywheel_model.0.safetensors
rica40325-dpo-1008-5162s-v5-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:16, 21.63it/s]
Loading 0:   3%|▎         | 10/363 [00:00<00:12, 28.08it/s]
Loading 0:   4%|▍         | 14/363 [00:00<00:14, 24.47it/s]
Loading 0:   6%|▌         | 20/363 [00:00<00:10, 33.22it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:15, 22.49it/s]
Loading 0:   7%|▋         | 27/363 [00:01<00:15, 21.71it/s]
Loading 0:   9%|▊         | 31/363 [00:01<00:13, 24.97it/s]
Loading 0:   9%|▉         | 34/363 [00:01<00:13, 25.20it/s]
Loading 0:  11%|█         | 39/363 [00:01<00:11, 28.13it/s]
Loading 0:  12%|█▏        | 43/363 [00:01<00:11, 28.08it/s]
Loading 0:  13%|█▎        | 48/363 [00:01<00:10, 30.94it/s]
Loading 0:  14%|█▍        | 52/363 [00:01<00:11, 28.22it/s]
Loading 0:  15%|█▌        | 56/363 [00:02<00:11, 27.75it/s]
Loading 0:  17%|█▋        | 61/363 [00:02<00:12, 24.14it/s]
Loading 0:  18%|█▊        | 64/363 [00:02<00:14, 20.67it/s]
Loading 0:  19%|█▉        | 69/363 [00:02<00:11, 25.31it/s]
Loading 0:  20%|█▉        | 72/363 [00:02<00:12, 22.93it/s]
Loading 0:  21%|██        | 77/363 [00:03<00:12, 23.63it/s]
Loading 0:  23%|██▎       | 84/363 [00:03<00:09, 30.16it/s]
Loading 0:  24%|██▍       | 88/363 [00:03<00:09, 28.85it/s]
Loading 0:  26%|██▌       | 93/363 [00:03<00:08, 31.50it/s]
Loading 0:  27%|██▋       | 97/363 [00:03<00:08, 29.93it/s]
Loading 0:  28%|██▊       | 101/363 [00:03<00:10, 24.00it/s]
Loading 0:  29%|██▊       | 104/363 [00:04<00:12, 21.25it/s]
Loading 0:  31%|███       | 111/363 [00:04<00:08, 28.32it/s]
Loading 0:  32%|███▏      | 115/363 [00:04<00:08, 27.98it/s]
Loading 0:  33%|███▎      | 120/363 [00:04<00:07, 30.59it/s]
Loading 0:  34%|███▍      | 124/363 [00:04<00:08, 29.42it/s]
Loading 0:  36%|███▌      | 129/363 [00:04<00:07, 31.00it/s]
Loading 0:  37%|███▋      | 133/363 [00:04<00:07, 28.87it/s]
Loading 0:  37%|███▋      | 136/363 [00:05<00:07, 28.76it/s]
Loading 0:  39%|███▉      | 141/363 [00:05<00:07, 30.57it/s]
Loading 0:  40%|███▉      | 145/363 [00:05<00:10, 21.67it/s]
Loading 0:  41%|████      | 149/363 [00:05<00:10, 20.84it/s]
Loading 0:  43%|████▎     | 156/363 [00:05<00:07, 27.69it/s]
Loading 0:  44%|████▍     | 160/363 [00:06<00:07, 27.30it/s]
Loading 0:  45%|████▌     | 165/363 [00:06<00:06, 29.66it/s]
Loading 0:  47%|████▋     | 169/363 [00:06<00:06, 28.09it/s]
Loading 0:  48%|████▊     | 174/363 [00:06<00:06, 29.39it/s]
Loading 0:  49%|████▉     | 178/363 [00:06<00:06, 28.00it/s]
Loading 0:  50%|█████     | 182/363 [00:06<00:07, 23.11it/s]
Loading 0:  51%|█████     | 185/363 [00:07<00:08, 20.24it/s]
Loading 0:  53%|█████▎    | 192/363 [00:07<00:06, 27.10it/s]
Loading 0:  54%|█████▎    | 195/363 [00:07<00:06, 25.17it/s]
Loading 0:  55%|█████▌    | 201/363 [00:07<00:05, 29.60it/s]
Loading 0:  56%|█████▋    | 205/363 [00:07<00:05, 27.61it/s]
Loading 0:  58%|█████▊    | 210/363 [00:07<00:05, 29.28it/s]
Loading 0:  59%|█████▉    | 214/363 [00:08<00:05, 28.46it/s]
Loading 0:  60%|██████    | 218/363 [00:08<00:04, 29.33it/s]
Loading 0:  61%|██████▏   | 223/363 [00:08<00:05, 25.41it/s]
Loading 0:  62%|██████▏   | 226/363 [00:08<00:05, 23.82it/s]
Loading 0:  63%|██████▎   | 230/363 [00:08<00:05, 22.81it/s]
Loading 0:  65%|██████▌   | 237/363 [00:08<00:04, 29.18it/s]
Loading 0:  66%|██████▋   | 241/363 [00:09<00:04, 28.13it/s]
Loading 0:  68%|██████▊   | 246/363 [00:09<00:03, 30.78it/s]
Loading 0:  69%|██████▉   | 250/363 [00:09<00:03, 29.50it/s]
Loading 0:  70%|███████   | 255/363 [00:09<00:03, 32.14it/s]
Loading 0:  71%|███████▏  | 259/363 [00:09<00:03, 29.20it/s]
Loading 0:  72%|███████▏  | 263/363 [00:09<00:04, 23.62it/s]
Loading 0:  73%|███████▎  | 266/363 [00:10<00:04, 20.89it/s]
Loading 0:  75%|███████▍  | 271/363 [00:10<00:03, 25.72it/s]
Loading 0:  76%|███████▌  | 275/363 [00:10<00:03, 23.48it/s]
Loading 0:  78%|███████▊  | 282/363 [00:10<00:02, 30.37it/s]
Loading 0:  79%|███████▉  | 286/363 [00:10<00:02, 28.75it/s]
Loading 0:  80%|████████  | 291/363 [00:10<00:02, 30.30it/s]
Loading 0:  81%|████████▏ | 295/363 [00:11<00:02, 28.72it/s]
Loading 0:  82%|████████▏ | 298/363 [00:11<00:02, 28.79it/s]
Loading 0:  83%|████████▎ | 303/363 [00:11<00:01, 30.54it/s]
Loading 0:  85%|████████▍ | 307/363 [00:11<00:02, 21.59it/s]
Loading 0:  86%|████████▌ | 311/363 [00:11<00:02, 21.07it/s]
Loading 0:  88%|████████▊ | 318/363 [00:11<00:01, 28.02it/s]
Loading 0:  89%|████████▊ | 322/363 [00:12<00:01, 27.87it/s]
Loading 0:  90%|█████████ | 327/363 [00:12<00:01, 29.86it/s]
Loading 0:  91%|█████████ | 331/363 [00:12<00:01, 27.85it/s]
Loading 0:  93%|█████████▎| 336/363 [00:12<00:00, 29.73it/s]
Loading 0:  94%|█████████▎| 340/363 [00:12<00:00, 27.96it/s]
Loading 0:  94%|█████████▍| 343/363 [00:12<00:00, 28.18it/s]
Loading 0:  95%|█████████▌| 346/363 [00:19<00:09,  1.71it/s]
Loading 0:  96%|█████████▌| 349/363 [00:19<00:06,  2.23it/s]
Loading 0:  97%|█████████▋| 353/363 [00:20<00:03,  3.18it/s]
Loading 0:  98%|█████████▊| 357/363 [00:20<00:01,  4.34it/s]
Loading 0: 100%|█████████▉| 362/363 [00:20<00:00,  6.49it/s]
                                                            

Job rica40325-dpo-1008-5162s-v5-mkmlizer completed after 124.37s with status: succeeded
Stopping job with name rica40325-dpo-1008-5162s-v5-mkmlizer
Pipeline stage MKMLizer completed in 124.88s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rica40325-dpo-1008-5162s-v5
Waiting for inference service rica40325-dpo-1008-5162s-v5 to be ready
Inference service rica40325-dpo-1008-5162s-v5 ready after 130.8676517009735s
Pipeline stage MKMLDeployer completed in 131.42s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.293952703475952s
Received healthy response to inference request in 1.6598515510559082s
Received healthy response to inference request in 1.812371015548706s
Received healthy response to inference request in 1.791203498840332s
Received healthy response to inference request in 1.7861707210540771s
5 requests
0 failed requests
5th percentile: 1.685115385055542
10th percentile: 1.7103792190551759
20th percentile: 1.7609068870544433
30th percentile: 1.787177276611328
40th percentile: 1.7891903877258302
50th percentile: 1.791203498840332
60th percentile: 1.7996705055236817
70th percentile: 1.8081375122070313
80th percentile: 1.9086873531341553
90th percentile: 2.1013200283050537
95th percentile: 2.197636365890503
99th percentile: 2.2746894359588623
mean time: 1.8687098979949952
Pipeline stage StressChecker completed in 10.95s
Shutdown handler de-registered
rica40325-dpo-1008-5162s_v5 status is now deployed due to DeploymentManager action
rica40325-dpo-1008-5162s_v5 status is now inactive due to auto deactivation removed underperforming models
rica40325-dpo-1008-5162s_v5 status is now torndown due to DeploymentManager action