submission_id: chaiml-albert-0823-dpo30_6942_v1
developer_uid: albert_chai
alignment_samples: 9432
alignment_score: -31.303730998571673
best_of: 8
celo_rating: 1099.12
display_name: albert_0823_dpo3000
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
is_internal_developer: True
language_model: ChaiML/albert_0823_dpo3000-1e5x2
max_input_tokens: 1024
max_output_tokens: 64
model_architecture: MistralForCausalLM
model_group: ChaiML/albert_0823_dpo30
model_name: albert_0823_dpo3000
model_num_parameters: 12772070400.0
model_repo: ChaiML/albert_0823_dpo3000-1e5x2
model_size: 13B
num_battles: 9432
num_wins: 3009
propriety_score: 0.5962145110410094
propriety_total_count: 951.0
ranking_group: single
status: torndown
submission_type: basic
timestamp: 2024-08-24T05:48:31+00:00
us_pacific_date: 2024-08-23
win_ratio: 0.3190203562340967
Download Preference Data
Resubmit model
Running pipeline stage MKMLizer
Starting job with name chaiml-albert-0823-dpo30-6942-v1-mkmlizer
Waiting for job on chaiml-albert-0823-dpo30-6942-v1-mkmlizer to finish
Stopping job with name chaiml-albert-0823-dpo30-6942-v1-mkmlizer
%s, retrying in %s seconds...
Starting job with name chaiml-albert-0823-dpo30-6942-v1-mkmlizer
Waiting for job on chaiml-albert-0823-dpo30-6942-v1-mkmlizer to finish
Stopping job with name chaiml-albert-0823-dpo30-6942-v1-mkmlizer
%s, retrying in %s seconds...
Starting job with name chaiml-albert-0823-dpo30-6942-v1-mkmlizer
Waiting for job on chaiml-albert-0823-dpo30-6942-v1-mkmlizer to finish
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║     _____            __           __                                ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║       /___/                                                         ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║                                                                     ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║  Version: 0.10.1                                                    ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║                                                                     ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║  belonging to:                                                      ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║                                                                     ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ║                                                                     ║
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission blend_lobuf_2024-08-22: ('http://chaiml-elo-alignment-run-3-v34-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"ValueError : [TypeError(\\"\'numpy.int64\' object is not iterable\\"), TypeError(\'vars() argument must have __dict__ attribute\')]"}')
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: Downloaded to shared memory in 102.971s
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpmfrdgw9t, device:0
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: quantized model in 41.900s
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: Processed model ChaiML/albert_0823_dpo3000-1e5x2 in 144.872s
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: creating bucket guanaco-mkml-models
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-albert-0823-dpo30-6942-v1
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-albert-0823-dpo30-6942-v1/special_tokens_map.json
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-albert-0823-dpo30-6942-v1/config.json
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-albert-0823-dpo30-6942-v1/tokenizer_config.json
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-albert-0823-dpo30-6942-v1/tokenizer.json
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-albert-0823-dpo30-6942-v1/flywheel_model.0.safetensors
chaiml-albert-0823-dpo30-6942-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:15, 22.55it/s]
Loading 0:   3%|▎         | 10/363 [00:00<00:12, 28.46it/s]
Loading 0:   4%|▍         | 14/363 [00:00<00:14, 24.66it/s]
Loading 0:   6%|▌         | 20/363 [00:00<00:10, 33.52it/s]
Loading 0:   7%|▋         | 24/363 [00:01<00:15, 21.29it/s]
Loading 0:   7%|▋         | 27/363 [00:01<00:15, 21.27it/s]
Loading 0:   9%|▉         | 32/363 [00:01<00:14, 22.57it/s]
Loading 0:  11%|█         | 39/363 [00:01<00:11, 29.18it/s]
Loading 0:  12%|█▏        | 43/363 [00:01<00:11, 28.74it/s]
Loading 0:  13%|█▎        | 48/363 [00:01<00:10, 30.86it/s]
Loading 0:  14%|█▍        | 52/363 [00:01<00:10, 29.38it/s]
Loading 0:  15%|█▌        | 56/363 [00:02<00:10, 29.88it/s]
Loading 0:  17%|█▋        | 61/363 [00:02<00:11, 25.50it/s]
Loading 0:  18%|█▊        | 64/363 [00:02<00:13, 22.72it/s]
Loading 0:  20%|█▉        | 71/363 [00:02<00:09, 29.43it/s]
Loading 0:  21%|██        | 75/363 [00:02<00:09, 28.97it/s]
Loading 0:  22%|██▏       | 79/363 [00:02<00:10, 28.29it/s]
Loading 0:  23%|██▎       | 84/363 [00:03<00:09, 30.71it/s]
Loading 0:  24%|██▍       | 88/363 [00:03<00:09, 29.36it/s]
Loading 0:  26%|██▌       | 93/363 [00:03<00:08, 31.51it/s]
Loading 0:  27%|██▋       | 97/363 [00:03<00:08, 30.17it/s]
Loading 0:  28%|██▊       | 101/363 [00:03<00:11, 23.55it/s]
Loading 0:  29%|██▊       | 104/363 [00:03<00:12, 21.26it/s]
Loading 0:  31%|███       | 111/363 [00:04<00:09, 27.98it/s]
Loading 0:  32%|███▏      | 115/363 [00:04<00:08, 27.73it/s]
Loading 0:  33%|███▎      | 120/363 [00:04<00:08, 30.14it/s]
Loading 0:  34%|███▍      | 124/363 [00:04<00:08, 29.07it/s]
Loading 0:  36%|███▌      | 129/363 [00:04<00:07, 31.46it/s]
Loading 0:  37%|███▋      | 133/363 [00:04<00:07, 30.23it/s]
Loading 0:  38%|███▊      | 137/363 [00:04<00:07, 30.24it/s]
Loading 0:  39%|███▉      | 142/363 [00:05<00:08, 25.85it/s]
Loading 0:  40%|███▉      | 145/363 [00:05<00:08, 24.57it/s]
Loading 0:  41%|████      | 149/363 [00:05<00:09, 23.34it/s]
Loading 0:  43%|████▎     | 156/363 [00:05<00:06, 30.16it/s]
Loading 0:  44%|████▍     | 160/363 [00:05<00:06, 29.15it/s]
Loading 0:  45%|████▌     | 165/363 [00:05<00:06, 31.06it/s]
Loading 0:  47%|████▋     | 169/363 [00:06<00:06, 29.59it/s]
Loading 0:  48%|████▊     | 174/363 [00:06<00:05, 31.61it/s]
Loading 0:  49%|████▉     | 178/363 [00:06<00:06, 29.94it/s]
Loading 0:  50%|█████     | 182/363 [00:06<00:07, 23.30it/s]
Loading 0:  51%|█████     | 185/363 [00:06<00:08, 21.04it/s]
Loading 0:  53%|█████▎    | 192/363 [00:07<00:06, 27.93it/s]
Loading 0:  54%|█████▍    | 196/363 [00:07<00:06, 27.70it/s]
Loading 0:  55%|█████▌    | 201/363 [00:07<00:05, 29.90it/s]
Loading 0:  56%|█████▋    | 205/363 [00:07<00:05, 28.93it/s]
Loading 0:  58%|█████▊    | 210/363 [00:07<00:04, 31.16it/s]
Loading 0:  59%|█████▉    | 214/363 [00:07<00:05, 29.77it/s]
Loading 0:  60%|██████    | 218/363 [00:07<00:04, 29.81it/s]
Loading 0:  61%|██████▏   | 223/363 [00:08<00:05, 25.13it/s]
Loading 0:  62%|██████▏   | 226/363 [00:08<00:05, 23.82it/s]
Loading 0:  63%|██████▎   | 230/363 [00:08<00:05, 22.80it/s]
Loading 0:  65%|██████▍   | 235/363 [00:08<00:04, 28.06it/s]
Loading 0:  66%|██████▌   | 239/363 [00:08<00:04, 25.16it/s]
Loading 0:  68%|██████▊   | 246/363 [00:08<00:03, 31.20it/s]
Loading 0:  69%|██████▉   | 250/363 [00:09<00:03, 29.54it/s]
Loading 0:  70%|███████   | 255/363 [00:09<00:03, 30.71it/s]
Loading 0:  71%|███████▏  | 259/363 [00:09<00:03, 28.97it/s]
Loading 0:  72%|███████▏  | 263/363 [00:09<00:04, 22.58it/s]
Loading 0:  73%|███████▎  | 266/363 [00:09<00:04, 20.60it/s]
Loading 0:  75%|███████▌  | 273/363 [00:10<00:03, 26.98it/s]
Loading 0:  76%|███████▌  | 276/363 [00:10<00:03, 25.13it/s]
Loading 0:  78%|███████▊  | 282/363 [00:10<00:02, 29.39it/s]
Loading 0:  79%|███████▉  | 286/363 [00:10<00:02, 28.22it/s]
Loading 0:  80%|████████  | 291/363 [00:10<00:02, 29.81it/s]
Loading 0:  81%|████████▏ | 295/363 [00:10<00:02, 28.20it/s]
Loading 0:  82%|████████▏ | 299/363 [00:10<00:02, 28.54it/s]
Loading 0:  84%|████████▎ | 304/363 [00:11<00:02, 24.54it/s]
Loading 0:  85%|████████▍ | 307/363 [00:11<00:02, 23.71it/s]
Loading 0:  86%|████████▌ | 311/363 [00:11<00:02, 22.90it/s]
Loading 0:  87%|████████▋ | 316/363 [00:11<00:01, 28.05it/s]
Loading 0:  88%|████████▊ | 320/363 [00:11<00:01, 25.70it/s]
Loading 0:  90%|█████████ | 327/363 [00:11<00:01, 32.27it/s]
Loading 0:  91%|█████████ | 331/363 [00:12<00:01, 30.31it/s]
Loading 0:  93%|█████████▎| 336/363 [00:12<00:00, 31.87it/s]
Loading 0:  94%|█████████▎| 340/363 [00:12<00:00, 30.52it/s]
Loading 0:  95%|█████████▍| 344/363 [00:19<00:09,  1.95it/s]
Loading 0:  96%|█████████▌| 348/363 [00:19<00:05,  2.61it/s]
Loading 0:  97%|█████████▋| 353/363 [00:19<00:02,  3.77it/s]
Loading 0:  98%|█████████▊| 357/363 [00:20<00:01,  4.89it/s]
                                                            

Job chaiml-albert-0823-dpo30-6942-v1-mkmlizer completed after 169.9s with status: succeeded
Stopping job with name chaiml-albert-0823-dpo30-6942-v1-mkmlizer
Pipeline stage MKMLizer completed in 183.78s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.29s
Running pipeline stage ISVCDeployer
Creating inference service chaiml-albert-0823-dpo30-6942-v1
Waiting for inference service chaiml-albert-0823-dpo30-6942-v1 to be ready
Failed to get response for submission blend_berib_2024-08-16: ('http://zonemercy-graft-cogent-v-7573-v6-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:43160->127.0.0.1:8080: read: connection reset by peer\n')
Inference service chaiml-albert-0823-dpo30-6942-v1 ready after 363.6550633907318s
Pipeline stage ISVCDeployer completed in 364.81s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.5372941493988037s
Received healthy response to inference request in 1.7065260410308838s
Received healthy response to inference request in 2.371098041534424s
Received healthy response to inference request in 1.707899808883667s
Received healthy response to inference request in 1.8485133647918701s
5 requests
0 failed requests
5th percentile: 1.7068007946014405
10th percentile: 1.707075548171997
20th percentile: 1.7076250553131103
30th percentile: 1.7360225200653077
40th percentile: 1.7922679424285888
50th percentile: 1.8485133647918701
60th percentile: 2.0575472354888915
70th percentile: 2.266581106185913
80th percentile: 2.4043372631073
90th percentile: 2.4708157062530516
95th percentile: 2.5040549278259276
99th percentile: 2.5306463050842285
mean time: 2.0342662811279295
Pipeline stage StressChecker completed in 11.05s
chaiml-albert-0823-dpo30_6942_v1 status is now deployed due to DeploymentManager action
chaiml-albert-0823-dpo30_6942_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-albert-0823-dpo30_6942_v1 status is now torndown due to DeploymentManager action

Usage Metrics

Latency Metrics