submission_id: chaiml-edit-data-dpo100-1e5_v6
developer_uid: albert_chai
alignment_samples: 10399
alignment_score: -1.0344709358415467
best_of: 8
celo_rating: 1237.4
display_name: chaiml-edit-data-dpo100-1e5_v1
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
is_internal_developer: True
language_model: ChaiML/edit_data_dpo100-1e5
max_input_tokens: 1024
max_output_tokens: 64
model_architecture: MistralForCausalLM
model_group: ChaiML/edit_data_dpo100-
model_name: chaiml-edit-data-dpo100-1e5_v1
model_num_parameters: 12772070400.0
model_repo: ChaiML/edit_data_dpo100-1e5
model_size: 13B
num_battles: 10399
num_wins: 5202
propriety_score: 0.7174638487208009
propriety_total_count: 899.0
ranking_group: single
status: torndown
submission_type: basic
timestamp: 2024-08-23T04:06:49+00:00
us_pacific_date: 2024-08-22
win_ratio: 0.5002404077315127
Download Preference Data
Resubmit model
Running pipeline stage MKMLizer
Starting job with name chaiml-edit-data-dpo100-1e5-v6-mkmlizer
Waiting for job on chaiml-edit-data-dpo100-1e5-v6-mkmlizer to finish
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║     _____            __           __                                ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║       /___/                                                         ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║                                                                     ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║  Version: 0.10.1                                                    ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║                                                                     ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║  belonging to:                                                      ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║                                                                     ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ║                                                                     ║
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission blend_nemek_2024-08-22: ('http://chaiml-elo-alignment-run-3-v34-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"ValueError : [TypeError(\\"\'numpy.int64\' object is not iterable\\"), TypeError(\'vars() argument must have __dict__ attribute\')]"}')
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: Downloaded to shared memory in 52.936s
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp_v6jriz3, device:0
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: quantized model in 41.065s
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: Processed model ChaiML/edit_data_dpo100-1e5 in 94.001s
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: creating bucket guanaco-mkml-models
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-edit-data-dpo100-1e5-v6
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-edit-data-dpo100-1e5-v6/special_tokens_map.json
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-edit-data-dpo100-1e5-v6/config.json
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-edit-data-dpo100-1e5-v6/tokenizer_config.json
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-edit-data-dpo100-1e5-v6/tokenizer.json
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-edit-data-dpo100-1e5-v6/flywheel_model.0.safetensors
chaiml-edit-data-dpo100-1e5-v6-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:15, 22.53it/s]
Loading 0:   3%|▎         | 10/363 [00:00<00:12, 28.19it/s]
Loading 0:   4%|▍         | 14/363 [00:00<00:14, 23.92it/s]
Loading 0:   6%|▌         | 20/363 [00:00<00:10, 32.48it/s]
Loading 0:   7%|▋         | 24/363 [00:01<00:16, 20.89it/s]
Loading 0:   7%|▋         | 27/363 [00:01<00:16, 20.74it/s]
Loading 0:   9%|▊         | 31/363 [00:01<00:13, 24.42it/s]
Loading 0:   9%|▉         | 34/363 [00:01<00:13, 25.01it/s]
Loading 0:  11%|█         | 39/363 [00:01<00:11, 28.09it/s]
Loading 0:  12%|█▏        | 43/363 [00:01<00:11, 27.45it/s]
Loading 0:  13%|█▎        | 48/363 [00:01<00:10, 30.08it/s]
Loading 0:  14%|█▍        | 52/363 [00:01<00:11, 28.22it/s]
Loading 0:  15%|█▌        | 56/363 [00:02<00:10, 28.74it/s]
Loading 0:  17%|█▋        | 61/363 [00:02<00:12, 24.29it/s]
Loading 0:  18%|█▊        | 64/363 [00:02<00:13, 21.73it/s]
Loading 0:  20%|█▉        | 71/363 [00:02<00:10, 28.54it/s]
Loading 0:  21%|██        | 75/363 [00:02<00:10, 27.72it/s]
Loading 0:  21%|██▏       | 78/363 [00:03<00:10, 25.95it/s]
Loading 0:  23%|██▎       | 84/363 [00:03<00:09, 30.18it/s]
Loading 0:  24%|██▍       | 88/363 [00:03<00:09, 29.03it/s]
Loading 0:  26%|██▌       | 93/363 [00:03<00:08, 31.19it/s]
Loading 0:  27%|██▋       | 97/363 [00:03<00:08, 29.82it/s]
Loading 0:  28%|██▊       | 101/363 [00:03<00:11, 23.27it/s]
Loading 0:  29%|██▊       | 104/363 [00:04<00:12, 21.21it/s]
Loading 0:  31%|███       | 111/363 [00:04<00:09, 27.98it/s]
Loading 0:  32%|███▏      | 115/363 [00:04<00:09, 27.49it/s]
Loading 0:  33%|███▎      | 120/363 [00:04<00:08, 30.12it/s]
Loading 0:  34%|███▍      | 124/363 [00:04<00:08, 28.78it/s]
Loading 0:  36%|███▌      | 129/363 [00:04<00:07, 30.40it/s]
Loading 0:  37%|███▋      | 133/363 [00:04<00:07, 29.11it/s]
Loading 0:  38%|███▊      | 137/363 [00:05<00:07, 29.37it/s]
Loading 0:  39%|███▉      | 142/363 [00:05<00:08, 25.38it/s]
Loading 0:  40%|███▉      | 145/363 [00:05<00:09, 23.93it/s]
Loading 0:  41%|████      | 149/363 [00:05<00:09, 23.09it/s]
Loading 0:  43%|████▎     | 156/363 [00:05<00:06, 29.84it/s]
Loading 0:  44%|████▍     | 160/363 [00:05<00:07, 28.85it/s]
Loading 0:  45%|████▌     | 165/363 [00:06<00:06, 31.12it/s]
Loading 0:  47%|████▋     | 169/363 [00:06<00:06, 29.90it/s]
Loading 0:  48%|████▊     | 173/363 [00:06<00:05, 32.04it/s]
Loading 0:  49%|████▉     | 177/363 [00:06<00:06, 27.97it/s]
Loading 0:  50%|█████     | 182/363 [00:06<00:07, 24.19it/s]
Loading 0:  51%|█████     | 185/363 [00:06<00:08, 21.85it/s]
Loading 0:  53%|█████▎    | 192/363 [00:07<00:05, 28.65it/s]
Loading 0:  54%|█████▍    | 196/363 [00:07<00:05, 28.10it/s]
Loading 0:  55%|█████▌    | 201/363 [00:07<00:05, 30.66it/s]
Loading 0:  56%|█████▋    | 205/363 [00:07<00:05, 29.16it/s]
Loading 0:  58%|█████▊    | 210/363 [00:07<00:05, 30.55it/s]
Loading 0:  59%|█████▉    | 214/363 [00:07<00:05, 28.82it/s]
Loading 0:  60%|██████    | 218/363 [00:08<00:05, 28.01it/s]
Loading 0:  61%|██████▏   | 223/363 [00:08<00:05, 24.03it/s]
Loading 0:  62%|██████▏   | 226/363 [00:08<00:05, 23.13it/s]
Loading 0:  63%|██████▎   | 230/363 [00:08<00:05, 22.25it/s]
Loading 0:  65%|██████▌   | 237/363 [00:08<00:04, 29.00it/s]
Loading 0:  66%|██████▋   | 241/363 [00:08<00:04, 28.44it/s]
Loading 0:  68%|██████▊   | 246/363 [00:09<00:03, 30.54it/s]
Loading 0:  69%|██████▉   | 250/363 [00:09<00:03, 29.55it/s]
Loading 0:  70%|███████   | 255/363 [00:09<00:03, 31.83it/s]
Loading 0:  71%|███████▏  | 259/363 [00:09<00:03, 30.24it/s]
Loading 0:  72%|███████▏  | 263/363 [00:09<00:04, 23.97it/s]
Loading 0:  73%|███████▎  | 266/363 [00:09<00:04, 21.69it/s]
Loading 0:  75%|███████▌  | 273/363 [00:10<00:03, 28.66it/s]
Loading 0:  76%|███████▋  | 277/363 [00:10<00:03, 28.10it/s]
Loading 0:  78%|███████▊  | 282/363 [00:10<00:02, 29.77it/s]
Loading 0:  79%|███████▉  | 286/363 [00:10<00:02, 28.79it/s]
Loading 0:  80%|████████  | 291/363 [00:10<00:02, 30.62it/s]
Loading 0:  81%|████████▏ | 295/363 [00:10<00:02, 28.96it/s]
Loading 0:  82%|████████▏ | 299/363 [00:10<00:02, 28.84it/s]
Loading 0:  84%|████████▎ | 304/363 [00:11<00:02, 25.00it/s]
Loading 0:  85%|████████▍ | 307/363 [00:11<00:02, 24.10it/s]
Loading 0:  86%|████████▌ | 311/363 [00:11<00:02, 23.24it/s]
Loading 0:  88%|████████▊ | 318/363 [00:11<00:01, 29.80it/s]
Loading 0:  89%|████████▊ | 322/363 [00:11<00:01, 28.99it/s]
Loading 0:  90%|████████▉ | 325/363 [00:11<00:01, 29.08it/s]
Loading 0:  91%|█████████ | 329/363 [00:12<00:01, 26.43it/s]
Loading 0:  93%|█████████▎| 336/363 [00:12<00:00, 32.25it/s]
Loading 0:  94%|█████████▎| 340/363 [00:12<00:00, 30.19it/s]
Loading 0:  95%|█████████▍| 344/363 [00:19<00:09,  2.06it/s]
Loading 0:  96%|█████████▌| 348/363 [00:19<00:05,  2.75it/s]
Loading 0:  97%|█████████▋| 353/363 [00:19<00:02,  3.95it/s]
Loading 0:  98%|█████████▊| 357/363 [00:19<00:01,  5.09it/s]
                                                            

Job chaiml-edit-data-dpo100-1e5-v6-mkmlizer completed after 116.21s with status: succeeded
Stopping job with name chaiml-edit-data-dpo100-1e5-v6-mkmlizer
Pipeline stage MKMLizer completed in 117.75s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.11s
Running pipeline stage ISVCDeployer
Creating inference service chaiml-edit-data-dpo100-1e5-v6
Waiting for inference service chaiml-edit-data-dpo100-1e5-v6 to be ready
Inference service chaiml-edit-data-dpo100-1e5-v6 ready after 211.69062304496765s
Pipeline stage ISVCDeployer completed in 212.72s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.9123036861419678s
Received healthy response to inference request in 1.8438315391540527s
Received healthy response to inference request in 1.9154367446899414s
Received healthy response to inference request in 1.43422532081604s
Received healthy response to inference request in 1.0959737300872803s
5 requests
0 failed requests
5th percentile: 1.1636240482330322
10th percentile: 1.2312743663787842
20th percentile: 1.366575002670288
30th percentile: 1.5161465644836425
40th percentile: 1.6799890518188476
50th percentile: 1.8438315391540527
60th percentile: 1.8712203979492188
70th percentile: 1.8986092567443849
80th percentile: 1.9129302978515625
90th percentile: 1.9141835212707519
95th percentile: 1.9148101329803466
99th percentile: 1.9153114223480225
mean time: 1.6403542041778565
Pipeline stage StressChecker completed in 8.90s
chaiml-edit-data-dpo100-1e5_v6 status is now deployed due to DeploymentManager action
chaiml-edit-data-dpo100-1e5_v6 status is now inactive due to auto deactivation removed underperforming models
chaiml-edit-data-dpo100-1e5_v6 status is now torndown due to DeploymentManager action

Usage Metrics

Latency Metrics