nothingiisreal-celeste-1_9070

submission_id: nothingiisreal-celeste-1_9070_v3
developer_uid: aaronday3
best_of: 8
celo_rating: 1185.41
display_name: nothingiisreal-celestev16
family_friendly_score: 0.0
formatter: {'memory_template': '<|im_start|>system\\n{memory}<|im_end|>\\n', 'prompt_template': '<|im_start|>user\\n{prompt}<|im_end|>\\n', 'bot_template': '<|im_start|>assistant\\n{bot_name}: {message}<|im_end|>\\n', 'user_template': '<|im_start|>user\\n{user_name}: {message}<|im_end|>\\n', 'response_template': '<|im_start|>assistant\\n{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 0.6, 'top_p': 1.0, 'min_p': 0.1, 'top_k': 60, 'presence_penalty': 0.15, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|im_end|>'], 'max_input_tokens': 512, 'best_of': 8, 'max_output_tokens': 64, 'reward_max_token_input': 256}
is_internal_developer: False
language_model: nothingiisreal/Celeste-12B-V1.6
max_input_tokens: 512
max_output_tokens: 64
model_architecture: MistralForCausalLM
model_group: nothingiisreal/Celeste-1
model_name: nothingiisreal-celestev16
model_num_parameters: 12772070400.0
model_repo: nothingiisreal/Celeste-12B-V1.6
model_size: 13B
num_battles: 12315
num_wins: 5927
ranking_group: single
reward_formatter: {'bot_template': '<|im_start|>assistant\\n{bot_name}: {message}<|im_end|>\\n', 'memory_template': '<|im_start|>system\\n{memory}<|im_end|>\\n', 'prompt_template': '<|im_start|>user\\n{prompt}<|im_end|>\\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '<|im_start|>user\\n{user_name}: {message}<|im_end|>\\n'}
reward_repo: ChaiML/reward_gpt2_medium_preference_24m_e2
status: torndown
submission_type: basic
timestamp: 2024-07-28T23:00:21+00:00
us_pacific_date: 2024-07-28
win_ratio: 0.48128298822574095
Download Preference Data
Resubmit model
Running pipeline stage MKMLizer
Starting job with name nothingiisreal-celeste-1-9070-v3-mkmlizer
Waiting for job on nothingiisreal-celeste-1-9070-v3-mkmlizer to finish
Failed to get response for submission sao10k-hydrochoerus-12b-v1-s1_v3: ('http://sao10k-hydrochoerus-12b-v1-s1-v3-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
nothingiisreal-celeste-1-9070-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║     _____            __           __                                ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║       /___/                                                         ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║                                                                     ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║  Version: 0.9.7                                                     ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║  https://mk1.ai                                                     ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║                                                                     ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║  The license key for the current software has been verified as      ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║  belonging to:                                                      ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║                                                                     ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║  Chai Research Corp.                                                ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ║                                                                     ║
nothingiisreal-celeste-1-9070-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission sao10k-hydrochoerus-12b-v1-s1_v3: ('http://sao10k-hydrochoerus-12b-v1-s1-v3-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission sao10k-hydrochoerus-12b-v1-s1_v3: ('http://sao10k-hydrochoerus-12b-v1-s1-v3-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
nothingiisreal-celeste-1-9070-v3-mkmlizer: Downloaded to shared memory in 27.872s
nothingiisreal-celeste-1-9070-v3-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp4mip4vlg, device:0
nothingiisreal-celeste-1-9070-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
nothingiisreal-celeste-1-9070-v3-mkmlizer: quantized model in 36.139s
nothingiisreal-celeste-1-9070-v3-mkmlizer: Processed model nothingiisreal/Celeste-12B-V1.6 in 64.012s
nothingiisreal-celeste-1-9070-v3-mkmlizer: creating bucket guanaco-mkml-models
nothingiisreal-celeste-1-9070-v3-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nothingiisreal-celeste-1-9070-v3-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/nothingiisreal-celeste-1-9070-v3
nothingiisreal-celeste-1-9070-v3-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/nothingiisreal-celeste-1-9070-v3/config.json
nothingiisreal-celeste-1-9070-v3-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/nothingiisreal-celeste-1-9070-v3/special_tokens_map.json
nothingiisreal-celeste-1-9070-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/nothingiisreal-celeste-1-9070-v3/tokenizer_config.json
nothingiisreal-celeste-1-9070-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/nothingiisreal-celeste-1-9070-v3/tokenizer.json
Failed to get response for submission sao10k-hydrochoerus-12b-v1-s1_v3: ('http://sao10k-hydrochoerus-12b-v1-s1-v3-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
nothingiisreal-celeste-1-9070-v3-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/nothingiisreal-celeste-1-9070-v3/flywheel_model.0.safetensors
nothingiisreal-celeste-1-9070-v3-mkmlizer: loading reward model from ChaiML/reward_gpt2_medium_preference_24m_e2
nothingiisreal-celeste-1-9070-v3-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 31.78it/s]
Loading 0:   3%|▎         | 12/363 [00:00<00:07, 47.18it/s]
Loading 0:   5%|▍         | 17/363 [00:00<00:07, 45.08it/s]
Loading 0:   6%|▌         | 22/363 [00:00<00:07, 46.48it/s]
Loading 0:   8%|▊         | 28/363 [00:00<00:07, 44.33it/s]
Loading 0:   9%|▉         | 33/363 [00:00<00:07, 42.68it/s]
Loading 0:  11%|█         | 40/363 [00:00<00:06, 48.62it/s]
Loading 0:  13%|█▎        | 46/363 [00:01<00:06, 46.05it/s]
Loading 0:  14%|█▍        | 51/363 [00:01<00:07, 44.49it/s]
Loading 0:  17%|█▋        | 60/363 [00:01<00:05, 50.59it/s]
Loading 0:  18%|█▊        | 66/363 [00:01<00:08, 35.01it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:07, 38.96it/s]
Loading 0:  21%|██        | 77/363 [00:01<00:07, 40.73it/s]
Loading 0:  23%|██▎       | 82/363 [00:01<00:07, 36.96it/s]
Loading 0:  25%|██▍       | 90/363 [00:02<00:06, 44.48it/s]
Loading 0:  26%|██▋       | 96/363 [00:02<00:06, 43.19it/s]
Loading 0:  28%|██▊       | 101/363 [00:02<00:06, 42.06it/s]
Loading 0:  30%|██▉       | 108/363 [00:02<00:05, 48.13it/s]
Loading 0:  31%|███▏      | 114/363 [00:02<00:05, 45.05it/s]
Loading 0:  33%|███▎      | 119/363 [00:02<00:05, 44.28it/s]
Loading 0:  34%|███▍      | 125/363 [00:02<00:05, 46.56it/s]
Loading 0:  36%|███▌      | 130/363 [00:02<00:05, 45.45it/s]
Loading 0:  37%|███▋      | 135/363 [00:03<00:04, 46.30it/s]
Loading 0:  39%|███▉      | 141/363 [00:03<00:04, 45.81it/s]
Loading 0:  40%|████      | 146/363 [00:03<00:06, 32.40it/s]
Loading 0:  41%|████▏     | 150/363 [00:03<00:06, 32.38it/s]
Loading 0:  43%|████▎     | 157/363 [00:03<00:05, 38.68it/s]
Loading 0:  45%|████▍     | 162/363 [00:03<00:04, 40.90it/s]
Loading 0:  46%|████▌     | 167/363 [00:04<00:05, 35.96it/s]
Loading 0:  48%|████▊     | 174/363 [00:04<00:04, 43.36it/s]
Loading 0:  49%|████▉     | 179/363 [00:04<00:04, 43.76it/s]
Loading 0:  51%|█████     | 184/363 [00:04<00:04, 43.95it/s]
Loading 0:  52%|█████▏    | 190/363 [00:04<00:03, 43.46it/s]
Loading 0:  54%|█████▎    | 195/363 [00:04<00:03, 42.23it/s]
Loading 0:  56%|█████▌    | 202/363 [00:04<00:03, 46.72it/s]
Loading 0:  57%|█████▋    | 208/363 [00:04<00:03, 45.04it/s]
Loading 0:  59%|█████▊    | 213/363 [00:05<00:03, 44.07it/s]
Loading 0:  61%|██████    | 220/363 [00:05<00:02, 50.07it/s]
Loading 0:  62%|██████▏   | 226/363 [00:05<00:04, 30.59it/s]
Loading 0:  64%|██████▎   | 231/363 [00:05<00:04, 31.94it/s]
Loading 0:  65%|██████▌   | 237/363 [00:05<00:03, 37.16it/s]
Loading 0:  67%|██████▋   | 242/363 [00:05<00:03, 38.16it/s]
Loading 0:  68%|██████▊   | 247/363 [00:05<00:03, 38.01it/s]
Loading 0:  70%|██████▉   | 253/363 [00:06<00:02, 39.23it/s]
Loading 0:  71%|███████   | 258/363 [00:06<00:02, 39.44it/s]
Loading 0:  73%|███████▎  | 265/363 [00:06<00:02, 45.01it/s]
Loading 0:  75%|███████▍  | 271/363 [00:06<00:02, 43.17it/s]
Loading 0:  76%|███████▌  | 276/363 [00:06<00:02, 41.25it/s]
Loading 0:  78%|███████▊  | 282/363 [00:06<00:01, 45.56it/s]
Loading 0:  79%|███████▉  | 287/363 [00:06<00:01, 45.18it/s]
Loading 0:  80%|████████  | 292/363 [00:06<00:01, 45.73it/s]
Loading 0:  82%|████████▏ | 298/363 [00:07<00:01, 44.64it/s]
Loading 0:  84%|████████▎ | 304/363 [00:13<00:22,  2.67it/s]
Loading 0:  85%|████████▍ | 308/363 [00:14<00:16,  3.39it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:11,  4.38it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:05,  7.18it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:03,  9.60it/s]
Loading 0:  91%|█████████ | 331/363 [00:14<00:02, 12.06it/s]
Loading 0:  93%|█████████▎| 338/363 [00:14<00:01, 16.81it/s]
Loading 0:  95%|█████████▍| 344/363 [00:14<00:00, 20.57it/s]
Loading 0:  96%|█████████▌| 349/363 [00:14<00:00, 23.68it/s]
Loading 0:  98%|█████████▊| 356/363 [00:15<00:00, 30.30it/s]
Loading 0: 100%|█████████▉| 362/363 [00:15<00:00, 33.16it/s]
                                                            
/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:957: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
nothingiisreal-celeste-1-9070-v3-mkmlizer:   warnings.warn(
nothingiisreal-celeste-1-9070-v3-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:785: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
nothingiisreal-celeste-1-9070-v3-mkmlizer:   warnings.warn(
nothingiisreal-celeste-1-9070-v3-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
nothingiisreal-celeste-1-9070-v3-mkmlizer: Saving duration: 0.346s
nothingiisreal-celeste-1-9070-v3-mkmlizer: Processed model ChaiML/reward_gpt2_medium_preference_24m_e2 in 7.860s
nothingiisreal-celeste-1-9070-v3-mkmlizer: creating bucket guanaco-reward-models
nothingiisreal-celeste-1-9070-v3-mkmlizer: Bucket 's3://guanaco-reward-models/' created
nothingiisreal-celeste-1-9070-v3-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/nothingiisreal-celeste-1-9070-v3_reward
nothingiisreal-celeste-1-9070-v3-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/nothingiisreal-celeste-1-9070-v3_reward/special_tokens_map.json
nothingiisreal-celeste-1-9070-v3-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/nothingiisreal-celeste-1-9070-v3_reward/config.json
nothingiisreal-celeste-1-9070-v3-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/nothingiisreal-celeste-1-9070-v3_reward/tokenizer_config.json
nothingiisreal-celeste-1-9070-v3-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/nothingiisreal-celeste-1-9070-v3_reward/merges.txt
nothingiisreal-celeste-1-9070-v3-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/nothingiisreal-celeste-1-9070-v3_reward/vocab.json
nothingiisreal-celeste-1-9070-v3-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/nothingiisreal-celeste-1-9070-v3_reward/tokenizer.json
nothingiisreal-celeste-1-9070-v3-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/nothingiisreal-celeste-1-9070-v3_reward/reward.tensors
Job nothingiisreal-celeste-1-9070-v3-mkmlizer completed after 105.11s with status: succeeded
Stopping job with name nothingiisreal-celeste-1-9070-v3-mkmlizer
Pipeline stage MKMLizer completed in 106.22s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.15s
Running pipeline stage ISVCDeployer
Creating inference service nothingiisreal-celeste-1-9070-v3
Waiting for inference service nothingiisreal-celeste-1-9070-v3 to be ready
Failed to get response for submission sao10k-hydrochoerus-12b-v1-s1_v3: ('http://sao10k-hydrochoerus-12b-v1-s1-v3-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Inference service nothingiisreal-celeste-1-9070-v3 ready after 111.31344676017761s
Pipeline stage ISVCDeployer completed in 113.57s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.398841142654419s
Received healthy response to inference request in 1.5437724590301514s
Failed to get response for submission sao10k-hydrochoerus-12b-v1-s1_v3: ('http://sao10k-hydrochoerus-12b-v1-s1-v3-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Received healthy response to inference request in 1.5366311073303223s
Received healthy response to inference request in 1.6887993812561035s
Received healthy response to inference request in 1.5497329235076904s
5 requests
0 failed requests
5th percentile: 1.538059377670288
10th percentile: 1.539487648010254
20th percentile: 1.5423441886901856
30th percentile: 1.5449645519256592
40th percentile: 1.5473487377166748
50th percentile: 1.5497329235076904
60th percentile: 1.6053595066070556
70th percentile: 1.6609860897064208
Failed to get response for submission sao10k-hydrochoerus-12b-v1-s1_v3: ('http://sao10k-hydrochoerus-12b-v1-s1-v3-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
80th percentile: 1.8308077335357666
90th percentile: 2.1148244380950927
95th percentile: 2.2568327903747556
99th percentile: 2.3704394721984863
mean time: 1.7435554027557374
Pipeline stage StressChecker completed in 9.55s
nothingiisreal-celeste-1_9070_v3 status is now deployed due to DeploymentManager action
nothingiisreal-celeste-1_9070_v3 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of nothingiisreal-celeste-1_9070_v3
Running pipeline stage ISVCDeleter
Checking if service nothingiisreal-celeste-1-9070-v3 is running
Tearing down inference service nothingiisreal-celeste-1-9070-v3
Service nothingiisreal-celeste-1-9070-v3 has been torndown
Pipeline stage ISVCDeleter completed in 5.19s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key nothingiisreal-celeste-1-9070-v3/config.json from bucket guanaco-mkml-models
Deleting key nothingiisreal-celeste-1-9070-v3/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key nothingiisreal-celeste-1-9070-v3/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key nothingiisreal-celeste-1-9070-v3/tokenizer.json from bucket guanaco-mkml-models
Deleting key nothingiisreal-celeste-1-9070-v3/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key nothingiisreal-celeste-1-9070-v3_reward/config.json from bucket guanaco-reward-models
Deleting key nothingiisreal-celeste-1-9070-v3_reward/merges.txt from bucket guanaco-reward-models
Deleting key nothingiisreal-celeste-1-9070-v3_reward/reward.tensors from bucket guanaco-reward-models
Deleting key nothingiisreal-celeste-1-9070-v3_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key nothingiisreal-celeste-1-9070-v3_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key nothingiisreal-celeste-1-9070-v3_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key nothingiisreal-celeste-1-9070-v3_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 6.17s
nothingiisreal-celeste-1_9070_v3 status is now torndown due to DeploymentManager action