developer_uid: chai_backend_admin
submission_id: chaiml-chaiml-sft-horror_v2
model_name: chaiml-chaiml-sft-horror_v2
model_group: ChaiML/chaiml-sft-horror
status: torndown
timestamp: 2024-09-17T22:28:09+00:00
num_battles: 3399780
num_wins: 1735452
celo_rating: 1259.93
family_friendly_score: 0.544482680036463
family_friendly_standard_error: 0.0023280206597160197
submission_type: basic
model_repo: ChaiML/chaiml-sft-horror
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
latencies: [{'batch_size': 1, 'throughput': 0.607831744496391, 'latency_mean': 1.6450946271419524, 'latency_p50': 1.6506541967391968, 'latency_p90': 1.8115070343017579}, {'batch_size': 3, 'throughput': 1.0754503370595012, 'latency_mean': 2.779811327457428, 'latency_p50': 2.7925580739974976, 'latency_p90': 3.0596812963485718}, {'batch_size': 5, 'throughput': 1.2203394610004932, 'latency_mean': 4.0819917953014375, 'latency_p50': 4.11495304107666, 'latency_p90': 4.608464026451111}, {'batch_size': 6, 'throughput': 1.24176054524199, 'latency_mean': 4.814535529613495, 'latency_p50': 4.808460354804993, 'latency_p90': 5.502043771743774}, {'batch_size': 8, 'throughput': 1.2120958356328986, 'latency_mean': 6.5549041414260865, 'latency_p50': 6.523899555206299, 'latency_p90': 7.45654022693634}, {'batch_size': 10, 'throughput': 1.2041006329524648, 'latency_mean': 8.262617356777191, 'latency_p50': 8.320612907409668, 'latency_p90': 9.369233751296997}]
gpu_counts: {'NVIDIA RTX A5000': 1}
display_name: chaiml-chaiml-sft-horror_v2
ineligible_reason: model is not deployable
is_internal_developer: True
language_model: ChaiML/chaiml-sft-horror
model_size: 13B
ranking_group: single
throughput_3p7s: 1.19
us_pacific_date: 2024-09-17
win_ratio: 0.5104600885939679
generation_params: {'temperature': 0.9, 'top_p': 0.9, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.5, 'frequency_penalty': 0.5, 'stopping_words': ['\n', '</s>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n##Try to make the conversation scary while stay in character##\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-chaiml-sft-horror-v2-mkmlizer
Waiting for job on chaiml-chaiml-sft-horror-v2-mkmlizer to finish
chaiml-chaiml-sft-horror-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-chaiml-sft-horror-v2-mkmlizer: ║     _____            __           __                                ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ║       /___/                                                         ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ║                                                                     ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ║  Version: 0.11.12                                                   ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ║                                                                     ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ║  belonging to:                                                      ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ║                                                                     ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ║                                                                     ║
chaiml-chaiml-sft-horror-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-chaiml-sft-horror-v2-mkmlizer: Downloaded to shared memory in 64.795s
chaiml-chaiml-sft-horror-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpp86pyzgl, device:0
chaiml-chaiml-sft-horror-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-chaiml-sft-horror-v2-mkmlizer: quantized model in 40.183s
chaiml-chaiml-sft-horror-v2-mkmlizer: Processed model ChaiML/chaiml-sft-horror in 104.978s
chaiml-chaiml-sft-horror-v2-mkmlizer: creating bucket guanaco-mkml-models
chaiml-chaiml-sft-horror-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-chaiml-sft-horror-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-chaiml-sft-horror-v2
chaiml-chaiml-sft-horror-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-chaiml-sft-horror-v2/config.json
chaiml-chaiml-sft-horror-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-chaiml-sft-horror-v2/special_tokens_map.json
chaiml-chaiml-sft-horror-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-chaiml-sft-horror-v2/tokenizer_config.json
chaiml-chaiml-sft-horror-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-chaiml-sft-horror-v2/tokenizer.json
chaiml-chaiml-sft-horror-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-chaiml-sft-horror-v2/flywheel_model.0.safetensors
chaiml-chaiml-sft-horror-v2-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:15, 23.15it/s]
Loading 0:   3%|▎         | 10/363 [00:00<00:11, 30.05it/s]
Loading 0:   4%|▍         | 14/363 [00:00<00:13, 26.49it/s]
Loading 0:   6%|▌         | 21/363 [00:00<00:08, 38.11it/s]
Loading 0:   7%|▋         | 26/363 [00:01<00:15, 22.43it/s]
Loading 0:   9%|▊         | 31/363 [00:01<00:12, 27.28it/s]
Loading 0:  10%|▉         | 35/363 [00:01<00:11, 29.09it/s]
Loading 0:  11%|█         | 39/363 [00:01<00:10, 30.22it/s]
Loading 0:  12%|█▏        | 43/363 [00:01<00:10, 29.78it/s]
Loading 0:  13%|█▎        | 48/363 [00:01<00:09, 32.70it/s]
Loading 0:  14%|█▍        | 52/363 [00:01<00:10, 31.03it/s]
Loading 0:  15%|█▌        | 56/363 [00:01<00:10, 30.43it/s]
Loading 0:  17%|█▋        | 61/363 [00:02<00:11, 26.93it/s]
Loading 0:  18%|█▊        | 64/363 [00:02<00:12, 23.32it/s]
Loading 0:  20%|█▉        | 71/363 [00:02<00:09, 30.57it/s]
Loading 0:  21%|██        | 75/363 [00:02<00:09, 30.30it/s]
Loading 0:  22%|██▏       | 79/363 [00:02<00:09, 29.76it/s]
Loading 0:  23%|██▎       | 84/363 [00:02<00:08, 32.06it/s]
Loading 0:  24%|██▍       | 88/363 [00:03<00:08, 30.58it/s]
Loading 0:  26%|██▌       | 93/363 [00:03<00:08, 33.02it/s]
Loading 0:  27%|██▋       | 97/363 [00:03<00:08, 31.64it/s]
Loading 0:  28%|██▊       | 101/363 [00:03<00:10, 25.65it/s]
Loading 0:  29%|██▊       | 104/363 [00:03<00:11, 22.21it/s]
Loading 0:  31%|███       | 111/363 [00:03<00:08, 29.26it/s]
Loading 0:  32%|███▏      | 115/363 [00:03<00:08, 28.72it/s]
Loading 0:  33%|███▎      | 120/363 [00:04<00:07, 31.76it/s]
Loading 0:  34%|███▍      | 124/363 [00:04<00:07, 30.40it/s]
Loading 0:  36%|███▌      | 129/363 [00:04<00:07, 32.09it/s]
Loading 0:  37%|███▋      | 133/363 [00:04<00:07, 29.84it/s]
Loading 0:  38%|███▊      | 137/363 [00:04<00:07, 30.34it/s]
Loading 0:  39%|███▉      | 142/363 [00:04<00:08, 27.10it/s]
Loading 0:  40%|███▉      | 145/363 [00:05<00:08, 25.05it/s]
Loading 0:  41%|████      | 149/363 [00:05<00:08, 24.16it/s]
Loading 0:  43%|████▎     | 156/363 [00:05<00:06, 31.05it/s]
Loading 0:  44%|████▍     | 160/363 [00:05<00:06, 29.11it/s]
Loading 0:  45%|████▌     | 165/363 [00:05<00:06, 32.08it/s]
Loading 0:  47%|████▋     | 169/363 [00:05<00:06, 31.18it/s]
Loading 0:  48%|████▊     | 174/363 [00:05<00:05, 33.20it/s]
Loading 0:  49%|████▉     | 178/363 [00:06<00:05, 31.29it/s]
Loading 0:  50%|█████     | 182/363 [00:06<00:06, 26.34it/s]
Loading 0:  51%|█████     | 185/363 [00:06<00:07, 22.89it/s]
Loading 0:  53%|█████▎    | 192/363 [00:06<00:05, 30.28it/s]
Loading 0:  54%|█████▍    | 196/363 [00:06<00:05, 29.30it/s]
Loading 0:  55%|█████▌    | 201/363 [00:06<00:05, 32.14it/s]
Loading 0:  56%|█████▋    | 205/363 [00:07<00:05, 30.98it/s]
Loading 0:  58%|█████▊    | 210/363 [00:07<00:04, 32.97it/s]
Loading 0:  59%|█████▉    | 214/363 [00:07<00:04, 31.34it/s]
Loading 0:  60%|██████    | 218/363 [00:07<00:04, 31.53it/s]
Loading 0:  61%|██████▏   | 223/363 [00:07<00:05, 27.26it/s]
Loading 0:  62%|██████▏   | 226/363 [00:07<00:05, 25.41it/s]
Loading 0:  63%|██████▎   | 230/363 [00:07<00:05, 24.19it/s]
Loading 0:  65%|██████▌   | 237/363 [00:08<00:04, 31.28it/s]
Loading 0:  66%|██████▋   | 241/363 [00:08<00:04, 30.34it/s]
Loading 0:  68%|██████▊   | 246/363 [00:08<00:03, 32.87it/s]
Loading 0:  69%|██████▉   | 250/363 [00:08<00:03, 31.02it/s]
Loading 0:  70%|███████   | 255/363 [00:08<00:03, 33.27it/s]
Loading 0:  71%|███████▏  | 259/363 [00:08<00:03, 31.84it/s]
Loading 0:  72%|███████▏  | 263/363 [00:09<00:03, 26.11it/s]
Loading 0:  73%|███████▎  | 266/363 [00:09<00:04, 22.44it/s]
Loading 0:  75%|███████▌  | 273/363 [00:09<00:03, 29.52it/s]
Loading 0:  76%|███████▋  | 277/363 [00:09<00:02, 29.13it/s]
Loading 0:  78%|███████▊  | 282/363 [00:09<00:02, 32.21it/s]
Loading 0:  79%|███████▉  | 286/363 [00:09<00:02, 31.34it/s]
Loading 0:  80%|████████  | 291/363 [00:09<00:02, 33.13it/s]
Loading 0:  81%|████████▏ | 295/363 [00:10<00:02, 31.64it/s]
Loading 0:  82%|████████▏ | 299/363 [00:10<00:02, 31.97it/s]
Loading 0:  84%|████████▎ | 304/363 [00:10<00:02, 27.68it/s]
Loading 0:  85%|████████▍ | 307/363 [00:10<00:02, 25.41it/s]
Loading 0:  86%|████████▌ | 311/363 [00:10<00:02, 24.21it/s]
Loading 0:  88%|████████▊ | 318/363 [00:10<00:01, 30.86it/s]
Loading 0:  89%|████████▊ | 322/363 [00:11<00:01, 29.74it/s]
Loading 0:  90%|█████████ | 327/363 [00:11<00:01, 32.40it/s]
Loading 0:  91%|█████████ | 331/363 [00:11<00:01, 31.05it/s]
Loading 0:  93%|█████████▎| 336/363 [00:11<00:00, 33.05it/s]
Loading 0:  94%|█████████▎| 340/363 [00:11<00:00, 31.49it/s]
Loading 0:  95%|█████████▍| 344/363 [00:18<00:09,  1.99it/s]
Loading 0:  96%|█████████▌| 348/363 [00:18<00:05,  2.67it/s]
Loading 0:  97%|█████████▋| 353/363 [00:18<00:02,  3.88it/s]
Loading 0:  98%|█████████▊| 357/363 [00:19<00:01,  5.03it/s]
                                                            

Job chaiml-chaiml-sft-horror-v2-mkmlizer completed after 128.42s with status: succeeded
Stopping job with name chaiml-chaiml-sft-horror-v2-mkmlizer
Pipeline stage MKMLizer completed in 131.40s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.30s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-chaiml-sft-horror-v2
Waiting for inference service chaiml-chaiml-sft-horror-v2 to be ready
Inference service chaiml-chaiml-sft-horror-v2 ready after 40.65250205993652s
Pipeline stage MKMLDeployer completed in 42.15s
run pipeline stage %s
Running pipeline stage StressChecker
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 3.1266896724700928s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.1326818466186523s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.9418690204620361s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.0844907760620117s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.7121562957763672s
5 requests
0 failed requests
5th percentile: 1.758098840713501
10th percentile: 1.8040413856506348
20th percentile: 1.8959264755249023
30th percentile: 1.9703933715820312
40th percentile: 2.0274420738220216
50th percentile: 2.0844907760620117
60th percentile: 2.103767204284668
70th percentile: 2.1230436325073243
80th percentile: 2.3314834117889407
90th percentile: 2.7290865421295165
95th percentile: 2.9278881072998044
99th percentile: 3.086929359436035
mean time: 2.199577522277832
Pipeline stage StressChecker completed in 13.44s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.21s
Shutdown handler de-registered
chaiml-chaiml-sft-horror_v2 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service chaiml-chaiml-sft-horror-v2-profiler
Waiting for inference service chaiml-chaiml-sft-horror-v2-profiler to be ready
Inference service chaiml-chaiml-sft-horror-v2-profiler ready after 190.4801106452942s
Pipeline stage MKMLProfilerDeployer completed in 190.86s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/chaiml-chaiml-sft-horror-v2-profiler-predictor-00001-deplo8z4bv:/code/chaiverse_profiler_1726612514 --namespace tenant-chaiml-guanaco
kubectl exec -it chaiml-chaiml-sft-horror-v2-profiler-predictor-00001-deplo8z4bv --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1726612514 && python profiles.py profile --best_of_n 8 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 64 --summary /code/chaiverse_profiler_1726612514/summary.json'
kubectl exec -it chaiml-chaiml-sft-horror-v2-profiler-predictor-00001-deplo8z4bv --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1726612514/summary.json'
Pipeline stage MKMLProfilerRunner completed in 1176.99s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service chaiml-chaiml-sft-horror-v2-profiler is running
Tearing down inference service chaiml-chaiml-sft-horror-v2-profiler
Service chaiml-chaiml-sft-horror-v2-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.15s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Pipeline stage %s skipped, reason=%s
Pipeline stage MKMLizer completed in 0.19s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.32s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-chaiml-sft-horror-v2
Waiting for inference service chaiml-chaiml-sft-horror-v2 to be ready
Received signal 2, running shutdown handler
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Pipeline stage %s skipped, reason=%s
Pipeline stage MKMLizer completed in 0.19s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.35s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-chaiml-sft-horror-v2
Ignoring service chaiml-chaiml-sft-horror-v2 already deployed
Waiting for inference service chaiml-chaiml-sft-horror-v2 to be ready
Inference service chaiml-chaiml-sft-horror-v2 ready after 10.560009002685547s
Pipeline stage MKMLDeployer completed in 11.93s
run pipeline stage %s
Running pipeline stage StressChecker
Pipeline stage %s skipped, reason=%s
Pipeline stage StressChecker completed in 0.19s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
Pipeline stage %s skipped, reason=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.19s
Shutdown handler de-registered
chaiml-chaiml-sft-horror_v2 status is now torndown due to DeploymentManager action